Data Engineer

Colombo - Sri Lanka

Monthly Salary: Not Disclosed

Posted on: Yesterday

Vacancies: 1 Vacancy

Job Summary

Job Description

Architect administer and optimize Databricks workspaces clusters jobs and workflows.
Design and develop scalable Azure-based data engineering solutions including Data Lake Data Factory Synapse Key Vault etc.
Build high-performance data pipelines using Python and PySpark.
Work with Hadoop-based platforms and distributed storage systems.
Implement and manage real-time data streaming solutions and ingestion pipelines.
Ensure proper configuration monitoring governance and performance tuning of data platforms.
Integrate CI/CD pipelines on Azure and work with containerization technologies (Docker/K8s).
Develop and optimize SQL queries stored procedures and data models using MS SQL.
Collaborate with architects data scientists and engineering teams to deliver enterprise-grade solutions.
Define best practices coding standards and data engineering frameworks across the organization.
Troubleshoot complex data issues and ensure high availability of mission-critical data systems.

Requirements

Minimum 10 years of total IT experience.
Minimum 6 years strong experience in the following areas: Databricks Administration Azure Cloud Python & PySpark Hadoop ecosystem Data streaming & data pipeline engineering MS SQL
Proven experience architecting large-scale distributed data systems.
Strong understanding of data lakes data warehousing and big data architecture patterns.

Nice-to-Have Skills

Experience with Apache Spark optimization and tuning.
Hands-on experience with CI/CD pipelines on Azure (GitHub Actions Azure DevOps etc.).
Knowledge of containers (Docker Kubernetes).
Understanding of MLOps or advanced analytics integrations.

Job Description Architect administer and optimize Databricks workspaces clusters jobs and workflows.Design and develop scalable Azure-based data engineering solutions including Data Lake Data Factory Synapse Key Vault etc.Build high-performance data pipelin...