Azure Data Engineer

Mumbai - India

Monthly Salary: Not Disclosed

Posted on: 10 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Description:

Experience in leading development of Data and Analytics products from Requirement Gathering State to Driving User Adoption
Develop and optimize ETL processes using Databricks and related tools like Apache Spark
Design efficient data processing systems and pipelines using Databricks APIs and other cloud services
Candidate with strong data transformation experience on Unity Catalog Delta Tables DLT
Strong proficiency in writing and optimizing SQL queries and working with databases
Ability to acquire specialized domain knowledge required to be more effective in all work activities BI & Data-warehousing concepts are a must.
Design develop and maintain scalable ETL/ELT pipelines using PySpark on Databricks.
Ingest and transform data from multiple structured and unstructured sources including cloud storage (Azure Data Lake AWS S3 etc.).
Optimize Spark jobs for performance and cost-efficiency on the Databricks platform.
Collaborate with data scientists analysts and stakeholders to understand data requirements and deliver high-quality solutions.
Implement best practices in data engineering including modular coding unit testing and version control (e.g. Git).
Automate data workflows and schedule jobs using Databricks Workflows or external orchestration tools (e.g. Airflow Azure Data Factory).
Ensure data quality integrity and governance in all data pipelines.
Participate in code reviews performance tuning and system monitoring.
Document solutions processes and configurations.
Responsible for data management activities related to the migration of on-prem sources to cloud system using Microsoft Azure PaaS Services and Azure Cloud Technologies including the creation and use of ingestion pipelines data lakes cloud-based data marts & data warehouses cloud-based semantic data services layer.

Job Description: Experience in leading development of Data and Analytics products from Requirement Gathering State to Driving User Adoption Develop and optimize ETL processes using Databricks and related tools like Apache Spark Design efficient data processing systems and pipelines...