Azure Data Engineering Lead

Trigent Software Private Limited

Job Location:

Hyderabad - India

Monthly Salary: Not Disclosed

Posted on: 30+ days ago

Vacancies: 1 Vacancy

Job Summary

Job Title: Azure Data Engineering Lead
Experience: 5 years
Location: Hyderabad (Hybrid WFO)
Notice Period: Immediate

Role Overview:

We are looking for a highly skilled Azure Data Engineering Lead to design develop and optimize scalable data solutions using Azure Databricks and Azure Data Factory (ADF). This role involves collaborating with cross-functional teams to deliver robust data pipelines enable advanced analytics and support enterprise data initiatives.

Key Responsibilities:

Data Pipeline Development:
- Build and maintain scalable ETL/ELT pipelines using ADF and Databricks notebooks.
- Integrate data from diverse sources including Azure Data Lake SQL DBs Blob Storage and external APIs.
Advanced Data Engineering:
- Develop distributed data processing workflows using PySpark Scala and Delta Lake in Databricks.
- Implement data quality checks error handling and logging mechanisms.
Automation & DevOps:
- Automate deployments using CI/CD pipelines with Azure DevOps and Terraform.
- Manage version control and collaborative development using Git.
Performance Optimization:
- Monitor and tune pipeline performance for cost-efficiency and scalability.
- Troubleshoot and resolve data pipeline failures and bottlenecks.
Collaboration & Documentation:
- Work with data scientists analysts and business stakeholders to understand data needs.
- Document architecture workflows and best practices for internal knowledge sharing.

Mandatory Skills:

Azure
Azure Databricks
PySpark

Good-to-Have Skills:

Delta Lake
Azure Data Factory / Mapping Data Flows
SQL & Python
Terraform
CI/CD Git Agile methodologies
Azure certifications (DP-203 AZ-104)

Job Title: Azure Data Engineering Lead Experience: 5 years Location: Hyderabad (Hybrid WFO) Notice Period: Immediate Role Overview: We are looking for a highly skilled Azure Data Engineering Lead to design develop and optimize scalable data solutions using Azure Databricks and Azure Data Factory (AD...