Data Engineer (Azure Databricks)

Bengaluru - India

Monthly Salary: Not Disclosed

Posted on: 8 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Description:

Design and develop high-performance Spark and Delta Lake workflows for data engineering and analytics workloads.
Lead the modernization and migration of ETL/ELT pipelines to Databricks lakehouse architecture.
Implement Databricks best practices across Unity Catalog governance data quality and observability.
Optimize clusters SQL warehouses autoscaling and serverless compute for performance and cost efficiency.
Perform advanced Spark performance tuning (partitioning shuffle optimization caching AQE skew mitigation I/O improvements).
Develop and maintain observability frameworks and dashboards for Spark jobs pipelines and compute usage.
Automate cluster provisioning job orchestration and CI/CD through reusable frameworks and scripts.
Analyze billing and utilization metrics to drive FinOps-based cost optimizations.
Evaluate and adopt emerging Databricks features such as Delta Live Tables Serverless Compute and Lakehouse Federation.
Partner with architects and data engineering teams to ensure scalability security and operational reliability.

Job Description: Design and develop high-performance Spark and Delta Lake workflows for data engineering and analytics workloads. Lead the modernization and migration of ETL/ELT pipelines to Databricks lakehouse architecture. Implement Databricks best practices across Unity Catalog govern...