Azure Databricks Data Engineer

We are looking for an experienced Azure Databricks Data Engineer to help build and operate scalable production-grade data platforms on Azure Databricks and Microsoft Azure.

The ideal candidate combines strong Spark engineering expertise with a production engineering mindset treating data pipelines as reliable enterprise-grade products rather than ad-hoc solutions. This role requires hands-on experience delivering secure governed high-performance batch and streaming data platforms at scale.

Tasks

• Design build and maintain scalable batch and streaming pipelines using bronze/silver/gold architectures
• Develop reusable transformation frameworks and production-grade Spark workloads
• Implement and optimize Delta Lake solutions including: ACID transactions Partitioning strategies Schema evolution OPTIMIZE/ZORDER Incremental and CDC processing patterns
• Configure and manage Unity Catalog permissions governance lineage and auditability
• Perform Spark performance tuning and observability monitoring
• Integrate Databricks with Azure services including ADLS ADF/Synapse Key Vault and Microsoft Entra ID
• Implement CI/CD pipelines and deployment automation using Azure DevOps or GitHub Actions
• Collaborate with architects analysts data scientists and engineers to align technical implementation with business needs
• Continuously improve platform performance reliability cost efficiency and standardization

Requirements

• Strong hands-on expertise with Azure Databricks engineering

• Advanced Apache Spark experience using:

• PySpark or Scala

• Spark SQL

• Deep understanding of Delta Lake architecture and optimization techniques

• Experience building scalable enterprise data platforms and real-time pipelines

• Strong governance and security experience in multi-team environments

• Experience with service principals managed identities lineage and auditability

• Strong troubleshooting and Spark performance optimization skills

• Azure integration experience across storage orchestration and security services

• Solid CI/CD and Infrastructure-as-Code exposure

• Comfortable working independently in consulting-style delivery environments