Senior Azure Databricks Engineer

Senior Azure Databricks Engineer

About the role:

We are looking for a Senior Azure Databricks Engineer with strong hands-on experience in both Databricks development and platform operations.

The ideal candidate should have a good end-to-end understanding of Databricks from developing Python/PySpark solutions to configuring clusters managing jobs and supporting production environments.

Responsibilities

Design develop and optimize data pipelines using Azure Databricks Python PySpark Spark SQL and Delta Lake.

Develop and maintain Databricks notebooks workflows and scheduled jobs.

Configure and manage Databricks clusters cluster policies autoscaling runtime versions and libraries.

Support Databricks workspace configuration infrastructure management and platform operations.

Implement Unity Catalog access controls data governance and data lineage.

Support Azure security Key Vault integration and secrets management.

Implement CI/CD pipelines and automate Databricks deployments.

Monitor and troubleshoot Databricks jobs clusters connectivity and production issues.

Optimize Spark performance compute resources and platform costs.

Collaborate with Data Engineering DevOps and Infrastructure teams.

Requirements

5 years of experience in Data Engineering or cloud data platforms including at least 3 years with Azure Databricks.

Strong Python fundamentals including data structures functions error handling and problem-solving.

Strong hands-on experience with Azure Databricks PySpark Spark SQL SQL ADLS Gen2 and Delta Lake.

Practical experience developing Databricks notebooks workflows and job scheduling.

Hands-on experience with Databricks cluster management autoscaling runtime configuration and cluster policies.

Experience with Unity Catalog access management and data lineage implementation and monitoring.

Good understanding of Databricks infrastructure workspace configuration and Azure cloud services.

Experience with CI/CD pipelines deployment automation and DevOps practices.

Knowledge of Azure networking security Key Vault and secrets management.

Experience troubleshooting production Databricks jobs clusters and platform issues.

Experience with Terraform for provisioning configuring and managing Azure Databricks infrastructure including workspaces clusters Unity Catalog and related Azure resources.

Good understanding of Spark architecture and performance optimization.

Strong communication and problem-solving skills.

Nice to Have

Experience with Bicep ARM Templates or other Infrastructure as Code tools.

Experience with Databricks Asset Bundles (DABs).

Experience with Azure Monitor Log Analytics and cost optimization.

Experience provisioning Databricks workspaces from scratch.

Experience in banking insurance or other regulated industries.

Azure or Databricks certifications.

Required Experience:

IC