Azure Databricks Platform Engineer

Senior Azure Databricks Platform Engineer

About the role:

We are looking for a Senior Azure Databricks Platform Engineer with strong hands-on experience in both Databricks development and platform operations. The candidate should be able to manage and support all the activities related to Databricks platform in conjunction with Azure cloud single handedly. Workspace creation maintenance deployment using DEVOPS CICD is integral for the role.

Responsibilities

Design develop and optimize data pipelines using Azure Databricks Python PySpark Spark SQL and Delta Lake.

Develop and maintain Databricks notebooks workflows and scheduled jobs.

Configure and manage Databricks clusters cluster policies autoscaling runtime versions and libraries.

Support Databricks workspace configuration infrastructure management and platform operations.

Implement Unity Catalog access controls data governance and data lineage.

Support Azure security Key Vault integration and secrets management.

Implement CI/CD pipelines and automate Databricks deployments.

Monitor and troubleshoot Databricks jobs clusters connectivity and production issues.

Optimize Spark performance compute resources and platform costs.

Collaborate with Data Engineering DevOps and Infrastructure teams.

Must have Requirements

5 years of experience in Platform Engineering on Azure data platforms including at least 3 years with Azure Databricks platform management.

The ideal candidate should have a good end-to-end understanding of Databricks from developing Python/PySpark solutions to configuring clusters managing jobs and supporting production environments. Workspace management cluster management Unity catalog implementation.

Strong Python fundamentals including data structures functions error handling and problem-solving.

Strong hands-on experience with Azure Databricks PySpark Spark SQL SQL ADLS Gen2 and Delta Lake.

Practical experience developing Databricks notebooks workflows and job scheduling.

Hands-on experience with Databricks cluster management autoscaling runtime configuration and cluster policies.

Experience with Unity Catalog access management and data lineage implementation and monitoring.

Good understanding of Databricks infrastructure workspace configuration and Azure cloud services.

Experience with CI/CD pipelines deployment automation and DevOps practices.

Knowledge of Azure networking security Key Vault and secrets management.

Experience troubleshooting production Databricks jobs clusters and platform issues.

Experience with Terraform for provisioning configuring and managing Azure Databricks infrastructure including workspaces clusters Unity Catalog and related Azure resources.

Good understanding of Spark architecture and performance optimization.

Strong communication and problem-solving skills.

Nice to Have

Experience with Bicep ARM Templates or other Infrastructure as Code tools.

Experience with Databricks Asset Bundles (DABs).

Experience with Azure Monitor Log Analytics and cost optimization.

Experience provisioning Databricks workspaces from scratch.

Experience in banking insurance or other regulated industries.

Azure or Databricks certifications.

Required Experience:

Senior IC