Senior Azure Databricks Engineer

We are Seeking experienced Databricks / Azure Consultants to support the design implementation and ongoing development of the Databricks data platform. The consultants will work with technology and data teams to build and enhance scalable data solutions using Databricks and Microsoft Azure with a strong focus on data engineering integration security automation and platform best practices.

Ideal candidates will have strong hands-on experience across the Databricks ecosystem including Databricks Workspace Unity Catalog Delta Lake Spark and MLflow along with experience developing solutions in Microsoft Azure. Candidates should be comfortable working in a large enterprise environment and collaborating across application development infrastructure cloud security architecture and data teams.

Key Skills and Experience

• Databricks Ecosystem: Hands-on experience with Databricks Workspace Unity Catalog Delta Lake Spark UI and MLflow.
• Programming & Data Engineering: Strong development experience with Python PySpark SQL and/or Java including the development and optimization of data processing and ETL/ELT pipelines.
• Cloud Platforms: Strong Microsoft Azure experience particularly Azure Databricks Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Data Factory. Experience with AWS including S3 and IAM is also desirable.
• Apache Spark: Experience developing troubleshooting and performance-tuning Spark-based workloads and data processing pipelines.
• DevOps & MLOps: Experience with Git CI/CD pipelines OpenShift Kubernetes and Terraform / Infrastructure as Code (IaC).
• Security & Governance: Experience implementing or supporting Databricks security access controls secrets management identity integration data governance and Unity Catalog.
• Automation & Operations: Experience with Databricks workflows job orchestration monitoring troubleshooting performance optimization and production support
• Enterprise Delivery: Ability to collaborate across technical teams and operate within established enterprise architecture security change-management and governance standards.
• Knowledge of Apache Airflow

Preferred Experience

• Experience designing and delivering production-grade Databricks solutions in a large enterprise environment particularly where security data governance re