Senior Databricks Specialist

Key Responsibilities

• Design and implement enterprise-scale data solutions using Azure Databricks.
• Define and maintain Databricks data architecture and platform standards.
• Design data ingestion and integration pipelines from multiple source systems.
• Develop scalable ETL/ELT pipelines using Databricks and Apache Spark.
• Design and implement Medallion Architecture (Bronze Silver Gold) where appropriate.
• Work with Delta Lake and optimize Delta tables and data pipelines.
• Design data models and transformation processes for analytical and operational reporting.
• Integrate Databricks with Microsoft Dynamics 365 / Dataverse and other enterprise applications.
• Develop and optimize batch and near-real-time data processing pipelines.
• Implement data quality validation monitoring lineage and governance processes.
• Optimize Spark jobs clusters SQL queries Delta tables and data pipelines for performance and cost.
• Implement appropriate data security and access controls.
• Work with Unity Catalog for data governance access control cataloging and lineage.
• Support Power BI and other analytical/reporting platforms through optimized data models and data services.
• Troubleshoot complex data engineering and performance issues.
• Establish development testing and production standards for Databricks.
• Support CI/CD and DevOps practices for Databricks solutions.
• Conduct technical reviews and mentor data engineers.
• Ensure data platform solutions meet security availability scalability and disaster recovery requirements.

Mandatory Technical Requirements

• Minimum 8 years of experience in data engineering/data architecture.
• Minimum 4 years of strong hands-on experience with Azure Databricks.
• Strong hands-on experience with Apache Spark and PySpark.
• Strong experience with Delta Lake.
• Strong experience designing enterprise data platforms and data pipelines.
• Strong knowledge of Databricks Medallion Architecture.
• Strong experience with Unity Catalog data governance access control and data lineage.
• Strong SQL skills and experience with large-scale data processing.
• Strong experience with Azure data services.
• Experience with Azure Data Factory / Azure Data Lake Storage Gen2.

Experience integrating Databricks with APIs enterprise applications CRM systems and other data sources