Senior Azure Databricks Expert

Job Title: Senior Azure Databricks Expert

Job Summary
This role involves designing building and optimizing enterprise-grade data platforms on Microsoft Azure. The ideal candidate will bring strong technical expertise in Azure Databricks Spark and data engineering along with the ability to lead teams mentor junior engineers and drive best practices across global cross-functional teams.

Key Responsibilities

• Provide technical leadership for Azure Databricks-based data transformation initiatives
• Design develop and maintain scalable data pipelines using Azure Databricks
• Build and optimize ETL/ELT solutions for enterprise data platforms
• Lead migration and modernization efforts from SQL Server and legacy systems to Azure
• Develop and optimize data processing workflows using Apache Spark PySpark and SQL
• Improve performance of Spark workloads through deep understanding of Spark architecture (drivers executors partitions stages tasks)
• Work with Azure Data Lake Storage Gen2 (ADLS Gen2) and Delta Lake for reliable data storage solutions
• Optimize Databricks clusters jobs and workloads for performance and cost efficiency
• Implement data governance and secure access using Unity Catalog and Azure security best practices
• Collaborate with architects DevOps teams and business stakeholders to deliver scalable solutions
• Troubleshoot and resolve production issues related to data pipelines and platform performance
• Support CI/CD pipelines and Infrastructure as Code (IaC) implementations
• Mentor junior engineers and provide technical guidance and best practices

Must-Have Skills

• Minimum 3 years of hands-on experience with Azure Databricks
• Strong expertise in Apache Spark PySpark and SQL
• Experience in building and optimizing ETL/ELT pipelines
• Hands-on experience with ADLS Gen2 and Delta Lake
• Strong understanding of Spark architecture and performance tuning
• Experience in migration from SQL Server or other relational databases
• Practical experience working in Azure cloud environments
• Strong problem-solving and troubleshooting skills

Good-to-Have Skills

• Experience with Unity Catalog and data governance frameworks
• Exposure to Azure services such as Azure Key Vault Azure Monitor and Networking
• Experience with CI/CD tools and DevOps practices
• Knowledge of Infrastructure as Code (ARM Terraform etc.)
• Experience working on enterprise-scale data platforms or analytics projects

Soft Skills

• Strong communication skills with B2 English proficiency
• Ability to work in globally distributed teams
• Leadership and mentoring capabilities
• Strong collaboration and stakeholder management skills
• Analytical mindset with attention to detail

Domain Knowledge

• Experience in enterprise data engineering and analytics environments
• Understanding of large-scale data platform modernization initiatives
• Exposure to consulting or client-facing environments is a plus
• Familiarity with Agile/Kanban delivery models (e.g. ServiceNow-based workflows)

Location: Pune(Remote)

Required Experience:

Senior IC