Databricks Platform Expert

Key Responsibilities

Databricks Platform Expert

• Manage configure and administer Databricks workspaces Clusters SQL Warehouses Serverless jobs and workspace objects.

• Implement and manage Unity Catalog including catalogs schemas tables access controls and data lineage.

• Optimize cluster policies auto-scaling strategies and cost management for Serverless and Classic compute.

• Serve as the SME for Databricks infrastructure governance and security best practices.

• Monitor workspace performance cluster stability logs job reliability and platform health.

• Implement CI/CD pipelines for notebooks jobs and Delta Live Tables using Git integration.

• Support user provisioning access controls (ACLs) secrets management and workspace SSO.

• Write efficient Spark (PySpark / SQL / Scala) code for ETL data transformations and pipeline optimizations.

• Assist data engineering teams with Spark job debugging performance tuning and code reviews.

• Build and maintain production-grade pipelines leveraging Delta Lake Databricks Jobs and DLT.

• Implement and manage RBAC SCIM provisioning AIM service principals and cluster access controls.

• Ensure compliance with enterprise data governance audit and logging requirements.

• Manage secrets Key Vault and enforce secure credential handling.

• Support audit reports compliance reviews and workspace security configuration.

• Monitor job failures cluster lifecycle performance and system events using Databricks logs and cloud-native monitoring tools (Azure Monitor).

• Create automated alerts and observability dashboards for platform usage cost and performance.

• Troubleshoot Databricks runtime issues library conflicts and Spark execution failures.

• Collaborate with cloud and network teams on VNet peering and private-link connectivity issues.

• Develop cost governance policies for cluster sizes job policies and SQL Warehouse tiers.

• Identify opportunities to reduce cost via autoscaling spot instances (classic clusters) and job consolidation.

Required Qualifications

• 46 years of experience working with Databricks as an administrator or data engineer.

• Strong expertise in Apache Spark programming (PySpark preferred; SQL or Scala is a plus).

• Hands-on experience with Databricks Jobs cluster configuration SQL Warehouses and Unity Catalog.

• Deep understanding of Delta Lake ACID transactions and lakehouse architecture.

• Experience with Git CI/CD and DevOps concepts for data engineering workflows.

• Knowledge of cloud platforms ( Azure).

• Familiarity with IAM networking basics monitoring tools and security patterns