Principal Engineer Databricks Data Engineer

Job Description :

• Audit the current Databricks workspace state including environments clusters Unity Catalog setup access controls and cost configuration.
• Implement and harden Unity Catalog including namespace structure access controls cluster policies and environment separation.  
• Define workspace standards naming conventions access request workflows and user/team onboarding processes.
• Establish data quality frameworks including SLAs certification criteria and data ownership models.
• Build automation templates for routine workspace operations such as cluster lifecycle management cost alerting and utilization monitoring.
• Stand up observability dashboards covering platform health cost job success rates and DQ SLA KPIs and monitor core data asset lineage across workspaces.
• Create runbooks for recurring operational tasks that can be executed by a mid-level/junior engineer without ongoing support from the senior resource.
• Conduct knowledge-transfer sessions with the platform engineering team.
• Provide final documentation covering architecture decisions operational standards open items and recommendations for the permanent hire profile.

Key Responsabilities.

• Evaluate and optimize Databricks environments including workspaces clusters catalogs permissions and security configurations.
• Define and implement governance standards to ensure secure scalable and compliant platform operations.
• Lead Unity Catalog adoption designing data structures access models and environment segregation strategies.
• Establish RBAC policies workspace standards naming conventions and user access processes.
• Drive data governance initiatives including data quality frameworks ownership models SLAs lineage and stewardship practices.
• Develop automation solutions and Infrastructure as Code (IaC) frameworks to streamline platform administration and reduce manual effort.
• Implement cost optimization monitoring and operational excellence practices across Databricks environments.
• Design observability and performance dashboards to track platform health utilization job success rates and data quality KPIs.
• Create operational documentation runbooks and support procedures to improve platform reliability and team readiness.
• Partner with security compliance infrastructure and data engineering teams to align platform capabilities with business needs.
• Provide training mentoring and knowledge transfer sessions to empower platform administrators and engineering teams.
• Recommend platform improvements and long-term strategies to enhance governance scalability and organizational maturity.

Qualifications :

Must have skills: Azure Data Factory Databricks Unity Catalog

Good to have skills: Data platform implementation Google Workspace Administration

Remote Work :

No

Employment Type :

Full-time