Senior Databricks with Unity Catalog expertise

Job Description

About the job youre considering

Platform Engineer for enterprise-level metadata management and data access initiatives. The ideal candidate will have deep expertise in platform administrationof Databricks and Unity Catalog and will drive strategic alignment across technical and business teams.

Your Role

• Implement data provisioning patterns based on business requirements following predefined processes policies standards and metadata management rules
• Create and manage distributed workspaces in Databricks set up workspace policies provision Databricks clusters and manage data infrastructure sizing and capacity
• Create Python notebooks implement data masking processes create UDFs (SQL/Python) troubleshoot data pipelines
• Ensure data security and compliance with regulations using Databricks and Privaceras features
• Navigate multi-step enterprise approval process across architecture security and governance teams
• Design and implement data architecture leveraging technologies such as Databricks Unity Catalog and Privacera.
• Develop optimize and manage data pipelines for ETL processes using Databricks with a focus on data integrity and quality
• Design and maintain data models and schemas incorporating Unity Catalog and Collibra data governance practices
• Establish a robust data governance strategy defining standards metadata management lineage and quality practices
• Operationalize Machine Learning models in Batch and Real Time Data Pipelines leveraging relevant governance setups
• Collaborate with cross-functional teams including data scientists engineers and analysts to translate business requirements into scalable solutions
• Foster collaboration and clarity in complex ambiguous environments.

Job Description - Grade Specific

Your skills and experience:

• Extensive hands-on platform engineering experience with Databricks and Unity Catalog
• Proven success in implementing distributed Databricks workspaces SQL and Python scripting and automation
• Experience with Azure infrastructure provisioning
• Familiarity with data governance frameworks and compliance standards
• Familiarity with Privacera data security and access control management
• Familiaritywith data requirements of common ML/AI use cases
• Awareness of data governance frameworks enterprise data compliance requirements metadata modeling data architecture and enterprise-scale data discovery solutions