Senior Data Platform Engineer (Azure Databricks)
Senior Data Platform Engineer (Azure Databricks)
Posted on :
24-07-2025
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
Position Summary :
As the (Senior) Data Platform Engineer (Azure Databricks) you will be responsible for building maintaining and evolving our Azure Databricks platform ensuring it is secure reliable scalable and cost-efficient. This role is pivotal in empowering our Data- & AIOps team by providing a robust automated and well-governed data environment. The ideal candidate will drive a "platform-as-a-product" mindset continuously improving our architecture operations and user experience.
Your Responsibilities :
Platform Architecture & Engineering:
- Design and implement architectural improvements including workspace migrations and robust disaster recovery (DR) strategies.
- Enhance and maintain the platform's security posture focusing on data governance data and compute isolation at logical and physical levels.
- Drive the evolution of the platform by researching prototyping and integrating new tools and features to enhance performance and scalability.
Data Governance & Security:
- Lead the expansion and utilization of Databricks Unity Catalog for granular data governance including row/column-level security data lineage tracking and metadata tagging.
- Conduct regular security and compliance checks to ensure the platform adheres to internal policies and industry standards.
Automation & Operations (DevOps & GitOps):
- Build and maintain resilient CI/CD pipelines using GitHub Actions to automate infrastructure (Terraform) and application deployments.
- Implement and manage a comprehensive observability stack (logging monitoring tracing alerting) to ensure high availability and proactive issue resolution.
- Apply FinOps principles to monitor analyze and optimize platform costs.
- Manage day-to-day platform administration including onboarding projects managing clusters and configuring jobs.
User Enablement & Self-Service:
- Develop and integrate user-friendly interfaces (e.g. via ServiceNow or Backstage) to enable self-service for access requests and resource provisioning.
- Create and maintain clear documentation including architecture diagrams runbooks and best-practice guides.
- Coach and train other colleagues championing a "platform-as-a-product" mindset focused on delivering value to internal users.
Your profile
Essential Qualifications:
- Bachelor's or Master's degree in Computer Science Mathematics Engineering or a related discipline.
- 5 years of professional experience in a platform engineering DevOps or SRE role with a focus on data-intensive platforms.
- Deep expertise in Microsoft Azure including ADLSGen2 Private Endpoints Azure Key Vault Azure Log Analytics and EntraID.
- Extensive hands-on experience with Azure Databricks administration operations security and performance tuning.
- Strong proficiency with Terraform and a solid understanding of GitOps principles.
- Advanced skills in shell scripting and experience with a scripting language like Python.
- Demonstrable experience building CI/CD pipelines with GitHub Actions. Familiarity with Databricks Asset Bundles is a plus.
- Proficient with Linux administration and troubleshooting.
- Practical experience implementing data governance solutions for data lakes especially with Unity Catalog.
- Solid understanding of Kubernetes (k8s) concepts and architecture.
Preferred Qualifications:
- Understanding of MLOps principles (e.g. feature stores model deployment strategies) to support ML engineering teams.
- Experience integrating with or building plugins for platforms like Backstage or ServiceNow.
Soft Skills and Cultural Fit:
- Excellent communication and collaboration skills to work effectively with technical teams and business stakeholders.
- Strong analytical thinking problem-solving abilities.
- Demonstrated high level of initiative self-motivation and a proactive self-starter mindset with a strong drive to independently identify and solve challenges.
- A focus on treating the platform as a product prioritizing the user experience and delivering tangible value to internal customers.
- A passion for continuous learning innovation knowledge sharing and driving excellence in data engineering.
- Ability to work effectively in a cross-functional fast-paced environment.
Employment Type
Full Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
