Data Platform Engineer
Data Platform Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
We are looking for Platform Automation Engineer to join our dynamic technology team and drive automation for Azure based data platform. As a You will be responsible for designing building and maintaining robust and scalable data platforms enabling self-service CI/CD and job orchestration for Databricks and related Azure services. You will work closely with data analysts and software engineers to ensure seamless data flow high data quality and optimal performance across our data infrastructure.
Primary responsibilities:
- Design build and maintain scalable data platforms.
- Automate deployment and configuration of Azure Databricks workspaces clusters jobs notebooks secrets and other components using Terraform Azure CLI or REST APIs.
- Automate data workflows and support CI/CD pipelines for deployment of notebooks libraries and jobs.
- Enable monitoring alerting and logging for Databricks workloads using Log Analytics Azure Monitor and audit logs.
- Implement security best practices including Unity Catalog secret management and networking controls.
- Collaborate with platform DevOps and data engineering teams to enable self-service capabilities.
- Monitor troubleshoot and ensure data system reliability.
Qualifications :
- Bachelors degree in computer science Computer Engineering relevant technical field or equivalent; Masters degree preferred.
Additional Information :
Skills
- Hands-on experience with Azure Databricks Databricks Unity Catalog jobs clusters REST APIs and workspace management
- Proficiency in IAC using Terraform and Databricks components.
- Experience with CI/CD tools - Azure DevOps GitHub Actions.
- Proficient in Python scripting and SQL.
- Understanding of ETL process.
- Knowledge of Databricks REST APIs job orchestration cluster policies and secrets management.
- Knowledge of Azure services - ADF Key Vault ADLS Gen2 Synapse.
Key Competencies:
- 4 years experience in data engineering ETL tools and working with large data sets.
- Minimum 3 years of working experience of distributed cluster.
- At least 2-3 years of Azure Databricks Cloud experience in Data Engineering
- Experience of Delta table ADLS DBFS ADF.
- Experience with Cloud based SQL Database: Azure SQL Editor
- Excellent problem-solving investigative and troubleshooting skills
- Experience with CI/CD tools such as Jenkins and Github
- Ability to work independently.
Soft Skills:
- Good Communication Skills
- Ability to coach and Guide young Data Engineers
- Decent Level in English as Business Language
Remote Work :
No
Employment Type :
Full-time
Employment Type
Full-time
