Azure Databricks DevOps Engineer
Azure Databricks DevOps Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Azure Databricks DevOps Engineer
Key Responsibilities:
-
Databricks Management: Deploy configure and maintain Azure Databricks environments ensuring scalability security and efficiency.
-
Infrastructure Automation: Implement and manage IaC using Terraform ARM templates or Bicep to automate cloud resource provisioning.
-
CI/CD Pipelines: Develop and manage CI/CD pipelines using tools like Azure DevOps GitHub Actions Jenkins to automate data pipeline deployments.
-
Monitoring & Logging: Set up and maintain monitoring solutions using Azure Monitor Log Analytics and Application Insights for proactive issue resolution.
-
Security & Compliance: Implement best practices for security governance and compliance including RBAC encryption and network security policies.
-
Data Integration & Optimization: Work with Data Engineers to optimize ETL pipelines ensuring high performance and cost efficiency.
-
Collaboration: Work closely with Data Scientists Engineers and Business Analysts to enable efficient data workflows.
-
Incident Management: Troubleshoot and resolve Databricks Azure Kubernetes Services (AKS) and networking issues efficiently.
-
Documentation & Best Practices: Maintain technical documentation and establish best practices for Azure Databricks and DevOps processes.
Required Skills & Qualifications:
-
Bachelors or Masters degree in Computer Science IT or a related field.
-
5 years of experience in DevOps Cloud Engineering or Data Engineering roles.
-
Strong handson experience with Azure Databricks Azure Data Factory and Azure Synapse Analytics.
-
Proficiency in Terraform ARM Templates or Bicep for infrastructure automation.
-
Experience with CI/CD tools such as Azure DevOps GitHub Actions or Jenkins.
-
Knowledge of Python Scala or PowerShell for automation and scripting.
-
Familiarity with containerization (Docker Kubernetes AKS).
-
Solid understanding of Azure networking security and identity management (IAM RBAC AAD).
-
Experience with monitoring and logging tools (Azure Monitor Log Analytics Prometheus Grafana).
-
Strong problemsolving skills and ability to work in an Agile/Scrum environment.
Preferred Qualifications:
-
Azure Certifications (e.g. AZ400 AZ104 DP203 DP900).
-
Experience with Big Data processing frameworks (Apache Spark Delta Lake).
-
Understanding of ML Ops and working with machine learning models in Databricks.
-
Knowledge of Eventdriven architectures (Azure Event Hubs Kafka Service Bus).
Employment Type
Full Time
