Azure Databricks DevOps Administrator
Azure Databricks DevOps Administrator
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Job Location: New York City New York Hybrid (3 days onsite)
Job Title: Azure Databricks DevOps Administrator
Duration: Long term Contract / CTH
Target Start Date: ASAP
Experience
As an Azure Databricks DevOps Administrator you will be responsible for managing and maintaining the Databricks platform ensuring optimal performance and security across workspaces clusters and jobs. This role is crucial in overseeing user administration managing Azure storage solutions and implementing CI/CD pipelines using Azure DevOps. By leveraging a deep understanding of Databricks architecture and proficiency in scripting languages the Azure DevOps administrator will automate tasks and enhance the efficiency of data operations. Strong communication skills and a commitment to highquality support will enable effective collaboration with crossfunctional teams directly contributing to the departments mission of delivering robust data solutions and insights.
Job Responsibilities
- Handson experience in Azure Cloud services Networking Concepts Security Cloud and onpremises system Deployment using Azure DevOps Azure Cloud Monitoring and Cost controls and Terraform handson experience
- CI/CD Pipeline Management: Design implement and manage Continuous Integration/Continuous Deployment (CI/CD) pipelines using Azure DevOps and GitHub. Ensure the pipelines are efficient reliable and scalable.
- Infrastructure as Code (IaC): Automate the provisioning and management of infrastructure with a focus on Azure Databricks and Azure Data Factory in a private network environment using tools like Terraform and ARM templates.
- Environment Management: Create and manage development testing and production environments ensuring consistency security and alignment with organizational requirements.
- Security: Implement security best practices throughout the CI/CD pipeline including secrets management secure code scanning and compliance with security standards.
- Monitoring & Logging: Set up and maintain monitoring and logging for applications and infrastructure using Azure Monitor Log Analytics and related tools to ensure system reliability and performance.
- Automation: Identify opportunities for automation to streamline processes reduce manual errors and improve operational efficiency.
- Policy Enforcement: Establish and enforce policies such as branch policies pull request reviews and pipeline approvals to maintain code quality and compliance with organizational standards.
- Manage and maintain Azure Databricks Platform workspaces clusters and jobs
- Oversee user administration including access controls and permissions
- Handle library installations runtime management and policy enforcement
- Implement/analyze cost control measures
- Administer Unity Catalog for data governance and security
- Collaborate with data engineers data scientists and analysts to optimize and streamline data workflows and analytical pipelines on the Databricks platform
- Manage Azure storage solutions including Blob Storage and Data Lake Storage
- Administer Azure Key Vault for secure storage of secrets and keys
- Configure and manage Azure Data Factory for data integration and ETL processes
- Implement and manage VNETs firewalls Azure policies and security best practices
- Set up budgets and alerts to monitor and control Azure costs and spend
- Configure alerts for proactive issue detection and resolution
- Databricks Lakehouse Monitoring
Minimum Qualifications
- Minimum of Bachelors degree in Computer science Information systems or Engineering
- Experience: 7 years of professional experience in DevOps with a strong focus on Azure. Knowledge of Data Platforms like Databricks Data Factory (preferred)
- Proficiency in scripting languages such as Python/PySpark PowerShell or Bash
- Experience in automating administrative tasks and workflows
- Knowledge of security best practices and compliance requirements
- Experience with ETL processes data pipelines and big data technologies
- Experience with backup and restore procedures for Databricks and Azure services
- Ability to troubleshoot and resolve issues in a timely and efficient manner
- Strong verbal and written communication skills
- Ability to document processes procedures and configurations clearly
- Commitment to providing highquality support to internal/external users and stakeholders
- Ability to understand and address the needs of internal and external customers
- Team player with the ability to work collaboratively with crossfunctional teams
- Flexibility to adapt to changing requirements and priorities
- Continuous Learning: Eagerness to learn new technologies and continuously improve skills to stay current in a rapidly evolving field.
Employment Type
Full Time
