Databricks/Python Architect
Databricks/Python Architect
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
- Design build and maintain scalable secure and high-performing Data Bricks platform infrastructure across cloud providers (AWS/Azure).
- Automate infrastructure provisioning and management using Infrastructure as Code (IaC) tools such as Terraform or CloudFormation.
- Implement and manage platform security including role-based access control (RBAC) service principals and data encryption.
- Monitor troubleshoot and optimize Data Bricks platform performance using CLI REST APIs and observability tools.
- Collaborate with data engineering teams to deploy monitor and optimize data pipelines streaming applications and ML models on Data Bricks.
- Integrate Delta Lake with existing data warehouses lakes or streaming systems for seamless data management.
- Support Spark-based applications by ensuring efficient resource use and resolving performance issues.
- Enable and manage CI/CD workflows for notebooks jobs and ML models using tools like Jenkins GitHub Actions or Azure DevOps.
- Enforce data governance and regulatory compliance using Unity Catalog and related tools for data quality and lineage.
- Design and implement robust logging monitoring and alerting systems for platform operations.
- Serve as subject matter expert for Data Bricks providing best practices guidance to data engineers and analysts.
- Work cross-functionally with DevOps IT and Data Science teams to ensure platform alignment with organizational objectives.
- Document platform architecture operational procedures and troubleshooting guides to streamline management and onboarding.
Required Skills and Qualifications:
- 10 years of overall experience with at least 3-4 years specifically in Data Bricks.
- Extensive experience in managing and optimizing Data Bricks environments and Spark workloads.
- Proficiency in cloud platforms (AWS Azure GCP) and their Data Bricks offerings.
- Skilled in IaC tools (Terraform CloudFormation Pulumi) and programming languages (Python Scala SQL).
- In-depth knowledge of Delta Lake data partitioning and schema evolution.
- Familiarity with MLOps tools such as MLflow and CI/CD tools (Git Jenkins Azure DevOps).
- Strong understanding of access control data security and compliance (GDPR HIPAA).
- Experience with monitoring and observability tools (Prometheus Datadog CloudWatch).
- Excellent problem-solving troubleshooting communication and documentation skills.
Preferred Qualifications:
- Data Bricks Certified Associate Developer or Architect.
- Knowledge of advanced data governance tools (Alation Collibra).
- Familiarity with containerization and orchestration (Docker Kubernetes).
Employment Type
Full-time
