Azure Databricks LeadArchitect
Azure Databricks LeadArchitect
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Title: Azure Databricks Lead/Architect
Location: Chicago IL Day 1 onsite & Hybrid Model)
Duration: Long Term
Mandatory skills
Required Skills & Experience:
- 12 years of professional experience in Data Engineering Architecture or Data Platforms.
- 5 years of handson experience with Azure Databricks Apache Spark and PySpark.
- Deep expertise in ETL design data modelling (Data Vault Star Schema Medallion architecture) and query optimization.
- Strong experience with SQL Python and Data Pipeline Orchestration.
- Handson experience in CI/CD practices GitHub Actions and version control tools.
- Expertise in Azure ecosystem (Azure Data Lake Storage Azure Data Factory Synapse Analytics Event Hub Kafka).
- Knowledge of structured streaming and realtime data processing.
- Experience implementing security and governance best practices for cloudbased data platforms.
Responsibilities:
- Architecture & Strategy:
- Design and develop scalable Lakehouse architectures that enable highperformance data ingestion processing and analytics.
- Define and enforce best practices for ETL data modeling and data governance within the Azure ecosystem.
- Architect and optimize Medallion Architecture (Bronze Silver Gold) for efficient data transformation and analytics.
- Ensure high availability security and performance of Azure Databricks environments.
- Define data infrastructure strategies ensuring alignment with business objectives and scalability requirements.
- Technical Leadership & Development:
- Provide architectural guidance and handson development support for ETL pipelines batch & streaming solutions in Azure Databricks using PySpark and SQL.
- Drive the implementation of Data Vault Star Schema and Data Warehouse modeling techniques.
- Implement CI/CD pipelines version control and automation to streamline deployment and testing.
- Lead the integration of Azure Data Services such as Azure Data Factory Azure Synapse Delta Lake and Delta Live Tables.
- Apply Chaos Engineering practices to ensure system resilience and fault tolerance.
- Collaboration & Mentorship:
- Work closely with Product Owners Data Engineers Data Scientists and Business Analysts to deliver highquality data solutions.
- Provide technical mentorship to engineering teams enabling them to follow best practices in Databricks development ETL and data pipeline management.
- Collaborate across departments to design functional and scalable solutions that balance business needs and technical feasibility.
- Analytics & Optimization:
- Conduct performance tuning and optimization of Databricks workloads Spark jobs and SQL queries.
- Define and implement monitoring frameworks to track data quality lineage and processing performance.
- Design efficient data governance access control and security policies for Databricks and Azure environments.
- Lead rootcause analysis and problemsolving for complex data engineering challenges.
Preferred Skills (Bonus Points):
- Experience with Snowflake and BI Tools (Power BI Tableau).
- Exposure to Graph Databases and Knowledge Graphs.
- Familiarity with Machine Learning or AIbased data solutions in Databricks.
- Experience in Python library development for reusable data processing functions.
Employment Type
Full Time
