Databricks Lead Data Architect

VDart Inc

Job Location:

Mississauga - Canada

Monthly Salary: Not Disclosed

Posted on: Yesterday

Vacancies: 1 Vacancy

Job Summary

Role: Databricks Lead / Data Architect
Location: Mississauga ON
Contract

About the Role

We are hiring a senior deeply hands-on Databricks Lead / Data Architect to drive the Databricks workstream of a large-scale data and AI modernization program for a major Canadian enterprise retail client. This is a build-and-lead role: you will own the technical direction of Databricks-based solutions end to end -architecture Lakehouse design data engineering migration of legacy ETL workloads and production operations - while remaining personally hands-on in code and design.

You will work side by side with the clients VP of Data & AI and the AVP of Data Platforms & Integration and their teams acting as the senior technical authority who turns strategy into delivered production-grade outcomes. The immediate focus is modernizing a large on-premise ETL estate (IBM DataStage) to an Azure-native Lakehouse on Azure Data Factory and Databricks and then scaling the platform to power enterprise analytics and AI use cases.

Key Responsibilities

Architect the Lakehouse: Design and own scalable secure Databricks Lakehouse architecture on Azure (Delta Lake Unity Catalog medallion bronze/silver/gold ADLS Gen2) aligned to enterprise standards.
Stay hands-on: Personally build and review PySpark / Spark SQL pipelines Delta Live Tables notebooks and orchestration - setting the engineering bar not just directing it.
Lead legacy migration: Drive conversion of complex legacy ETL (DataStage) workloads to Databricks/PySpark and ADF including patterns accelerators and reusable frameworks for code conversion and validation.
Own performance & cost: Optimize cluster configuration job performance partitioning and cost; establish FinOps and right-sizing practices on Databricks.
Embed governance: Implement data governance lineage quality and access control through Unity Catalog and Purview; ensure security privacy and compliance by design.
Enable analytics & AI: Design Gold-layer semantic models and feature pipelines that serve BI (Power BI) advanced analytics and ML/GenAI use cases (MLflow Azure ML).
Lead the squad: Provide technical leadership and mentoring to data engineers; define best practices coding standards CI/CD (Azure DevOps) and review processes.
Partner with the client: Work closely with the clients VP (Data & AI) AVP (Data Platforms & Integration) platform architects and business stakeholders to translate requirements into delivery roadmaps and measurable outcomes.

Required Qualifications (Must-Have)

12 years in data engineering / data platform architecture with 4 years of deep hands-on Databricks delivery.

Expert-level Databricks: Spark (PySpark & Spark SQL) Delta Lake Delta Live Tables Unity Catalog Workflows performance tuning and cluster/cost optimization.
Strong Azure data stack: Azure Data Factory ADLS Gen2 Azure Key Vault Azure DevOps (CI/CD) and Azure networking/security fundamentals.
Proven migration track record: Led at least one large-scale migration from legacy ETL (e.g. DataStage Informatica Teradata) to a cloud lakehouse including complex transformation logic.
Lakehouse design depth: Medallion architecture dimensional & semantic modelling SCD handling surrogate keys and data quality / reconciliation frameworks.
Engineering rigor: CI/CD version control (Git) automated testing/validation observability and production support of mission-critical pipelines.
Leadership with hands-on credibility: Demonstrated ability to lead engineers and engage senior client stakeholders while still contributing code and designs directly.
Location: Based in the Greater Toronto Area / Ontario eligible to work in Canada and able to work on a hybrid basis with client-site presence as required.

Preferred / Nice-to-Have

Databricks certifications (e.g. Databricks Certified Data Engineer Professional / Solutions Architect) and relevant Azure certifications (DP-203 AZ-305).
Experience in retail supply chain merchandising or financial-services data domains.
Familiarity with IBM DataStage DB2 Oracle and legacy on-prem ETL estates.
Exposure to agentic AI / GenAI patterns MLOps/LLMOps and AI-assisted code migration tooling.
Experience operating a warm-standby DR and high-availability data platform.

Role: Databricks Lead / Data Architect Location: Mississauga ON Contract About the Role We are hiring a senior deeply hands-on Databricks Lead / Data Architect to drive the Databricks workstream of a large-scale data and AI modernization program for a major Canadian enterprise retail client. This is...

Role: Databricks Lead / Data Architect
Location: Mississauga ON
Contract

About the Role

Key Responsibilities

Architect the Lakehouse: Design and own scalable secure Databricks Lakehouse architecture on Azure (Delta Lake Unity Catalog medallion bronze/silver/gold ADLS Gen2) aligned to enterprise standards.
Stay hands-on: Personally build and review PySpark / Spark SQL pipelines Delta Live Tables notebooks and orchestration - setting the engineering bar not just directing it.
Lead legacy migration: Drive conversion of complex legacy ETL (DataStage) workloads to Databricks/PySpark and ADF including patterns accelerators and reusable frameworks for code conversion and validation.
Own performance & cost: Optimize cluster configuration job performance partitioning and cost; establish FinOps and right-sizing practices on Databricks.
Embed governance: Implement data governance lineage quality and access control through Unity Catalog and Purview; ensure security privacy and compliance by design.
Enable analytics & AI: Design Gold-layer semantic models and feature pipelines that serve BI (Power BI) advanced analytics and ML/GenAI use cases (MLflow Azure ML).
Lead the squad: Provide technical leadership and mentoring to data engineers; define best practices coding standards CI/CD (Azure DevOps) and review processes.
Partner with the client: Work closely with the clients VP (Data & AI) AVP (Data Platforms & Integration) platform architects and business stakeholders to translate requirements into delivery roadmaps and measurable outcomes.

Required Qualifications (Must-Have)

12 years in data engineering / data platform architecture with 4 years of deep hands-on Databricks delivery.

Expert-level Databricks: Spark (PySpark & Spark SQL) Delta Lake Delta Live Tables Unity Catalog Workflows performance tuning and cluster/cost optimization.
Strong Azure data stack: Azure Data Factory ADLS Gen2 Azure Key Vault Azure DevOps (CI/CD) and Azure networking/security fundamentals.
Proven migration track record: Led at least one large-scale migration from legacy ETL (e.g. DataStage Informatica Teradata) to a cloud lakehouse including complex transformation logic.
Lakehouse design depth: Medallion architecture dimensional & semantic modelling SCD handling surrogate keys and data quality / reconciliation frameworks.
Engineering rigor: CI/CD version control (Git) automated testing/validation observability and production support of mission-critical pipelines.
Leadership with hands-on credibility: Demonstrated ability to lead engineers and engage senior client stakeholders while still contributing code and designs directly.
Location: Based in the Greater Toronto Area / Ontario eligible to work in Canada and able to work on a hybrid basis with client-site presence as required.

Preferred / Nice-to-Have

Databricks certifications (e.g. Databricks Certified Data Engineer Professional / Solutions Architect) and relevant Azure certifications (DP-203 AZ-305).
Experience in retail supply chain merchandising or financial-services data domains.
Familiarity with IBM DataStage DB2 Oracle and legacy on-prem ETL estates.
Exposure to agentic AI / GenAI patterns MLOps/LLMOps and AI-assisted code migration tooling.
Experience operating a warm-standby DR and high-availability data platform.