Lead Data Engineer Azure Data Platform

Toronto - Canada

Monthly Salary: Not Disclosed

Posted on: 2 hours ago

Vacancies: 1 Vacancy

Job Summary

Role: Lead Data Engineer - Azure Data Platform.

Location: Toronto ON (Onsite).
Duration: Long Term Contract.

Key Responsibilities:

Data Platform Architecture:

Architect end-to-end data pipelines using Azure Data Factory Azure Databricks Azure Synapse Analytics and Azure Data Lake Gen2 for enterprise-scale data processing.
Develop production-grade PySpark transformations and Python ETL processes handling petabyte-scale datasets with complex business logic.
Lead data modeling initiatives implementing star schemas dimensional modeling and SCD Type 2 for analytics and reporting platforms.

Team Leadership & Technical Excellence:

Mentor and lead data engineering teams of 5-10 engineers through code reviews technical standards and architectural decision-making.
Implement enterprise data quality frameworks using Great Expectations Azure Monitor and Azure Purview for governance and compliance.

Real-Time & Advanced Analytics:

Design real-time streaming solutions leveraging Azure Event Hubs Stream Analytics and Delta Lake for low-latency analytics use cases.
Optimize Spark jobs for cost-efficiency performance and scalability across Azure Databricks clusters.

Required Technical Expertise:

Azure Platform (5 years): Data Factory Databricks Synapse Analytics Data Lake Gen2 Event Hubs Stream Analytics Purview.
Python/PySpark: Pandas NumPy PySpark DataFrames/SQL Delta Lake MLflow.
Data Modeling: Star Schema Dimensional Modeling SCD Type 2 Data Quality Frameworks.
Orchestration: Airflow (MWAA) Data Factory Pipelines Databricks Workflows.
SQL & Performance: Advanced T-SQL Spark SQL Query Optimization.
DevOps & IaC: Terraform ARM Templates Azure DevOps CI/CD.
Monitoring: Azure Monitor Log Analytics Great Expectations Monte Carlo.

Leadership Requirements:

Lead developer experience managing data engineering teams through complete SDLC with demonstrated success mentoring junior engineers and driving technical excellence.
Executive stakeholder communication: Translate complex technical concepts into business value during requirement sessions and roadmap presentations.
Agile/Scrum mastery: Lead sprint ceremonies using JIRA Confluence and cross-functional collaboration with data scientists analysts and business teams.

Experience Profile:

7 years data engineering experience including 5 years hands-on Azure platform expertise.
Multiple long-term enterprise projects (1.5 years each) building production data platforms.
Hands-on Python and PySpark demonstrated across 3 major projects with complex transformations.

Role: Lead Data Engineer - Azure Data Platform. Location: Toronto ON (Onsite). Duration: Long Term Contract. Key Responsibilities: Data Platform Architecture: Architect end-to-end data pipelines using Azure Data Factory Azure Databricks Azure Synapse Analytics and Azure Data Lake Gen2 for en...