Senior Azure Data Engineer Databricks

About the Role and Job:

Job Title: Senior Azure Data Engineer - Databricks

Location: Toronto ON

Model: Hybrid (4 Days Onsite)

Job Type: Full time/ Contract

This role will lead complex data engineering initiatives influence architectural decisions and partner closely with analytics data science and platform teams.

Key Responsibilities

Data Architecture & Leadership

Design implement and evolve Medallion Architecture (Bronze Silver Gold) using Delta Lake on Azure Databricks.

Lead data architecture decisions establish engineering standards and define best practices.

Review designs and code from team members ensuring scalability performance and data quality.

Data Engineering & Development

Build and maintain high volume fault tolerant ETL/ELT pipelines using Azure Databricks and Azure Data Factory.

Develop optimized PySpark transformations for large scale batch and streaming data.

Implement reusable frameworks for ingestion validation transformation and enrichment.

Data Ingestion & Streaming

Architect and develop ingestion pipelines for structured semi structured and unstructured data.

Implement batch and real time streaming pipelines using Azure Event Hubs / Kafka / Structured Streaming.

Support incremental loads and CDC based data ingestion strategies.

Performance Optimization & Reliability

Optimize Spark workloads using partitioning caching Z Ordering and file sizing strategies.

Tune Delta Lake tables for performance and cost efficiency.

Monitor pipeline health troubleshoot failures and improve data platform reliability.

Cloud Security & Governance

Leverage Azure services including ADLS Gen2 Synapse Analytics Key Vault Azure Monitor and networking/security patterns.

Implement role based access control secrets management and data protection best practices.

Align data solutions with governance compliance and auditing requirements.

Collaboration & Mentorship

Work closely with data scientists analysts product owners and business stakeholders.

Enable advanced analytics and machine learning workloads.

Mentor junior and mid level data engineers and contribute to team skill growth.

Required Skills & Qualifications

7 years of data engineering experience with strong Azure cloud expertise.

Hands on expertise with Azure Databricks Delta Lake and Medallion Architecture.

Advanced proficiency in PySpark and Python.

Strong experience with Azure Data Factory for orchestration.

Deep understanding of data modeling data lakes and big data processing.

Proven ability to design scalable production grade data platforms.

Strong communication skills and experience working in agile environments.

Preferred Qualifications

Experience implementing CI/CD pipelines for data platforms (Azure DevOps GitHub Actions).

Exposure to streaming technologies (Kafka Event Hubs Structured Streaming).

Familiarity with data governance cataloging and lineage tools (e.g. Azure Purview).

Experience supporting ML and advanced analytics workloads.

Prior experience in financial services healthcare retail or large scale enterprise environments.