Principal Data Engineer – Databricks
Job Location:
Monthly Salary:
Experience Required:
Posted on:
Yesterday
Posted on:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Principal Data Engineer Databricks Spark Delta Lake PySpark Data Lakehouse AWS/Azure
Job Description
Location: Toronto
Work Model: Onsite (4 days/week)
Key Requirements
1218 years of overall Data Engineering experience.
8 years of experience with Enterprise Data Warehouse and Data Lake platforms.
5 years of hands-on experience with Databricks and Apache Spark at scale.
Strong experience modernizing legacy Cloudera platforms (CDH/CDP Hive HBase Impala Spark) to Databricks Lakehouse.
Experience redesigning ingestion transformation and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
Experience ensuring data reconciliation audit integrity and consistency during migration.
Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
Experience implementing layered architectures including:
Raw/Landing Layer
Curated/Conformed Layer
Semantic/Consumption Layer
Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
Strong experience with finance and risk data models including:
General Ledger
Sub-ledger
Financial Hierarchies
Credit Risk Models
Liquidity Risk Models
Market Risk Models
Experience enabling reporting use cases including aggregation drill-down and drill-back capabilities.
Experience building and managing semantic/consumption layers for BI reporting and analytics.
Ability to define business metrics dimensions hierarchies and KPIs.
Experience with Databricks SQL Delta Tables and dbt or similar frameworks.
Strong experience developing and optimizing large-scale data pipelines using:
PySpark
Spark SQL
Delta Lake
Experience implementing Medallion Architecture:
Bronze Layer
Silver Layer
Gold Layer
Experience optimizing workloads using Z-ORDER OPTIMIZE caching and cluster configurations.
Experience implementing data governance data quality frameworks reconciliation controls and exception handling.
Experience establishing data lineage and metadata management.
Knowledge of data security access control and compliance standards.
Experience with cloud platforms such as AWS or Azure.
Experience with CI/CD pipelines using:
Git
Terraform
Jenkins
Azure DevOps
Familiarity with orchestration tools such as:
Apache Airflow
Databricks Workflows
Experience with dbt is a plus.
Ability to act as a technical authority and lead architecture decisions.
Experience mentoring senior engineers and establishing engineering standards.
Strong stakeholder management skills with finance risk analytics and governance teams.
Ability to translate complex data structures into business-ready insights.
Nice to Have
Experience in Banking Financial Services Insurance (BFSI) Capital Markets or regulatory reporting.
Exposure to:
SAP Finance
Oracle Financials
SAP S/4HANA
Experience supporting AI/ML workloads.
Databricks or cloud certifications.
Key Responsibilities
Lead Cloudera to Databricks transformation initiatives.
Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
Build scalable high-performance data pipelines and modern data architectures.
Drive data modernization governance quality and security initiatives.
Support regulatory management and analytical reporting platforms.
Provide technical leadership mentor engineering teams and establish best practices.
Required Skills:
Sailpoint
