Director, Data Engineering

Requisition ID: 268951

Thanks for your interest in ScotiaTech Scotiabanks new and innovative Technology hub in Bogota.

Join a purpose driven winning team that promotes creativity and innovation in a fast-paced environment where were always committed to results in an inclusive diverse and high-performing culture.

Purpose
We are seeking a skilled and visionary Data Engineering Director to lead a dynamic team responsible for driving our data development initiatives as we transition from on-premises infrastructure and legacy cloud environments to Azure Databricks as the target enterprise data platform. This pivotal role will oversee the strategy design and execution of scalable lakehouse architectures secure data ingestion patterns and high-performing data pipelines using Azure Databricks Delta Lake Unity Catalog Azure Data Lake Storage and related Azure services. The purpose of this position is to champion the modernization of our data ecosystem enabling advanced analytics AI/ML readiness real-time insights stronger governance and operational efficiency on the Azure platform. Reporting directly to senior leadership the Data Engineering Director will collaborate with cross-functional teams to deliver a future-ready data platform that powers innovation regulatory compliance and business growth. This position will oversee our Data Engineering operations for GBS Bogota serving IB.

Accountabilities

Strategic Leadership: Define and execute the data engineering roadmap for migrating on-premises data systems to the cloud aligning with organizational objectives and timelines.

Team Management: Lead mentor and grow a high-performing team of data engineers fostering a culture of collaboration innovation and technical excellence.

Azure Databricks Migration Oversight: Oversee the end-to-end migration of data pipelines data stores and analytical workloads from on-premises and legacy platforms to Azure Databricks ensuring secure scalable and cost-effective delivery.

Lakehouse Architecture Design: Design and implement robust scalable and secure lakehouse architectures on Azure Databricks including Delta Lake Unity Catalog medallion architecture patterns and integration with Azure Data Lake Storage and downstream analytics platforms.

Data Pipeline Development: Direct the creation and optimization of ETL/ELT batch streaming and incremental data pipelines using Azure Databricks Apache Spark SQL Python/PySpark and orchestration capabilities across Azure and Databricks workflows.

Collaboration: Partner with data scientists analysts IT teams and business stakeholders to understand requirements and deliver tailored data solutions.

Quality Governance and Security: Establish and enforce best practices for data governance quality security lineage access controls encryption compliance and auditability using Unity Catalog Azure security controls and enterprise data management standards.

Technology Evaluation: Assess and recommend Azure-native and Databricks-native capabilities including Lakeflow Delta Lake Unity Catalog Azure Data Factory Azure Monitor CI/CD tooling and DevOps automation to enhance data processing storage governance and accessibility.

Performance Monitoring: Implement monitoring optimization and cost-management strategies for Azure Databricks workloads including cluster/serverless compute selection Spark job tuning pipeline observability SLA tracking and incident response.

Budget & Resource Management: Manage team resources project timelines and budgets effectively to deliver on commitments.

Education / Experience / Other Information

Education:
Bachelors degree in Computer Science Data Science Information Technology Business Administration or a related field.

Experience:
Minimum of 10 years of experience in data engineering with at least 5 years in a leadership or management role.
Proven track record of successfully leading large-scale data migration projects from on-premises systems to cloud platforms.
Hands-on experience with Microsoft Azure and Azure Databricks as the target data platform including Azure Data Lake Storage Azure Data Factory Azure Synapse or Fabric integration Azure Monitor Microsoft Entra ID and enterprise networking/security patterns.
Strong knowledge of Azure Databricks Apache Spark Delta Lake Unity Catalog SQL Python/PySpark ETL/ELT design batch and streaming ingestion data warehousing and lakehouse architecture patterns.

Skills:
Exceptional leadership and communication skills with the ability to translate technical concepts to non-technical stakeholders. Deep understanding of data modeling ETL/ELT processes cloud architecture principles Azure Databricks operating models secure coding practices data governance CI/CD DevOps performance tuning and cost optimization for enterprise-scale data platforms.

Ideal:
Education: Masters degree in a relevant technical discipline or an MBA with a focus on technology management.

Experience:
o Prior experience as a Data Engineering Director or similar senior-level role in a cloud-first organization.
o Expertise in Azure-first or hybrid cloud environments with Azure Databricks as the primary platform for lakehouse analytics AI/ML enablement and enterprise data product delivery.
o Background in industries with complex data needs (e.g. finance healthcare or tech).
o Advanced familiarity with Azure Databricks capabilities such as Unity Catalog Delta Lake medallion architecture Lakeflow Jobs Spark Declarative Pipelines Auto Loader Structured Streaming workflow orchestration and integration with Power BI or Microsoft Fabric.

Certifications: Microsoft Azure Data Engineer Associate Azure Solutions Architect Expert Databricks Certified Data Engineer Databricks Certified Professional or equivalent cloud/data platform certifications are a strong plus.

Working Conditions
Work in a standard office-based environment with standard hours.

#LI-HYBRID

Location(s): Colombia : Bogota : Bogota

ScotiaTech is a business unit within ScotiaGBS a Scotiabank Group company located in Bogota Colombia. The ScotiaTech hub was created to support different technology systems and processes of the Bank. We offer an inclusive positive work environment and competitive benefits.

At ScotiaTech we value the unique skills and experiences each individual brings and are committed to creating and maintaining an inclusive and accessible environment for everyone. Candidates must apply directly online to be considered for this role. We thank all applicants for their interest in a career at ScotiaTech; however only those candidates who are selected for an interview will be contacted.

Note: All postings in will remain live for a minimum of 5 days.