Data Engineering Manager
Share
Job Location:
Mississauga - Canada
Monthly Salary:
Not Disclosed
Posted on:
Yesterday
Vacancies:
1 Vacancy
Job Summary
About Upshop:
Upshopis the foremost provider of a SaaS platform designed to streamline forecasting ordering production and inventory optimization processes for food retailers. Its unified platform simplifies and enhances associate tasks promoting smarter and more interconnected operations across Fresh Center DSD and eCommerce departments. With over 450 retailers and 50000 stores relying on its mission-critical operations platform globally customers have witnessed substantial enhancements in sales shrinkage reduction food safety and sustainability throughout their stores.
AtUpshop we believe that great businesses are built by great people. Our People function is at the heart of our companys growth ensuring we attract develop and retain A Players who drive our mission forward.
Our Values:
- Extremely Accountable
- Customer Obsessed
- Always Innovating
- Demand Excellence
- Biased for Action
Data Engineering Manager
Role Overview
We are seeking an experienced Data Engineering Manager to lead the technical evolution of our Databricks Lakehouse. You will be responsible for the architecture scaling and reliability of our data products ensuring that the platform provides high-quality actionable data for retail and supply chain analytics. This is a leadership role that requires a balance of advanced systems design and hands-on operational rigor.
Key Responsibilities
Architecture & Platform Ownership
- Lakehouse Strategy: Own the end-to-end design and optimization of the Databricks Lakehouse architecture leveraging Unity Catalog and Delta Lake for multi-tenant isolation.
- Scalable Ingestion: Drive multi-tenant ingestion strategies from Azure Table Storage Cosmos DB and APIs focusing on watermark logic and idempotent pipelines.
- Framework Development: Lead the evolution of in-house PySpark-based ETL frameworks to support complex transformations and downstream consumption.
Operational Excellence & Data Observability
- Data Quality Validation: Implement comprehensive data validation frameworks and automated checks to ensure accuracy completeness and consistency across the Lakehouse.
- Observability & Monitoring: Drive the strategy for end-to-end data observability using monitoring and alerting to proactively identify pipeline drifts or failures before they impact stakeholders.
- FinOps & Performance: Improve cost transparency through DBU tracking and compute profiling (Serverless vs. Classic); drive performance tuning via clustering and partitioning strategies.
DevOps & Technical Governance
- Deployment Integrity: Oversee CI/CD pipelines and Databricks Asset Bundles (DABs) to maintain parity between UAT and Production environments.
- Governance Standards: Implement version control rollback strategies and deployment governance to ensure a stable and auditable production environment.
Cross-Functional Leadership
- Strategic Partner: Partner with BI Data Science and Product teams to bridge business requirements into scalable production-grade data solutions.
- Team Growth: Mentor and lead a distributed team of engineers focusing on high-velocity delivery and technical excellence.
- Tenant Logic: Support customer onboarding by designing tenant-specific transformation logic that scales without increasing architectural complexity.
Qualifications
- 8 years in Data Engineering; 3 years in a leadership/management role.
- Expert in Workflows Unity Catalog and Delta Lake.
- Deep proficiency in PySpark SQL and Azure Data ecosystem (Storage Cosmos DB).
- Proven experience managing multi-tenant or enterprise-scale platforms.
- Strong background in CI/CD Infrastructure-as-Code and Data Modeling.
Preferred Skills
- Experience with retail or supply chain data forecasting.
- Expertise in optimizing serverless compute environments.
- Experience maintaining production SLAs for customer-facing data dependencies.
Required Experience:
Manager
Key Skills
About Company
Upshop delivers total supermarket and grocery store operations by connecting Fresh, Center, eCommerce, and DSD department operations.
