Data Engineering Leadership
Job Location:
Monthly Salary:
Posted on:
Yesterday
Posted on:
Vacancies:
1 Vacancy
Vacancies:
Department:
Job Summary
The search
Our client is seeking multiple technical leaders to contribute at both the Staff and Lead Engineer level. These are high-visibility high-impact opportunities with a talent-dense homegrown bootstrapped globally competitive Canadian tech success story. The company finds itself squarely in the middle of the disruption of search by AI/LLMs the fragmentation of the media landscape and in a chapter of rapid transformational growth.
The shared mandate is to modernize how massive volumes of data are processed modeled and made available to products and customers. The work spans petabyte-scale batch and streaming pipelines low-latency APIs analytical platforms and the architecture that connects raw event data to reliable production-grade data products.
Staff Engineer candidates may bring a stronger centre of gravity in backend and API systems or in data processing and platform engineering. For the Lead Engineer search the client is seeking a hands-on technical and people leader with the affinity and aptitude to guide a distributed team through transformation.
The client
Our client builds a high throughput system powering hundreds of billions of daily transactions each completed within milliseconds across globally distributed infrastructure designed for reliability and efficiency. They have been quietly bootstrapping and growing in line with revenue for over 2 decades. They operate at a planetary scale with an exchange platform with thousands of nodes that handles nearly 500 billion daily auctions and is trending towards 1 trillion daily auctions in the next 4 years. To the present day the company has not raised money from venture capitalists. It maintains its independence from external stakeholders enabling it to chart its course maintain a long-term perspective and build an enduring sustainable business that currently employs 600 team members globally.
Data is central to nearly every part of its platform. High-volume event and transactional data powers customer reporting billing marketplace analytics experimentation machine-learning workflows and product experiences. The organization is evolving from centralized batch-oriented reporting toward a platform-driven architecture combining batch streaming and asynchronous processing with APIs becoming a primary interface to data.
This is not conventional business-intelligence work. The challenge is to process and serve enormous datasets with strong correctness guarantees predictable latency and disciplined infrastructure economics. Engineers work across the full path from edge and transport through compute storage query execution and serving balancing latency cost complexity and reliability at every layer.
Our client has been stubbornly racking and stacking infrastructure around the world for the duration of their existence a habit that allows for them to price themselves at the cost of electricity while their competitors are mired in rising cloud infrastructure costs. This long-term somewhat contrarian thinking puts them in a position to offer stability and career longevity evidenced by robust benefits that include RRSP matching.
If youre looking for a role that offers an opportunity to innovate and optimize systems at a massive scale creating a lasting impact in an environment that values technical excellence and resilience this could be for you.
Role overviews
In our clients context both Staff Engineers and Leads remain hands-on. This is an intentional and informed cultural practice tailored to the challenge ahead of them. Leads are expected to understand the systems deeply make informed architectural decisions and help their teams execute.
As a Staff Engineer you will be a primary owner and technical leader for data-intensive systems spanning processing platform and product-facing services. You will drive cross-team architecture define reusable platform patterns lead complex multi-system initiatives and influence the long-term evolution of data processing and access across the organization. Depending on your background your work may lean toward one or both of the following domains:
Data products and APIs: This team builds and operates high-scale services exposing aggregated and near real-time data for reporting analytics and customer-facing products; shaping API contracts query abstraction and execution caching response design performance scalability and reliability. This team operates as a full-stack data engineering group owning the path from data generation and aggregation through query execution APIs and user-facing delivery. On this team you can expect to:
- Design high-scale APIs for multidimensional reporting and analytical workloads
- Define contracts for metrics dimensions filters schemas and aggregations
- Improve dynamic query generation caching response shaping and execution performance
- Connect APIs to batch streaming and near real-time data sources
- Build primarily in Go or a comparable backend language such as Java
- Improve observability incident response reliability and consumer experience
- Lead initiatives involving Product Data Platform ML and Application Engineering
Data processing and platform: This team builds and evolves large-scale batch and streaming pipelines that transform raw event data into clean canonical business-ready datasets; improving data models workflow orchestration correctness observability runtime and compute efficiency. On this team you can expect to:
- Build large-scale Spark pipelines and streaming systems using Kafka and Flink
- Transform raw event data into canonical production-grade datasets
- Design partitioning storage aggregation and query strategies for enormous datasets
- Support historical backfills incremental processing and real-time availability
- Help decompose centralized processing into domain-oriented streams and workloads
- Move appropriate services and processing patterns from Hadoop toward Kubernetes
- Improve data quality correctness observability throughput and compute efficiency
- Participate in the long-term design of globally distributed data infrastructure
As a Lead Engineer you will lead a distributed team of data and software engineers responsible for high-scale low-latency data processing pipelines analytical platforms and reporting practice this means:
- Set the technical direction for the teams products systems and architectures while remaining comfortable working directly with the code and technology
- Hire mentor and develop engineers creating an environment of accountability collaboration and meaningful technical growth
- Partner with Product to establish roadmaps goals forecasts and measurable technical and business outcomes
- Manage scope and sequencing across more opportunities than available resources communicating progress and risks early and coordinating dependencies across teams
- Own the quality health and outcomes of the teams platforms including operational response technical debt reliability and continuous improvement
The shared mandate
- Platform architecture: Define and evolve patterns across edge transport compute storage modeling query execution and serving
- Batch and streaming systems: Build systems supporting historical processing backfills incremental updates and near real-time availability without creating unnecessary architectural complexity
- Data products and interfaces: Provide stable flexible access to metrics dimensions filters aggregations and canonical datasets through well-designed APIs and analytical platforms
- Performance and economics: Optimize query planning data layout partitioning caching execution throughput and infrastructure use so capacity does not need to grow linearly with data volume
- Correctness and trust: Design for deduplication late-arriving data schema evolution data contracts reconciliation and the integrity of business-critical reporting and billing datasets
- Reliability and observability: Own monitoring alerting incident response root-cause analysis workflow health and the mechanisms that make system behaviour visible
- Cross-functional influence: Work across Data Engineering Data Systems Application Engineering Product analytics machine learning and experimentation teams to turn platform capabilities into business outcomes
Tech stack
- Processing and streaming: Spark Scala Kafka Flink Airflow
- Data services and APIs: Go Java REST gRPC SQL
- Compute and storage: Kubernetes Hadoop/HDFS Ceph
- Query and analytics: Trino Vertica Druid and StarRocks
- Reporting and BI: Looker Superset Redash
The precise mix will vary by team and assignment. Depth in comparable technologies and the ability to reason from first principles matter more than matching every tool.
Key responsibilities
- Vision and strategic direction: Identify high-leverage technical opportunities and translate them into a coherent direction for the platform and business
- System design and delivery: Lead the design and implementation of distributed data systems operating under heavy workloads and demanding latency requirements
- Data platform evolution: Help move the organization from centralized batch-oriented systems toward workload-aware architectures combining batch streaming and asynchronous processing
- API and query architecture: Define durable interfaces schemas query abstractions and serving patterns for flexible access to large-scale multidimensional data
- Pipeline and model ownership: Build canonical datasets and processing workflows supporting reporting billing products analytics experimentation and machine-learning use cases
- Operational excellence: Improve observability service health incident response workflow reliability technical debt management and post-incident learning
- Organizational leadership: Coordinate across teams unblock dependencies communicate trade-offs clearly and build alignment around technical and product outcomes
Your know-how
- You have significant experience designing building and operating data-intensive systems at scale
- You have a deep understanding of distributed-systems fundamentals including partitioning consistency failure modes state throughput latency and cost trade-offs
- You have experience with large-scale data processing and modeling (the client leans on Spark currently)
- You have experience with streaming technologies such as Kafka and Flink including incremental processing late data deduplication and replay
- You have experience designing stable data contracts schemas canonical datasets and transformation layers for business-critical use cases
- You have experience with large-scale storage data warehouse or analytical query systems and a strong grasp of query performance data layout and workload-aware optimization
- You have experience with workflow orchestration and the production deployment or operation of services on Kubernetes
- You have a track record of end-to-end ownership: navigating ambiguity debugging complex cross-system issues making sound trade-offs and improving systems after they enter production
- You have an excellent command of English and experience collaborating with neighbouring engineering disciplines and stakeholders beyond R&D
For API-oriented work you will call backend engineering skills (ideally but not necessarily with Go) and experience with REST/gRPC design versioning performance and high-concurrency services.
Additional know-how for the Lead Engineer roles
- You have 3 years of experience leading and managing distributed teams
- You have an affinity for mentorship and creating an environment that nurtures a balance of innovation and accountability
- You have experience partnering with Product Management and internal stakeholders to shape roadmaps forecasts deliverables timelines and measurable outcomes
- You are experienced making tradeoffs and experience exercising judgment to prioritize a broad portfolio when initiatives outnumber resources including thoughtful sequencing and proactive communication of progress delays and risk
Interested in learning more
If you are interested in building data systems at a scale few Canadian companies can offer while helping shape a consequential platform transformation we would love to hear from you. Please send your resume or LinkedIn profile to with Data Engineering as the subject or apply using the following link. One of our partners will be in touch shortly!
Required Experience:
IC
