Data Engineering Operations Lead

TransUnions Job Applicant Privacy Notice

What Well Bring:

What Youll Bring:

Key Responsibilities:

• Own and manage daily Data Engineering operations including data ingestion workflows pipeline monitoring and incident resolution.
• Oversee end-to-end pipeline health - proactively monitor triage and resolve failures bottlenecks and data quality issues across all ingestion layers.
• Organize and prioritize the teams daily workload - assign tasks track progress and remove blockers to ensure smooth operational delivery.
• Maintain and improve ingestion pipelines from various data sources into data lakes warehouses and real-time streaming systems.
• Act as the first point of escalation for pipeline failures SLA breaches and data anomalies - driving root cause analysis and permanent fixes.
• Define and enforce operational standards - runbooks alerting thresholds on-call procedures and incident management practices.
• Coordinate with upstream data providers and downstream consumers to manage dependencies and communicate pipeline status.
• Lead daily stand-ups sprint planning and work organization for the Data Engineering Ops function under the wider Data Engineering practice
• Mentor and guide junior engineers on operational best practices debugging and pipeline development.
• Collaborate with data scientists analysts and product partners to onboard new data sources and meet ingestion SLAs.
• Drive continuous improvement in pipeline reliability observability and efficiency.
• Implement best practices in data governance data quality monitoring and compliance across all pipelines.
• Design build test and deploy Data solutions at scale including data lakes data warehouses and real-time analytics.
• Lead technical delivery on use cases plan and delegate tasks to junior team members and oversee work from inception to final product.
  

Skills & Experience:

Essential:

• Bachelors degree in Computer Science Engineering Statistics or a related field
• Minimum 8 years of data engineering experience (ideally 10) with at least 3 years in senior/lead roles.
• 6 years of experience in Big Data technologies(e.g. Spark Hive Hadoop Databricks).
• Advanced proficiency with Apache Spark including PySpark and SparkSQL tuning and performance optimisation experience is fundamental.
• Proficiency in Python Pandas (Scala/Java knowledge is desirable).
• Working knowledge of Apache Hive.
• Strong SQL knowledge and experience(T-SQL working with SQL Server SSMS).
• Expertise in designing and implementing scalable data pipelines and ETL processesusing the GCP data stack including BigQuery Dataflow Pub/Sub Cloud Storage Cloud Composer Cloud Functions Dataproc (Spark).
• Excellent knowledge of data engineering concepts and best practices.
• Proven ability to lead mentor inspire and support junior team members.
• Ability to lead technical deliverables autonomouslyand guide junior data engineers.
• Ability to organize and manage daily engineering workload - task assignment prioritization and delivery tracking.
• Strong attention to detail and adherence to best practices.
• Experience with batch real-time streaming and ETL processes including incident resolution and pipeline recovery.
• Experience building and managing ETL workflows using Apache Airflow including DAG creation scheduling and error handling.
• Source control with Git.
• Knowledge of CI/CD conceptsand experience designing CI/CD for data pipelines.
• Experience designing logical data models and physical data models including data warehouse and data mart designs.
• Knowledge of Delta Lake concepts and common data formats Lakehouse architecture.
• Software engineering principles including OOP design patterns SDLC Agile TDD and performance optimization.
  
  

Desirable:

• Relevant certifications(e.g. Google Cloud Professional Data Engineer).
• Experience with streaming services such as Kafkais a plus.
• R & Sparklyr experienceis a plus.
• Knowledge of MLOps concepts AI/ML lifecycle management and MLflow.
• Harness experienceis a plus.

Impact Youll Make:

Impact Youll Make:

TransUnion a place to grow:

We know that it is unrealistic to expect candidates to have each and every aspect of the essential and/or desirable skills listed above if there is something you cant tick off right now good you can learn here!

Impact you will make:

Enable Decision Making across the organization using data driven culture.

Whats In It for you
At TransUnion you will be joining a friendly forward thinking global business.

As well as a competitive salary & bonus scheme our benefits package includes up 26 days annual leave (plus bank holidays) a generous contributory pension scheme private health care and a host of other employee lifestyle benefits.

That is in addition to a variety of physical mental and financial fitness wellbeing programmes such as lunchtime yoga boxing classes mindfulness app access daily dedicated away from keyboard time to ensure colleagues take a break and our diversity forums and networking groups.

This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week.

We are committed to being a place where diversity is not only present but also embraced. As an equal opportunity employer all qualified applicants will receive consideration for employment without regard to race colour religion sex national origin age disability status veteran status genetic information marital status citizenship status sexual orientation gender identity or any other characteristic protected by law.

TransUnion Job Title