Data Engineer

Anticipated Contract End Date/Length: December 31 2026
Work set up: Hybrid (23 days per week in office)
Clearance required: BPSS checks. Candidates must be eligible for BPSS clearance

Our client in the Information Technology and Services industry is looking for a Data Engineer with strong Spark Scala and Python skills to develop maintain and troubleshoot data orchestration workflows and reusable engineering components. The role will support the processing transformation validation and delivery of large-scale financial datasets while contributing to platform engineering production support monitoring automation and operational resilience. The successful candidate will work across application platform database infrastructure and downstream teams to maintain reliable data processes and ensure compliance with technology security and change standards.

Responsibilities

• Develop maintain and troubleshoot data orchestration workflows using Apache Airflow for extract generation data publishing feed delivery and operational batch processes.
• Build and support reusable engineering components using Python Scala and Sharedview Spark to process transform and validate large-scale financial datasets.
• Design write and optimize SQL/PostgreSQL and Starburst/Trino queries for metadata processing data validation reconciliation and operational investigation.
• Support Output Gateway production issues including failed DAGs delayed extracts missing files data mismatches checksum failures logs status updates and downstream delivery exceptions.
• Contribute to platform engineering and release activities using GitLab Docker and controlled CI/CD practices across DEV UAT and production-like environments.
• Enhance monitoring logging alerting audit tracking and automation to improve operational stability supportability and evidence for control processes.
• Work with application platform database infrastructure and downstream teams to diagnose issues end-to-end and implement safe controlled fixes.
• Maintain a strong risk and control mindset by ensuring changes are tested reviewed documented and released in accordance with technology security and change standards.

Qualifications :

• Strong skills and experience in data engineering using Spark Scala and Python.
• Experience developing maintaining and troubleshooting Apache Airflow workflows for data orchestration and operational batch processes.
• Hands-on experience building reusable engineering components to process transform and validate large-scale financial datasets.
• Proficiency in SQL and PostgreSQL including query design optimization metadata processing data validation and reconciliation.
• Experience supporting production data workflows and troubleshooting failed DAGs delayed extracts missing files data mismatches checksum failures and downstream delivery exceptions.
• Familiarity with GitLab Docker and controlled CI/CD practices across development UAT and production-like environments.
• Understanding of monitoring logging alerting audit tracking and automation to improve operational stability and supportability.
• Ability to collaborate with application platform database infrastructure and downstream teams to diagnose and resolve technical issues.
• Strong awareness of risk controls testing documentation and technology security and change standards.

Desirable Skills / Preferred Qualifications

• Experience with Starburst/Trino AWS S3 concepts and distributed query behavior for large-dataset validation and troubleshooting.
• Exposure to React JS or API-driven operational user interfaces for configuration monitoring and support workflows.
• Familiarity with OpenShift Kubernetes Helm or container platform deployments in enterprise environments.
• Understanding of financial data controls auditability reconciliation data lineage operational resilience and data quality expectations.

Additional Information :

All your information will be kept confidential according to EEO guidelines.

Candidates must be legally authorized to live and work in the country where the position is based without requiring employer sponsorship.

HelloKindred is committed to fair transparent and inclusive hiring practices. We assess candidates based on skills experience and role-related requirements.

We appreciate your interest in this opportunity. While we review every application carefully only candidates selected for an interview will be contacted.

HelloKindred is an equal opportunity employer. We welcome applicants of all backgrounds and do not discriminate on the basis of race colour religion sex gender identity or expression sexual orientation age national origin disability veteran status or any other protected characteristic under applicable law.

Remote Work :

No

Employment Type :

Contract