Data Engineer

Job Role: Data Engineer

Role Overview and Expected Deliverables

The engineer will design build and operate scalable secure and analytics ready data pipelines and models across GCP BigQuery and AWS delivering:
- Airflow orchestrated data workflows and automations.
- DBT Cloud transformations testing frameworks and environment governance.
- Robust CI/CD and Infrastructure as Code (IaC) deployments (GitHub Actions Bitbucket Bamboo Terraform or CloudFormation).
- Performance optimisation across data pipelines SQL BigQuery and AWS data services.
- High quality technical documentation production support incident response and handover to internal teams.
- Close collaboration with cross functional stakeholders including analytics product and platform engineering.

Core Technical Requirements

Cloud & Platform Engineering:
- Strong proficiency across AWS and GCP.
- On AWS: IAM roles and policies Lambda S3 RDS Redshift.
- On GCP: BigQuery workload optimisation cost management.

Data Engineering Tooling:
- Hands on experience building data pipelines for data warehouse environments.
- Strong SQL capability including analytical and window functions.
- Advanced dbt Cloud experience (models SCDs tests macros release management).
- Airflow workflow design including idempotency backfills and monitoring.

DevOps CI/CD & IaC:
- Experience with CI/CD (GitHub Bitbucket GitHub Actions Bamboo).
- Infrastructure as Code using Terraform or CloudFormation.
- Familiarity with data quality frameworks and automated testing of pipelines.

Programming & Modelling:
- Proficiency in Python.
- Strong data modelling expertise: facts/dimensions SCD patterns performance optimisation.

Security Data Governance & Operational Resilience:
- Solid grounding in cloud security best practices and least privilege IAM.
- Understanding of data access controls regulated environments and governance expectations.

Delivery Approach & Ways of Working

- Experience working within Agile delivery frameworks.
- Use of JIRA and Confluence for delivery management and documentation.
- Proven track record collaborating effectively in multi disciplinary teams across engineering product and analytics.

Specialisations

- BigQuery performance tuning and cost optimisation.
- DBT Cloud governance (contracts tests macros release processes).
- Airflow workflow optimisation and monitoring.
- Cross cloud pipelines and data movement (AWS GCP).
- Security IAM and data access design for regulated financial services environments.

Experience & Other

- 8 or more years experience in data engineering or equivalent technical roles.
- Strong problem solving capability comfortable troubleshooting pipeline failures and performance issues.
- Strong communication and stakeholder management skills.
- Self starter able to lead components independently while partnering closely with internal teams.
- Experience supporting production environments including incident response and root cause analysis.
- All intellectual property rights retained by client