Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
Vacancies:
Job Summary
The Data Engineering team is responsible for a fundamental data system processing 50 billion rows of data per day and feeding directly into trading decisions. The Data Engineer will be designing implementing and maintaining this system keeping it reliable resilient and low-latency.
We are looking for a highly technical engineer who is highly technical with strong experience working in MPP platforms and/or Spark big data (e.g. weather forecasts AIS pings satellite imagery ) and developing resilient and reliable data pipelines. You will be responsible for data pipelines end to end: acquisition loading transformation implementing business rules/analytics and delivery to the end user (trading desks / data science / AI).
You will partner closely with business stakeholders and engineering teams to understand their data requirements and deliver the necessary data infrastructure to support their activities. Given our scale you should bring a deep focus on performance optimisation improving data access times and reducing latency.
This role will require strong coding skills in SQL and Python and a deep understanding on how to leverage the AWS stack.
Strong communication is essential: you should be comfortable translating technical concepts to non-technical users as well as turning business requirements into clear actionable technical designs.
Qualifications :
Essential
- 5 years in the data engineering space
- Proficient with MPP Databases (Snowflake Redshift Big Query Azure DW) and/or Apache Spark
- Proficient at building resilient data pipelines for large datasets
- Deep AWS or cloud understanding across core and extended services.
- 2 years experience working with at least 3 of the following: ECS EKS Lambda DynamoDB Kinesis AWS Batch ElasticSearch/OpenSearch EMR Athena Docker/Kubernetes
- Proficient with Python and SQL and with good experience with data modelling
- Experience with a modern orchestration tools (Airflow / Prefect / Dagster / similar)
- Comfortable working in a dynamic environment with evolving requirements
Desirable
- Exposure to trading and/or commodity business
- Snowflake experience
- DBT experience
- Infrastructure as Code (Terraform Cloud Formation Ansible Serverless)
- CI/CD Pipelines (Jenkins / GIT / BitBucket Pipelines / similar)
- Database/SQL tuning skills
- Basic data science concepts
Remote Work :
No
Employment Type :
Full-time
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
We are a leader in the energy sector with a presence across the spectrum; from oil through to power, renewables and carbon credits. Every day we use our expertise to distribute energy around the world. We source from producers, refiners and intermediaries and deliver to refineries, ut ... View more
