Data Engineer/ Scientist (1595) KG
Data Engineer/ Scientist (1595) KG
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Introduction
Job Description
WHAT WILL BE YOUR ROLE AND RESPONSIBILITIES
- Design develop and maintain data ingestion pipelines for companyconnected fleet focusing on Call Detail Records (CDRs).
- Utilise AWS Glue for ETL processes to transform and load data into the analytics platform.
- Implement efficient querying solutions using AWS Athena.
- Develop serverless applications and workflows using AWS Lambda.
- Monitor and optimise data pipelines using AWS CloudWatch.
- Manage and maintain PostgreSQL RDS databases for data storage and retrieval.
- Create interactive dashboards and reports using AWS QuickSight.
- Leverage Terraform extensively to define deploy and manage AWS infrastructure as code ensuring scalability and reliability.
- Use Git and GitHub for version control collaboration and maintaining code repositories.
- Implement and manage GitHub CI/CD pipelines to automate testing deployment and delivery processes.
- Ensure data security and compliance with companyinformation classification requirements.
- Stay updated with the latest data engineering tools technologies and industry trends.
- Identify opportunities for process improvements and automation to enhance pipeline efficiency.
- Collaborate with crossfunctional teams to understand data requirements and deliver solutions.
- Develop and maintain technical documentation for data engineering processes and solutions.
Minimum Requirements
WHICH QUALIFICATIONS/EXPERIENCE DO WE NEED FOR THE ROLE
- Bachelors or Masters degree in Computer Science Software Engineering or a related field.
- Minimum of 3 years experience as a Data Engineer.
- At least 2 years of experience working with AWS services.
- Proven experience in building and maintaining data pipelines for largescale datasets.
- Agile working experience
ESSENTIAL SKILLS REQUIREMENTS:
- Proficiency in Python 3.x for data processing and automation.
- Experience with AWS Glue for ETL processes.
- Strong knowledge of AWS Athena for querying large datasets.
- Handson experience with AWS Lambda for serverless computing.
- Familiarity with AWS EC2 for scalable computing resources.
- Expertise in AWS CloudWatch for monitoring and logging.
- Proficiency in working with PostgreSQL RDS for database management.
- Experience with AWS QuickSight for data visualisation and reporting.
- Strong understanding of data ingestion pipelines particularly for Call Detail Records (CDRs).
- Proficiency in Git and GitHub for version control and collaboration including experience with GitHub CI/CD pipelines.
- Any additional responsibilities assigned in the Agile Working Model (AWM) Charter
ADVANTAGEOUS SKILLS REQUIREMENTS:
- Extensive experience with Terraform for infrastructure as code including creating managing and optimising AWS resources.
- Experience with other AWS services such as S3 Kinesis and DynamoDB.
- Knowledge of data formats such as Parquet AVRO JSON and CSV.
- Experience with Docker for containerisation.
- Understanding of Big Data technologies and frameworks.
- Familiarity with Agile working models and tools like JIRA and Confluence.
- Experience with data quality tools such as Great Expectations.
- Knowledge of REST API development and integration.
- Strong analytical skills for troubleshooting and optimising data pipelines.
- Experience in developing technical documentation and artefacts.
Package & Remuneration
Salary Market Related
Employment Type
Contract
