Senior Data Engineer
Senior Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Duties & Responsibilities
Our client in the financial industry is looking for a skilled and experienced Senior Data Engineer on a 12 month contract in Cape Town to build and maintain their data infrastructure on the AWS cloud.
The successful candidate will be responsible for designing developing and deploying robust and scalable data pipelines ensuring data quality and reliability for their financial services and/or healthcare operations. You will also have strong programming skills expertise in data warehousing and ETL/ELT processes and handson experience with AWS data services.
Responsibilities:
- Design build and maintain scalable and reliable data pipelines using AWS services (e.g. Glue Step Functions Data Pipeline Kinesis) and other relevant technologies.
- Develop and implement ETL/ELT processes to transform and load data from various sources into our data warehouse and data lake on AWS (e.g. Redshift S3.
- Ensure data quality integrity and performance throughout the data lifecycle.
- Implement data monitoring and alerting systems to proactively identify and resolve data pipeline issues.
- Collaborate with Data Architects to implement data models and architectural designs on the AWS platform.
- Work closely with Data Analysts to understand their data needs and provide them with clean and accessible data.
- Automate datarelated tasks and processes to improve efficiency and reduce manual effort.
- Optimize data storage and processing solutions for performance and costeffectiveness on AWS.
- Implement and maintain data security and compliance measures in accordance with industry regulations (e.g. HIPAA GDPR POPIA).
- Develop and maintain comprehensive documentation for data pipelines and infrastructure.
- Troubleshoot and resolve datarelated issues in a timely manner.
- Stay uptodate with the latest data engineering tools and technologies on the AWS platform.
- Mentor and guide junior data engineers as needed.
NonNegotiables:
- A bachelors degree in computer science Software Engineering or a related technical field.
- 58 years of experience as a Data Engineer with a strong focus on building data pipelines on the AWS cloud.
- Proficiency in at least one programming language such as Python Scala or Java.
- Strong experience with SQL and working with relational databases (e.g. PostgreSQL MySQL SQL Server).
- Handson experience with AWS data services such as S3 Redshift Glue Athena Kinesis Lambda and RDS.
- Experience with ETL/ELT tools and frameworks.
- Understanding of data warehousing concepts and best practices.
- Experience with data pipeline orchestration tools (e.g. Apache Airflow AWS Step Functions).
- Familiarity with data streaming technologies (e.g. Kafka Kinesis Streams) is a plus.
- Experience with infrastructureascode (IaC) tools like Terraform or CloudFormation is a plus.
- Excellent communication and collaboration skills.
- Familiarity with healthcare data standards (e.g. HL7 FHIR) or financial services regulations (e.g. GDPR CCPA) is a significant advantage (tailor based on the primary sector).
Required Experience:
Senior IC
Employment Type
Full-Time
