Data Engineer

Durban - South Africa

Monthly Salary: Not Disclosed

Posted on: 30+ days ago

Vacancies: 1 Vacancy

Job Summary

Our client is seeking an experienced and passionate Data Engineer to design and build robust data infrastructure that facilitates large data volumes. The role requires a blend of database admin skills (work with and manage relational databases) and data engineering (taking data from various datasets and collections to clean store transform and analyse). You will work closely with business analysts and software engineers to ensure efficient data pipelines quality data sources and scalable solutions that support key business decisions and innovation.

Responsibilities:

Assembling large complex sets of data that meet non-functional and functional business requirements
Design develop monitor and maintain scalable data pipelines and ETL processes
Build required infrastructure for optimal extraction transformation and loading of data from various data sources using Integration and SQL technologies often these are Cloud based
Identifying designing and implementing internal process improvements including redesigning infrastructure for greater scalability optimizing data delivery and automating manual processes
Building analytical tools to utilise the data pipeline providing actionable insight into key business performance metrics
Ensure data quality consistency integrity and security across all systems
Drive continuous improvement of data engineering best practices and tooling

Required Skills and Experience:

Bachelors Degree in Computer Science Engineering Mathematics or related field
5-7 years of experience in a database management data engineering or similar role
Proficiency in programming languages such as Python or Scala
Strong proficiency in SQL and experience with relational databases (e.g. MSSQL PostgreSQL MySQL)
Hands-on experience with No-SQL database technologies
Experience in database optimization and performance tuning required
Good understanding of Data Integration patterns
Exposure to BI tooling such as Power BI or Yellowfin is advantageous
Experience setting up MS SQL Replication and Data Archiving strategies will be beneficial
Experience with cloud platforms (e.g. AWS GCP Azure) and services like S3 Lambda Redshift BigQuery or Snowflake
Familiarity with big data technologies like Apache Spark Data Bricks and Hive
Familiarity with data modelling warehousing concepts and data governance practices
Exposure to Data cleansing and de-duplication techniques will be beneficial

Advantageous:

Experience with stream processing tools (e.g. Kafka Spark Streaming Flink)
Knowledge of containerization (Docker) and orchestration tools (Kubernetes)
Understanding of CI/CD principles and infrastructure-as-code
Exposure to machine learning workflows and MLOps

Please send your CV to emailprotected or for more information contact us on (VoIP) or .

In sending your CV you confirm that you have read and understood our POPI Policy found on our website . Should you be unsuccessful for this particular position you have no objection to us retaining your personal information in our database which you confirm is true correct and up to date. Should a suitable opportunity arise in the future we will contact you and request your permission to submit your information.

Responsibilities:

Assembling large complex sets of data that meet non-functional and functional business requirements
Design develop monitor and maintain scalable data pipelines and ETL processes
Build required infrastructure for optimal extraction transformation and loading of data from various data sources using Integration and SQL technologies often these are Cloud based
Identifying designing and implementing internal process improvements including redesigning infrastructure for greater scalability optimizing data delivery and automating manual processes
Building analytical tools to utilise the data pipeline providing actionable insight into key business performance metrics
Ensure data quality consistency integrity and security across all systems
Drive continuous improvement of data engineering best practices and tooling

Required Skills and Experience:

Bachelors Degree in Computer Science Engineering Mathematics or related field
5-7 years of experience in a database management data engineering or similar role
Proficiency in programming languages such as Python or Scala
Strong proficiency in SQL and experience with relational databases (e.g. MSSQL PostgreSQL MySQL)
Hands-on experience with No-SQL database technologies
Experience in database optimization and performance tuning required
Good understanding of Data Integration patterns
Exposure to BI tooling such as Power BI or Yellowfin is advantageous
Experience setting up MS SQL Replication and Data Archiving strategies will be beneficial
Experience with cloud platforms (e.g. AWS GCP Azure) and services like S3 Lambda Redshift BigQuery or Snowflake
Familiarity with big data technologies like Apache Spark Data Bricks and Hive
Familiarity with data modelling warehousing concepts and data governance practices
Exposure to Data cleansing and de-duplication techniques will be beneficial

Advantageous: