Data Engineer

POSITION GENERAL DUTIES AND TASKS :
Job Description: We are seeking a skilled Data Engineer to do the migration of our existing data warehouse and data model from MariaDB to the Hadoop Big Data Cloudera onpremise platform. The ideal candidate must be proficient in SQL Hive SQL Spark and data modeling. Additionally they should possess a strong understanding of the production deployment process including design development testing UAT and production deployment. Experience of scheduling jobs using Autosys is essential for this role.


Key Responsibilities
Migrate existing data warehouse and data model from MariaDB to Hadoop Big Data Cloudera onpremises platform.
Develop and optimize SQL Hive SQL and Spark scripts to ensure efficient data processing.
Design and implement data models to support business requirements and optimize performance.
Collaborate with crossfunctional teams to understand data requirements and ensure data integrity throughout the migration process.
Develop and execute test plans to validate data accuracy and system performance.
Coordinate with stakeholders (internal) to plan and execute production deployments.
Schedule and monitor jobs using Autosys to ensure timely execution and minimize downtime.
Provide technical expertise and guidance to team members throughout the migration project.
Document processes procedures technical specifications and best practices to facilitate knowledge sharing and ensure scalability.
Create and maintain unit test case documents to ensure code quality and reliability.


Qualifications
Bachelors (or Higher) degree in computer science Engineering or a related field.
Proven experience of 5 years in data engineering and migration projects for big data (Hortonworks / Cloudera)
Strong handson experience on Cloudera and related ecosystem components.
Strong experience in implementing ETL (Extract Transform Load) processes.
Highly proficient in SQL Hive SQL Spark and data modeling.
Strong understanding of the production deployment process.
Experience with scheduling jobs using Autosys or similar.
Experience with version control systems such as Bitbucket GIT etc.
Ability to troubleshoot and resolve data related issues efficiently.
Good communication and interpersonal skills
Experience with Dataiku is a plus but not mandatory.