Data Engineer

Role Overview

We are looking for experienced Data Engineers to design develop and maintain scalable data pipelines and data infrastructure supporting Data Science Machine Learning analytics and business applications.

Key Responsibilities

Design develop and maintain scalable ETL/ELT data pipelines.

Integrate data from multiple internal and external data sources.

Develop data transformation and processing workflows.

Build and optimize SQL queries and data models.

Ensure data quality consistency availability and reliability.

Collaborate with Data Scientists and ML Engineers to provide high-quality datasets.

Develop and maintain data ingestion and processing workflows.

Troubleshoot and optimize data pipelines and processing jobs.

Contribute to data architecture and cloud-based data solutions.

Implement appropriate data security governance and quality practices.

Document data pipelines processes and technical solutions.

Required Technical Skills

Strong proficiency in Python.

Strong proficiency in SQL.

Hands-on experience with ETL/ELT pipelines.

Good understanding of relational and non-relational databases.

Experience with data warehousing and data modeling.

Knowledge of Spark/PySpark.

Experience with workflow orchestration tools such as Airflow or equivalent.

Exposure to AWS Azure or GCP.

Understanding of distributed data processing and scalable data architectures.

Preferred Skills

Cloud data platforms.

Databricks or equivalent platforms.

Kafka or other data-streaming technologies.

CI/CD and DevOps practices.

Experience working with large-scale datasets.

Required Skills:

PythonSQLETL/ELTSpark/PySparkAirflowAWSAzureGCPData modelingData qualityData governanceDistributed systemsCI/CD