Data Engineer

Design develop and maintain scalable and reliable data platforms and pipelines for analytics and operational use cases
Build distributed data processing workflows using Apache Spark pySpark and Databricks
Develop and evolve cloud-native data solutions leveraging Azure and AWS services
Model and optimize data structures for analytics reporting and downstream consumption
Integrate data from multiple heterogeneous sources while ensuring data quality consistency and traceability
Implement data quality controls schema validation lineage tracking and monitoring processes
Optimize performance scalability reliability and cost-efficiency of data pipelines and infrastructure
Collaborate closely with scientists analysts and engineering teams to translate business and research requirements into technical solutions
Contribute to CI/CD processes infrastructure automation and engineering best practices
Research and introduce new technologies and approaches within the data engineering ecosystem

Qualifications :

3 years of commercial experience in Data Engineering
2 years of hands-on production experience with Databricks
Strong expertise with Delta Lake including medallion architecture and table optimization
Advanced proficiency in Python pySpark and SQL
Deep practical knowledge of Databricks components including Unity Catalog Lakeflow Jobs / Workflows and job optimization
Hands-on experience with AWS services including S3 IAM KMS and at least one of AWS Batch ECR EventBridge Lambda or API Gateway
Experience designing event-driven and asynchronous data pipelines
Strong knowledge of Infrastructure as Code and CI/CD practices
Practical experience with Terraform and Databricks Asset Bundles
Experience working with Git workflows pull request reviews and automated testing
Understanding of secure authentication approaches such as OIDC and service principals
Practical experience with MLflow experiment tracking model registry and reproducibility practices
Strong understanding of data quality lineage schema contracts and provenance concepts
Experience with data quality frameworks such as Great Expectations Soda pyDeequ or Databricks DQX
Experience monitoring and optimizing data platform reliability and cloud costs
Upper-Intermediate or higher English level

WILL BE A PLUS:

Experience packaging workloads using Docker and running batch workloads in AWS
Familiarity with ECR and containerized data processing solutions
Experience with experiment optimization libraries such as scikit-optimize BoTorch Ax or scipy
Understanding of ML feature engineering and leakage prevention techniques
Experience working in regulated or high-security environments
Knowledge of data governance compliance and data residency requirements
Experience disabling or restricting AI-assisted tooling/features in secure environments

Additional Information :

Remote Work :

Yes

Employment Type :

Full-time