Lead Data Engineer

Overview:

Our client is one of the fastest-growing companies a leader in digital transformation Blockchain and AI technologies with global clients from the education healthcare and manufacturing industries. With aggressive plans to expand into many international markets they are looking to grow their team!

Responsibilities:

• Design and deliver cloud-native data solutions on AWS across ingestion storage processing orchestration serving and monitoring layers.
• Design develop and maintain robust ETL/ELT pipelines for multi-layer data ingestion.
• Create and manage scalable feature stores tailored for healthcare analytics.
• Collaborate closely with ML and analytics teams to operationalize ML workflows.
• Support production data platforms in terms of performance reliability and scalability.

Technical Requirements:

• 5 years of proven experience in data engineering building complex production-grade data pipelines and architecture.
• Extensive experience with the Python data ecosystem specifically Pandas and NumPy for large-scale data processing and transformation.
• Strong proficiency in SQL and relational database management including schema design and query optimization.
• Hands-on experience with AWS data and compute services including S3 Lambda and Batch for building scalable serverless data pipelines.
• Proficient with boto3 and AWS Wrangler (awswrangler) for programmatic interaction with AWS services and S3-based data lakes including partitioned data structures.
• Proficiency in Bash scripting and navigating Linux environments for automation and operational tooling.
• Skilled in code optimization including profiling memory-efficient processing and refactoring for performance in data workloads.
• A clear understanding of Data Quality Data Privacy and Data Governance principles.
• Working understanding of MLOps practices including model deployment monitoring versioning and integrating ML inference into production data pipelines.

Good to have:

• Previous experience with IoT Data
• Familiarity with privacy-preserving architecture and data compliance standards.