Lead Data Engineer ETL

We areseekinga highly skilled and experiencedData Engineering Leadwith strongexpertisein AWS data services and retail data this role you will lead the design development and optimization of scalable data pipelines responsible for ingesting and transforming data fromMMS (Merchandise Management Systems)andPOS (Point of Sale)systems into a centralizedOperational Data Store (ODS)to support downstream applications and analytics use cases.

You will play a critical role in building a robust high-performance data platform using AWS-native services ensuring data quality reliability and real-time or near real-time availability for business operations.

Responsibilities:-

• Design and implement scalableETL/ELT pipelinesto ingest data fromMMS / POS or third practice systemsinto AWS-based data platforms.

• Build andmaintaina centralizedOperational Data Store (ODS)usingCouchbaseor similarNoSQL technologiesoptimizedfor low-latency application access.

• Develop andoptimizedata processing workflows using ApacheSpark /PySparkand AWS services such as AWS Glue and Amazon EMR.

• Createdenormalized API-ready data modelsaligned with downstream microservices and application consumption patterns.

• Implementidempotent processing CDC merge (upsert) strategies and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.

• Leverage Amazon S3 Amazon Redshift or Amazon Aurora for efficient storage and querying of structured and semi-structured data.

• Implement data ingestion patterns (batch and streaming) using tools like Amazon Kinesis or AWS Lambda where applicable.

• Apply performance tuning and optimization techniques to improve pipeline efficiency scalability and cost-effectiveness.

• Define and enforcedata governance data quality and metadata managementstandards across the data platform.

• Collaborate with DevOps teams to design and maintainCI/CD pipelinesusing tools like AWSCodePipeline GitLab or similar.

• Conductpeer reviewsand provide technical leadership and mentorship to the data engineering team.

• Collaborate withmicroservices and application teamsto ensure seamless integration with the ODS via APIs and event streams.

Requirements:-

• 5 years of experience indata engineering with at least 2 years in alead role.

• Strong hands-on experience with AWS data services such as AWS Glue Amazon S3 Amazon Redshift and Amazon EMR.

• ProficiencyinPySpark/ Sparkfor large-scale data processing and optimization.

• Experience designing and implementingODS layers using NoSQL databases preferablyCouchbase or similar (DynamoDB MongoDB).

• StrongexpertiseinETL/ELT design patterns data ingestion and transformation pipelines.

• Experience working withMMS POS or retail transaction datais highly preferred.

• Hands-on experience withCI/CD pipelines(GitLab AWSCodePipeline or similar).

• Good understanding ofdata governance data quality and metadata frameworks.

• Experience supportingmicroservicesarchitectureswith data platforms.

• Strong problem-solving skills andabilitytooptimizecomplex data workflows.

• Excellent communication and stakeholder management skills.

Ability to work in a fast-paced agile environment and manage multiple priorities.