Data Engineer (1744)

Role: AWS Data Engineer

Preferred Skills

.

ETL Development

.

AWS ETL Services

.

Data Pipeline Development

.

SQL & Python Programming

.

Data Warehousing

Experience: 4-8 years

Detailed Responsibilities & Skills

Must Have Technical Skills

Strong hands-on experience with:

.

AWS ETL Services (Glue Lambda Step Functions)

.

AWS AppFlow (mainly ODATA connector)

.

AWS Database Migration Service (DMS)

.

OData Services and API Integration

.

Amazon Redshift

.

Data Lake & Amazon S3

.

Python PySpark SQL Data Modeling

.

Data Modeling & ETL Development

.

Data Quality & Performance Optimization

.

Enterprise Data Integration (SAP Oracle Azure Oracle Cloud)

Key Responsibilities

.

Design develop and maintain scalable ETL pipelines using AWS Glue Lambda Step Functions and Redshift.

.

Build and orchestrate AWS workflows using Master Parent and Child Step Functions to execute Glue jobs and Redshift stored procedures.

.

Develop optimized Glue jobs using Python and PySpark ensuring efficient DPU utilization and performance tuning.

.

Design and implement data ingestion pipelines from multiple enterprise data sources including SAP Oracle Database Oracle Cloud Azure and other heterogeneous systems.

.

Capture change data / delta data to ingestion layer.

.

Store transform and manage data in Amazon S3 (Object Store Iceberg Tables) and Amazon Redshift.

.

Perform data cleansing transformation and validation to ensure high data quality and reliability.

.

Identify opportunities to improve ETL performance data quality and overall pipeline efficiency.

.

Design and implement data models including fact and dimension tables to support reporting and analytics.

.

Transform unstructured and semi-structured data into structured datasets suitable for downstream consumption.

.

Develop BI-ready datasets and understand KPI calculations across multiple business domains.

.

Collaborate with business and analytics teams to understand reporting requirements and translate them into technical solutions.

.

Troubleshoot production issues and support enhancements for existing data pipelines.

Additional Responsibilities

.

Design and develop data ingestion pipelines using AWS AppFlow AWS DMS OData services APIs and batch ingestion frameworks.

.

Integrate enterprise applications such as SAP (OData/CDS Services) Oracle Database Oracle Cloud Azure and other heterogeneous sources with AWS data platforms.

.

Configure and optimize AWS DMS for full load and Change Data Capture (CDC) migration scenarios.

.

Build and maintain AWS AppFlow integrations for secure and automated data movement between SaaS applications and AWS services.

.

Troubleshoot and optimize data ingestion performance across AppFlow DMS Glue and Step Functions.

Required Experience

.

4–8 years of experience in Data Engineering with strong AWS expertise.

.

Proven experience building enterprise-scale ETL pipelines on AWS.

.

Strong proficiency in Python PySpark and SQL.

.

Experience working with Amazon Redshift as a developer including stored procedures and performance optimization.

.

Hands-on experience with AWS Step Functions for workflow orchestration.

.

Experience in AWS Glue job development optimization and Glue Crawlers.

.

Experience handling large datasets and optimizing ETL workloads.

.

Good understanding of data modeling concepts including star schema fact tables and dimension tables.

.

Familiarity with cloud-based data lakes and modern data architectures.

.

Experience integrating data from enterprise applications such as SAP Oracle Azure and Oracle Cloud.

.

Understanding of KPI calculations and preparation of analytics-ready datasets.

Good to Have

.

Knowledge of Apache Iceberg on Amazon S3.

.

Experience with CI/CD for AWS data pipelines.

.

Exposure to Infrastructure as Code (CloudFormation/Terraform).

.

Understanding of data governance data cataloging and security best practices.

.

Experience with monitoring and troubleshooting AWS data services.

Soft Skills

.

Strong analytical and problem-solving skills.

.

Good communication and stakeholder management.

.

Ability to work independently as well as within cross-functional teams.

.

Strong documentation and knowledge-sharing practices.

.

Ability to work in an Agile development environment.