Lead Data Engineer - Python, AWS, ETL - R
Lead Data Engineer - Python, AWS, ETL - R
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Primary Skills
- Athena SNS SQS CloudWatch Macie Spark - Scala Kinesis CloudFormation EMR Open Search DynamoDB Amazon API Gateway SCT Redshift DMS Oozie
Job requirements
- Location: Chicago IL - Hybrid with 2-3 days on-site in a week
- Design and implement scalable ETL/ELT pipelines using AWS Glue Spark (PySpark) and Step Functions.
- Work with structured and semi-structured data using Athena S3 and Lake Formation to enable efficient querying and access control.
- Develop and deploy serverless data processing solutions using AWS Lambda and integrate them into pipeline orchestration.
- Perform advanced SQL and PL/SQL development for data transformation analysis and performance tuning. Build data lakes and data warehouses using S3 Aurora and Athena.
- Implement data governance security and access control strategies using AWS tools including Lake Formation CloudFront EBS/EFS and IAM. Develop and maintain metadata lineage and data cataloging capabilities.
- Participate in data modeling exercises for both OLTP and OLAP environments.
- Work closely with data scientists analysts and business stakeholders to understand data requirements and deliver actionable insights. Monitor debug and optimize data pipelines for reliability and performance.
- Strong experience with AWS data services: Glue Athena Step Functions Lambda Lake Formation S3 EC2 Aurora EBS/EFS CloudFront.
- Proficient in PySpark Python SQL (basic and advanced) and PL/SQL.
- Solid understanding of ETL/ELT processes and data warehousing concepts.
- Familiarity with modern data platform fundamentals and distributed data processing.
- Experience in data modeling (conceptual logical physical) for analytical and operational use cases.
- Experience with orchestration and workflow management tools within AWS.
- Strong debugging and performance tuning skills across the data stack.
Employment Type
Full-Time
