AWS Data Engineer

Torrance, CA - USA

Monthly Salary: Not Disclosed

Posted on: 4 days ago

Vacancies: 1 Vacancy

Job Summary

AWS Glue Developing ETL pipelines working with Glue Catalog and transforming data using PySpark

AWS Lambda Writing serverless functions (Python ) to process move or transform data

AWS S3 Managing storage organizing Qualtrics input/output files and optimizing data retrieval

AWS Step Functions Orchestrating Glue and Lambda jobs into automated workflows

AWS Athena Querying structured data stored in S3 for quick analysis

AWS CloudWatch & Logging Monitoring job execution and debugging errors

AWS MWAA airflow for Jobs orchestration

2. Data Processing & ETL Skills

Data Transformation & Aggregation Merging dealer master and dealer employee files into a single hierarchy file

ETL Workflow Automation Automating ingestion transformation and report generation

Data Format Handling Processing CSV JSON or XML files from Qualtrics

3. API & Integration Skills

Qualtrics API Extracting raw data managing survey responses and automating data pull

AWS Event-Driven Processing Triggering Lambda jobs based on new files arriving in S3

Integration with External Systems Connecting AWS with CRM ERP or BI tools

4. Data Analysis & Reporting

Data Validation & Quality Checks Ensuring accuracy in processed files before Qualtrics ingests them Report File Generation Formatting data outputs as per business-defined templates

Basic SQL & Athena Queries Extracting insights from processed data

5. Programming & Scripting

Python Writing data processing scripts for Lambda Glue (PySpark) and automation

SQL Querying and managing structured datasets

Shell Scripting (Bash) Automating file movement and data preparation

6. DevOps & Security (Nice to Have)

IAM Permissions & Role Management Ensuring AWS services have correct access

Infrastructure as Code (IaC) Using Terraform or CloudFormation for AWS resource setup

CI/CD for Data Pipelines Automating deployment of Lambda and Glue scripts

Pi-square technologies is a Michigan (USA) Headquartered Automotive Embedded Engineering Services company Synergy Partner for major OEMs and Tier 1s and their implementation partners in Automotive Embedded Product Development Projects Requirements Analysis Software Design Software Implementation Efficient Build Release Process and turnkey software V & V Services. We have more than 20 years of industry expertise with specialization in the latest cutting-edge automotive technologies such as Infotainment connected vehicles Cyber security OTA and Advanced Safety/ Body electronics.

AWS Glue Developing ETL pipelines working with Glue Catalog and transforming data using PySpark AWS Lambda Writing serverless functions (Python ) to process move or transform data AWS S3 Managing storage organizing Qualtrics input/output files and optimizing data retrieval AWS Step F...