AWS Data

Toronto - Canada

Monthly Salary: Not Disclosed

Posted on: 8 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Responsibilities
Lead the architectural design and development of a scalable reliable and flexible metadata-driven data ingestion and extraction framework on AWS using Python Pyspark SQL and various AWS technologies.
Design and implement a customizable data processing framework using Python or . This framework should be capable of handling diverse scenarios and evolving data processing requirements.
Implement data pipeline for data Ingestion transformation and extraction leveraging the AWS Cloud Services
Seamlessly integrate a variety of AWS services including S3 Kafka AWS Glue Amazon Redshift Lambda SQL QS/SNS/Cloudwatch/Step function/CDK Athena EC2 RDS (Oracle Postgres MySQL) AWS Crawler to construct a highly scalable and reliable data ingestion and extraction pipeline.
Facilitate configuration and extensibility of the framework to adapt to evolving data needs and processing scenarios.
Develop and maintain rigorous data quality checks and validation processes to safeguard the integrity of ingested data.
Implement robust error handling logging monitoring and alerting mechanisms to ensure the reliability of the entire data pipeline.
Automate repetitive tasks and build reusable frameworks to improve efficiency.

Primary Skill:

Relevant years of hands-on experience in data engineering with a proven focus on data ingestion and extraction using Python.
Extensive AWS experience is mandatory with proficiency in Lambda Refshift SQS SNS AWS IAM AWS Step Functions Cloud Watch CDK S3 and RDS (Oracle Aurora Postgres).
Good experience working with both relational and non-relational/NoSQL databases is required.
Strong SQL experience is necessary demonstrating the ability to write complex queries from scratch.
Strong scripting experience with the ability to build intricate data pipelines using AWS serverless architecture.
Strong Pyspark/Python experience