Job Title: Data Engineer
Location: Boston MA(Hybrid)
Job Type: Contract
Client: State Street
Note: Need 10 of experience resume and accepts only H1b visa
Responsibilities:
- Data Pipeline Development: Design and implement scalable data pipelines using Python SQL and PySpark to process large volumes of data efficiently.
- AWS Infrastructure Management: Manage and optimize AWS services such as S3 Redshift ECS and Lambda to support data processing and storage needs.
- Databricks Integration: Leverage Databricks to enhance data processing capabilities including ETL (Extract Transform Load) processes and advanced analytics.
- Performance Optimization: Continuously monitor and optimize data pipelines to ensure high performance and reliability.
- Collaboration: Work closely with data scientists analysts and other engineers to understand data requirements and deliver solutions that meet business needs.
- Security and Compliance: Ensure all data processing activities comply with security and data protection regulations.
Mandatory Skills:
- Should have 5 years of experience as Data Engineer.
- Proficiency in Python SQL and AWS services.
- Extensive experience with PySpark for big data processing.
- Understanding of AWS ECS and Lambda.
- Familiarity with data warehousing solutions like Redshift or Snowflake.
- Knowledge of data modeling ETL processes and data lakes.
Good to Have Skills:
- Exposure to Databricks for data processing and analytics. (This is a big plus)