Sr. Pyspark Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
2 hours ago
Vacancies:
1 Vacancy
Job Summary
Job Title: PySpark Data Engineer
Experience: 3 Years
Location: Hyderabad
Job Summary:
We are looking for a skilled and experienced PySpark Data Engineer to join our growing data engineering team. The ideal candidate will have 6 years of experience in designing and implementing data pipelines using PySpark AWS Glue and Apache Airflow with strong proficiency in SQL. You will be responsible for building scalable data processing solutions optimizing data workflows and collaborating with cross-functional teams to deliver high-quality data assets.
Requirements
Key Responsibilities:
- Design develop and maintain large-scale ETL pipelines using PySpark and AWS Glue.
- Orchestrate and schedule data workflows using Apache Airflow.
- Optimize data processing jobs for performance and cost-efficiency.
- Work with large datasets from various sources ensuring data quality and consistency.
- Collaborate with Data Scientists Analysts and other Engineers to understand data requirements and deliver solutions.
- Write efficient reusable and well-documented code following best practices.
- Monitor data pipeline health and performance; resolve data-related issues proactively.
- Participate in code reviews architecture discussions and performance tuning.
Requirements
- 3 years of experience in data engineering roles.
- Strong expertise in PySpark for distributed data processing.
- Hands-on experience with AWS Glue and other AWS data services (S3 Athena Lambda etc.).
- Experience with Apache Airflow for workflow orchestration.
- Strong proficiency in SQL for data extraction transformation and analysis.
- Familiarity with data modeling concepts and data lake/data warehouse architectures.
- Experience with version control systems (e.g. Git) and CI/CD processes.
- Ability to write clean scalable and production-grade code.
Benefits
- Comprehensive Medical Coverage:Health insurance of INR 7.0 Lakhs for you and your family (up to 6 members) ensuring complete peace of mind.
- Robust Protection Plans:Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
- Retirement Benefits:PF and Gratuity provided as per standard government regulations.
- Flexible Work Options:Enjoy hybrid work arrangements & flexible working hours
- Generous Leave Policy:21 days of annual leave in addition to 10 company-declared holidays.
- Employee Well-being Spaces:Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
Required Skills:
Key Responsibilities: Design develop and maintain large-scale ETL pipelines using PySpark and AWS Glue. Orchestrate and schedule data workflows using Apache Airflow. Optimize data processing jobs for performance and cost-efficiency. Work with large datasets from various sources ensuring data quality and consistency. Collaborate with Data Scientists Analysts and other Engineers to understand data requirements and deliver solutions. Write efficient reusable and well-documented code following best practices. Monitor data pipeline health and performance; resolve data-related issues proactively. Participate in code reviews architecture discussions and performance tuning. Requirements 3 years of experience in data engineering roles. Strong expertise in PySpark for distributed data processing. Hands-on experience with AWS Glue and other AWS data services (S3 Athena Lambda etc.). Experience with Apache Airflow for workflow orchestration. Strong proficiency in SQL for data extraction transformation and analysis. Familiarity with data modeling concepts and data lake/data warehouse architectures. Experience with version control systems (e.g. Git) and CI/CD processes. Ability to write clean scalable and production-grade code.
Company Industry
IT Services and IT Consulting
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
