Pyspark Engineer5 YearsHyderabad

Overview:

Key Responsibilities:

• Designing and implementing PySparkbased data processing and analytics solutions.
• Optimizing and tuning PySpark jobs for performance and scalability.
• Collaborating with data scientists and analysts to understand their requirements and implement efficient data workflows.
• Developing and maintaining ETL processes using PySpark to integrate data from multiple sources.
• Creating and managing data pipelines for realtime and batch processing.
• Troubleshooting and resolving issues related to data processing and pipeline execution.
• Implementing best practices for data engineering and ensuring data quality and reliability.
• Collaborating with crossfunctional teams to support datarelated initiatives and projects.
• Participating in code reviews and providing technical guidance to junior team members.
• Staying updated with the latest advancements in PySpark and Big Data technologies.

Required Qualifications:

• Bachelors or Masters degree in Computer Science Engineering or a related field.
• Minimum of 5 years of handson experience in PySpark development and Big Data technologies.
• Proficiency in Python programming for data analysis and manipulation.
• Strong understanding of SQL and experience working with relational databases.
• Experience in building and optimizing data pipelines for largescale data processing.
• Solid understanding of distributed computing principles and cluster management.
• Knowledge of data warehousing concepts and best practices.
• Experience with cloud platforms such as AWS Azure or GCP is a plus.
• Excellent problemsolving and analytical skills with a detailoriented mindset.
• Strong communication and collaboration abilities to work effectively in a team environment.

big data,python,sql,data engineering