Senior Data Engineer PySpark
Senior Data Engineer PySpark
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Software Requirements:
- PySpark
- Hadoop
- Spark
- Python
- Unix
- HDFS
- PyArrow
- SQL
Overall Responsibilities:
- Develop and optimize data processing workflows using PySpark.
- Implement and manage scalable data pipelines within Hadoop and Spark ecosystems.
- Collaborate with data scientists analysts and stakeholders to accurately capture data requirements and deliver effective solutions.
- Ensure data quality and integrity across all processes and workflows.
- Monitor and resolve performance issues in big data applications.
- Stay updated with advancements in big data technologies.
Technical Skills:
Categorywise:
PySpark and Spark:
- Proficiency in PySpark for sophisticated data processing tasks.
- Experience in Spark development including performance optimization techniques.
Hadoop and HDFS:
- Advanced proficiency with Hadoop ecosystems.
- Skilled in leveraging HDFS for efficient data storage and retrieval.
Programming and Libraries:
- Strong programming foundation in Python.
- Familiarity with supporting libraries such as PyArrow.
Unix:
- Basic experience with Unix systems for executing data processing tasks.
Databases (SQL):
- Solid working knowledge of SQL for database management and querying.
Experience:
- Minimum 7 years of experience in big data environments.
- Extensive exposure to PySpark and Spark development.
- Significant experience in data engineering and managing large datasets.
DaytoDay Activities:
- Develop test and deploy scalable data processing pipelines using PySpark.
- Execute data extraction transformation and loading (ETL) activities.
- Collaborate with crossfunctional teams to gather requirements and deliver efficient solutions.
- Optimize and troubleshoot existing data workflows and pipelines.
- Document processes workflows and data models comprehensively.
- Continuously monitor and enhance data processing performance.
Qualifications:
- Bachelors or masters degree in computer science Data Engineering or a related field.
- Relevant certifications in big data technologies are advantageous.
Soft Skills:
- Exceptional problemsolving and analytical skills.
- Strong communication abilities to collaborate effectively with team members and stakeholders.
- Demonstrated ability to work independently as well as in a team setting.
- High attention to detail with a proactive approach to identifying and solving issues.
- Flexibility and adaptability to evolving technologies and methodologies.
SYNECHRONS DIVERSITY & INCLUSION STATEMENT
Diversity & Inclusion are fundamental to our culture and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity Equity and Inclusion (DEI) initiative Same Difference is committed to fostering an inclusive culture promoting equality diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger successful businesses as a global company. We encourage applicants from across diverse backgrounds race ethnicities religion age marital status gender sexual orientations or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements mentoring internal mobility learning and development programs and more.
All employment decisions at Synechron are based on business needs job requirements and individual qualifications without regard to the applicants gender gender identity sexual orientation race ethnicity disabled or veteran status or any other characteristic protected by law.
Required Experience:
Senior IC
Employment Type
Full-Time
