pyspark
Job Location:
Monthly Salary:
Posted on:
10 hours ago
Posted on:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Roles & Responsibilities:
Professional & Technical Skills:
- Expected to be an SME collaborate and manage the team to perform.
- Responsible for team decisions.
- Engage with multiple teams and contribute on key decisions.
- Provide solutions to problems for their immediate team and across multiple teams.
- Lead the implementation of best practices to improve software quality and delivery timelines.
- Mentor junior team members to support their professional growth and skill development.
- Coordinate cross-functional efforts to align software development with organizational goals.
- Collaborate with other teams to integrate Spark jobs into the overall data pipeline.
Professional & Technical Skills:
- Must To Have Skills: Proficiency in PySpark.
- Strong programming skills in Python and experience with distributed computing frameworks with Knowledge of Java and Snowflake would be an added advantage.
- Experience in designing and optimizing data processing pipelines for large-scale datasets with Extensive experience in design build and deployment of Python-based applications Experience in various AWS services such as EMR API Gateway RDS instance and Lambda.
- Familiarity with cloud platforms and integration of PySpark applications within cloud environments.
- Ability to troubleshoot and resolve performance bottlenecks in data processing workflows.
- Knowledge of software development lifecycle and agile methodologies.
- Hands-on experience in relational databases preferably Oracle and PostgreSQL and writing complex SQL queries.
- Ability to Understand complex data sets and ETL processes and how they can be optimized using Spark with Monitoring and tuning data loads and queries.
