Pyspark Engineer

Bengaluru - India

Monthly Salary: Not Disclosed

Posted on: 11 hours ago

Vacancies: 1 Vacancy

Job Summary

Key Responsibilities

Design develop and maintain scalable data pipelines using PySpark for processing large volumes of structured and unstructured data.

Develop and manage ETL/ELT workflows using Informatica BDM.

Work closely with data architects analysts and business stakeholders to understand data requirements and translate them into technical solutions.

Optimize data processing jobs to ensure high performance and reliability. Implement data quality data governance and data validation processes. Integrate data from multiple sources including databases APIs and enterprise systems. Support data warehouse and data lake initiatives within the compliance with banking regulatory and security standards. Troubleshoot and resolve data pipeline issues and performance bottlenecks. Participate in code reviews testing and deployment processes to maintain high development standards.

Required Skills & Qualifications

6 years of experience in Data Engineering or related roles.

Strong hands-on experience with PySpark for large-scale data processing.

Solid experience with Informatica BDM (Big Data Management).

Experience working with Big Data technologies and distributed data processing frameworks.

Strong knowledge of SQL and data modeling concepts.

Experience with data integration ETL development and workflow orchestration.

Familiarity with cloud platforms or big data ecosystems (Hadoop/Spark environments) is a plus.

Good understanding of data governance security and compliance requirements in banking.

Strong analytical problem-solving and communication skills.

Domain Requirement

Mandatory experience in the Banking or Financial Services domain preferably working with enterprise data platforms regulatory reporting systems or large-scale financial datasets.

Preferred Qualifications

Experience with data lake architectures and data warehouse platforms.

Exposure to Agile development methodologies.

Knowledge of Python-based data engineering frameworks and automation.

Required Experience:

Key ResponsibilitiesDesign develop and maintain scalable data pipelines using PySpark for processing large volumes of structured and unstructured data.Develop and manage ETL/ELT workflows using Informatica BDM.Work closely with data architects analysts and business stakeholders to understand data re...