Python with Spark Developer (5.1-7 years) Chennai

About Us

Capco a Wipro company is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe we support 100 clients acrossbanking financial and Energy sectors. We are recognized for our deep transformation execution and delivery.

WHY JOIN CAPCO

You will work on engaging projects with the largest international and local banks insurance companies payment service providers and other key players in the industry. The projects that will transform the financial services industry.

MAKE AN IMPACT

Innovative thinking delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners we deliver disruptive work that is changing energy and financial services.

#BEYOURSELFATWORK

Capco has a tolerant open culture that values diversity inclusivity and creativity.

CAREER ADVANCEMENT

With no forced hierarchy at Capco everyone has the opportunity to grow as we grow taking their career into their own hands.

DIVERSITY & INCLUSION

We believe that diversity of people and perspective gives us a competitive advantage.

MAKE AN IMPACT

WHY JOIN CAPCO

You will work on engaging projects with some of the largest banks in the world on projects that will transform the financial services industry. We offer: A work culture focused on innovation and creating lasting value for our clients and employees Ongoing learning opportunities to help you acquire new skills or deepen existing expertise A flat non-hierarchical structure that will enable you to work with senior partners and directly with clients A diverse inclusive meritocratic culture

We offer:

• A work culture focused on innovation and creating lasting value for our clients and employees
• Ongoing learning opportunities to help you acquire new skills or deepen existing expertise
• A flat non-hierarchical structure that will enable you to work with senior partners and directly with clients

Python Developer PySpark 57 Years Chennai

Position Purpose:

• Overall 5-7 years of experience as a Python Developer with Spark.

• The Python/PySpark Developer will design develop test and maintain highperformance dataprocessing pipelines using strong Python pyspark and SQL skills.

• You will transform raw market trade and client data into trusted consumable datasets that feed downstream analytics reporting and machinelearning models used by the CEFS ClientEngagementandFinancialSecurities franchise.

• Sound knowledge of Agile (Scrum/Kanban).

Responsibilities

Direct Responsibilities

• Design & developrobust ingestion transformation and enrichment pipelines withPythonPySpark andSQL.

• Write and optimizecomplex SQL queries analytical UDFs and window functions for data aggregation and reporting.

• Collaboratewith CEFS data architects data scientists and business analysts to translate functional requirements into technical specifications.

• Unittest integratetest and reviewcode.

• Maintain CI/CD pipelines(Git Jenkins Docker) for automated build test and deployment of jobs.

• Monitorproduction workloads and troubleshoot performance bottlenecks memory issues and job failures.

• Documentdata lineage pipeline design and operational runbooks in Confluence/SharePoint.

• Keep up to date with latest technologies and trends and provide input expertise and recommendations.

Contributing Responsibilities

• Contribute towards innovation (e.g. AI/ML); suggest new technical practices for efficiency improvement.

• Participate inAgile ceremonies(sprint planning daily standups retrospectives) and help groom the backlogs.

• Mentor junior engineers and champion best practices in Python coding Spark optimization and dataengineering patterns.

• Evaluate emerging technologies and deliver proofofconcepts for CEFS.

Technical & Behavioral Competencies

• Resourceful to quickly understand complexities involved and provide the way forward.

• Good experience in technical analysis of n-tier applications with multiple integrations using object oriented APIs & Microservices approaches.

• Strong knowledge about design patterns and development principles.

• Inclination and prior experience of working across SQL Python and ETL.

• Strong Hands-on experience - Python (NumPy pandas Python Frameworks Restful APIs MS-SQL or Oracle.

• PySpark- DataFrames Spark SQL Structured Streaming performance tuning (partitioning caching broadcast joins).

• Advanced SQL complex queries stored procedures query optimization.

• Good Knowledge and experience to use Python packages such as Pandas NumPy etc. for cleaning up of Data Data Wrangling Analysis of Data and Visualization of Data.

• Good experience in development and maintenance of code/scripts in both functional and technical specifications of all applications component bug fixing and production support.

• Good knowledge on Linux/Unix environment (basic commands shell scripting etc.) testing phases documentation and new framework.

• Some experience of working with build tools like Maven & DevOps tools like Bitbucket Jenkins.

• Knowledge of Agile Scrum DevOps.

• Development experience in Data Engineering environment.

• Ability & willingness to learn & work on diverse technologies (languages frameworks and tools).

• Self-motivated good interpersonal skills and inclination to constantly upgrade new technologies and frameworks.

• Good communication and co-ordination skills.

Specific Qualifications:

• Good to have knowledge of front-end technologies preferably Flask.

Skills Referential (Required knowledge skills and abilities)

Technical Skills:

• Python

• Pyspark

• SQL

• ETL

Behavioral Skills:

• Ability to synthetize / simplify

• Ability to collaborate / Teamwork

• Attention to detail / rigor

• Ability to deliver / Results driven

Education Level: Bachelors degree or equivalent

Location: Chennai