Lead Software Engineer - Pyspark data bricks Data Engineer
Lead Software Engineer - Pyspark data bricks Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Job Description
Join us as we embark on a journey of collaboration and innovation where your unique skills and talents will be valued and celebrated. Together we will create a brighter future and make a meaningful difference.
As a Lead Data Engineer at JPMorgan Chase within the Compliance Technology Team which is aligned to Corporate Technology division you are an integral part of an agile team that works to enhance build and deliver data collection storage access and analytics solutions in a secure stable and scalable way. As a core technical contributor you are responsible for maintaining critical data pipelines and architectures across multiple technical areas within various business functions in support of the firms business objectives.
Job responsibilities
- Provides recommendationsandinsight on data management governance proceduresandintricacies applicable to the acquisition maintenance validation andutilization of data
- Designs and delivers trusted data collection storage access and analytics data platform solutions in a secure stable and scalable way
- Defines database back-uprecovery and archivingstrategy
- Generates advanced data models for one or more teams using firm wide tooling linear algebra statistics and geometrical algorithms
- Approves data analysis toolsandprocesses
- Creates functional andtechnicaldocumentationsupporting bestpractices
- Advises junior engineers and technologists
- Evaluates and reports onaccesscontrol processes todetermineeffectiveness of dataasset security
- Adds to team culture of diversity equity inclusion and respect
Required qualifications capabilities and skills
- Formal training or certification on Data Engineeringconcepts and 5 years applied experience
- Demonstrated leadership abilities including leading and mentoring a team of data engineers.
- Expertise in designing and implementing scalable data architectures and pipelines.
- Strong proficiency in programming languages such as Python and PySpark.
- Experience with AWS cloud platform including their data services.
- In-depth knowledge of data warehousing solutions and ETL processes.
- Ability to work collaboratively with cross-functional teams including data scientists analysts and business stakeholders.
- Experience with data governance and ensuring data quality and integrity.
- Proficiency in Big Data technologies with a strong focus on Performance Optimization using best practices
- Excellent communication skills with the ability to convey complex technical concepts to non-technical audiences.
- Experience with version control systems like Git and CI/CD pipelines for data engineering workflows.
Preferred qualifications capabilities and skills
- Experience with Databricks using PySpark
- Solid understanding of the Parquet file format.
- Experience with ETL tools and experience in ETL process
- Familiarity with machine learning frameworks and integrating them into data pipelines.
Employment Type
Full-Time
