Lead Software Engineer Pyspark and Scala

We have an opportunity to impact your career and provide an adventure where you can push the limits of whats possible.

As a Lead Software Engineer at JPMorganChase within the Consumer and Community Banking youare an integral part of an agile team that works to enhance build and deliver trusted market-leading technology products in a secure stable and scalable way. As a core technical contributor you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

Job responsibilities

• Lead the design development and maintenance of robust scalable cloud-based data processing pipelines and infrastructure ensuring adherence to engineering standards governance frameworks and industry best practices.
• Architect and refine data models for large-scale datasets optimizing for efficient storage high-performance retrieval and advanced analytics while upholding data integrity and quality.
• Partner with cross-functional teams to translate complex business requirements into effective scalable data engineering solutions that drive organizational value.
• Champion a culture of innovation and continuous improvement proactively identifying and implementing enhancements to data infrastructure processing workflows and analytics capabilities.
• Define and execute data strategy including the development of enterprise data models and the management of end-to-end data infrastructurefrom design and construction to installation and ongoing maintenance of large-scale processing systems.
• Drive data quality initiatives ensure seamless data accessibility for analysts and data scientists and maintain strict compliance with data governance and regulatory requirements.
• Align data engineering practices with business objectives ensuring solutions are both technically sound and strategically relevant.
• Author review and approve technical requirements and architectural designs and lead process re-engineering efforts to deliver cost-effective high-impact business solution

Required qualifications capabilities and skills

• Formal training or certification on software engineering concepts and 5 years applied experience
• Expert in at least one distributed data processing framework (Spark)
• Expert in at least one cloud data Lakehouse platforms (AWS Data lake services or Databricks if not Hadoop)
• Expert in at least one scheduling/orchestration tools ( Airflow alternatively AWS Step Functions or similar)
• Expert with relational and NoSQL databases.
• Expert in data structures data serialization formats (JSON AVRO Protobuf or similar) and big-data storage formats (Parquet Iceberg or similar)
• Proficiency in microservices architecture serverless computing and distributed cluster computing tools such as Docker Kubernetes etc.
• Proficiency in one or more data modelling techniques (Dimensional Data Vault Kimball Inmon etc.)
• Experience with test-driven development (TDD) or behavior-driven development (BDD) practices as well as working with continuous integration and continuous deployment (CI/CD) tools.
• Experience organizing and leading design workshops coding sessions and hackathons to promote a culture of excellence and innovation in data engineering.
• Expertise in architecting reusable future-ready design patterns that address diverse use cases across the organization.
• Expertise in working with streaming platforms like Kafka MQ etc.

• Hands-on experience with Infrastructure as Code (IaC) tools preferably Terraform; experience with AWS CloudFormation is also valued.
• Proficiency in cloud-based data pipeline technologies such as Spinnaker or similar platforms.
• Strong working knowledge of the Snowflake data platform.
• Experience in budgeting and resource allocation for data engineering projects.
• Proven ability to manage vendor relationships effectively.