Lead Software Engineer Apache Spark, AWS

We have an opportunity to impact your career and provide an adventure where you can push the limits of whats possible.

As a Lead Software Engineer at JPMorgan Chase within the Engineer Services Platform (ESP) in the Chief Technology Office (CTO) you are an integral part of an agile team that builds and operates a core capability in the firms artifact engineering ecosystem that enables secure stable and scalable consumption of open source software. As a core technical contributor you will lead critical technology solutions that optimize open source software usage across the firm by strengthening governance controls and delivering actionable transparency through usage and compliance health metrics (per internal open source governance documentation).

Job responsibilities

• Design develop and troubleshoot platform services that govern open source acquisition usage and compliance at scale
• Design and implement scalable data platforms and pipelines (batch and streaming) with strong reliability and data quality controls.
• Build event-driven/CDC-based architectures to enable near real-time data movement and downstream consumption patterns.
• Develop and operate pipelines using technologies such as Spark Kafka and Airflow focusing on maintainability and observability.
• Design distributed data systems with strong understanding of partitioning/sharding scalability and performance optimization.
• Build solutions on AWS and cloud-native data services; ensure security governance and cost-aware design.
• Optimize end-to-end performance (ingestion processing storage query) including SLAs and operational readiness.
• Partner with stakeholders to define data contracts integration patterns and operational runbooks for production support.
• Contribute to engineering excellence via design reviews code reviews and best practices for pipeline testing and operational stability.

Required qualifications capabilities and skills

• Formal training or certification on software engineering concepts and 5 years applied experience.
• Hands-on practical experience delivering system design application development testing and operational stability for platform services used by many engineering teams.
• Advanced in one or more programming language(s) with demonstrated ability to build production services and integrations used in critical developer workflows.
• Proficiency in automation and continuous delivery methods including designing and operating integrations that can enforce policy decisions in CI/CD pathways (per internal AMS integration expectations).
• Proficient in all aspects of the Software Development Life Cycle with strong engineering hygiene around secure coding peer review observability and incident response.
• Strong data engineering skills including hands-on experience designing and building data pipelines and well-versed with the Medallion Architecture (Bronze Silver and Gold layers) to support reliable scalable and audit-ready data flows for compliance telemetry and open source health metrics.
• Advanced understanding of agile methodologies and modern engineering practices such as CI/CD application resiliency and security particularly as they apply to supply chain controls and governance automation.
• Demonstrated proficiency in software applications and technical processes within a technical discipline such as cloud-native services distributed systems developer platforms or security tooling.
• In-depth knowledge of the financial services industry and its IT systems including expectations for traceability control alignment and audit-ready evidence.
• Practical cloud-native experience including service design deployment patterns and runtime operations.

Preferred qualifications capabilities and skills

• Experience with NoSQL technologies such as Cassandra MongoDB or DynamoDB.
• Familiarity with big data technologies such as Spark Scala and HDFS.
• Familiarity with modern infrastructure technology and architecture including cloud virtualization and object stores.
• Experience implementing or operating open source governance controls including license risk classification approaches (e.g. Red/Yellow/Green models) and approval workflows for restricted licenses (per internal license governance guidance including examples such as AGPL-3.0 and enforcement expectations).
• Experience building compliance and posture dashboards that translate technical telemetry into operational health metrics and actionable insights for engineering and governance stakeholders.