Lead Software Engineer PythonPySparkDatabricksAWS

We have an opportunity to impact your career and provide an adventure where you can push the limits of whats possible.

As a Lead Software Engineer - Python/PySpark/Databricks/AWS at JPMorganChase within the Corporate Technology team youare an integral part of an agile team that works to enhance build and deliver trusted market-leading technology products in a secure stable and scalable way. As a core technical contributor you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

Job responsibilities

• Executes creative software solutions design development and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems
• Develops secure high-quality production code and reviews and debugs code written by others
• Identifies opportunities to eliminate or automate remediation of recurring issues to improve overall operational stability of software applications and systems
• Leads evaluation sessions with external vendors startups and internal teams to drive outcomes-oriented probing of architectural designs technical credentials and applicability for use within existing systems and information architecture
• Leads communities of practice across Software Engineering to drive awareness and use of new and leading-edge technologies
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality delivery speed and operational outcomes (e.g. AI-assisted code review/refactoring test strategy acceleration incident/root-cause analysis support) while establishing consistent validation standards (secure coding peer review automated testing) and promoting reuse of effective patterns across the team.
• Applies knowledge of tools within the Software Development Life Cycle toolchain including enterprise-authorized AI-assisted development and automation capabilities to improve the value realized by automation.

Required qualifications capabilities and skills

• Formal training or certification on software engineering concepts and 5 years applied experience
• Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g. for coding code review test acceleration troubleshooting) with the ability to set team expectations for validating AI outputs for correctness performance and security.
• Strong understanding of responsible AI use in engineering workflows including data sensitivity considerations secure handling of inputs/outputs and adherence to resiliency and security expectations; experience coaching engineers on safe compliant adoption within delivery practices
• Hands-on practical experience delivering system design application development testing and operational stability
• Experience building and operating Databricks Lakehouse solutions hosted on Amazon Web Services (AWS) including Amazon S3 Identity and Access Management (IAM) Key Management Service (KMS) basic networking concepts (VPC/security groups) and logging/auditing.
• Experience using Delta Lake (ACID-compliant tables partitioning strategies schema evolution) and Apache Spark on Databricks including performance optimization (cluster sizing skew mitigation join strategies caching and file sizing/compaction).
• Experience delivering batch and streaming data pipelines (Structured Streaming incremental processing backfills late-arriving data handling) and implementing governance/security controls in Databricks (e.g. Unity Catalog table/column-level permissions credential passthrough where applicable) with operational ownership including monitoring/alerting incident response root-cause analysis (RCA) and service level objective/service level agreement (SLO/SLA) management.
• Advanced in one or more programming language(s) including Python PySpark
• Advanced understanding of agile methodologies such as CI/CD Application Resiliency and Security
• Architect Databricks Lakehouse solutions including bronze/silver/gold (or equivalent) layering and domain-oriented data products; implement resilient scalable ingestion from AWS sources into Databricks using batch and streaming patterns (including CDC where required).
• Build maintainable pipelines using Delta Live Tables (DLT) and/or Databricks Jobs/Workflows with modular design documentation and runbooks; ensure production readiness through retries checkpointing idempotency safe re-runs and defined replay/backfill procedures; implement testing practices including unit/integration tests data quality checks and contract testing; Apply governance-by-design controls (least privilege PII classification auditing lineage/metadata controlled sharing/consumption); optimize Spark/Delta performance and cost (cluster right-sizing storage layout job/warehouse spend); lead design/code reviews and mentor engineers; partner cross-functionally with stakeholders and security/platform teams; deliver CI/CD and infrastructure-as-code for Databricks AWS with promotion across environments and strong version control/code review discipline.

• AI experience