Senior Manager of Software Engineering Databricks, AWS

As a Manager of Software Engineering at JPMorganChase within Corporate Sector Enterprise Technology youare an integral part of an agile team that works to enhance build and deliver trusted market-leading technology products in a secure stable and scalable way. As a core technical contributor you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

Job Responsibilities:

• Lead architecture and delivery of high-throughput low-latency data pipelines using Databricks and Apache Spark (Core SQL Structured Streaming).
• Establish lakehouse patterns with Delta Lake (ACID transactions schema evolution time travel Z-ordering compaction) and ensure performance at scale.
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality delivery speed and operational outcomes (e.g. AI-assisted code review/refactoring test strategy acceleration incident/root-cause analysis support) while establishing consistent validation standards (secure coding peer review automated testing) and promoting reuse of effective patterns across the team.
• Applies knowledge of tools within the Software Development Life Cycle toolchain including enterprise-authorized AI-assisted development and automation capabilities to improve the value realized by automation.
• Own Databricks cluster strategy and setup: runtime selection autoscaling driver/executor sizing Spark configs unit scripts cluster policies pools and instance profiles.
• Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.
• Design secure data ingestion and transformation frameworks leveraging Databricks services: Design delta or unmanaged tables Create tasks for data ingestion process Create DAGs using Airflow to orchestrate creation of trusted and refined data.
• Enforce data quality lineage and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.
• Drive Spark performance engineering: partitioning strategies file sizing AQE broadcast joins shuffle tuning caching spill/memory control and job right-sizing to optimize cost.
• Build reusable libraries frameworks and APIs in Python and/or Java; oversee unit integration and data validation testing.
• Implement CI/CD for data projects (Git-based workflows) Terraform Infrastructure deployments environment promotion and automated deployments; champion engineering standards and code reviews.

Required qualifications capabilities and skills:

• Formal training or certification on software engineering concepts and 5 years applied experience.
• 10 years of professional software/data engineering experience including substantial production work with Spark on Databricks or EMR.
• Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g. for coding code review test acceleration troubleshooting) with the ability to set team expectations for validating AI outputs for correctness performance and security.
• Strong understanding of responsible AI use in engineering workflows including data sensitivity considerations secure handling of inputs/outputs and adherence to resiliency and security expectations; experience coaching engineers on safe compliant adoption within delivery practices
• Strong proficiency in Python and/or Java for data processing platform tooling and automation.
• Hands-on Databricks expertise (Delta Lake Unity Catalog Workflows Repos/notebooks SQL Warehouses).
• Proven track record architecting and operating ETL/ELT pipelines (batch and streaming) with schema design/evolution SLAs and reliability engineering.
• Deep skills in Spark performance tuning and Databricks cluster setup/optimization.
• Strong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).
• CI/CD and automation tooling for data (Git workflows artifact management) and testing frameworks (pytest JUnit).
• Security-first mindset: roles/instance profiles secret management encryption-at-rest/in-transit and network controls.

Preferred qualifications capabilities and skills:

• Experience with Delta Live Tables and advanced governance (catalogs grants auditing) in Databricks.
• AWS networking knowledge (VPC subnets routing security groups) and data egress controls.
• Experience with Terraform for Infra deployments
• Cost optimization experience: autoscaling strategies spot vs on-demand auto-termination storage layouts and compaction.
• Observability for data systems (freshness/completeness metrics lineage SLAs alerting).
• Drive databricks performance tuning through liquid clustering or partitioning keys familiarity with Airflow Genie Streamlit and React
• Demonstrated leadership in code quality reviews testing strategy CI/CD and technical mentorship; excellent communication with stakeholders.