Software Engineer III Databricks IAM

Plano, TX - USA

Monthly Salary: Not Disclosed

Posted on: 14 hours ago

Vacancies: 1 Vacancy

Job Summary

Description

We have an opportunity to impact your career and provide an adventure where you can push the limits of whats possible.

As a Lead Software Engineer at JPMorgan Chase within the Chief Technology Office - Identity and Access Management - Data Service Engineering team youare an integral part of an agile team that works to enhance build and deliver trusted market-leading technology products in a secure stable and scalable way. As a core technical contributor you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

Job responsibilities

Executes creative software solutions design development and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems
Architect and deliver high-throughput low-latency data pipelines on Databricks and Apache Spark (Core SQL Structured Streaming)
Develops secure high-quality production code and reviews and debugs code written by others
Define and implement Lakehouse patterns with Delta Lake (ACID schema evolution time travel Z-ordering compaction) to achieve performance at scale
Manage Databricks compute and cluster configurations: runtime selection autoscaling driver/executor sizing Spark configs init-scripts cluster policies pools and instance profiles including orchestrate jobs with Databrick workflows to integrate with AWS
Design secure data ingestion and transformation on AWS using S3 (data lake and lifecycle) Glue (catalog/ETL) IAM and Secrets Manager (RBAC/credentials) CloudWatch (logging/metrics/alerting) Lambda (serverless utilities) and Kinesis/Kafka/MSK (streaming)
Implement data quality lineage and governance with Unity Catalog and/or Glue Catalog - embed expectations and validations within pipelines
Optimize spark performance and cost via partitioning strategies file sizing AQE broadcast joins shuffle tuning caching spill/memory management and job right - sizing
Leads evaluation sessions with external vendors startups and internal teams to drive outcomes-oriented probing of architectural designs technical credentials and applicability for use within existing systems and information architecture
Collaborate with platform security and networking teams to enforce encryption network controls and least-privilege access; ensure compliance with organizational policies
Add to team culture of diversity opportunity inclusion and respect

Required qualifications capabilities and skills

Formal training or certification on software / data engineering concepts and 3 years applied experience
Hands-on Databricks expertise across Delta Lake Unity Catalog Workflows Repos/notebooks and SQL Warehouses including cluster setup/optimization and Spark performance tuning
Advanced in one or more programming language(s) framework(s) and tools (i.e. Python and or Java Spark and Databricks)
Proficient in all aspects of the Software Development Life Cycle and proficient in automation and continuous delivery methods
Advanced understanding of agile methodologies such as CI/CD Git-based workflows and Terraform IaC Application Resiliency and Security
Experience with incident responses and root - cause analysis; maintaining SLAs observability and runbooks to drive continuous improvement in reliability and cost efficiency participating in technical design reviews and collaborating with product owners and cross - functional teams on roadmaps and backlog prioritization
Demonstrated proficiency in software applications and technical processes within a technical discipline (e.g. cloud artificial intelligence machine learning mobile etc.)
Strong AWS experience with S3 IAM (including roles/instance profiles) Glue CloudWatch Kinesis/MSK and DynamoDB
Proven track record architecting and operating ETL/ELT pipelines (batch and streaming) covering schema design/evolution SLAs and reliability engineering
Strong SQL and analytics data modeling (dimensional/star schemas) aligned with Lakehouse best practices
Program with a security-first mindset in secret management encryption at rest/in transit and network controls

Preferred qualifications capabilities and skills

Experience with Databricks: Delta Live Tables and advanced governance (catalogs grants auditing)
Practical cloud native experience - using AWS networking and egress: VPC subnets routing security groups and data egress controls
Analyze IaC CI/CD and test automation: Terraform for infrastructure deployments; Git workflows artifact management; testing frameworks (pytest JUnit).
Knowledge in cost optimization with autoscaling strategies spot vs on-demand auto-termination storage layouts and compaction
Drive real-time ingestion with Kafka/MSK or Kinesis Data Streams/Firehose
Comprehend observability metrics lineage SLAs and alerting for data systems
Experience in financial services or similarly regulated environments

Required Experience:

DescriptionWe have an opportunity to impact your career and provide an adventure where you can push the limits of whats possible.As a Lead Software Engineer at JPMorgan Chase within the Chief Technology Office - Identity and Access Management - Data Service Engineering team youare an integral part o...

Description

We have an opportunity to impact your career and provide an adventure where you can push the limits of whats possible.

Job responsibilities

Executes creative software solutions design development and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems
Architect and deliver high-throughput low-latency data pipelines on Databricks and Apache Spark (Core SQL Structured Streaming)
Develops secure high-quality production code and reviews and debugs code written by others
Define and implement Lakehouse patterns with Delta Lake (ACID schema evolution time travel Z-ordering compaction) to achieve performance at scale
Manage Databricks compute and cluster configurations: runtime selection autoscaling driver/executor sizing Spark configs init-scripts cluster policies pools and instance profiles including orchestrate jobs with Databrick workflows to integrate with AWS
Design secure data ingestion and transformation on AWS using S3 (data lake and lifecycle) Glue (catalog/ETL) IAM and Secrets Manager (RBAC/credentials) CloudWatch (logging/metrics/alerting) Lambda (serverless utilities) and Kinesis/Kafka/MSK (streaming)
Implement data quality lineage and governance with Unity Catalog and/or Glue Catalog - embed expectations and validations within pipelines
Optimize spark performance and cost via partitioning strategies file sizing AQE broadcast joins shuffle tuning caching spill/memory management and job right - sizing
Leads evaluation sessions with external vendors startups and internal teams to drive outcomes-oriented probing of architectural designs technical credentials and applicability for use within existing systems and information architecture
Collaborate with platform security and networking teams to enforce encryption network controls and least-privilege access; ensure compliance with organizational policies
Add to team culture of diversity opportunity inclusion and respect

Required qualifications capabilities and skills

Formal training or certification on software / data engineering concepts and 3 years applied experience
Hands-on Databricks expertise across Delta Lake Unity Catalog Workflows Repos/notebooks and SQL Warehouses including cluster setup/optimization and Spark performance tuning
Advanced in one or more programming language(s) framework(s) and tools (i.e. Python and or Java Spark and Databricks)
Proficient in all aspects of the Software Development Life Cycle and proficient in automation and continuous delivery methods
Advanced understanding of agile methodologies such as CI/CD Git-based workflows and Terraform IaC Application Resiliency and Security
Experience with incident responses and root - cause analysis; maintaining SLAs observability and runbooks to drive continuous improvement in reliability and cost efficiency participating in technical design reviews and collaborating with product owners and cross - functional teams on roadmaps and backlog prioritization
Demonstrated proficiency in software applications and technical processes within a technical discipline (e.g. cloud artificial intelligence machine learning mobile etc.)
Strong AWS experience with S3 IAM (including roles/instance profiles) Glue CloudWatch Kinesis/MSK and DynamoDB
Proven track record architecting and operating ETL/ELT pipelines (batch and streaming) covering schema design/evolution SLAs and reliability engineering
Strong SQL and analytics data modeling (dimensional/star schemas) aligned with Lakehouse best practices
Program with a security-first mindset in secret management encryption at rest/in transit and network controls

Preferred qualifications capabilities and skills

Experience with Databricks: Delta Live Tables and advanced governance (catalogs grants auditing)
Practical cloud native experience - using AWS networking and egress: VPC subnets routing security groups and data egress controls
Analyze IaC CI/CD and test automation: Terraform for infrastructure deployments; Git workflows artifact management; testing frameworks (pytest JUnit).
Knowledge in cost optimization with autoscaling strategies spot vs on-demand auto-termination storage layouts and compaction
Drive real-time ingestion with Kafka/MSK or Kinesis Data Streams/Firehose
Comprehend observability metrics lineage SLAs and alerting for data systems
Experience in financial services or similarly regulated environments

Required Experience:

Key Skills

Spring
.NET
C/C++
Go
React
OOP
C#
Data Structures
JavaScript
Software Development
Java
Distributed Systems

Apply Now

About Company

JPMorganChase

JPMorganChase, one of the oldest financial institutions, offers innovative financial solutions to millions of consumers, small businesses and many of the world’s most prominent corporate, institutional and government clients under the J.P. Morgan and Chase brands. Our history spans ov ... View more

View Profile View Profile

AI AutoApply

Apply to 100+ jobs with one click