Lead AWS Data Engineer

Playing an essential role in the U.S. economy Fannie Mae is foundational to housing finance. Here your expertise can help fuel purpose-driven innovation that expands access to homeownership and affordable rental housing across the country. Join Fannie Mae to grow your career and help people find a place to call home.

Job Description

We are seeking an experienced Lead AWS Data Engineer to lead the design development and optimization of large-scale AWS-based Data Lake and data pipeline solutions. The ideal candidate will have deep expertise in AWS data services (EMR Glue S3 Athena Lambda) and PySpark-based data processing along with strong experience in data modeling performance optimization and data pipeline orchestration.

This role requires strong technical leadership and stakeholder collaboration with the ability to analyze complex datasets perform data mapping across systems and translate business requirements into scalable data engineering solutions.

THE IMPACT YOU WILL MAKE

The Lead AWS Data Engineer role will offer you the flexibility to make each day your own while working alongside people who care so that you can deliver on the following responsibilities:

Data Engineering & Architecture

• Design build and maintain scalable AWS Data Lake architectures using services such as S3 EMR Glue Athena and Lambda.
• Develop and optimize data pipelines and ETL/ELT workflows using PySpark AWS Glue and EMR.
• Implement high-performance distributed data processing solutions for large-scale datasets.
• Develop frameworks for data ingestion transformation validation and publishing within the data lake ecosystem.

Performance Optimization

• Diagnose and resolve EMR cluster performance issues including memory management Spark job optimization partitioning strategies and resource allocation.
• Optimize Spark/PySpark workloads for cost and performance.
• Implement monitoring and performance tuning strategies for data processing pipelines.

Data Analysis & Integration

• Analyze complex datasets across multiple systems to support data mapping transformation and integration.
• Define and implement data quality checks and validation frameworks.
• Collaborate with data architects and analysts to develop efficient data models and data flows.

Leadership & Collaboration

• Act as a technical lead for data engineering initiatives and mentor junior engineers.
• Work closely with business stakeholders product owners and data consumers to gather requirements and translate them into technical solutions.
• Provide guidance on data architecture best practices and standards.

Workflow & Automation

• Build and maintain workflow orchestration solutions using tools such as Airflow Step Functions or Glue Workflows.
• Automate deployment and management of data pipelines using CI/CD practices and infrastructure-as-code.

Minimum Required Experiences:

• 4 years of experience in Data Engineering / Big Data development.
• 4 years of hands-on experience with AWS data services.
• Strong experience with AWS EMR AWS Glue S3 Data Lakes Athena / Redshift / Lakehouse architectures.
• Expertise in PySpark and Spark-based distributed processing.
• Basic understanding or exposure to Generative AI concepts and AWS Bedrock services.
• Strong experience building large-scale data pipelines.
• Proven experience with EMR performance tuning and debugging.
• Experience with data mapping and integration across heterogeneous datasets.
• Strong SQL and data modeling skills.
• Excellent communication and stakeholder management skills.

Desired Experiences:

• Bachelor degree or equivalent
• 10 years of experience in Data Engineering / Big Data development.
• 5 years of hands-on experience with AWS data services.
• Hands-on experience with AWS Bedrock including working with foundation models and building GenAI-powered data solutions.
• Experience integrating AI/ML or Generative AI capabilities into data pipelines or analytics platforms.
• AWS Certification (e.g. AWS Certified Data Analytics Machine Learning Specialty or AI/ML-related certifications) preferred.
• Experience with AWS Step Functions.
• Experience with Data Lake governance tools (Lake Formation Glue Catalog).
• Knowledge of data security and compliance frameworks.
• Experience implementing CI/CD pipelines for data platforms.

Target Pay Range: $141000.00 - $184000.00 a year

Internal Job Title: Information Security Technology - Data Engineering - Lead Associate

#LI-JM1

Qualifications

Education:

The future is what you make it to be. Discover compelling opportunities at most roles employees are expected to work onsite on a regular basis at their designated office -office work cadence is determined by your manager. Proximity within a reasonable commute to your designated office location is preferred unless the job is noted as open to remote.

Fannie Mae is an equal opportunity employer and considers qualified applicants for employment without regard to race color religion sex national origin disability age sexual orientation gender identity/gender expression marital or parental status or any other protected factor. Fannie Mae is committed to providing reasonable accommodations to qualified individuals with disabilities who are employees or applicants for employment unless to do so would cause undue hardship to the company. If you need assistance using our online system and/or you need a reasonable accommodation related to the hiring/application process please complete this form.

The hiring range for this role is set forth below. Final salaries will generally vary within that range based on factors that include but are not limited to skill set depth of experience certifications and other relevant qualifications. This position is eligible to participate in a Fannie Mae incentive program (subject to the terms of the program). As part of our comprehensive benefits package Fannie Mae offers a broad range of Health Life Voluntary Lifestyle and other benefits and perks that enhance an employees physical mental emotional and financial well-being. See more here.

Requisition compensation:

to