Data Engineer

Role: Data Engineer

Client: MD State

Reference# 71

Number of Positions: Multiple

Location: Primary place of performance is HYBRID with 40% onsite requirement in Baltimore City MD. One week per month onsite is required. On-site attendance may also be required during system rollout activities in addition to the monthly requirement. May require travel to other offices in the Baltimore area as needed.

About the Role: The client is seeking a hands-on Data Engineer to design develop and optimize large-scale data pipelines in support of our Enterprise Data Warehouse (EDW) and Data Lake solutions. This role requires deep technical expertise in coding pipeline orchestration and cloud-native data engineering on AWS. The Data Engineer will be directly responsible for implementing ingestion transformation and integration workflows ensuring data is high-quality compliant and analytics-ready. This role may support multiple projects as needed.

Position Description: Responsible for designing building and maintaining data pipelines and infrastructure to support data-driven decisions and analytics. The individual is responsible for the following tasks:

A. Design develop and maintain data pipelines and extract transform load (ETL) processes to collect process and store structured and unstructured data

B. Build data architecture and storage solutions including data lakehouses data lakes data warehouse and data marts to support analytics and reporting

C. Develop data reliability efficiency and quality checks and processes

D. Prepare data for data modeling

E. Monitor and optimize data architecture and data processing systems

F. Collaboration with multiple teams to understand requirements and objectives

G. Administer testing and troubleshooting related to performance reliability and scalability

H. Create and update documentation

Role and Responsibilities:

Hands-On Data Pipeline Development

• Design code and deploy ETL/ELT pipelines across bronze silver and gold layers of the Data Lakehouse.
• Build ingestion pipelines for structured (SQL) semi-structured (JSON XML) and unstructured data using PySpark/Python programming language using AWS Glue or EMR.
• Implement incremental loads deduplication error handling and data validation.
• Actively troubleshoot debug and optimize pipelines for scalability and cost efficiency.
• EDW & Data Lake Implementation
• Develop dimensional data models (Star Schema Snowflake Schema) for analytics and reporting.
• Build and maintain tables in Iceberg Delta Lake or equivalent OTF formats.
• Optimize partitioning indexing and metadata for fast query performance.
• Healthcare Data Integration
• Build ingestion and transformation pipelines for EDI X12 transactions (etc.).
• Implement mapping and transformation of EDI data with FHIR and HL7 frameworks.
• Work hands-on with AWS Health Lake (or equivalent) to store and query healthcare data.
• Data Quality Security & Compliance
• Develop automated validation scripts to enforce data quality and integrity.
• Implement IAM roles encryption and auditing to meet HIPAA and CMS compliance standards.
• Maintain lineage and governance documentation for all pipelines.
• Collaboration & Delivery
• Work closely with the Lead Data Engineer analysts and data scientists to deliver pipelines that support enterprise-wide analytics.
• Actively contribute to CI/CD pipelines Infrastructure-as-Code (IaC) and automation.
• Continuously improve pipelines and adopt new technologies where appropriate.

Education: This position requires a bachelors or masters degree from an accredited college or university with a major in computer science statistics mathematics economics or a related field. Three (3) years of equivalent experience in a related field may be substituted for the Bachelors degree.

General Experience: The proposed candidate must have a minimum of three (3) years of experience as a data engineer.

Specialized experience: The candidate should have experience as a data engineer or in a similar role with a strong understanding of data architecture and ETL processes. The candidate should be proficient in programming languages for data processing and knowledgeable about distributed computing and parallel processing.

• 3 years of hands-on experience in building deploying and maintaining data pipelines on AWS or equivalent cloud platforms.
• Strong coding skills in Python and SQL (Scala or Java a plus).
• Proven experience with Apache Spark (PySpark) for large-scale processing.
• Hands-on experience with AWS Glue S3 Redshift Athena EMR Lake Formation.
• Strong debugging and performance optimization skills in distributed systems.
• Hands-on experience with Iceberg Delta Lake or other OTF table formats.
• Experience with Airflow or other pipeline orchestration frameworks.
• Practical experience in CI/CD and Infrastructure-as-Code (Terraform CloudFormation).
• Practical experience with EDI X12 HL7 or FHIR data formats.
• Strong understanding of Medallion Architecture for data lake houses.
• Hands-on experience building dimensional models and data warehouses.
• Working knowledge of HIPAA and CMS interoperability requirements.