Sr. Databricks Data Engineer

Halvik Corp delivers a wide range of services to 13 executive agencies and 15 independent agencies. Halvik is a highly successful WOB business with more than 50 prime contracts and 500 professionals delivering Digital Services Advanced Analytics Artificial Intelligence/Machine Learning Cyber Security and Cutting-Edge Technology across the US Government. Be a part of something special!

Position Overview

The Senior Databricks Data Engineer / Technical Lead will support the U.S. Department of Transportation (DOT) data modernization effort by designing building migrating and operating secure scalable data pipelines in a Databricks Lakehouse. This is a highly hands-on role focused on legacy and external-source migration metadata-driven ingestion Medallion Architecture data quality and reconciliation production operations and transition to the AWS/OneDOT environment.

Core Responsibilities

• Strategic Execution: Lead the design and implementation of reusable ingestion and transformation patterns for legacy and external data sources into Databricks including support for migration to the AWS/OneDOT environment.
• Operational Oversight: Build and operate Bronze Silver and Gold pipelines; manage batch and incremental loads orchestration dependencies retries/recovery monitoring alerting and production support.
• Collaboration: Partner with the Technical Project Manager Data Architects business/data SMEs BI developers Cloud/DevSecOps security and governance teams to translate requirements into implementable data solutions.
• Technical Performance: Develop and optimize Python PySpark and SQL workloads using Databricks Delta Lake Unity Catalog Auto Loader and/or Lakeflow capabilities and Databricks Jobs/Workflows with a focus on reliability performance and cost.
• Data Quality & Governance: Implement source-to-target reconciliation data validation schema evolution lineage access controls documentation and reusable engineering standards for production data pipelines.

Minimum Requirements

• Education: Bachelors degree in computer science Data Engineering Information Technology Engineering or a related field; equivalent relevant experience may be considered.
• Experience: Minimum of 7 years of professional data engineering experience including significant hands-on experience with Databricks/Spark and production data pipelines.
• Technical Proficiency: Advanced Python PySpark and SQL skills; strong experience with Databricks Delta Lake AWS/S3 batch/file ingestion (CSV/JSON) REST APIs Git/CI/CD and data pipeline orchestration.
• Compliance: Must meet applicable DOT contract security and suitability requirements and demonstrate secure handling of PII and other sensitive data including access controls and secrets management.

• Certifications: Relevant Databricks and/or AWS data engineering certification preferred; equivalent demonstrated hands-on experience will be considered.

Preferred Expertise

• Demonstrated success modernizing legacy data platforms (Sybase/SAP IQ or similar) into Databricks including source-to-target reconciliation cutover and production stabilization.
• Experience with Unity Catalog metadata/configuration-driven ingestion Auto Loader/Lakeflow Databricks Jobs/Workflows lineage/cataloging tools and scalable multi-source ingestion frameworks.
• Federal/DOT delivery experience; familiarity with Agile/Kanban production O&M cloud cost optimization and data governance. Streaming and AI/ML data pipeline exposure is a plus.