Databricks Lead

Index Analytics LLC is a rapidly growing Baltimore-based small business providing health-related consulting services to the federal government. At the center of our company culture is a commitment to instilling a dynamic and employee-friendly place to work. We place a priority on promoting a supportive and collegial team environment and enhancing staff experience through career development and educational opportunities.

Position Overview

The incumbent willleadthe development and implementation of Databricks pipelines across multiple environments with access from the Clients AWS Government Cloud. As part of a cross-functional project team that includes data engineers database architects analysts and analytic SMEs the Databricks Lead will implement a scalable Databricks environment capable of ingesting diverse data sources to support advanced data analytics.

Responsibilities

• Architect develop and implement data models and data engineering pipelines to support ingestion of diverse CMS data sources applying complex business rules derived from Python/R code reviews and enable analytics through optimized Apache Spark datasets while maintaining strong security and governance practices.
• Engineer and manage endtoend data processing workflows including receiving data files in multiple formats applying validated business logic loading transformed data into production pipelines delivering recurring and adhoc data extracts to meet stakeholder and program needs.
• Support and administer the Databricks environment including user and access control management for clusters jobs notebooks DDL deployments optimizing performance monitoring usage and delivery of cost and utilization reporting.
• Ensure data quality and maintain documentation conducting comprehensive QA validation on all ingested and delivered datasets managing codebooks and user training guides as well as maintaining complete metadata sourcetotarget mappings and technical artifacts according to established schedules.

• U.S. citizen or otherwise authorized to work in the United States and able to demonstrate physical residency in the U.S. for at least three (3) of the past five (5) years.
• Bachelors degree or higher in computer science or relevant discipline with eight (8) or more years of professional work experience including six (6) or more years of data engineering experience designing and building pipelines across a variety of database technologies. Databricks certified Data Engineer certifications are a plus.
• Three (3) or more years of handson experience with the Databricks Platform and developer tools including performance tuning use of Databricks CLI and REST APIs and cost optimization for Databricks compute and storage.
• Two (2) or more years of handson programming experience with SAS Python JavaScript or similar tools.
• Working experience with Databricks integrations including Delta Lake LakeFlow and Lakehouse Federation.
• Proven experience with Databricks Unity Catalog and related governance/security frameworks as well as working knowledge of database security including Access Control Lists (ACLs) Unity Catalog permissions and RBAC controls.
• Proficiency with cloud platforms such as AWS or Azure including database deployment automation and CI/CD pipelines in an AWS environment.
• Experience working with unstructured and semistructured data (XML JSON Parquet video audio images PDF) is a plus.
• Strong communication skills and commitment to continuous learning.

Attention Candidates

Were dedicated to ensuring a safe and transparent recruitment process for all candidates and have implemented robust measures to protect your personal information. Please be aware that all employment-related communications will originate from a secure portal () or a corporate email address (). If you have any concerns please dont hesitate to reach out to us at.

If you are selected for an interview please be advised that Index Analytics LLC reserves the right to prohibit the use of artificial intelligence (AI) tools including but not limited to AI-generated responses real-time transcription or automated assistance during the interview process. We value authentic interactions and the opportunity to engage directly with candidates. Any unauthorized use of AI may result in disqualification from consideration.

The salary range provided represents the estimated compensation for new hires in this position applicable across all locations. Actual offers may vary based on factors such as the candidates skills qualifications experience and market conditions. Index complements its base salary offering with a competitive package that includes health and retirement benefits discretionary bonuses and reimbursement for professional development opportunities.

Index Analytics provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race color religion age sex national origin disability status genetics protected veteran status sexual orientation gender identity or expression or any other characteristic protected by federal state or local laws. This policy applies to all terms and conditions of employment including recruiting hiring placement promotion termination layoff recall transfer leaves of absence compensation and training.