Senior Data Engineer
Share
Job Location:
Louisville, KY - USA
Monthly Salary:
Not Disclosed
Posted on:
22 hours ago
Vacancies:
1 Vacancy
Job Summary
Our Company
BrightSpring Health Services
Overview
We are seeking an experienced Senior Data Engineer to join our team in building a next generation AI-enabled Health Cloud Platform. This platform powers intelligent clinical and pharmacy solutions using Generative AI Agentic AI Robotic Process Automation (RPA) and Retrieval-Augmented Generation (RAG) with domain-specific vector databases.
As a Senior Data Engineer you will design and optimize data pipelines data lakes and analytics frameworks that enable scalable AI/ML workflows. You will work closely with AI architects data scientists and product teams to ensure high-quality secure and compliant data infrastructure for healthcare applications.
Responsibilities
- Design build and maintain data pipelines for ingesting transforming and storing large-scale healthcare datasets.
- Implement ETL/ELT workflows using Databricks Spark and Delta Lake for high performance data processing.
- Optimize data architecture for AI/ML workloads including support for vector databases and RAG pipelines.
- Collaborate with AI teams to integrate structured and unstructured data into predictive and generative AI models.
- Ensure compliance with healthcare data standards (FHIR HL7 RxNorm) and regulatory requirements (HIPAA).
- Develop and enforce data governance security and quality frameworks.
- Monitor and improve data pipeline performance scalability and reliability.
- Mentor junior engineers and contribute to best practices in data engineering and MLOps.
Qualifications
- Bachelors or Masters degree in Computer Science Data Engineering or related field.
- 10 years of experience in data engineering with strong expertise in Databricks Spark and distributed data systems.
- Proficiency in Python SQL and experience with data modeling and data lakehouse architectures.
- Hands-on experience with cloud platforms (Azure preferred; AWS/GCP acceptable).
- Experience with vector databases (FAISS Weaviate Pinecone) and RAG pipelines.
- Familiarity with MLOps tools (MLflow Airflow or similar) and data versioning.
- Strong understanding of healthcare data standards and compliance frameworks.
- Excellent problem-solving skills and ability to work in a fast-paced mission-driven environment.
- Knowledge of Generative AI and Agentic AI frameworks (LangChain AutoGen).
- Familiarity with data security privacy and regulatory compliance in healthcare.
- Contributions to open-source projects or experience in large-scale data environments.
- Travel up to 25%
About our Line of Business
Required Experience:
Senior IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Welcome to SpringHealth Behavioral Health & Integrated Care where we assist those we serve in reaching their highest level of independence.
