Senior Data Engineer AI and ML frameworks
Senior Data Engineer AI and ML frameworks
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Data Standardization and Transformation:
Convert diverse data structures from various EHR systems into a unified format based on FHIR standards
Map and normalize incoming data to the FHIR data model ensuring consistency and completeness
Kafka Integration:
Consume and process events from the Kafka stream produced by the Data Writer Module
Deserialize and validate incoming data to ensure adherence to required standards
Data Segmentation:
Separate data streams for warehousing and AI model training applying specific preprocessing steps for each purpose
Prepare and validate data for storage and machine learning model training
Error Handling and Logging:
Implement robust error handling mechanisms to track and resolve data mapping issues
Maintain detailed logs for auditing and troubleshooting purposes
Data Ingestion and Processing:
Use LLMs to extract structured data from EHRs research articles and clinical notes
Ensure semantic consistency and interoperability during data ingestion
Knowledge Graph Construction:
Integrate extracted data into a knowledge graph representing entities and relationships for semantic data integration
Implement contextual understanding and querying of complex relationships within the knowledge graph (KG)
Advanced Predictive Modeling:
Leverage KGs and LLMs to enhance data interoperability and predictive analytics
Develop frameworks for contextualized insights and personalized medicine recommendations
Feedback Loop:
Continuously update the knowledge graph with new data using LLMs ensuring uptodate and relevant insights.
Work Closely with CrossFunctional Teams
Collaborate with data scientists AI specialists and software engineers to design and implement data processing solutions
Communicate eectively with stakeholders to align on goals and deliverables
Contribute to Engineering Culture:
Foster a culture of innovation collaboration and continuous improvement within the engineering team
Qualifications :
- Deep understanding of patterns and software development practices for eventdriven architectures
- Handson experience with stateful stream data processing solutions (Kafka or similar streaming platforms)
- Strong knowledge of data serialization/deserialization using various data formats (at minimum JSON and Avro) and integration with schema registries
- Proven Python software development expertise with experience in data processing and integration (most of the software is written in Python)
- Practical experience building endtoend solutions with Apache Flink or a similar platform
- Experience with containerization and orchestration using Kubernetes (K8s) and Helm especially on Google Kubernetes Engine (GKE)
- Familiarity with Google Cloud Platform (GCP) or a similar cloud platform
- Handson experience implementing data quality solutions for schemaonread or schemaless data
- Handson experience integrating with Apache Kafka particularly the Confluent Platform
- Familiarity with AI and ML frameworks
- Proficiency in SQL and experience with both relational and NoSQL databases
- Experience with graph databases like Neo4j or RDFbased systems
- Experience in the healthcare domain and familiarity with healthcare standards such as FHIR and HL7 for data interoperability
WOULD BE A PLUS:
- Experience with web data sing
Additional Information :
PERSONAL PROFILE
Strong problemsolving skills with the ability to design innovative solutions for complex data integration and processing challenges
Excellent communication skills with the ability to articulate complex technical concepts and work eectively with various stakeholders
Commitment to improving healthcare through datadriven solutions and technology
Stay abreast of the latest technologies and industry trends while continually improving your skills and knowledge
Ability to work in a collaborative environment valuing diverse perspectives and contributing to a positive team culture
Remote Work :
Yes
Employment Type :
Fulltime
Employment Type
Remote
