Principal AI Data Engineer

Contract role: Principal AI Data Engineer

Key Responsibilities

• Develop and evaluate AI/GenAI/AgenticAI prototypes using tools like Copilot Studio AI Foundry and Copilot Analyst Agent Mosiac AI Genie AgentBricks MLflow with a focus on quick wins and enterprise integration.
  
• Build and tune Retrieval-Augmented Generation (RAG) systems including embedding model selection prompt engineering and traceable evaluation.
  
• Design and deploy basic AI agents using frameworks such as LangChain AutoGen and smolagents
  
• Communicate complex AI concepts clearly to business stakeholders and cross-functional teams.
  
• Collaborate on E platform enhancements and work within its current limitations.
  
• Deploy models and applications using Azure OpenAI Azure AI Foundry Databricks Mosaic Gateway and Docker.
  
• Follow DevOps best practices including CI/CD pipelines testing linting and GitHub workflows.
  
• Write modular reusable code using OOP design patterns in Python (Pydantic PyTorch etc.).
  
• Operate in agile teams and contribute to sprint planning reviews and retrospectives.
  
• Deliver hands on GenAI/AgenticAI systems used directly by commercial teams within Trading & Supply taking solutions from prototype to production
  
• Apply engineering skills (emphasis on Databricks) and research skills across experimentation rapid prototyping and iterative delivery. Someone who puts emphasis on reproducibility and open source manages large-scale text and structured datasets on Databricks.
  
• Build AI capability manage stakeholders and communicate effectively to ensure alignment between business needs and AI solutions and a quick understanding of commercial operations that happen in T&S
  
• Design and run evaluation and testing frameworks for GenAI systems including benchmarking reproducibility checks and structured model assessments
  
• Build solutions using Databricks infrastructure Genie MLflow (deployment and tracing and evaluations) LangChain and LangGraph and integrate them into scalable AI workflows and architectures
  
• Contribute to system planning architectural design and structured testing to ensure long term reliability performance and maintainability
  
• Preferably also someone who can set the building blocks and lead building out the backlog
  

Required Skills

• Bachelor or Master or equivalent in Statistics Mathematics Econometrics or similar discipline with at least 8-12 years experience on data science/AI projects.
  
• Deep understanding of LLM families (GPT Llama Claude Mistral) and their reasoning capabilities.
  
• Strong experience with Databricks- DLT Delta Lake concepts UC governance.
  
• Solid understanding of streaming technologies (e.g. Spark Structured Streaming Autoloader)
  
• Programming skills in Python SQL or Scala.
  
• Proficiency in data modelling ETL/ELT processes and data architecture.
  
• Strong analytical background with problem-solving skills.
  
• Performance tuning concepts like watermarking late data handling parallelism & checkpointing.
  
• Hands-on expertise in ADF and Qlik Replicate for data ingestion and replication.
  
• Experience working in Azure cloud environments.
  
• Experience with GenAI evaluation frameworks and benchmarking methodologies.
  
• Experience in MS Copilot AI Foundry Databricks (MosiacAI MLflow Agentbricks Genie)
  
• Strong Git practices and collaborative coding standards.
  
• A passion for and expertise in practicing data science to solve real-world problems.
  
• Excellent oral and written communication skills.
  
• Strong interpersonal skills and enthusiasm for teamwork as well as the ability to work independently.
  
• Familiarity with the enterprise AI platforms and governance models is a plus.
  
• Strong decision-making abilities using data-driven insights to make informed choices that align with organizational goals.
  
• Skills in managing conflicts and facilitating effective resolutions to maintain a positive and productive team dynamic.
  
• Ability to engage with and manage expectations of various stakeholders including executives project managers and other teams.
  
• Proficiency in identifying potential risks in data projects and implementing strategies to mitigate them.
  
• Strong commitment and ownership of project delivery.