Cloud Economics & Intelligence

Prudentials purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured for our people customers and partners. We provide a platform for our people to do their best work and make an impact to the business and we support our peoples career ambitions. We pledge to make Prudential a place where you can Connect Grow and Succeed.

Key Responsibilities

Design and implement generative AI solutions using Databricks Genie focusing on natural language querying and interactive AI experiences.

Build RAG-based applications integrating retrieval mechanisms with LLMs for improved accuracy and relevance.

Develop and maintain AI workflows using the Databricks Mosaic AI Framework (training fine-tuning inference).

Engineer end-to-end AI pipelines: data ingestion (PySpark SQL) processing (DLT) and deployment (CI/CD Asset Bundles).

Deploy AI products with intuitive UIs using Streamlit or Dash.

Integrate APIs and build advanced agent systems using ReAct Reflection Planning and Deep Agent frameworks.

Optimize performance and minimize cloud costs on Azure and GCP through monitoring and cost-saving strategies.

Leverage LangChain/LangGraph for orchestrating complex AI chains and multi-agent workflows.

Ensure code quality testing documentation and production readiness.

Technical Skills & Qualifications

35 years of hands-on experience in AI/ML engineering with a focus on generative AI.

Proficiency in PySpark and SQL for large-scale data processing.

Hands-on experience with Databricks Genie for AI querying and interaction.

Expertise in Prompt Engineering and optimizing LLM performance.

Strong knowledge of Databricks Mosaic AI Framework (model development serving monitoring).

Experience building RAG solutions (vector stores embeddings retrieval pipelines).

Proficiency with LangChain and LangGraph for AI orchestration and agent workflows.

CI/CD for AI/ML using Databricks Asset Bundles and automated deployment pipelines.

API integration and Agent-to-Agent communication patterns.

Advanced agent frameworks: ReAct Reflection Planning Deep Agent systems.

Databricks Lakehouse architecture and Delta Live Tables (DLT) for real-time pipelines.

Deployment of AI apps using Streamlit or Dash.

Good understanding of Azure and GCP infrastructure and cost optimization strategies (e.g. auto-scaling spot instances resource tagging).

Bachelors degree in Computer Science Data Science Engineering or related field (or equivalent experience).

Preferred

Experience with MLOps practices and production monitoring of AI systems.

Familiarity with Databricks Unity Catalog Model Registry and Feature Store.

Background in vector databases (e.g. FAISS Pinecone Databricks Vector Search).

Professional & Soft Skills

Excellent communication skills ability to explain complex AI concepts to technical and non-technical stakeholders.

Strong problem-solving and analytical thinking in fast-paced environments.

Collaborative team player with experience in cross-functional projects.

Attention to detail in code documentation and system design.

Proactive learner stays updated with advancements in generative AI and Databricks features.

Ownership mindset takes responsibility for deliverables from development to production.

Prudential is an equal opportunity employer. We provide equality of opportunity of benefits for all who apply and who perform work for our organisation irrespective of sex race age ethnic origin educational social and cultural background marital status pregnancy and maternity religion or belief disability or part-time / fixed-term work or any other status protected by applicable law. We encourage the same standards from our recruitment and third-party suppliers taking into account the context of grade job and location. We also allow for reasonable adjustments to support people with individual physical or mental health requirements.