Associate Principal, AI Engineer
San Diego, CA - USA
Job Summary
Location
This role is located at our HQ in San Diego CA.
Position Summary
The Assoc Principal AI Engineer is the most senior individual contributor on the AI Engineering team responsible for the design development andproductionizationof the most complex AI systems in the organization. This is a deeply technical hands-on role for an engineer who has spent years in thetrenchesbuilding training fine-tuning and shipping AI systems at scale and is now ready to set technical direction across multiple teams.
The role combines applied research with production engineering. The Assoc Principal AI Engineer translates the latest advances in foundation models agentic systems and machine learning into robust observable and economicallyviableproduction systems. They write code design systems lead the hardest technical decisions and shape the engineering culture thatdetermineshow AI gets built across the company.
Key Responsibilities
Technical Leadership
- Set the technical direction for AI Engineering across foundation model integration fine-tuning pipelines RAG systems agentic workflows and evaluation infrastructure.
- Own the most complex and ambiguous AI engineering problems in the company frominitialdesign through production deployment and ongoing optimization.
- Establish engineering standards for model development prompt management evaluation deployment and observability that the rest of the AI organization adopts.
- Lead architecture reviews and serve as the senior technical reviewer for high-stakes AI initiatives.
AI Systems Development
- Design and build production-grade Generative AI systems including retrieval-augmented generation multi-agent orchestration tool-using agents and domain-adapted models.
- Develop fine-tuning distillation and post-training pipelines using techniques such as SFT DPO RLHF and parameter-efficient methods (LoRAQLoRA adapters).
- Architect and implement vector retrieval systems semantic search and hybrid retrieval pipelinesoptimizedfor accuracy latency and cost.
- Build robust evaluation frameworks covering automated metrics LLM-as-judge human review regression testing and safety evaluations.
Platform and Infrastructure
- Design and build the AI platform that powers internal teams including model serving infrastructure prompt and prompt-template management experiment tracking and feature stores.
- Optimizeinference performance across latency throughput and cost including quantization batching caching speculative decoding and intelligent routing across model providers.
- EstablishLLMOpspractices for continuous evaluation drift detection prompt versioning rollback strategies and incident response.
- Partner with platform and infrastructure teams to ensure AI workloads run reliably on GPU and accelerator hardware across cloud environments.
Research to Production
- Stay current with the rapidly evolving AI research landscape andidentifywhich advances translate into production value for the business.
- Prototype emerging techniques (new model architectures training methods agent frameworks) and lead the path from experiment to production system.
- Contribute to internal technical strategy on build versus buy decisions for foundation models vector databases agent frameworks and AI tooling.
Cross-Functional Influence
- Partner with product data science research and business stakeholders to scope AI initiatives and shape solutions that deliver measurable business impact.
- Mentor senior and staff engineers raising the technical bar across the AI organization.
- Represent AI Engineering in executive forums customer conversations vendor evaluations and industry engagements.
- Author technical documents designdocs and (whereappropriate)external publications that contribute to the broader AI community.
Required Qualifications
- 12 years of software engineering experience with 6 years focused on machine learning or AI systems and 2 years buildingproductionGenerative AI applications.
- Demonstrated ownership of large-scale AI systems in production including responsibility for latency cost accuracy and reliability outcomes.
- Deep hands-onexpertisein Python and modern ML frameworks (PyTorch TensorFlow JAX Hugging Face Transformers).
- Strong command of LLM application development including RAG architectures prompt engineering function calling structured outputs and agentic patterns.
- Experience with model fine-tuning evaluation and deployment lifecycles across at least one major cloud platform (GCP Azure or AWS).
- Proven ability to design distributed systems including familiarity with vector databases message queues container orchestration and observability stacks.
- Bachelors orMasters degree in Computer Science Machine Learning Statistics ora relatedquantitative discipline. PhD welcomed but notrequired.
Preferred Qualifications
- Experience training fine-tuning or post-training foundation models using techniques such as SFT DPO RLHF RLAIF or constitutional methods.
- Familiarity with agentic frameworks (LangChainLangGraphAutoGenCrewAI custom orchestration) and multi-agent system design patterns.
- Background in Voice AI speech systems multimodal models or computer vision applied at production scale.
- Contributions toopen sourceAI projects peer-reviewed publications or notable conference presentations.
- Experience in regulated or high-stakes domains (life sciences healthcare financial services) where accuracy safety and governance requirements are stringent.
- Familiarity with responsible AI practices including red-teaming jailbreak resistance content safety bias evaluation and AI governance frameworks.
- Typically requires a minimum of 15 years of related experience with a Bachelors degree; or 12 years and a Masters degree; or a PhD with 8 years experience; or equivalent experience.
Technical Skill Profile
Foundation Models and LLMs:GPT-4 class models Claude Gemini open-weight models (Llama Mistral Qwen) fine-tuning techniques instruction tuning alignment methods.
AI Frameworks and Tooling:PyTorch Hugging Face TransformersLangChainLangGraphLlamaIndexDSPy RayvLLMTensorRT-LLM model serving frameworks.
RAG and Retrieval:Vector databases (PineconeWeaviatepgvector Vertex AI Vector Search) embedding models reranking hybrid search chunking strategies query understanding.
Evaluation and Observability:RAGASDeepEval custom eval harnessesLangSmith Weights and BiasesArizeOpenTelemetryfor AI workloads.
Programming and Engineering:Python (expert) oneadditionalsystems language (Go Java or Rust) SQL distributed systems microservices API design.
Cloud and Platform:GCP (Vertex AIAlloyDB GKE) Azure (Azure AI Foundry AKS) AWS (Bedrock SageMaker EKS) Docker Kubernetes Terraform.
Data:Streaming and batch pipelineslakehousearchitectures feature stores data versioning (DVCLakeFS) high-throughput ETL.
Engineering Competencies
- Technical Depth:Expert-level mastery of AI engineering with the ability tooperatefrom research papers down to production code.
- Systems Thinking:Comfort designing systems that span multiple services data stores model providers and failure modes.
- Pragmatism:Strong instinct for when to build when to buy and when to wait witha track recordof avoiding over-engineering.
- Communication:Ability to explain complex AI concepts to executives write design docs that drive decisions and influence peers across disciplines.
- Builders Mindset:Genuine enjoyment of writing code and solving hard technical problems not just reviewing or directing others.
- Curiosity and Continuous Learning:Active engagement with the AI research landscape and a habit of trying new things.
We are a company deeply rooted in belonging promoting an inclusive environment where employees feel valued and empowered to contribute to our mission. Built on a strong foundation Illumina has always prioritized openness collaboration and seeking alternative perspectives to propel innovation in genomics. We are proud to confirm a zero-net gap in pay regardless of gender ethnicity or race. We also have several Employee Resource Groups (ERG) that deliver career development experiences increase cultural awareness and offer opportunities to engage in social responsibility. We are proud to be an equal opportunity employer committed to providing employment opportunity regardless of sex race creed color gender religion marital status domestic partner status age national origin or ancestry physical or mental disability medical condition sexual orientation pregnancy military or veteran status citizenship status and genetic information. Illumina conducts background checks on applicants for whom a conditional offer of employment has been made. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable local state and federal laws. Background check results may potentially result in the withdrawal of a conditional offer of employment. The background check process and any decisions made as a result shall be made in accordance with all applicable local state and federal laws. Illumina prohibits the use of generative artificial intelligence (AI) in the application and interview process. If you require accommodation to complete the application or interview process please contact To learn more visit: The position will be posted until a final candidate is selected or the requisition has a sufficient number of qualified applicants.
Required Experience:
IC