Sr. Lead AI Engineer

Job Title: Sr. Lead AI Engineer
Location: Palo Alto CA

This role is based in our new Palo Alto hub. We operate a hubfirst model with regular inoffice collaboration expected.

About the Role

At Klaviyo we believe the future of software isnt just better productivity tools for humansits systems that can run optimize and adapt themselves based on outcome and reward signals. Weve already built the infrastructure and applications that sit between businesses and consumersserving 183000 customers billions of consumer profiles and hundreds of billions of messages and conversion events. Now were investing heavily in Marketing AI and Service AI: the shared services and agentic capabilities that power AInative experiences across Klaviyo.

As a Sr. Lead AI Engineer in the AI & Analytics organization youll be the technical leader for a set of highimpact AI services and agentic systems. Youll own architecture and execution for complex distributed backend systems; guide other engineers through design and implementation; and partner closely with product machine learning and data science to turn AI ideas into reliable scalable production capabilities. This is a hands-on tech lead role not a peoplemanager role.

Youll also be located in our new Palo Alto office. This hub is a center of gravity for AI at Klaviyowith employees in this hub all having machine learning and AI knowledge and experiences but also tightly connected to our other U.S. and international R&D hubs. Youll help shape how Palo Alto collaborates with Boston and global teams and open up clear paths for growth into broader technical leadership staff/principallevel scope or future people leadership if thats the direction you want to go.

How Youll Make a Difference

• Define and improve the engineering architecture for Service and Marketing AI capabilities. Lead the design of scalable lowlatency backend systems and APIs that power our AI products and agents for 183K customers handling billions of events and interactions.
• Build and harden AI serving systems. Lead the development of services that host and orchestrate AI models (LLMs tools evaluators retrieval systems) with clear contracts SLOs and observability so other teams can depend on them.
• Evolve our agentic architecture. Drive the technical roadmap for Service and Marketing AI agentsimproving tool use autonomy and reliability so agents can safely take actions on behalf of customers and internal users with minimal human intervention.
• Set engineering standards for AI services. Establish best practices for evaluation safety/guardrails prompt and model versioning offline and online tests and incident response for AIbacked systems; hold the bar for design reviews and code quality.
• Lead crossteam technical work. Act as the primary technical interface between Service / Marketing AI and other product/platform teamsclarifying ownership boundaries aligning on interfaces and unblocking dependencies across hubs and time zones.
• Mentor and uplevel engineers. Coach senior and midlevel engineers through design implementation and operational best practices helping them grow into stronger system owners and future tech leads.
• Help build the Palo Alto hub. Shape local engineering rituals help with hiring and onboarding and create healthy patterns for crosshub collaboration so work and career opportunities are not constrained by geography.
• Measure what matters. Define and track key metrics for your systemsavailability latency costtoserve agent success rates eval scores and customer adoptionand use those insights to drive roadmap and technical decisions.
• Transform engineering and product development workflows by putting AI at the center building & utilizing agentic coding tools and AI-first working.

Who You Are

• Seasoned backend engineer & tech lead. 8 years of professional software engineering experience with a strong focus on backend and distributed systems; youve led complex projects or areas endtoend and acted as the goto technical owner for key services.
• Handson with generative & agentic AI in production. Youve built and shipped generative or agentic AI applications (e.g. LLMbacked flows toolusing agents retrievalaugmented systems) and are comfortable with prompt design fewshot approaches finetuning and evaluation.
• Deep experience with largescale distributed systems. Youve architected and operated reliable services async processing pipelines and distributed task queues (e.g. Celery Kafka SQS RabbitMQ Redis) supporting highthroughput workloads.
• Strong Python and data tooling fluency. Proficient in Python and modern backend frameworks (FastAPI Django or similar) with experience using big data tools such as Spark/Hadoop and ORMs like SQLAlchemy/Alembic.
• Productiongrade cloud experience. Comfortable with AWS and Kubernetes CI/CD pipelines observability and operational best practices; you treat infrastructure choices as part of the system design not an afterthought.
• Evaluation and qualityobsessed. Youve designed human and automated evals for AI systems know how to instrument for quality and understand how to balance latency cost and response quality in realworld usage.
• Technical leader not just individual contributor. You influence architecture beyond your own code facilitate design reviews rally crossfunctional partners and help teams converge on pragmatic decisions in ambiguous spaces.
• Collaborative and customerfirst. Youre comfortable collaborating directly with PMs ML engineers and customers; you can translate between customer needs product goals and technical constraints and you care deeply about the customer experience.
• Youve already been effectively practicing agentic coding in your daily work. Youre hungry to responsibly explore new AI tools and workflows finding ways to make your work smarter and more efficient.
• Bonus: You use AI tools to accelerate exploration shorten iteration cycles and bring sharper ideas to the tablefor yourself and your teammates.

Nice to Haves

• Prior experience training and deploying ML models (including RL or RLHF) into production systems that drove measurable business impact.
• Experience building shared AI/ML platforms or services used by multiple product teams.
• Background in marketing tech ecommerce or other domains where customer data personalization and experimentation are central.

We use Covey as part of our hiring and / or promotional process. For jobs or candidates in NYC certain features may qualify it as an AEDT. As part of the evaluation process we provide Covey with job requirements and candidate submitted applications. We began using Covey Scout for Inbound on April 3 2025.

Please see the independent bias audit report covering our use of Covey here

Required Experience:

Senior IC