Lead AI Engineer

Job Title: Lead AI Engineer Marketing & Service AI
Location: Palo Alto CA

This role is based in our new Palo Alto hub. We operate a hubfirst model with regular inoffice collaboration expected.

About the Role

At Klaviyo we believe the future of software isnt just better productivity tools for humansits systems that can run optimize and adapt themselves based on outcome and reward signals. Weve already built the infrastructure and applications that sit between businesses and consumersserving 183000 customers billions of consumer profiles and hundreds of billions of messages and conversion events. Now were investing heavily in Marketing AI and Service AI: the shared services and agentic capabilities that power AInative experiences across Klaviyo.

As a Lead AI Engineer in the AI & Analytics organization youll play a key role in designing and building scalable backend systems and user experiences that power our AI products and agentic solutions. Youll own complex services endtoend contribute to architecture for highimpact AI features and partner closely with product managers machine learning engineers and data scientists to turn AI ideas into reliable scalable production capabilities. This is a handson backendheavy role with opportunities to influence architecture async processing pipelines distributed systems and morewithout being the overall area tech lead.

Youll be based in our new Palo Alto office a center of gravity for AI at Klaviyosurrounded by teammates with deep ML and AI experience and tightly connected to our other U.S. and international R&D hubs. Youll collaborate daily with engineers in Boston and other locations and youll have a clear path to grow into broader technical leadership staff/principal scope or future people leadership if thats the direction you want to gowithout being constrained by hub location.

How Youll Make a Difference

• Design and build core AI services. Implement scalable lowlatency backend systems and APIs that power our Marketing and Service AI capabilities for 183K customers handling billions of events and interactions.
• Scale AI data and inference pipelines. Develop robust reliable and scalable data collection and processing pipelines so our generative and agentic models have the features context and feedback they need to perform well in production.
• Build and harden AI serving systems. Implement services that host and orchestrate AI models (LLMs tools evaluators retrieval systems) with strong contracts logging and observability so other teams can depend on them.
• Evolve our agentic architecture. Help evolve how our agents plan call tools and react to feedbackshipping improvements that increase autonomy reliability and safety for both customerfacing and internal workflows.
• Apply and refine best practices for AI systems. Use and help improve standards for evaluation safety/guardrails prompt and model management offline/online tests and incident response for AIbacked systems.
• Collaborate across teams. Work closely with product ML data and platform teams to clarify requirements align on interfaces and unblock dependencies for the services you ownespecially across hubs and time zones.
• Mentor and uplevel teammates. Provide thoughtful code reviews share patterns and examples and help mid-level and junior engineers grow stronger in building distributed and AIpowered systems.
• Help shape the Palo Alto hub. Participate in interviewing onboarding and local engineering rituals; contribute ideas that make Palo Alto a highly collaborative highbar hub that works seamlessly with other locations.
• Measure what matters. Instrument your services and use metricsavailability latency costtoserve agent success rates eval scores and customer adoptionto guide your own technical decisions and influence the team roadmap.
• Youve already been effectively practicing agentic coding in your daily work. Youre hungry to responsibly explore new AI tools and workflows finding ways to make your work smarter and more efficient.

Who You Are

• Experienced backend engineer. 57 years of professional software engineering experience with a strong focus on backend and distributed systems; youve led complex projects endtoend within a team and owned services in production.
• Handson with generative & agentic AI in production. Youve built and shipped generative or agentic AI applications (e.g. LLMbacked flows toolusing agents retrievalaugmented systems) and are comfortable with prompt design fewshot approaches finetuning and evaluation.
• Strong distributed systems and async background. Youve built reliable services async processing pipelines and distributed task queues (e.g. Celery Kafka SQS RabbitMQ Redis) that support highthroughput workloads.
• Fluent in Python and data tooling. Proficient in Python and modern backend frameworks (FastAPI Django or similar) with experience using big data tools such as Spark/Hadoop and ORMs like SQLAlchemy/Alembic.
• Comfortable in cloudnative environments. Experience with AWS and Kubernetes CI/CD pipelines observability and operational best practices; you understand how infrastructure choices affect reliability latency and cost.
• Evaluation and qualityminded. Youve created human and/or automated evals for AI systems instrumented for quality and understand how to balance latency cost and response quality in realworld usage.
• Collaborative and customerfirst. You enjoy working directly with PMs engineers in other teams and sometimes customers; you care deeply about the customer experience and use data and feedback to guide decisions.
• Growthoriented teammate. You communicate clearly give and receive feedback well and contribute to a team culture that is humble ambitious and valuesaligned.
• Youve already experimented with AI in work or personal projects and youre excited to dive in and learn fast. Youre hungry to responsibly explore new AI tools and workflows finding ways to make your work smarter and more efficient.
• Youve already been effectively practicing agentic coding in your daily workor youre eager to deepen that practiceusing AI assistants and tools to scaffold code run experiments and iterate faster while keeping yourself in the drivers seat.
• Bonus: You use AI tools to accelerate exploration shorten iteration cycles and bring sharper ideas to the tablefor yourself and your teammates.

Nice to Haves

• Prior experience training and deploying your own ML models (including RL or RLHF) into production systems that drove measurable business impact.
• Experience building shared AI/ML platforms or services used by multiple product teams.
• Background in marketing tech ecommerce or other domains where customer data personalization and experimentation are central.

We use Covey as part of our hiring and / or promotional process. For jobs or candidates in NYC certain features may qualify it as an AEDT. As part of the evaluation process we provide Covey with job requirements and candidate submitted applications. We began using Covey Scout for Inbound on April 3 2025.

Please see the independent bias audit report covering our use of Covey here

Required Experience:

IC