Principal Engineer

We are seeking a Sr. Staff / Principal AI Systems Engineer to build scale and operate the AI Agent Platform that powers reasoning-driven assistants and autonomous agents across Freshworks and to make that platform operate itself. You will own the systems architecture and engineering of a multi-tenant agent runtime that serves agentic workloads at high throughput and low latency and you will pioneer an Agentic AIOps approach where autonomous agents monitor diagnose and remediate the platform in production.

This is a hands-on systems engineering role at the staff/principal level with a strong platform-scale and agentic-operations center of gravity. Youll design how thousands of agents are orchestrated served observed and kept healthy at enterprise scale and youll set the technical direction which the broader engineering organization builds against. The level (Sr. Staff vs. Principal) will be calibrated to your scope of technical ownership and organizational impact.

Key Responsibilities:

Scalable Agent Platform Systems

• Architect and build the core AI Agent Platform agent runtime orchestration tool/API invocation state and memory management and the retrieval/knowledge services agents reason over
• Design for scale and efficiency: high-throughput multi-tenant serving concurrency and queueing for agent workloads model/inference routing caching and cost-aware execution
• Build the control plane and systems primitives other teams use to define deploy version and operate agents safely
• Drive latency throughput and cost optimization across the agentic request path (planning retrieval tool calls generation)

Agentic AIOps & Autonomous Operations

• Architect agentic operations workflows where autonomous agents observe platform telemetry reason about anomalies perform root-cause analysis and execute remediation shifting operations from human-driven to agent-driven
• Design multi-agent operational loops (detection diagnosis remediation) that collaborate escalate and hand off to on-call humans with clear rationale and audit trails
• Build closed-loop self-healing for the platform: auto-detection and repair of failing agents degraded tools/connectors stale knowledge failed ingestion and retrieval/index drift
• Define guardrails confidence thresholds and human-in-the-loop controls that make autonomous remediation safe at multi-tenant scale
• Apply LLMs to operations directly incident summarization runbook generation on-call copilots and natural-language querying of platform telemetry

Observability for Agentic Systems

• Instrument the platform end to end: distributed tracing across planning-retrieval-tool-generation loops metrics structured logging and event correlation so multi-agent behavior is explainable and debuggable
• Define golden signals for both system health and agent quality task success rate tool-call accuracy grounding/hallucination rates latency cost-per-task throughput as first-class telemetry the operating agents act on
• Establish SLOs/SLIs and error budgets for agent workflows with alerting that feeds the agentic-ops layer

Reliability SRE & Distributed Systems

• Engineer the platform as a resilient event-driven cloud-native distributed system (Kubernetes streaming pipelines microservices) across regions and tenants
• Drive SRE practices capacity planning graceful degradation failover chaos/resilience testing blameless incident response and progressively automate them through agents
• Build for operability first: every component designed to be observed diagnosed and acted on autonomously

Knowledge & Retrieval (the domain agents operate over)

• Guide the RAG/retrieval and knowledge services agents depend on ensuring health freshness and quality are continuously monitored and remediated by the agentic-ops layer
• Oversee enterprise content ingestion and sync (Confluence SharePoint Google Drive Salesforce KB ServiceNow KB) and multi-modal retrieval at platform scale

Technical Leadership

• Set technical direction for the agent platform and agentic operations; mentor engineers and drive cross-team systems architecture decisions.
• Partner with product and platform leadership on the long-term strategy for self-operating enterprise-grade agentic AI

Qualifications :

Required

• 10 years building production software with deep experience designing and operating large-scale distributed systems and platforms
• Proven experience building scalable AI/agent platforms or high-throughput ML serving systems in production orchestration multi-tenancy latency/cost optimization
• Hands-on experience designing agentic or autonomous workflows multi-agent reasoning tool/API invocation planning loops applied to real production problems
• Strong AIOps and SRE background: observability tooling (OpenTelemetry Prometheus Grafana distributed tracing) SLOs/error budgets anomaly detection incident management and closed-loop automation with human-in-the-loop safeguards
• Hands-on experience applying LLMs to production workflows (reasoning decision support summarization)
• Strong proficiency in Python and a systems language (Go/Java); cloud-native architecture (Kubernetes event-driven microservices streaming pipelines).
• Working familiarity with RAG and knowledge systems (retrieval embeddings knowledge graphs) sufficient to architect over them depth here is a plus not the primary bar
• For Principal level: demonstrated org-wide technical influence setting architecture direction across teams and driving large ambiguous systems initiatives end to end

Preferred

• ML-driven anomaly detection alert correlation and predictive operations at scale
• SRE leadership operating AI/ML or data-intensive platforms
• Familiarity with agentic frameworks (LangChain LangGraph) and vector/graph stores
• AI safety and governance grounding for autonomous enterprise systems.
• Contributions to open-source agentic observability or AIOps frameworks

Additional Information :

Please note this is a hybrid role with onsite expectations of 3x/week (Tues - Thurs) from our San Mateo CA headquarters. 

The annual base salary range for this position is $216000- $298000. This role is also eligible for a target bonus. 

Compensation is based on a variety of factors including but not limited to location experience job-related skills and level. 

Freshworks offers multiple options for dental medical vision disability and life insurance. Equity ESPP flexible PTO flexible spending commuter benefits and wellness benefits are also offered. Freshworks also offers adoption and parental leave benefits. 

At Freshworks we have fostered an environment that enables everyone to find their true potential purpose and passion welcoming colleagues of all backgrounds genders sexual orientations religions and ethnicities. We are committed to providing equal opportunity and believe that diversity in the workplace creates a more vibrant richer environment that boosts the goals of our employees communities and business. Fresh vision. Real impact. Come build it with us.

Remote Work :

No

Employment Type :

Full-time