Splunk Enterprise Architecture and Design

Project Role : Operations Engineer
Project Role Description : Support the operations and/or manage delivery for production systems and services based on operational requirements and service agreement.
Must have skills : Splunk Enterprise Architecture and Design Event management with AIOPS Splunk Enterprise Observability & ITSI
Good to have skills : NA
Minimum 5 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
A Tools & Platforms Site Reliability Engineer (SRE) ensures the reliability availability performance and continuous improvement of the infrastructure engineering tooling estate spanning observability platforms infrastructure-as-code tooling CI/CD pipelines ITSM platforms internal developer portals secret management and AI-augmented operations tooling. The role applies a software engineering discipline to platform operations building automated remediation establishing SLIs and SLOs for tooling platforms reducing toil through systematic automation and owning reliability outcomes end to end across the four tooling pillars.
At Level 7 / 8 this individual operates at the intersection of platform engineering SRE practice and AI operations not just keeping platforms running but continuously raising their reliability ceiling. A distinctive aspect of this role is ownership of LLMOps reliability ensuring AI-augmented operations tooling (runbook automation pipelines agentic ITSM workflows RAG knowledge bases and AI alert correlation services) meets production-grade SLOs in a regulated financial services environment.
Observability SRE
ELK/Splunk
OpenTelemetry
SLI/SLO/Error Budget
IaC & Automation SRE
Terraform
Ansible/Chef
GitHub Actions/ArgoCD
HashiCorp Vault
Policy-as-Code
ITSM & DevOps SRE
ServiceNow
xmatters
Backstage IDP
Jira/Confluence
CMDB Reliability
AI Ops SRE
LLMOps Reliability
Agentic ITSM SRE
AI Alert Pipeline SRE
RAG Platform SRE
Model Observability

Roles & Responsibilities:
Own reliability of observability platforms Splunk defining and maintaining SLIs SLOs and error budgets for metrics pipelines alerting systems and dashboard availability across all infrastructure tiers
Engineer auto-remediation for common observability failures scraper restarts index rollover failures ingest pipeline blockages reducing MTTR and eliminating repetitive manual toil
Implement and govern OpenTelemetry instrumentation standards across the infrastructure estate ensuring telemetry coverage is comprehensive consistent and production-grade
Drive observability-as-code adoption dashboards alert rules SLO definitions and recording rules version-controlled and deployed through GitOps pipelines with automated testing
Perform capacity planning and performance analysis for observability platforms managing cardinality growth storage retention query performance and ingest throughput at scale
Lead blameless post-mortems for observability platform failures producing structured RCA with systemic preventive actions that address root causes rather than symptoms
AI-Augmented Operations SRE
Own reliability of LLMOps pipelines monitoring model API health (OpenAI Anthropic Claude Google Gemini) prompt execution success rates token consumption latency SLOs and cost anomaly alerting for AI-augmented operations tooling
Engineer reliability for agentic ITSM workflows LangChain LlamaIndex CrewAI including agent execution health tool call success rates human-in-the-loop handoff reliability and automated failure recovery
Build observability for RAG knowledge base platforms vector database (Pinecone Weaviate ChromaDB) availability retrieval latency SLOs embedding pipeline health and index freshness monitoring
Implement AI alert correlation reliability ensuring LLM-based alert grouping pipelines maintain accuracy and availability SLOs with fallback to rule-based alerting during AI platform degradation
Define and enforce LLMOps governance frameworks prompt version control model evaluation pipelines output quality monitoring and FSI compliance controls (audit logging data residency) for AI operations tooling
Lead blameless post-mortems for AI tooling failures diagnosing model degradation hallucination events pipeline failures and agent workflow breakdowns with preventive actions that meet FSI audit standards
Professional & Technical Skills:
Certifications
-Terraform Associate or Professional
Splunk Professional
-AWS DevOps Engineer Pro or GCP DevOps Engineer
HashiCorp Vault Associate
-Certified Kubernetes Administrator (CKA)
ITIL Foundation or Practitioner
Must-Have Technical Skills
-Observability SRE: Splunk SLI/SLO/error budget engineering OpenTelemetry ELK/Splunk pipeline reliability and observability-as-code practices
-IaC Reliability: Terraform state backend health drift detection automation module registry SRE and policy-as-code pipeline reliability across AWS and GCP
-CI/CD SRE: GitHub Actions ArgoCD pipeline health monitoring runner auto-scaling deployment success rate SLOs and automated rollback engineering
-Vault Reliability: HA cluster monitoring seal/unseal automation certificate lifecycle management and lease renewal automation for secrets infrastructure
-LLMOps Reliability: Model API health monitoring (OpenAI Anthropic Gemini) prompt execution SLOs token/cost anomaly alerting and AI pipeline auto-remediation
- RAG Platform SRE: Vector database availability (Pinecone Weaviate ChromaDB) retrieval latency SLOs embedding pipeline health and index freshness monitoring
-Automation & Toil Reduction: Python SRE automation scripting event-driven remediation infrastructure SDK integration (boto3 GCP client) and operational workflow engineering
-Incident Management: P1/P2 bridge leadership blameless post-mortems structured RCA error budget reviews and SLA-governed resolution in FSI environments
-Performance & Capacity: Platform capacity trending SLO burn rate alerting cardinality management and proactive capacity interventions across observability and AI tooling
Preferred / Advantageous
Experience with chaos engineering or game day exercises for platform tooling resilience validating failure modes in observability CI/CD or AI pipeline infrastructure
Familiarity with eBPF-based observability (Cilium Pixie) for deep platform telemetry and service mesh reliability engineering
Exposure to model serving infrastructure Triton vLLM or similar for AI/ML platform reliability beyond API-based LLM tooling
Background in SRE or platform engineering within financial services or other highly regulated industries

Additional Information:
SLOs for all platform pillars observability IaC CI/CD ITSM and AI tooling are consistently met with error budgets actively managed and reliability improving measurably quarter-on-quarter
Toil across the tooling estate decreases consistently manual intervention patterns are replaced by automated observable workflows and the teams time shifts toward reliability engineering rather than repetitive operations
Major platform incidents are managed with clear ownership rapid mobilisation blameless RCA outputs and systemic fixes that prevent recurrence
LLMOps and AI-augmented operations tooling meets production SLOs model API failures agent workflow breakdowns and RAG pipeline degradation are detected early remediated automatically where possible and escalated with full context when not
Engineering teams across Cloud Network Security Database and Voice towers rely on platform tooling that is observable self-healing and consistently available the Tools & Platforms SRE is the reason it stays that way
- The candidate should have minimum 5 years of experience in Splunk Enterprise Architecture and Design.
- A 15 years full time education is required.