AI OPS Lead

Maintain Dynatrace Davis AI alerting rules service-level problem notification routing (data-plane problems AI Squad Slack infra problems Infra Squad Slack P1 PagerDuty) custom metric event alerting (guardrailtriggerrate > 3 baseline budgetburnrate > 2 baseline) Build and maintain platform runbooks Vault sealed recovery (step-by-step unseal procedure key shard holder contacts) node failure response (which pods need manual rescheduling StatefulSet recovery order) Kafka broker failure (controller election verification partition rebalance monitoring) Participate in on-call rotation incident response per runbook escalation to Lead for P1 incidents post-mortem documentation (5-why root cause timeline contributing factors action items) SLO burn rate monitoring and alerting Operate Argo Rollouts canary deployments monitor canary health via Dynatrace AnalysisRun results intervene on stalled rollouts execute manual rollback when auto-rollback does not trigger within threshold verify <2-minute rollback SLA with stopwatch measurements Manage GitLab Container Registry image retention policy configuration (keep last 10 versions per service delete untagged images weekly) storage quota monitoring registry mirror configuration for air-gapped deployment scenarios Kubernetes Core RKE2 / Rancher cluster provisioning on bare-metal node configuration (kubelet args containerd config iDRAC integration for OOB management) control plane HA etcd operations (snapshot restore member add/remove) RKE2 upgrade procedure (worker nodes first then control plane one node at a time) Workload objects Deployment StatefulSet (stable network identity ordered pod management) DaemonSet (OneAgent OTel Collector must run on every node) Job/CronJob (schema migrations daily backup snapshots) init containers sidecar injection (Istio Vault Agent gVisor) Storage PersistentVolume / PersistentVolumeClaim lifecycle StorageClass provisioner configuration volume expansion CSI driver operation (Rook-Ceph CSI local-path provisioner for scratch storage) Velero backup and restore snapshot CRDs Networking Service types (ClusterIP NodePort LoadBalancer ExternalName) DNS resolution (CoreDNS config tuning) Ingress vs Gateway API EndpointSlice network debugging (kubectl exec curl tcpdump via network debug pods Hubble for Cilium) RBAC and security.