Splunk Engineer

Duration: 3 months

Location: Remote

Rate: $50/hr on W2

Keeping a multi-site Splunk Enterprise (indexer clustering SHC) healthy: upgrades/patching daily/weekly health checks capacity & license management DR tests.

Onboarding data cleanly and securely: forwarders/syslog/HEC; sourcetypes props/transforms timestamping/line-breaking field extractions retention.

Improving performance and reliability: monitor ingestion/search performance queues storage/bucket health; remove bottlenecks; tune searches and data models.

Enabling users: create/optimize SPL searches dashboards alerts; advise engineers SREs and SecOps on best practices and troubleshooting.

The most important duties are

Operate and harden a multi-site Splunk Enterprise environment (indexer clustering SHC deployer/deployment server RBAC app lifecycle).

Monitor and tune ingestion search and storage (RF/SF validation; bucket health; NFS tuning; queue depths).

Lead data onboarding projects across on-prem SaaS cloud (Azure/AWS) K8s; ensure auditability and data-handling policy compliance.

Build/optimize SPL dashboards alerts; coach consumers on SPL and performance patterns (tstats accelerations base/inline searches).

Maintain DR posture and execute/verify failovers.

What this job needs to be successful is (traits and characteristics)

3 5 years administering Splunk Enterprise at multi-TB/day scale including indexer clustering and SHC in multi-site deployments.

Expert SPL and performance tuning (tstats data models/accelerations search optimization).

Deep data-onboarding skills (forwarders/syslog/HEC) and mastery (timestamps line-breaking field extraction value normalization).

Strong Linux admin scripting (bash Python); networking/TLS fundamentals.

Experience with NFS-backed indexers (operational tuning/gotchas).

Clear communicator with a customer-enablement mindset; documents well; bias for automation.

Nice-to-have: Splunk Architect cert; experience with ES ITSI MLTK and SOAR; familiarity with data-science/ML concepts (to partner with teams not to lead research).

The simplest and easiest way to see that this job is done well is

Cluster health green: RF/SF consistently met; successful failover tests.

Low ingest error rate and low data latency to index; stable license utilization.

Search KPIs: median and P95 search times within agreed SLOs; reduced scheduler/skipped search rates.

Clean data: correct timestamps low unknown sourcetypes stable field extraction accuracy.

User outcomes: growing self-service usage actionable dashboards/alerts and satisfied internal customers (shorter MTTR for incidents).

No audit/compliance exceptions related to Splunk data handling or access controls.

Basic qualifications

3 5 years hands-on Splunk Enterprise administration at scale (multi-TB/day) including indexer clustering SHC deployer/DS license mgmt.

Strong SPL and performance tuning (tstats DMs accelerations base/inline searches).

Data onboarding expertise: forwarders/syslog/HEC; props/transforms; timestamping/line-breaking; field extractions; retention planning.

Linux scripting (bash/Python); networking/TLS fundamentals.

Experience operating with NFS-backed indexers.