Data Engineering Tech Lead
Share
Job Location:
Monthly Salary:
Experience Required:
Posted on:
2 hours ago
Vacancies:
1 Vacancy
Job Summary
Role: Data Engineering Tech Lead
Job Category:GCB4
Location:UK
Key Responsibilities:
Design implement and maintain data pipelines to ingest and process OpenShift telemetry (metrics logs traces) at scale.
Stream OpenShift telemetry via Kafka (producers topics schemas) and build resilient consumer services for transformation and enrichment.
Engineer data models and routing for multi-tenant observability; ensure lineage quality and SLAs across the stream layer.
Integrate processed telemetry into Splunk for visualisation dashboards alerting and analytics to achieve Observability Level 4 (proactive insights).
Implement schema management (Avro/Protobuf) governance and versioning for telemetry events.
Build automated validation replay and backfill mechanisms for data reliability and recovery.
Instrument services with OpenTelemetry; standardise tracing metrics and structured logging across platforms.
Use LLMs to enhance observability capabilities (e.g. query assistance anomaly summarization Runbook generation).
Collaborate with platform SRE and application teams to integrate telemetry alerts and SLOs.
Ensure security compliance and best practices for data pipelines and observability platforms.
Document data flows schemas dashboards and operational Runbook.
Required Skills:
Hands-on experience building streaming data pipelines with Kafka (producers/consumers schema registry Kafka Connect/KSQL/KStream).
Proficiency with OpenShift/Kubernetes telemetry (OpenTelemetry Prometheus) and CLI tooling.
Experience integrating telemetry into Splunk (HEC UF source types CIM) building dashboards and alerting.
Strong data engineering skills in Python (or similar) for ETL/ELT enrichment and validation.
Knowledge of event schemas (Avro/Protobuf/JSON) contracts and backward/forward compatibility.
Familiarity with observability standards and practices; ability to drive toward Level 4 maturity (proactive monitoring automated insights).
Understanding of hybrid cloud and multi-cluster telemetry patterns.
Security and compliance for data pipelines: secret management RBAC encryption in transit/at rest.
Good problem-solving skills and ability to work in a collaborative team environment.
Strong communication and documentation skills.
Required Skills:
data Engineer
Company Industry
IT Services and IT Consulting
Key Skills
- Administrative Skills
- Facilities Management
- Biotechnology
- Creative Production
- Design And Estimation
- Architecture
