Kafka Administrator

New York City, NY - USA

Monthly Salary: Not Disclosed

Posted on: 3 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Title: Kafka Admin

Location: NYC NY (3 days onsite minimum)

Duration: 6 months

Job Summary:

The ideal candidate will be responsible for designing deploying maintaining and scaling Kafka clusters in mission-critical environments while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform.

Responsibilities

Design implement and maintain highly available Kafka clusters in production environments.
Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
Administer security protocols and access controls to protect the platform.
Monitor cluster health and performance using our established monitoring stack (Prometheus Grafana)
Troubleshoot and resolve cluster-related incidents such as broker outages replication lag.
Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
Assist with planning and managing cluster resources to support future growth.
Develop Kafka topics partitions ACLs schema registry and optimize throughput and reliability.
Manage a tune of Kafka Connect Kafka Streams and Mirror Maker as needed for data pipelines.
Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL Ubuntu).
Automate Kafka provisioning configuration and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g. Ansible Terraform).
Deep knowledge of Kafka ecosystem components like ZooKeeper Kafka Connect Schema Registry MirrorMaker KRaft (Kafka without ZooKeeper).
Build custom Kafka tooling and dashboards for producers and consumers monitor lag and health metrics.
Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
Perform root cause analysis and resolution of platform-level incidents and outages.
Develop and maintain Linux shell scripts for automation of deployments monitoring and maintenance tasks.

Qualifications we seek in you!

Qualifications

BE/B Tech/MCA
Excellent written and verbal communication skills

Qualifications/ Skills

Working experience with OpenShift Kubernetes Cloud services to deploy Kafka clusters using vendor provided helm charts & docker images
Kafka Resiliency dc/dr setup & observability for Kafka nodes
Data ingestion & streaming pipeline setup for other data platforms to Kafka & Kafka to data sources like Solr
Five or more years of Kafka administration experience ideally with Confluent Kafka
Proficiency in Linux Python or Bash scripting
Hands-on knowledge of Kafka monitoring tools and experience with security protocols.

Job Title: Kafka Admin Location: NYC NY (3 days onsite minimum) Duration: 6 months Job Summary: The ideal candidate will be responsible for designing deploying maintaining and scaling Kafka clusters in mission-critical environments while also supporting the Linux-based infrastructure that forms the ...