Platform Engineer- Kafka with Linux

Role: Platform Engineer- Kafka with Linux.

Location: New York City NY (Onsite).

Duration: 12 Months Contract.

Job Description:

• Core Platform Engineer with deep expertise in Apache Kafka and Linux systems engineering.
• The ideal candidate will be responsible for designing deploying maintaining and scaling Kafka clusters in mission-critical environments while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform.

Responsibilities:

• Design implement and maintain highly available Kafka clusters in production environments.
• Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
• Administer security protocols and access controls to protect the platform.
• Monitor cluster health and performance using our established monitoring stack (Prometheus Grafana)
• Troubleshoot and resolve cluster-related incidents such as broker outages replication lag.
• Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
• Assist with planning and managing cluster resources to support future growth.
• Develop Kafka topics partitions ACLs schema registry and optimize throughput and reliability.
• Manage a tune Kafka Connect Kafka Streams and MirrorMaker as needed for data pipelines.
• Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL Ubuntu).
• Automate Kafka provisioning configuration and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g. Ansible Terraform).
• Deep knowledge of Kafka ecosystem components like ZooKeeper Kafka Connect Schema Registry MirrorMaker KRaft (Kafka without ZooKeeper).
• Build custom Kafka tooling and dashboards for producers and consumers monitor lag and health metrics.
• Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
• Perform root cause analysis and resolution of platform-level incidents and outages.
• Develop and maintain Linux shell scripts for automation of deployments monitoring and maintenance tasks.

Minimum Qualifications:

• BE/B Tech/MCA.
• Excellent written and verbal communication skills.
• Min 5 years of experience.
• Experience in Finance Domain.

Preferred Qualifications/ Skills:

• Strong hands-on experience with Apache Kafka (2.x or 3.x) including cluster administration and tuning.
• In-depth experience with Linux system internals kernel tuning filesystems and performance troubleshooting.
• Experience in managing Kafka brokers in production environments with millions of messages per second.
• Deep knowledge of Kafka ecosystem components like ZooKeeper Kafka Connect Schema Registry MirrorMaker KRaft (Kafka without ZooKeeper).
• Scripting and automation skills in Shell Python or Go.
• Experience in container orchestration platforms like Docker and Kubernetes.
• Familiarity with SSL/TLS SASL RBAC and encryption in-transit/at-rest for Kafka.
• Expertise in monitoring and alerting for Kafka and Linux systems.
• Strong understanding of distributed systems message queues and event-driven architectures.