Kafka Developer

The Kafka Developer is responsible for designing developing and maintaining event-driven data pipelines and streaming solutions within the project ecosystem. The role focuses on enabling scalable reliable and loosely coupled microservices using Apache Kafka tightly integrated with Spring Boot applications CI/CD pipelines and cloud platforms.

Day-to-Day Responsibilities

• Design and develop Kafka producers and consumers using Java and Spring Kafka to support event-driven workflows.
• Collaborate with Full Stack and backend teams to define event models message contracts and topic strategies.
• Design and manage Kafka topics partitions replication factors and retention policies based on business and performance needs.
• Implement event-driven integration patterns enabling asynchronous communication between microservices.
• Develop stream processing applications using Kafka Streams including filtering transformations aggregations and joins.
• Implement schema management using Avro JSON or Protobuf ensuring backward and forward compatibility.
• Apply Kafka security best practices including SSL/TLS SASL authentication ACLs and secure data transmission.
• Design robust error-handling mechanisms including retries dead-letter topics and idempotent message processing.
• Optimize Kafka performance by tuning producer/consumer configurations partition strategies and batching.
• Integrate Kafka-based services with Spring Boot microservices deployed on AWS VMware TAS or container platforms.
• Contribute to CI/CD pipelines for automated build test and deployment of Kafka applications.
• Monitor Kafka clusters and applications using Prometheus Grafana ELK or native Kafka metrics.
• Perform production support and BAU activities including troubleshooting message lag consumer failures and cluster issues.
• Conduct root cause analysis for incidents and implement preventive improvements.
• Participate in Agile ceremonies including sprint planning daily stand-ups reviews and retrospectives.
• Maintain technical documentation including topic catalogs event schemas and integration guidelines.
• Continuously improve system reliability scalability and maintainability by adopting industry best practices.
• Stay updated on Kafka ecosystem enhancements and streaming technologies and proactively suggest improvements.

Required Experience:

IC