Sr. Engineer, Software – Kafka

Job Description:

Sr. Engineer Software - designs develops and delivers scalable software solutions that enable highly available and secure systems across the enterprise.

Collaborating closely with data engineers this role architect implements complex data pipelines real-time event-driven messaging frameworks and distributed microservices communication using Apache Kafka and messaging queue technologies.

The Engineer brings deep expertise in Kafka Python and Java object-oriented programming UNIX environments and Bash shell scripting along with hands-on experience building RESTful API endpoints open-source messaging platforms and end-to-end features spanning real-time API and data layers.

Success is measured by the quality scalability and reliability of software delivered and the ability to drive technical outcomes across teams.

What You ll Do

Build real-time data processing pipelines that support low-latency high-throughput and fault-tolerant distributed systems

Develop and maintain event-driven messaging frameworks using Apache Kafka or similar technologies to enable microservices to communicate data through events

Automate manual processes and operational workflows using Bash shell scripting in UNIX environments

Design develop test and deploy high-quality software solutions using Python and Java Build scalable high-performing

RESTful APIs and microservices aligned with fault-tolerant distributed systems Implement and support event-driven and message-based architectures using Kafka or similar technologies

Build scalable ETL frameworks using object-oriented

Python and Java programming languages Contribute to system and application architecture including service design data flows and integration patterns Participate in code reviews agile ceremonies and technical design discussions

Experience deploying and managing Kafka in containerized environments using Docker and Kubernetes

Proficiency with Kafka Connect for data integration and Schema Registry for Avro/Protobuf/JSON schema management

Troubleshoot and resolve complex software issues across development and production environments

Experience developing Kafka Producer and Consumer APIs using Spring Boot and Spring Kafka Proficiency building and exposing

RESTful API endpoints for seamless service integration Experience contributing to or building open-source messaging platforms

Develop and maintain CI/CD pipelines automation and deployment workflows Write and maintain clear technical documentation including design documents and system specifications

Continuously evaluate and adopt modern technologies to improve system performance reliability and maintainability

Daily use of AI productivity tools (Claude Cursor) is required across the software development lifecycle including design API and microservice development code generation code review testing debugging deployment documentation and production support.

Design build and operate AI agents that automate software engineering tasks such as generating boilerplate code for new microservices creating unit and API tests analyzing logs triaging incidents and supporting CI/CD pipelines.

Build and integrate AI-enabled capabilities into backend services and APIs using foundation models prompt engineering and retrieval-augmented generation (RAG) patterns.

Implement audit logging observability and human-in-the-loop controls for AI agents and AI-assisted workflows running in Tier-0 production environments.

What You ll Bring

Bachelor s degree in computer science Engineering or a related field or equivalent practical experience

6 years of software engineering experience building and supporting enterprise-grade applications

Strong experience working in Agile or Scrum-based development environments

Demonstrated ability to work independently on complex technical problems while collaborating across teams Strong analytical problem-solving and communication skills

Expertise in Apache Kafka including brokers topics partitions consumer groups and Kafka Streams

Experience implementing Kafka ACLs and role-based access control (RBAC) Familiarity with foundation models prompt engineering retrieval-augmented generation (RAG) and AI agent development applied to full-stack application development including enterprise reusable solutions

Must Have Skills

Proficiency with messaging and event-driven systems such as Kafka

Expert-level proficiency in Python or Java object-oriented programming

Strong experience with Python or Java object-oriented programming

Hands-on experience with UNIX/Linux environments and expert-level Bash shell scripting for automation and pipeline orchestration

Automate manual processes and operational workflows using Bash shell scripting in UNIX environments

Build responsive accessible and performant applications delivering end-to-end features across API and data layers Experience building scalable REST APIs and microservices

Knowledge of TLS/SSL encryption and SASL authentication for securing Kafka clusters

Experience with Git GitLab and CI/CD automation Experience working with relational databases and writing complex SQL Experience with containerization and orchestration tools such as Docker and Kubernetes

Experience using observability and telemetry tools such as Splunk AppDynamics or Grafana

Hands-on experience with AI productivity tools (Claude and Cursor or similar IDE) and

working knowledge of foundation models prompt engineering retrieval-augmented generation (RAG) and AI agent development.

Nice to Have Experience with

Databricks Snowflake or cloud-based data platforms Network Domain experience

Experience working in cloud-native environments (Azure AWS or GCP)

Experience contributing to system architecture or platform-level design decisions