Senior Data Engineer (Real-Time Data Processing & Streaming)

We are looking for an experienced Senior Data Engineer to join Endava and contribute to the design development and maintenance of scalable high-performance data processing solutions.

In this role you will focus on real-time data processing streaming technologies and event-driven architectures working with complex data pipelines that ingest transform enrich and distribute data from connected physical devices to downstream applications and storage systems.

You will collaborate closely with Tech Leads software engineers and cross-functional teams to build reliable efficient and maintainable data processing solutions in a cloud-native environment.

Key Responsibilities:

• Design develop and maintain scalable real-time and near-real-time data processing pipelines.

• Implement streaming data processing solutions using technologies such as Apache Kafka Apache Flink and Apache Spark.

• Develop and maintain Java-based services and integration components for processing data received from physical devices.

• Implement complex data transformations mappings validations and enrichments to support business and technical requirements.

• Build and maintain event-driven integrations using Kafka Kafka Streams Kafka Connect and messaging protocols such as AMQP.

• Ensure reliable data delivery to downstream consumers applications and storage systems.

• Implement and maintain MongoDB collections data structures and related application-level integrations following architectural guidelines.

• Develop deploy and maintain cloud-native services primarily on AWS leveraging Kubernetes and managed cloud services.

• Contribute to the modernization and migration of legacy data processing components toward Kafka-based streaming architectures.

• Collaborate with technical leads and engineering teams to translate architectural designs into robust production-ready implementations.

• Troubleshoot performance reliability and data consistency issues across distributed data processing systems.

• Contribute to code quality automated testing CI/CD practices and continuous improvement of engineering processes.

Qualifications :

Required Skills and Experience:

• Strong professional experience in Data Engineering with a focus on streaming data processing and distributed systems.

• Advanced programming skills in Java (11) including experience with the Spring framework.

• Hands-on experience with Apache Kafka including Kafka Streams Kafka Connect and the development of Kafka-based integrations.

• Practical experience with streaming and distributed processing frameworks such as Apache Flink Apache Spark or Spark Streaming.

• Solid understanding of event-driven architectures asynchronous communication patterns and messaging systems including AMQP.

• Experience designing and implementing data transformation enrichment and integration pipelines.

• Experience developing and integrating microservices and distributed applications.

• Hands-on experience with AWS cloud services particularly S3 SQS and EKS.

• Experience deploying and operating containerized applications using Docker and Kubernetes.

• Working knowledge of MongoDB including collections document structures and Java-based data access.

• Familiarity with Redis and its use in distributed applications.

• Experience with Git-based development workflows and CI/CD pipelines preferably using GitLab and GitLab CI/CD.

• Strong troubleshooting and problem-solving skills particularly in distributed high-throughput data processing environments.

• Ability to work collaboratively with technical leads and cross-functional engineering teams while taking ownership of implementation and delivery.

Nice to Have:

• Experience with Apache Pulsar or Apache Storm particularly in legacy system maintenance or modernization initiatives.

• Familiarity with Microsoft Azure services and hybrid or multi-cloud environments.

• Experience processing telemetry IoT or other event data generated by physical devices.

• Knowledge of SMTP-based integrations.

• Experience migrating legacy messaging and stream processing solutions to modern Kafka-based architectures.

• Understanding of performance optimization fault tolerance monitoring and observability in distributed streaming systems.

Additional Information :

Discover some of the global benefits that empower our people to become the best version of themselves:

• Finance: Competitive salary package share plan company performance bonuses value-based recognition awards referral bonus;  
• Career Development: Career coaching global career opportunities non-linear career paths internal development programmes for management and technical leadership;
• Learning Opportunities: Complex projects rotations internal tech communities training certifications coaching online learning platforms subscriptions pass-it-on sessions workshops conferences;
• Work-Life Balance: Hybrid work and flexible working hours employee assistance programme;
• Health: Global internal wellbeing programme access to wellbeing apps;
• Community: Global internal tech communities hobby clubs and interest groups inclusion and diversity programmes events and celebrations.

At Endava were committed to creating an open inclusive and respectful environment where everyone feels safe valued and empowered to be their best. We welcome applications from people of all backgrounds experiences and perspectivesbecause we know that inclusive teams help us deliver smarter more innovative solutions for our customers. Hiring decisions are based on merit skills qualifications and potential. If you need adjustments or support during the recruitment process please let us know.

At Endava were committed to creating an open inclusive and respectful environment where everyone feels safe valued and empowered to be their best. We welcome applications from people of all backgrounds experiences and perspectivesbecause we know that inclusive teams help us deliver smarter more innovative solutions for our customers. Hiring decisions are based on merit skills qualifications and potential. If you need adjustments or support during the recruitment process please let us know.

Remote Work :

No

Employment Type :

Full-time