Debezium and Kafka Connect Application Developer

Eindhoven - Netherlands

Monthly Salary: Not Disclosed

Posted on: 2 hours ago

Vacancies: 1 Vacancy

Job Summary

Role ID#34013 is posted to you on SF. Kindly work on the position and submit profiles at the earliest. This is a sub-contracting role in NL.

Role: Debezium and Kafka Connect Application Developer

Location: DAf Eindhoven Netherlands

Type: 1 yr contract Tenure as of now is one year project.

Mode: Fully Onsite

Cost rate : upto 45 EUR per hour

Role Overview

We are seeking an experienced Developer/Technical Application Manager to design build and maintain robust data ingestion pipelines. The core of this role involves leveraging Debezium for Change Data Capture (CDC) to stream data into our datalakehouse.

You will be responsible for the end-to-end lifecycle of these pipelines operating within an AWS environment using Docker containers on ECS and ECR. This is a hybrid role requiring strong development capabilities alongside operational management (Application Management) to ensure high availability and data integrity of our data ingestion pipelines.

Key Responsibilities

Pipeline Development: Take ownership of the CDC ingestion framework utilizing Kafka connectors (Debezium Iceberg sink S3 sink).
Containerized Infrastructure Management: Deploy and manage Debezium and Kafka Connect workers using Docker containers orchestrating on AWS ECS (Elastic Container Service) and ECR.
Data Lake Integration: Manage data ingestion into AWS S3 utilizing Parquet and Apache Iceberg formats.
Infrastructure as Code: Use Terraform to provision and manage AWS resources supporting the data platform.
CI/CD: Build and maintain deployment pipelines using GitHub and GitHub Actions.
Operational Excellence: Monitor pipeline health troubleshoot connectivity issues and ensure the reliability of the Kafka ecosystem.
Optional: Support and optimize workflow orchestration using Airflow where applicable.

Candidate Profile

Must-Have Experience (Non-Negotiable)

Apache Kafka & Kafka Connect: Multiple years of hands-on experience configuring deploying and managing Kafka Connect clusters in a production environment.

Containerization: Extensive experience with Docker is required. You must be comfortable building images and managing container lifecycles.

AWS Compute: Proven experience running containers on AWS ECS and managing images via AWS ECR.

Key Technical Requirements

Infrastructure as Code: Strong proficiency with Terraform for provisioning AWS resources.

CI/CD: Experience creating and maintaining workflows in GitHub and GitHub Actions.

Optional: Familiarity with Apache Airflow for scheduling and monitoring is a plus but not mandatory.

Highly Desirable (Strong Plus)

Debezium: While Kafka Connect experience is a must specific experience configuring Debezium connectors for various databases (SQL Server PostgreSQL etc.) is a massive advantage.

Open Table Formats: Strong knowledge of Apache Iceberg (or Hudi/Delta Lake) and storing data as Parquet on AWS S3 is significantly preferred.

Baseline Skills (Assumed)

Programming/Scripting: Proficiency in Python and Bash scripting.

Cloud Fundamentals: General knowledge of AWS core services (IAM VPC CloudWatch).

Optional: Basic understanding of Object-Oriented (OO) principles and software design patterns.

Tech Stack Summary

Streaming: Apache Kafka Kafka Connect Debezium

Compute/Containerization: AWS ECS AWS ECR Docker

Storage/Format: AWS S3 Apache Iceberg Parquet

DevOps: Terraform GitHub Actions

Languages: Python Bash

Optional Orchestration: Apache Airflow

Role ID#34013 is posted to you on SF. Kindly work on the position and submit profiles at the earliest. This is a sub-contracting role in NL. Role: Debezium and Kafka Connect Application Developer Location: DAf Eindhoven Netherlands Type: 1 yr contract Tenure as of now is one year project. Mod...