Data Engineer Ab initio

Data Engineer (Ab Initio OzoneCH Flink)

ROLE: Streaming Data Engineer Ab Initio OzoneCH Flink Job Title: Data Engineer (Ab Initio OzoneCH Flink) Location: Berkley Heights NJ

Role Overview: We are seeking a highly skilled resource to design and implement high-performance event-driven data pipelines ensuring low-latency data processing and high availability system for the large credit card processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE EME Conduct>It) to build stateful services that ingest filter and transform data from sources like Kafka or message queues pushing updates to dashboards or downstream databases in near-real-time. Key Responsibilities:

Create complex Ab Initio continuous flow graphs including stateful joins sliding time windows and aggregations.

Implement event-driven data pipelines using Kafka MQ and file streams.

Ensure the resilience of continuous flows including checkpointing and recovery to guarantee exactly-once processing.

Apply advanced Ab Initio components (e.g. Reformat Rollup Join Partition) to ensure low-latency performance.

Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues

Develop ETL pipelines for batch and real-time data ingestion and transformation.

Implement and ensure data validation data security integrity and compliance across big data platforms.

Monitor and troubleshoot performance issues in large-scale clusters.

Collaborate with data scientists analysts and application teams to deliver high-quality data solutions.

Automate workflows and improve operational efficiency using scripting and orchestration tools. Required Skills & Qualifications:

Deep understanding of credit card process system

Deep knowledge of GDE (Graphical Development Environment) EME (Enterprise Meta>Environment) Conduct>It and Continuous Flows.

Understanding of Kafka message queues and real-time stateful services.

Proficiency in Unix/Linux shell scripting SQL and database technologies (e.g. Oracle Teradata).

Experience in Java Scala Python or Kafka is plus.

Familiarity with Linux/Unix environments and shell scripting.

Understanding of data security governance and compliance standards.

Experience with cloud-based big data platforms

Exposure to containerization (Docker Kubernetes) for big data workloads.

Knowledge of CI/CD pipelines for data engineering projects.

Behavioral Skills:

Good Communication skills

5 days Work from Office at Berkley Heights NJ

Team Player

Ability to work in a changing environment

Strong problem solving and analytical skills

Ability to work independently or within a team

Manage day-to-day challenges and communicate developmental risks with the technical team Qualifications:

Bachelors degree in computer science Software Engineering or a related field.

Proficiency in business process modeling and documentation tools.

Product implementation experience is preferre