Java Spark Engineer

Must haves: Java (core Java any version) Kafka Spark and AWS (Lake Formation preferred).

Nice to have: Machine Learning Python (scripting) PySpark Apache Flink Scala.
Seeking Two Senior level Data Engineers with Cloud and Streaming experience for CCB Data Technology. This team works on multiple downstream applications. There is only some ETL work the main function of this team is to build out tools and frameworks for the department internally. These tools and frameworks are used to assist internal teams in measuring productivity. The ideal candidate will have started their career as a Java Developer but moved into Data Engineering. Candidates should have significant experience with streaming and AWS. They will be working with Lake Formation in AWS and will use Kafka and Spark for data streaming. They must have experience working in a large environment with high volume real time data streaming.

This team will eventually incorporate ML technology but the position will not use ML early on in the project. They are also adding Apache Flink is another tool.

The interviews will consist of technical discussions on Java AWS Kafka and Spark streaming. Candidates will be asked to explain the complexities of the environments they worked in how to handle concurrency how they moved data in the past what problems they ran into and how they found solutions. There may also be basic Java questions to make sure they understand Java code.

Must haves: Java (core Java any version) Kafka Spark and AWS (Lake Formation).

Nice to have: Machine Learning Python (scripting) PySpark Apache Flink Scala.

Position can be located in Jersey City San Francisco or Palo Alto CA 3 days/week. CTH 603 level for both positions. 34 interviews.

Job Description:

Must have hands on experience :
min. 10 year programing experience
Cloud AWS
Spark performance tuning
Machine learning ops

Key Responsibilities:
1. Design & Develop spark data processing pipelines that can apply complex transformations on large volumes of data
2. Help teams with Performance tuning of complex spark jobs processing huge volumes of data in limited time
3. Develop and enhance common data processing frameworks focusing on efficiency scalability and code reusability
4. Implement and optimize streaming data processing pipelines using Kafka Flink
5. Provide mentorship and technical guidance to junior team members promoting best practices in data processing & Streaming
6. Stay current with the latest trends and technologies in Java Spark streaming and big data processing AWS services
7. Lead code reviews ensuring high coding standards and practices

Qualifications:
1. Bachelors or Masters degree in computer science engineering or related field
2. 10 years of relevant professional experience in software development with a focus on Java spark streaming and batch high volume data processing
3. Proven experience in performance tuning of Spark jobs in largescale data environments
4. Strong background in streaming data technologies and realtime data processing using kafka spark flink
5. Experience building common frameworks/libraries particularly for data processing
6. Exceptional problem solving skills and algorithmic thinking
7. Experience with AWS cloud platform and services (including SNS SQS Event bridge lambda glue lake formation etc.)
8. Knowledge of Docker Kubernetes and other containerization and orchestration tools
9. Candidate little short on above experience can still apply if they have experience in working with MLOps with specific expertise in implementing and managing offline/online/inline feature stores using vendor products like Sagemaker Tecton Feast Databricks etc.