Data Engineer Scala

We are seeking an experienced professional with a strong background in Big Data solutions and cloud technologies particularly in Google Cloud Platform (GCP). The ideal candidate will have a comprehensive understanding of Hadoop ecosystem migration from onpremises to cloud environments and possess a solid grasp of SQL data security principles and continuous integration/continuous deployment (CI/CD) processes using GitHub Actions.

Requirements

• Analyze develop and implement comprehensive database designs data models and logical data specifications.
  
• Expertise in scheduling and orchestration tools such as Oozie and Airflow.
  
• Proficiency in agile methodology contributing to a collaborative fastpaced and highly skilled team environment.
  
• Execute migration projects for the Hadoop ecosystem from onpremises infrastructure to cloud platforms with a preference for experience in GCP.
  
• Develop and optimize big data pipelines in the cloud ensuring best practices in programming with Pyspark/Scala.
  
• Implement and maintain data security protocols to safeguard against vulnerabilities.
  

Essential Skills:

• 810 years of experience in delivering Big Data solutions with significant expertise in cloudbased deployments preferably on GCP.
  
• Proven track record in deploying applications built with Spark/Scala in cloud environments.
  
• Advanced proficiency in programming with Pyspark/Scala adhering to industry best practices.
  
• Extensive knowledge of Hadoop Spark Python Shell scripting and similar frameworks.
  
• At least 2 years of experience in developing realtime data processing pipelines using Kafka Pub/Sub Spark Streaming.
  
• Strong capabilities in SQL query formulation and optimization of Hadoop and Spark jobs.
  

Desirable Skills:

• Familiarity with agile methodologies.
  
• Advanced skills in GCP and experience with cloud migration projects.
  
• Excellent communication and teamwork abilities.
  
• Additional Advantage
  
• Proven experience in migrating the Hadoop ecosystem from onpremises to cloud especially to GCP will be considered a significant advantage.
  

Additional Information:

• Location: We have a strong preference for candidates who are currently based in the Netherlands or within Europe. This facilitates a smoother transition and integration into our team given our operational locations and work culture.
  
  
• Visa Sponsorship: For international candidates currently not in the Netherlands this job is not applicable. However this offer extends to those who have already been in the Netherlands for a minimum of one year. This criterion helps ensure that potential candidates are already familiar with the Dutch work environment and regulatory landscape easing the onboarding process.
  
  
• Availability: Ideal candidates will be available to start immediately or within 30 days. We value the ability to mobilize quickly and integrate efficiently into our dynamic work environment contributing to our ongoing projects without delay
  

We are seeking an experienced professional with a strong background in Big Data solutions and cloud technologies, particularly in Google Cloud Platform (GCP). The ideal candidate will have a comprehensive understanding of Hadoop ecosystem migration from on-premises to cloud environments and possess a solid grasp of SQL, data security principles, and continuous integration/continuous deployment (CI/CD) processes using GitHub Actions. Requirements Analyze, develop, and implement comprehensive database designs, data models, and logical data specifications. Expertise in scheduling and orchestration tools such as Oozie and Airflow. Proficiency in agile methodology, contributing to a collaborative, fast-paced, and highly skilled team environment. Execute migration projects for the Hadoop ecosystem from on-premises infrastructure to cloud platforms, with a preference for experience in GCP. Develop and optimize big data pipelines in the cloud, ensuring best practices in programming with Pyspark/Scala. Implement and maintain data security protocols to safeguard against vulnerabilities. Essential Skills: 8-10 years of experience in delivering Big Data solutions, with significant expertise in cloud-based deployments, preferably on GCP. Proven track record in deploying applications built with Spark/Scala in cloud environments. Advanced proficiency in programming with Pyspark/Scala, adhering to industry best practices. Extensive knowledge of Hadoop, Spark, Python, Shell scripting, and similar frameworks. At least 2 years of experience in developing real-time data processing pipelines using Kafka, Pub/Sub, Spark Streaming. Strong capabilities in SQL query formulation and optimization of Hadoop and Spark jobs. Desirable Skills: Familiarity with agile methodologies. Advanced skills in GCP and experience with cloud migration projects. Excellent communication and teamwork abilities. Additional Advantage Proven experience in migrating the Hadoop ecosystem from on-premises to cloud, especially to GCP, will be considered a significant advantage. Additional Information: Location: We have a strong preference for candidates who are currently based in the Netherlands or within Europe. This facilitates a smoother transition and integration into our team given our operational locations and work culture. Visa Sponsorship: For international candidates currently not in the Netherlands, this job is not applicable. However, this offer extends to those who have already been in the Netherlands for a minimum of one year. This criterion helps ensure that potential candidates are already familiar with the Dutch work environment and regulatory landscape, easing the onboarding process. Availability: Ideal candidates will be available to start immediately or within 30 days. We value the ability to mobilize quickly and integrate efficiently into our dynamic work environment, contributing to our ongoing projects without delay