Big Data Engineer

Main Responsibilities:

• Contribute to the design and construction of the company data lake
• Collect, store, process, and support the analysis of huge sets of data, both structured and unstructured
• Choosing optimal solutions to use in big data use cases, then maintain, implement, monitor and integrate them with the data and IT architecture used across the company
• Build and teach the company about big data technologies, participate actively in the journey setup, from the discovery phase until the corporate data-centric transformation
• Build solutions with key concepts: security and privacy by design

Job Requirements

Requirements:

• Knowledge of the Linux operating system (OS, networking, process level)
  Understanding of Big Data technologies (Hadoop, Hbase, Spark, Kafka, Flume, Hive, etc)
• Understanding of one or more object-oriented programming languages (Java, C++, C#, Python)
• Fluent in at least one scripting language (Shell, Python, Ruby, etc.)
• Experience with at least one Hadoop distribution (Cloudera, MapR or preferably Hortonworks)
• Experience building complex data processing pipelines using continuous integration tools
• Experience with Cassandra, MongoDB or equivalent NoSQL databases
• Experience developing in an Agile environment
• Bachelor’s / Master’s / PhD degree in Computer Science or related field
• 3+ years of building data pipelines experience or equivalent
• Nice to have: Experience in designing big data/distributed systems
• Nice to have: Experience creating and driving large scale ETL pipelines