Big Data Engineer

Job Description

• Define the architecture, scope and deliver various Big Data solutions.
• Support other teams by providing guidance on data modeling, data usage, processing and how they can best leverage the platform
• Build scalable data pipelines to ingest data from a variety of data sources, identify critical data elements and define data quality rules.
• Leverage Spark/Hadoop ecosystem knowledge to design and develop capabilities to deliver innovative and improved data solutions.
• Provide insights on area of improvements including Data Governance, best practices, large scale processing
• Support the bug fixing and performance analysis along the data pipeline

Job Requirements

• 5+ years of experience of the following
• Software engineer, with strong skills in at least one programming language is mandatory, preferably Scala or Java or Python
• Experience with Spark on Hadoop, EMR etc
• Experience working with real time data processing using Kafka, Spark Streaming or similar technology
• Experience with NoSQL Databases (Cassandra , Elasticsearch,…etc. ).
• Experience with distributed systems and design/implementation for reliability, availability, scalability and performance
• Experience with IoT Enablement Applications & Platforms like (thingsBoard , thingswork , etc) is a huge plus.
• IoT Wireless Communication Technologies (LoRA, Sigfox, NB-IoT) is a plus.
• Creative and innovative approach to problem-solving
• Prefer to Have:
• Familiarity with containerized platform like Docker and Kubernetes
• Experience working with Hive, Presto or other querying frameworks