Senior Data Engineer

Responsibilities

• Create and maintain, with the team, an optimal data pipeline architecture. Build, deploy, and maintain distributed data analytics systems using Hadoop/Spark, Python, Java , Go , Scala, and C/C++
• Assemble large, complex data sets that meet functional / non-functional requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal loading of data from a wide variety of data sources using SQL and big data technologies.
• Work with stakeholders including the Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Identify gaps and improve the platform’s quality, robustness, and speed
• Meet customer requirements in terms of availability, reliability, functionality, and performance

Job Requirements

• Results-oriented, determined and having a self-driven attitude
• Deep Understanding of CS (Data Structure , Algorithms , Data Sterilization , Discreate Mathematics , CLI , Shell Scripting , networking, and corn jobs )
• Proven Experience in programming languages like (Java , Python , Scala)
• Deep Understanding Concurrent Distributed systems (Times, Clocks ,Ordering of events , Broadcasting and Replication )
• Deep understanding of relational and non-relational databases
• Deep understanding Data warehouse
  • Concepts (Data Modeling, OLAP, OLTP , Denormalization ,Different Schemas)
  • Engines like Snowflake , Hive , Impala , Google Big Query , Amazon Redshift
• Deep understanding Data Lakes (Delta Lake , Databricks , Amazon Athena , Azure data lake)
• Deep understanding of Data Processing Engines (PIG , Spark , Flink , NIFI , Kafka , Storm)
• Workflow Managing systems (Airflow , Luigi , Google Composer)
• Basics understanding and implementing Machine Learning (TensorFlow, Spark MiLB)
• Experience building and optimizing ‘big data’ data pipelines, architectures, and data sets. In both batch and real-time data integration.
• Working knowledge of stream processing, and highly scalable ‘big data’ data stores and messaging systems (Rabbit MQ , Pub Sub , Amazon SNS)
• Excellent communication skills, both verbally and in writing, to be able to align with various stakeholders and cross border squads
• Understanding data privacy and security (data governance, key management, encryption)
• Understanding Cluster Computing (AWS , Azure, GCP) is preferred.
• Understand software testing (Unit Testing , Functional testing , Integration testing) is preferred.
• Understanding identity and access management (active directory) is preferred.