Senior Data Engineer
Senior Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
We are looking for a passionate and experienced Big Data Engineer for driving the design and
development of core platform frameworks with the mega knowledge of Spark Python/Scala
and Kafka Streams that enable the delivery and construction processes for the Data
Management Data Discovery and Analytics group and using emerging big data technologies.
Title: Lead Data Engineer - Big Data
Location: Gurugram
Experience: 4-7 Yrs.
Education: Bachelor s / Master s in Software Engineering
Responsibilities
Gather and process raw data at scale.
Design and develop data applications using selected tools and frameworks.
Read extract transform stage and load data to selected tools and frameworks.
Build and execute data warehousing mining and modeling activities using agile
development techniques.
Perform tasks such as writing scripts calling APIs writing SQL queries etc.
Deploy and monitor products on AWS Cloud platform
Monitoring data performance and modifying infrastructure as needed.
Developing ETL flows based on Hadoop/Big Data stack technology: Apache Nifi Hive
Spark KafkaAWS GLUE AWS LakeFormationAWS S3.
Troubleshooting and performance optimization for data processing flows data models.
Able to develop and maintain Python based Rest api s.
Sounds Like You
4 years of Big data development experience with working proficiency in Java/Scala
/Python to write data pipelines and data processing layers.
Strong in Data Structures & Algorithms Analytical and problem-solving skills.
Proficiency in Python with exposure to any web frameworks.
Proficiency with Spark Hadoop v2 MapReduce HDFS.
Experience with building stream-processing systems using solutions such as Storm or
Spark-Streaming.
Experience with the integration of data from multiple data sources.
Experience with SQL NoSQL databases.
Experience with various messaging systems such as Kafka or RabbitMQ.
Experience with confluent packages like Kafka connect schema registry.
Experience with Cloudera/MapR/Hortonworks.
Experience with AWS big data solutions like GLUERedshiftLakeformation.
Experience in Data Lake platforms like HudiIceberg or Delta.
Good knowledge of Big Data querying tools such as Pig Hive or Impala
Knowledge of various ETL techniques and frameworks such as Flume.
delta,kafka,data structures,iceberg,etl,aws glue,hadoop,scala,aws,hudi,aws lake formation,stream-processing,impala,rabbitmq,hdfs,pig,kafka streams,aws redshift,spark,hive,nosql,data warehousing,apache nifi,mapreduce,big data,sql,python,data lake,algorithms,data processing
Employment Type
Full Time
