Experienced in the designing and development of Hadoop-based Big Data applications and worked in ecosystem technologies like HDFS, Hive, Sqoop, Apache Spark, SQL and AWS for more than 3 years. Proficient in developing an…
Experienced in the designing and development of Hadoop-based Big Data applications and worked in ecosystem technologies like HDFS, Hive, Sqoop, Apache Spark, SQL and AWS for more than 3 years. Proficient in developing and implementing Spark DataFrame-based data processing workflows using Scala or Python programming languages. Experienced in optimizing Spark RDD performance by tuning various configuration settings, such as memory allocation, caching, and serialization. Skilled in handling semi structured/serialized data processing using hive (AVRO,PAQUET,JSON,ORC). Processed large-scale structured and semi-structured data sets, including filtering, mapping, reducing, grouping, and aggregating data. Experienced in using Sqoop to import and export data between Hadoop clusters and datalakes such as Amazon S3. Experience working with Spark SQL in production environments and implementing performance monitoring. Worked with Spark DataFrame APIs and SQL syntax and ability to write complex SQL queries and DataFrame operations. Experienced in ETL (Extract, Transform, Load) testing methodologies and processes and having knowledge about data integration and consolidation processes in ETL pipelines. Familiarity with Hive metastore and its role in managing table metadata and schema evolution. Experienced in using Spark DataFrame transformations and actions to process large-scale structured and semi-structured data sets, including (Filtering, Mapping, Reducing, Grouping, and Aggregating data). Migrating the jobs to AWS. with services s3 , EMR for spark jobs ,Athena for Business analytics and EC2 for scheduling. Data extraction and loading from cloud-based platforms, such as Amazon s3. Knowledgeable about data integration and consolidation processes in ETL pipelines. Skilled in using Sqoop to import and export data between Hadoop clusters and data lakes such as Amazon S3. Familiarity with Spark dataFrame APIs and SQL syntax and ability to write complex SQL queries and dataFrame operations to solve business problems
3 My total years of exp and Relevant exp .
I got chance to work on different Big Data Stack.Like(Hadoop,hdfs,hive,spark,sqoop,AWS). Recently i started migrating project AWS.
Before I used to in the data ingestion team where we used RDBMS as a source and we sqoop the data to HDFS and we processed it using hive and write to HDFS as avro. We use avro because of schema evolution. We have multiple RDBMS table in which we run multiple Sqoop Jobs and do the processing.
Later for example (1 year ago). I started working with Data application team where I have spark rigorously. In the data application. We have so many WEB apis coming with complex json json with different data models we almost run 7 spark jobs for different use cases like Customer data cleansing, Prediction Model spark jobs with currency conversions and few of the spark jobs do joins with AVRO data which generated during data ingestion and we write data to different HDFS directories as per the requirement also with complex nested data generation.
My business uses impala to do analytics on processed data.
In the recent times we started migrating the jobs to AWS. with services s3 , EMR for spark jobs,Athena for Business analytics and ec2 for scheduling.
We have a done POC on EMR step executions run those spark jobs using EMR command Runner.
Extracting data from multiple WEB API’s in JSON format with different data models and converting it into RDD data frame.
Processing hierarchical data like Multiline, Struct, Array and flattening data to performing operations.
Migrating the jobs to AWS. with services s3 , EMR for spark jobs ,Athena for Business analytics and EC2 for scheduling.
Develop our code in IDE and commit that code to the GIT BASH and run Jenkins job with CI/CD Pipeline which enables JAR in the production environment.
Creating EMR cluster with EC2 nodes for AWS project development and runs automation using step execution and command runner script.
Implemented partitioning, dynamic partitioning and bucketing in Hive for efficient data access using spark.
Skilled in leveraging Hive partitioning, bucketing, indexing, and caching features to improve query performance and reduce data processing overhead.
Working with Hive in production environments and implementing performance monitoring and alerting systems to detect and resolve performance issues.
Developing and implementing Spark data processing workflows using scala and optimized performance by Dynamic memory allocation, caching and serialization.
Data platform sizing, tuning, optimization and system landscape integration in large-scale, enterprise deployments.
Familiarity with Hive metastore and its role in managing table metadata and schema evolution.
Querying Hive tables using SQL queries and performing data analysis using tools like Apache Spark.
Developing Spark applications to implement complex data transformations and aggregations for batch processing jobs, leveraging Spark SQL and dataFrames.
Designed and implemented end-to-end data integration solutions using Sqoop for large-scale data migrations from on-premise databases to Hadoop clusters.
نظام التوظيف التشغيلي المعتمد على الذكاء الاصطناعي أولاً، الذي يوحّد ويؤتمت دورة التوظيف بالكامل للمؤسسات الحديثة حول العالم.
د.جوب منطقة حرة ذ.م.م. 2026 © جميع الحقوق محفوظة
