Overall 3.5 years of experience in data engineering handling Big Data Ecosystem with technologies including Spark,HDFS, Sqoop, Hive, AWS, SQL Exposure on cloud technology of AWS – S3 ,EMR. Proficient in using Spark RDD b…
Overall 3.5 years of experience in data engineering handling Big Data Ecosystem with technologies including Spark,HDFS, Sqoop, Hive, AWS, SQL Exposure on cloud technology of AWS – S3 ,EMR. Proficient in using Spark RDD based data processing using Scala programming language Expertise in using Spark DataFrame transformations and actions to process large-scale structured and semi-structured data sets, including filtering, mapping, reducing, grouping, and aggregating data. Familiarity in using Spark SQL to process large-scale structured and semi-structured data sets, including querying, filtering, mapping, reducing, grouping, and aggregating data. Worked with different data serialization formats (Avro, Parquet, JSON, etc.) in Spark Enhanced performance by Implemented Spark partitioning and caching strategies. Worked on Spark functions for complex data processing. Proficient in Hive data types, schemas, partition, buckets and file formats Experienced in efficiently using Hive managed and external table with respect to the business requirement. Ability to optimize Hive queries for performance, scalability and efficiency using techniques such as indexing, partitioning, caching, etc. Knowledge of Hive table formats, including ORC, Parquet, and Avro, and their advantages and disadvantages for different use cases. Experienced in importing , exporting and writing sqoop commands for transfer of data between Hadoop and relational databases using Sqoop. Proficient in configuring and using Sqoop jobs for incremental data transfers using Sqoop's incremental import feature. Used Sqoop for Import of data with different file formats like csv, avro and parquet Worked with import and export of data from cloud platform storage services like Amazon S3 using Sqoop Have optimized Sqoop configurations for maximum performance in various data transfer scenarios. Team player with strong sense of ownership.
<p>Processed and analysed large volumes of structured and unstructured data using Spark<br>
Used Dataframe to query various data sources such as csv, parquet, avro, json<br>
Performed various operations like joins, filters, maps, etc to deliver the data as per business requirements<br>
Performed data cleansing, transformations and aggregations using Spark<br>
Achieved 50% improvement in data processing speed and 30% reduction in resource consumption by optimizing Spark performance and tuning spark parameters<br>
Collaborated with data-scientists and business analysts to understand the data requirements and deliver insights<br>
Developed custom Spark functions for complex data processing using scala<br>
Designed and optimized Spark jobs for join operations<br>
Performed operations using AWS S3 Storage buckets, importing and exporting data to and from S3 using local terminal<br>
Proficient in setting AWS EC2 instance creation and configuration for optimal performance<br>
Hands-on experience with real-time scenarios with AWS EMR</p>
<p>Developed Spark applications for distributed data processing<br>
Created and managed RDDs (Resilient Distributed Distributeds) for data transformations<br>
Utilized spark sql for structured data manipulation and analysis<br>
Designed and implemented Spark jobs using Scala<br>
Managed Spark libraries and dependencies<br>
Performed data cleansing and pre-processing using Spark transformations</p>
<p>Worked in querying Hive tables using SQL-like syntax and performing data analysis<br>
Enhanced Hive query performance by tuning various configuration settings, such as memory allocation, parallelism, and compression<br>
Performed hive partitions for faster query performance<br>
Used Sqoop to load data into Hive tables for further processing and analysis<br>
Performed imports of different types of serialized data using Sqoop</p>
<p>Have created and managed Hive tables, including managed, external, and partitioned tables<br>
Experienced in designing efficient data models using Hive tables and partitions to optimize query performance<br>
Used different Hive table formats, including ORC, Parquet, and Avro as per their advantages and disadvantages for different use cases<br>
Performed hive schema evolution with avro file format<br>
Involved in setting up and configuring Sqoop-based data import and export solutions for large-scale data warehousing projects<br>
Worked on creating and managing Sqoop jobs automation for efficient import of data from RDBMS<br>
Used Sqoop to import and export data between Hadoop clusters and data lakes such as Amazon S3</p>
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
