4 years of experience as a Data Engineer and Python Developer, specializing in crafting data-intensive applications within the Hadoop Ecosystem. My expertise extends to Big Data analytics, Cloud Data Engineering, Data Vi…
4 years of experience as a Data Engineer and Python Developer, specializing in crafting data-intensive applications within the Hadoop Ecosystem. My expertise extends to Big Data analytics, Cloud Data Engineering, Data Visualization, Reporting, and the implementation of Data Quality Solutions.
Proficient in managing data systems in Cloudera, Hortonworks, Hadoop, and AWS with a focus on highly distributed, large-scale data.
Hands-on experience with a wide range of Hadoop ecosystem components, including Hadoop, Hive, Pig, Sqoop, HBase, Cassandra, Spark, Spark Streaming, Spark SQL, Oozie, Zookeeper, Kafka, Flume, MapReduce, Yarn, and Scala.
Skilled in working with NoSQL databases like DynamoDB, MongoDB, and HBase, using REST APIs for real-time data processing.
Strong understanding of Spark architecture and components, with expertise in Spark Core, Spark SQL, and Spark Streaming, enabling the development of PySpark and Spark-Scala applications for analytics and stream processing.
Extensive use of Spark Data Frames API for analytics on Hive data, and proficiency in using Spark-SQL with diverse data sources such as JSON, Parquet, and Avro.
Proficient in Python scripting, including statistical functions with NumPy and data visualization using Matplotlib and Pandas.
Expertise in developing complex HiveQL queries and Hive User Defined Functions (UDFs) for data extraction.
Experienced in optimizing and performance tuning Hive jobs using partitioning and bucketing techniques.
Skilled in translating Hive/SQL queries into Spark transformations using Data Frames and Datasets.
Proficient in orchestrating automated workflows using Oozie.
Experience in configuring Zookeeper for cluster coordination and data consistency.
Familiarity with AWS services, including S3, EC2, SQS, RDS, EMR, Kinesis, Lambda, Event Bridge, Glue, Redshift, Athena, DynamoDB, Elasticsearch, Service Catalog, CloudWatch, and IAM.
Experience in data migration from AWS S3 to Snowflake using Snowpipe.
Proficient in leveraging the Azure cloud platform, encompassing HDInsight, Data Lake, Data Bricks, Blob Storage, Data Factory, Synapse, SQL, SQL DB, DWH, and Data Storage Explorer.
Demonstrated expertise in the design and implementation of data pipelines using Azure Data Factory, facilitating efficient data ingestion, transformation, and loading processes.
Extensive experience in utilizing Apache Airflow to author workflows in the form of directed acyclic graphs (DAGs). This includes visualizing both batch and real-time data pipelines in production, monitoring progress, and troubleshooting issues as needed.
Hands-on experience with popular visualization tools such as Tableau and Power BI, enabling effective data representation and analysis.
Excellent communication, interpersonal, and problem-solving skills, with a strong ability to work effectively in a team and adapt quickly to new technologies and environments.
Effectively utilized Spark technologies, including Spark RDD, Data Frame API, Data set API, Data Source API, Spark SQL, and Spark Streaming in various real-time projects. Demonstrated expertise in handling Spark Context, Spark-SQL, Data Frame, Pair RDD, and Spark YARN.
Established a centralized Data Lake on the AWS Cloud, leveraging core services such as S3, EMR, Redshift, and Athena. This initiative significantly improved data storage and accessibility for real-time processing requirements.
Created Python-based Spark Applications tailored to manage data from diverse sources, including RDBMS and Streaming, to meet the dynamic demands of real-time data processing projects.
Enhanced Hadoop algorithms by developing PySpark scripts, resulting in a notable improvement in runtime efficiency for real-time data processing.
Configured and monitored Apache Airflow Directed Acyclic Graphs (DAGs) to facilitate smooth data migration from S3 buckets to Snowflake data warehousing in real-time.
Implemented Lambda functions to perform various tasks like creating ad-hoc tables, adding schema, structuring data in S3, validating, filtering, sorting, and transforming Dynamo DB data. Transformed data was promptly loaded into a PostgreSQL database in real-time.
Designed and executed ETL processes using AWS Glue and Python for the real-time migration of campaign data from external sources (S3, ORC/Parquet/Text files) into AWS Redshift.
Configured Snowpipe for efficient data ingestion from S3 buckets, storing incoming data in Snowflake's staging area, and utilized micro-batching for real-time processing of a large volume of files on the Snowflake cloud.
Managed Data Marts in the Data Warehouse, implementing structures like Star Schema and Snowflake Schema with Type II Slowly Changing Dimensions (SCD) for real-time historical data retention.
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
