Data Engineer with 3+ years’ experience managing large-scale data environments, optimizing data pipelines, and architecting resilient Data Warehouses. Proficient driving data-driven strategies, enhancing operational effi…
Data Engineer with 3+ years’ experience managing large-scale data environments, optimizing data pipelines, and architecting resilient Data Warehouses. Proficient driving data-driven strategies, enhancing operational efficiency through Big Data technologies. Seeking to contribute knowledge elevates data-driven initiatives to a challenging role.
● Created Azure Data Factory pipelines to import data from text files & on-premises databases into Azure SQL data warehouse, reducing
data ingestion time by 30%.
● Engineered data pipelines for ETL processes, loading data from various sources into Azure data services like Azure Data Lake, Azure SQL
Database, or Azure Synapse Analytics, managing 10TB+ of data. Enabled in-depth analysis and led to a remarkable 10% increase in
customer retention.
● Pioneered the implementation of Databricks notebooks using Python (PySpark) and Spark SQL for transforming the data stored in Azure
Data Lake store Gen2 from Raw to Stage and Curated zones.
● Collaborated with cross-functional teams to identify and resolve data quality issues, leading to a significant decrease in data errors by
50% and contributing to a 70% increase in stakeholder satisfaction.
● Transformed Hive/SQL queries into efficient Spark transformations using Spark RDDs and Python, resulting in a 40% reduction in data
processing time and improved overall data pipeline performance.
● ETL/ELT technologies and scripting languages are used to develop and maintain workflows for real-time data processing.
<p>Created Azure Data Factory pipelines to import data from text files and on-premises databases into Azure SQL data warehouse, reducing data ingestion time by 30%.<br>
Engineered data pipelines for ETL processes, loading data from various sources into Azure data services like Azure Data Lake, Azure SQL Database, or Azure Synapse Analytics, managing 10TB+ of data.<br>
Enabled in-depth analysis and led to a remarkable 10% increase in customer retention.<br>
Pioneered the implementation of Databricks notebooks using Python (PySpark) and Spark SQL for transforming the data stored in Azure Data Lake store Gen2 from Raw to Stage and Curated zones.<br>
Collaborated with cross-functional teams to identify and resolve data quality issues, leading to a significant decrease in data errors by 50% and contributing to a 70% increase in stakeholder satisfaction.<br>
Transformed Hive/SQL queries into efficient Spark transformations using Spark RDDs and Python, resulting in a 40% reduction in data processing time and improved overall data pipeline performance.<br>
Used ETL/ELT technologies and scripting languages to develop and maintain workflows for real-time data processing.</p>
<p>Led development of an open-source tool that prints 50+ Cloudera-hosted Hadoop Cluster health and monitoring metrics such as Hadoop, HDFS, Hive, HBase, Zookeeper, Kafka, Apache Spark, and MapReduce using Shell Scripting, Cloudera API, and Python Libraries.<br>
Conducted auditing, inspecting, and visualizing S3 server access logs using AWS Lambda and AWS Athena.<br>
Automated resulting scripts and workflow using Apache Airflow and shell scripting to ensure daily execution in production.<br>
Utilized AWS Glue for transformations and AWS Lambda, SQS, and SNS to automate processes and worked on AWS Data Pipeline to configure data loads from S3 to Redshift.<br>
Spearheaded development and implementation of data pipelines in Snowflake, automating processes and reducing manual intervention by 99%, while empowering business users with enhanced data ownership.<br>
Worked on Source/Version control Tools using Github, validate the change sets code changes, Check-in/Out, and versioning and developed CI/CD pipelines using Jenkins on AWS to create, test, and deploy code to higher environments.<br>
Authored in-depth analysis of network element functionality trends by executing advanced python and SQL queries on data warehouse (Snowflake) including predictive modeling using scikit-learn.<br>
Conducted 50+ code reviews and mentored 12 junior developers on various projects.</p>
<p>Analyzed S2TM with table descriptions, schema details for both source and target, data types of both source and target while mapping columns, transformations, and Teradata target table structure.<br>
Employed Pig and Hive Scripts to transform raw data based on given transformation provided in S2TM.<br>
Prepared Source and target DDL's based on S2TM and designed tables accordingly in Hive and Hbase.<br>
Solved performance issues in Pig scripts and Hive Scripts with understanding of Joins, Groups, and aggregation.<br>
Developed Tableau dashboards for real-time insights and utilized Apache Airflow for ETL orchestration, ensuring data consistency.</p>
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
