A detail-oriented and results-driven Data Engineer with 3+ years of experience in designing, implementing, and maintaining data pipelines and infrastructure. Analyze or transform stored data by writing Spark Jobs using P…
A detail-oriented and results-driven Data Engineer with 3+ years of experience in designing, implementing, and maintaining data pipelines and infrastructure. Analyze or transform stored data by writing Spark Jobs using Python and Scala on AWS and Azure, Hive Scripts, Amazon AWS services ELT tool based on business requirements and having good working knowledge on Hive, Spark, Databricks, Python, DBT, Snowflake, Scala, Hbase, Sqoop, Oozie. Created Azure Data Factory pipelines to import data from text files & on-premises databases (SQL Server and Oracle) into Azure SQL data warehouse and reduce data ingestion time by 30%. Good understanding of data analysis, data validation, data cleaning, data verification, data quality, metadata management, and master data management. Experience in writing complex SQL scripts using Statistical Aggregate functions and Analytical functions to support ETL in using sparkSQL and cloud data warehouse. Extensively worked in ETL process consisting of data transformation, data sourcing, mapping, conversion along with data modelling. Designed and implemented scalable infrastructure and data pipelines using technologies like Apache Spark, TensorFlow, or PyTorch to support large-scale machine learning workflows. Experience in setting up workflow using Apache Airflow and Oozie workflow engine for managing and scheduling Spark jobs. Well-Versed in Designing, Architecting, and implementing scalable cloud-based web applications using AWS and Azure.
• Developing Azure data factory pipelines by creating datasets, source and destination linked services to move the data from oracle database to
Azure Data Lake Store Raw Zone.
• Designing and implementing data pipelines to extract, transform, and load (ETL) data from various sources into Azure data services like Azure Data
Lake, Azure SQL Database, or Azure Synapse Analytics to manage 10TB+ of data, enabling in-depth analysis that led to a remarkable 10% increase
in customer retention.
• Creating Databricks notebooks using Python (PySpark) and Spark SQL for transforming the data that is stored in Azure Data Lake store Gen2 from
Raw to Stage and Curated zones.
• Working with various big data file formats such as Parquet, CSV and JSON in different scenarios of building data pipelines in Azure Data Factory
and Azure Synapse Pipelines.
• Implementing with Visual studio and Repos in Azure DevOps to do code commits and code migrations across dev, test and prod environments.
• Developing and implementing machine learning models for predictive analytics, recommendation systems, or anomaly detection tasks.
• Collaborated with cross-functional teams to identify and resolve data quality issues, leading to a significant decrease in data errors by 50% and
contributing to a 70% increase in stakeholder satisfaction.
• Utilizing Apache Spark SQL and data frame methods to execute data transformations and aggregations on difficult semi- structured data.
• Using Power BI as a reporting tool to connect with Synapse for generating daily reports.
Developing Azure data factory pipelines by creating datasets, source and destination linked services to move the data from oracle database to Azure Data Lake Store Raw Zone.
Designing and implementing data pipelines to extract, transform, and load (ETL) data from various sources into Azure data services like Azure Data Lake, Azure SQL Database, or Azure Synapse Analytics to manage 10TB+ of data, enabling in-depth analysis that led to a remarkable 10% increase in customer retention.
Creating Databricks notebooks using Python (PySpark) and Spark SQL for transforming the data that is stored in Azure Data Lake store Gen2 from Raw to Stage and Curated zones.
Working with various big data file formats such as Parquet, CSV and JSON in different scenarios of building data pipelines in Azure Data Factory and Azure Synapse Pipelines.
Implementing with Visual studio and Repos in Azure DevOps to do code commits and code migrations across dev, test and prod environments.
Developing and implementing machine learning models for predictive analytics, recommendation systems, or anomaly detection tasks.
Collaborated with cross-functional teams to identify and resolve data quality issues, leading to a significant decrease in data errors by 50% and contributing to a 70% increase in stakeholder satisfaction.
Utilizing Apache Spark SQL and data frame methods to execute data transformations and aggregations on difficult semi-structured data.
Using Power BI as a reporting tool to connect with Synapse for generating daily reports.
Developed various functionalities for Data Ingestion framework using Python resulting in a 30% reduction in data processing time.
Created the Json configuration files for cleansing the data through generic Spark framework.
Designed and Developed ETL Processes in AWS Glue to migrate data from external sources like S3, and Parquet/Text Files into AWS Redshift.
Conceptualized and executed complex Spark SQL Jobs, achieving a 40% improvement in processing large-scale data sets for Data Marts.
Successfully implemented SCD2 logic using Spark for master reference data, enhancing historical data accuracy by 15%.
Good exposure in writing SQL queries and in data analysis and fixing data issues for various facts and dimensions for data warehouse and data marts for star schema and snowflake schema.
Experience in solving the performance issues in PySpark and created test case documents.
Used AWS Glue for transformations and AWS Lambda, SQS and SNS to automate the process & worked on AWS Data Pipeline to configure data loads from S3 to Redshift
Designing and implementing complex data pipelines using Apache Airflow, streamlining data processing, orchestrating workflows, and ensuring robust data quality and reliability across diverse data sources.
Utilized and worked on Source/Version control Tools using Github, validate the change sets code changes, Check-in/Out and versioning and developed CI/CD pipelines using Jenkins on AWS to create, test, and deploy the code to higher environment.
Assisted with the design and implementation of data governance and security policies and procedures, including data encryption and access controls.
Experience in analysing ETL requirements and identify solution elements for the enhancement requests and change requests.
Built complex ETL/ELT pipelines for data processing in azure cloud using Azure Data Factory.
Prepared the Mapping from the Source Systems to the Target Reporting table with the required metrics.
Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs on Scala and Python.
Created stored procedures to load the data into fact and dimension tables in Azure SQL Data warehouse.
Created complex queries based on the business requirement using PySpark/SparkSQL in azure databricks.
Migration of on-premises data (SQL Server) to Azure Data Lake Store (ADLS) using Azure DataFactory (ADF).
Developed ETL workflows using Apache Spark and Python, resulting in a 30% reduction in data processing time and improved data accuracy.
Analyzed complex HiveQL queries to address the concerns regarding any error values/data mismatch between source and sink while validating the reports by users.
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
