Azure Data Engineer with 8 years of IT experience, including 6 years of hands-on experience in Azure Data Engineering, designing, developing, and optimizing scalable data engineering solutions across cloud and on-premise…
Azure Data Engineer with 8 years of IT experience, including 6 years of hands-on experience in Azure Data Engineering, designing, developing, and optimizing scalable data engineering solutions across cloud and on-premises environments. Experienced in building enterprise-scale ETL/ELT pipelines, data ingestion and transformation frameworks, distributed data processing solutions, and lakehouse architectures. Strong experience in data pipeline development, Spark performance optimization, data partitioning, parallel processing, pipeline orchestration, data modeling, data quality, validation, monitoring, and production troubleshooting. Experienced in designing reliable and scalable data solutions to support enterprise analytics and downstream reporting requirements. Hands-on experience with data governance, metadata management, data lineage, access control, CI/CD, automated deployments, and workflow orchestration across development and production environments.
<p>Designed and delivered end-to-end data engineering and forecasting solutions, managing 10–12 production pipelines across Azure Data Factory and Databricks.<br>
Designed and maintained end-to-end Azure Data Factory (ADF) pipelines for data ingestion, transformation, orchestration, and data delivery across Dev, QA, and Prod environments.<br>
Integrated Control-M for enterprise workflow scheduling and reliable data processing.<br>
Developed and optimized PySpark and Spark workloads using partitioning, parallel processing, and multithreading techniques to improve pipeline and model execution performance.<br>
Designed Delta Lake-based data processing solutions integrated with ADLS Gen2 and Snowflake, following Medallion Architecture (Bronze, Silver, Gold) to support scalable downstream analytics.<br>
Developed PySpark-based data and ML pipelines supporting time-series forecasting using XGBoost, ARIMA, and Holt-Winters models.<br>
Integrated MLflow and LLM APIs into data workflows to support automated business summarization use cases.<br>
Implemented data validation, data quality, metadata management, data lineage, and governance practices using Azure Purview and Databricks Unity Catalog.<br>
Managed CI/CD pipelines using Azure DevOps and Git to support automated deployment and version control across development and production environments.</p>
<p>Built and maintained 7-9 ETL/ELT pipelines processing 300M–500M records per month, integrating data from 5–6 source systems into Snowflake and ADLS for enterprise analytics across P&G and telecom domains.<br>
Developed data ingestion and transformation workflows using PySpark, Azure Databricks, Snowflake, and ADLS, with UC4 for workflow scheduling and orchestration.<br>
Optimized PySpark jobs in Azure Databricks using window functions, advanced aggregations, partitioning, and execution tuning to address processing bottlenecks.<br>
Developed and optimized Snowflake data models and loading workflows to support enterprise BI analytics and reporting requirements.<br>
Implemented data validation rules, error handling, and logging within PySpark pipelines to improve data reliability and downstream reporting quality.<br>
Maintained version-controlled ETL code using SVN.</p>
<p>Monitored and supported ADF and Hadoop-based data pipelines.<br>
Troubleshot production issues and performed incident analysis and root-cause troubleshooting to maintain reliable data processing.<br>
Performed production troubleshooting to improve pipeline stability and operational reliability.</p>
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
