Data Engineer with 5+ years of experience designing and building scalable, cloud-native data pipelines on Microsoft Fabric and Azure Databricks. Skilled in ingesting data from RDBMS, REST APIs, Salesforce, Snowflake Shar…
Data Engineer with 5+ years of experience designing and building scalable, cloud-native data pipelines on Microsoft Fabric and Azure Databricks. Skilled in ingesting data from RDBMS, REST APIs, Salesforce, Snowflake Share, configuration files, and Azure Storage into Lakehouse and Data Lake / Snowflake environments. Strong hands-on expertise in Delta Lake and Fabric Lakehouse-based incremental ingestion frameworks with ACID transactions and MERGE/UPSERT logic, built on Medallion Architecture (Bronze, Silver, Gold). Proficient in PySpark, Spark SQL, Azure Data Factory (ADF), Fabric Data Pipelines and Notebooks, with a strong focus on performance optimization, data modeling, and delivering reliable, high-quality data solutions for analytics and reporting.
<p>Designed and implemented data pipelines to onboard multiple source systems into a Fabric Lakehouse following Medallion Architecture (Bronze, Silver, Gold).</p><p> Built the transformation, cleansing, and data modeling logic aligned with business requirements using Fabric Notebooks and Data Pipelines, producing curated Gold-layer datasets for analytics and reporting.</p><p> Designed and built scalable ingestion pipelines to load data from RDBMS, REST APIs, Snowflake Share, CFG files, and Azure Storage into Azure Data Lake and Snowflake.</p><p> Architected end-to-end ingestion pipelines using Azure Databricks for structured and semi-structured data.</p><p> Implemented a Delta Lake-based ingestion framework with ACID transactions, schema evolution, and incremental load patterns.</p><p> Developed incremental PySpark pipelines with MERGE/UPSERT logic, reducing full refresh cycles.</p><p> Automated workflows using Databricks Jobs and Workflows, minimizing manual intervention.</p><p> Built complex PySpark transformation pipelines for data cleansing, enrichment, auditing, and SCD-like processing.</p><p> Applied Databricks and Spark optimization techniques to improve performance and optimize compute costs.</p><p> Developed optimized SQL queries and stored procedures for validation and analytical transformations.</p><p> Collaborated with analytics, BI, and governance teams and contributed to documentation and reusable frameworks.</p><p> Designed data pipelines following Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake to support incremental processing, data quality, and analytics-ready datasets.</p><p> Implemented data quality and audit frameworks with monitoring and exception handling.</p>
<p>Worked with Azure SQL Database, Azure Cosmos DB, and ADLS for scalable data storage and processing.<br>
Developed ADF pipelines to migrate on-prem SQL Server data to ADLS, improving storage flexibility by 30%.<br>
Boosted distributed query performance by 20% using Broadcast Hash Joins.<br>
Implemented a partitioning strategy that improved large-table join operations by 25%.<br>
Applied caching techniques to accelerate multi-day queries, achieving a 30% performance improvement.<br>
Supported data modeling by designing and optimizing schemas, improving retrieval speed by 20%.<br>
Built automated data workflows, reducing manual effort by 50%.<br>
Developed Spark RDD transformations and Spark Core operations for distributed data processing.<br>
Reduced processing time by 30% using optimized Spark and Python transformations.<br>
Designed transformation workflows (cleaning, filtering, aggregation, enrichment) using PySpark, SQL, and Python.<br>
Collaborated with functional teams to troubleshoot, optimize, and enhance data pipelines.<br>
Tuned and monitored pipelines for performance and reliability.<br>
Wrote optimized SQL queries, stored procedures, joins, subqueries, and correlated subqueries.</p>
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
