Results-driven Azure Data Engineer with 7 years of overall experience specializing in building enterprise-scale data pipelines, lakehouse architectures, and cloud data platforms. Proven expertise in Azure Data Factory, D…
Results-driven Azure Data Engineer with 7 years of overall experience specializing in building enterprise-scale data pipelines, lakehouse architectures, and cloud data platforms. Proven expertise in Azure Data Factory, Databricks, PySpark, ADLS Gen2, and Delta Lake and end-to-end Power BI reporting solutions. Hands-on experience across retail, finance, and logistics domains executing large-scale ETL/ELT workflows, incremental data loading, schema enforcement, and Medallion Architecture. Certified Microsoft Fabric Data Engineer Associate (DP-700) with proven capability in delivering high-throughput data solutions.
<p>Architected a metadata-driven ETL pipeline on Azure Data Factory to ingest e-commerce data from CSV, Excel, JSON, and REST API sources into ADLS Gen2 without hardcoding source-specific logic.</p><p> Designed a 4-layer dynamic pipeline hierarchy (Master → Source Type → Cloud Provider → File Type) using nested Lookup and Switch activities driven entirely by SQL-based configuration to trigger downstream Databricks notebooks.</p><p> Built a SQL Server control framework with trigger, job, and logging tables, and developed stored procedures for centralized execution tracking and failure traceability.</p><p> Implemented dynamic Copy Activity mappings and Azure Databricks (PySpark and SQL) transformations to handle schema variations (column renaming, delimiter/sheet detection, deduplication) across client files.</p><p> Standardized all ingested data into Snappy-compressed Parquet and Delta formats within a curated Data Lake zone (ADLS Gen2) using PySpark, enabling faster, analytics-ready consumption for Power BI reporting.</p><p> Designed and developed interactive Power BI dashboards using Star Schema modeling, complex DAX measures and deliver actionable business insights directly from curated Delta Lake tables.</p><p> Conducted end-to-end pipeline testing and debugging across Databricks notebooks, Linked Services, Datasets, and parameters, ensuring reliable data flow from raw ingestion to the curated layer and Power BI reports.</p>
<p>Executed end-to-end relational data extraction, transformation, and load (ETL) routines using SQL Server, MySQL, and complex stored procedures.<br>
Built dynamic, metadata-driven Azure Data Factory (ADF) pipelines with Self-Hosted Integration Runtime (SHIR) to extract transactional data securely from MySQL databases.<br>
Designed time-partitioned folder hierarchies in ADLS Gen2 utilizing Snappy-compressed Parquet format, reducing storage consumption and improving query execution throughput.<br>
Implemented automated data quality verification scripts and checksum reconciliation checks, maintaining strict data accuracy across staging and production tables.<br>
Developed PySpark scripts for data standardization, type casting, missing value handling, and aggregated business metric generation.<br>
Set up automated error handling, centralized logging, and alerting mechanisms to instantly notify technical teams of pipeline failures or memory bottlenecks.</p>
<p>Managed operational database procedures, data extraction, and ETL processes using MySQL and SQL scripts.<br>
Executed data transformation procedures and maintained log monitoring routines across UNIX environments.<br>
Performed root-cause analysis (RCA) for data inconsistencies, production query failures, and system outages.</p>
<p>Managed campus IT infrastructure, network availability, database backup schedules, and hardware maintenance protocols.</p>
<p>Designed and maintained production data pipelines in Azure Data Factory and Databricks, processing baseline daily volumes of 4–5 GB and handling burst loads up to 25 GB/day during peak retail campaigns.</p><p> Implemented Medallion Architecture (Bronze, Silver, and Gold layers) across Azure Data Lake to deliver clean, validated, and analytics-ready datasets for enterprise reporting.</p><p> Developed incremental ETL workflows using watermark tables, Delta MERGE/UPSERT logic, and schema validation, drastically reducing duplicate records and resource overhead.</p><p> Optimized PySpark workloads using broadcast joins, caching, and custom bucketing while leveraging Delta Lake features such as ACID transactions and Time Travel for data reliability.</p><p> Configured event-driven pipeline triggers via Azure Event Grid and Blob storage events for seamless end-to-end data processing and automated monitoring.</p>
نظام التوظيف التشغيلي المعتمد على الذكاء الاصطناعي أولاً، الذي يوحّد ويؤتمت دورة التوظيف بالكامل للمؤسسات الحديثة حول العالم.
د.جوب منطقة حرة ذ.م.م. 2026 © جميع الحقوق محفوظة
