Senior Data Engineer with close to 10 years building, migrating and operating production Spark batch and near-real-time streaming pipelines in highly governed enterprise environments. Hands-on with PySpark, Spark SQL, Ka…
Senior Data Engineer with close to 10 years building, migrating and operating production Spark batch and near-real-time streaming pipelines in highly governed enterprise environments. Hands-on with PySpark, Spark SQL, Kafka and Kafka Connect, CDC, Python and SQL on Azure Databricks (Delta Lake, Delta Live Tables, Auto Loader), ADLS Gen2 and Cloudera/Hadoop. Currently on a regulated banking data platform for Emirates NBD (via ValueLabs): led the SAP HANA decommissioning, migrating ~180 daily legacy jobs to PySpark across 5 banking entities, and delivered Kafka streaming into Oracle with CI/CD. Owns daily and monthly production loads with row-count and reconciliation checks. Builds medallion lakehouse layers and reusable config-driven frameworks; uses AI tools (Claude Code) in day-to-day delivery. Databricks Data Engineering Professional certified; reviews code and mentors engineers.
<p>Led the SAP HANA decommissioning, migrating BDM workflows to PySpark for 5 banking entities, eliminating SAP licence costs and automating ~180 daily jobs with SQL loads into Oracle.<br>
Built a near-real-time Kafka streaming pipeline on the bank’s centralised streaming platform: consumed events, flattened nested payloads, republished to a new topic and loaded into Oracle via Kafka Connect, deployed through CI/CD with Git.<br>
Engineered an end-to-end ETL pipeline for Wholesale Banking customer deals using PySpark, Kafka and an ODS, standardising approvals and tracking deal expiries for business cost savings.<br>
Designed a parameterised PySpark framework configured through simple YAML files, letting the team replicate ETL jobs quickly and with zero defects.<br>
Manage daily and monthly Cloudera ETL loads across finance sources and pricing structures via Airflow and Oozie, with row-count and reconciliation checks; uses Claude Code for faster delivery with no critical issues.<br>
Parsed nested MongoDB JSON (read-only) with PySpark.</p>
<p>Built ADLS Gen2 medallion layers (raw, curated) in Parquet over a 100 GB to 1 TB estate, ingesting via ADF, Synapse Pipelines and PySpark, cutting cost and improving reliability and data quality.<br>
Built feature-based ADF pipelines for a new dealer analytics web app, processing 100 GB daily.<br>
Used Azure Synapse Analytics for ~3 years, building major parts of the warehouse: dedicated and serverless SQL pools, Synapse Pipelines, Spark notebooks, dimensional models (fact/dimension), T-SQL stored procedures and warehouse loads, tuned via distribution, indexing and partitioning.<br>
Designed and delivered the migration of enterprise data from legacy Hadoop tables to an Azure Cosmos DB application database, reducing query retrieval times by 40%.<br>
Worked with Business Analysts and Backend Developers to define requirements and design optimised JSON schemas for Cosmos DB.<br>
Also worked with Unity Catalog and Snowflake.</p>
<p>Built batch and streaming IoT pipelines with PySpark and Spark Streaming across Azure cloud and on-premises deployments, processing semi-structured telemetry from ActiveMQ and Event Hubs.<br>
Designed a framework to deploy machine learning models on unstructured data (text, images, video), routing predictions to Cosmos DB and MongoDB for downstream analytics.<br>
Developed automated PySpark and SQL ETL jobs to clean and transform data from ADLS Gen2 raw to curated layers (Parquet), scheduled via Rundeck, Azure Pipelines and Azure Data Factory.<br>
Ran Azure Databricks PySpark notebooks orchestrated from Azure Data Factory, and worked with HDInsight, Unity Catalog, Kubernetes with Helm and Docker Compose.<br>
Built Python Flask REST API endpoints for CRUD operations across target data systems, saving an estimated 15 hours of manual effort per week.</p>
<p>Built a Master Data Management (MDM) solution for PwC with the team, translating complex functional requirements into matching and survivorship rules to identify the golden record.<br>
Developed PySpark ETL workflows to migrate CSV, JSON and MySQL data into an on-premises Cloudera HDFS environment in Parquet format.<br>
Created HDFS-based tables and SQL transformations for the curated layer, scheduled with Cron.<br>
Automated data API testing with Python and Selenium to validate Oracle DB to MongoDB data loads.<br>
Contributed to data warehouse schema design and resolved defects through VSTS (Azure DevOps).</p>
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
