نبذة عني
On the data engineering side, I have experience orchestrating Spark workflows in Databricks. Experienced with Linux/Python/SQL/Docker/Terraform, and various AWS/Azure/GCP cloud services. I also have professional experien…
On the data engineering side, I have experience orchestrating Spark workflows in Databricks. Experienced with Linux/Python/SQL/Docker/Terraform, and various AWS/Azure/GCP cloud services. I also have professional experience with setting up ETL pipelines to integrate over 15 systems including Finance, ERP (JD Edwards) and Procurement systems into Vertica (a column-store database) with classical data vault modelling (Hub-Link-Satellite). I have integrated data ingestion from real-time/streaming sources (Kafka) using Databricks/Spark, and have experience with data orchestration (Databricks Workflows). I was the lead for the planning, development and execution of a 100TB data migration into Databricks.
Aside from Databricks I have also picked up Snowflake + DBT, having gone through Roberto Zagni’s Data Engineering with DBT book to learn best practices for production data modelling/engineering.
On the generative AI/LLM side, I have experience with the development cycle + deployment. Can handle deploying the LLM infrastructure on GCP/Azure/AWS cloud, developing and serving containerized python backend applications that use LLM, setting up a test-driven process for agentic AI development. I can also optimize the LLM model to reduce costs/inference times using optimization techniques such as KV caching & quantization. I have experience with both a classical RAG architecture and also voicebot applications.
الخبرة
Part-time research for Digivate Labs (Azure Gen AI)
Part-time research looking into and advising on generative AI app hosting using the new Microsoft Foundry.
Primarily focusing on meeting UAE data residency requirements for prospective clients.
Exploring alternatives for fine-tuning that respect UAE data residency requirements.
Exploring alternatives for voice-to-voice inference processing because the Microsoft Foundry Voice Live API does not support regional deployment for UAE-north.
Technical Consultant
Worked primarily as a data engineer with a heavy focus on Databricks.
Later involved with research and development for a low-latency speech-to-speech LLM app development on GCP.
Set up an unstructured text streaming pipeline to process scraped websites and apply a Databricks registered ML model for classification.
Set up an Azure SQL database and pushed the text data into stream using Azure Logic Apps and Azure Event Hub.
Automated reports for management.
Built a batch pipeline in Databricks to ingest invoice PDF files and extract data using the open-AI 4o model configured in Databricks Mosaic Gateway.
Parsed PDF invoice files using OCR technology into unstructured text and extracted structured field data using an LLM.
Read through, analyzed and implemented Databricks’ unofficial X12 EDI Parser from GitHub to ingest EDI data for X12 835 and 837 transaction types.
Refactored complex Python pipelines including geospatial data into Databricks PySpark, reducing execution time by 60% in some cases.
Did a proof-of-concept for an AWS DynamoDB ingestion pipeline on Databricks.
Familiarity with the AWS SDK for Python (Boto3 library).
Refactored a pipeline that wrote data into Aerospike.
Designed workflows to migrate and validate 100TB of data from AWS S3 into Databricks Delta Tables.
Discussed AWS/Azure cloud architectures with clients to understand their current systems and requirements and how their pipelines could be migrated to Databricks.
Implemented a phone voicebot solution integrating Twilio and Gemini Flash 2.5 Live via a websocket developed using the Google gen-AI Python SDK.
Deployed the solution as a container on Google Cloud Run.
Implemented bidirectional audio byte streaming to enable low latency conversations.
Set up a FreePBX/Asterisk instance on a Linux VM as an autodialer component routing the call to a telephony service via SIP.
Technical Consultant
Set up an unstructured text streaming pipeline to process scraped websites and apply a Databricks registered ML model for classification. Set up an Azure SQL database and pushed the text data into stream using Azure Logic Apps and Azure Event Hub (which provides Kafka connector protocol for data sinks). Automated reports for management., Batch pipeline in Databricks to ingest invoice pdf files and extract data using open-AI 4o model configured in Databricks Mosaic Gateway. PDF invoice files parsed using OCR technology into unstructured text from which structured field data such as invoice number, invoice line item, billing address, quantity, tax type & tax amount were extracted using an LLM., Read through, analyzed and implemented Databricks’ (unofficial) X12 EDI Parser (available on Github) in order to implement a pipeline to ingest EDI data for X12 835 and 837 transaction types., Refactored complex python pipelines including Geospatial data into Databricks PySpark, in some cases reducing the execution time by 60%., Did a proof-of-concept for an AWS DynamoDB (NoSQL) ingestion pipeline on Databricks. Familiarity with the AWS SDK for Python (Boto3 library). Also refactored a pipeline that wrote data into Aerospike (another NoSQL key-value database)., Designed workflows to migrate and validate 100TB of data from AWS S3 into Databricks Delta Tables., Discussed AWS/Azure cloud architectures with clients to understand their current systems and requirements and how their pipelines could be migrated to Databricks., Implemented a phone voicebot solution which integrated Twilio & Gemini Flash 2.5 Live via a websocket developed using the Google gen-AI python SDK. The solution was deployed as a container on Google Cloud Run and involved bidirectional audio byte streaming in order to enable low latency conversations. Set up a FreePBX/Asterisk instance on a Linux VM as an autodialer component routing the call to a telephony service via SIP.
Data Engineer
Worked as a data engineer.
Performed system integrations.
Designed ETL pipelines.
Coordinated with software vendors to discuss API requirements for a newly launched datahub project.
Participated in discussions with various departments including Operations, Finance, HR, and Procurement to understand their data models.
Designed pipelines to integrate over 15 systems with a proprietary datahub platform.
Was responsible for the initial implementation of the company’s datahub project.
Automated reports for management.
Designed a psychometric report in PowerBI for onboarding new employees.
Involved in integration of a new procurement system (GEP) as a replacement for the outdated customized procurement modules in JDE.
Exposed to various JDE modules and tools such as orchestrators.
Worked with functional basics of procurement such as Bidding/PR/PO, 2-way voucher and 3-way voucher matching, and approval processes.
Data Engineer
Participated in discussions with various departments (Operations/Finance/HR/Procurement) to understand their data models., Designed pipelines to integrate over 15 systems with a proprietary datahub platform and was responsible for the initial implementation of the company’s datahub project., Automated reports for management., Designed a psychometric report in PowerBI for onboarding new employees., Involved in integration of a new procurement system (GEP) as a replacement for the outdated (customized) procurement modules in JDE. Exposure to various JDE modules and tools such as orchestrators, and also functional basics of procurement such as Bidding/PR/PO, 2-way voucher & 3-way voucher matching, approval processes.
Data Science Intern
Developed a neural network model from scratch to detect clients’ risk of default.
The model performed competitively with existing models.
Established an automated system to periodically analyze new data on a daily basis.
Performed data preprocessing including data normalization, one-hot encoding, imputation, and data wrangling.
Performed customer segmentation on Mashreq’s client base using traditional machine learning techniques such as K-means clustering to enable more specific targeting for advertisements and notifications.
Data Science Intern
Developed a neural network model from scratch to detect clients’ risk of default which performed competitively with existing models. Established an automated system to periodically analyze new data on a daily basis., Performed data preprocessing including data normalization, one-hot encoding, imputation, and data wrangling., Performed customer segmentation on Mashreq’s client base using traditional machine learning techniques such as K-means clustering to enable more specific targeting for advertisements and notifications.
Intern
Researched and analyzed companies and their financial sheets to evaluate their eligibility for corporate loans.
Assisted the Relationship Managers on client visits.
Observed relationship managers discuss new opportunities and follow up with clients.
Assisted with business proposals and handled relations with other banks during time in the financial institutions department.
Worked with wholesale banking relationship managers to identify and develop new markets to increase the asset book.