Abdul Mannan

Abdul Mannan

Principal Data Engineer | Expert Vetted on Upwork
Pakistan
Urdu, English

About Me

Databricks Certified Data Engineer with 13+ years of experience building large-scale Data Lakes, real-time pipelines, and analytics platforms across Fintech, Cybersecurity, Health Care, EDtech, and social media domains. …

Experience

Freelance Data Engineer

Upwork
Jan 2021 - Present · 5 years 6 months

Design, develop, and optimize multiple data warehouses, data marts, and data lake(house) using Databricks and Spark.
Optimize ETL performance, cost, and execution time using PySpark on Databricks (AWS, GCP, and Azure).
Design cost-efficient and performant solutions, solving complex problems on AWS and Azure cloud.

Freelance Data Engineer

Upwork
Jan 2021 - Present · 5 years 6 months

Design develop, and optimize multiple data warehouses, data marts, and data lake(house) using Databricks and Spark. Optimize ETL performance, cost, and execution time using PySpark on Databricks (AWS, GCP, and Azure) Design cost-efficient & performant solutions, solving complex problems on AWS & Azure cloud.

Principal Data Engineer

Gettr · Lahore, Pakistan
Sep 2022 - May 2024 · 1 year 7 months

Tech company focused on social mediaRedesign Datalake infrastructure, leveraging Databricks and SparkDeveloped intricate dimensional models for various data entities including Posts, UsersSuccessfully overhauled the end-to-end real-time user rewards system

Principal Data Engineer

Gettr - Remote
Oct 2022 - May 2024 · 1 year 7 months

Redesign Datalake infrastructure, leveraging Databricks and Spark.
Developed intricate dimensional models for various data entities including Posts and Users.
Successfully overhauled the end-to-end real-time user rewards system.

Sr. Data Engineer

Visionet · Lahore, Pakistan
Mar 2021 - Oct 2022 · 1 year 7 months

IT consulting and services company Design, build, and optimize ETL data pipelines in Spark on DatabricksLead the offshore development team and ensure client requirements are met in a timely manner.

Sr. Data Engineer

Visionet
Mar 2021 - Oct 2022 · 1 year 7 months

Design, build, and optimize ETL data pipelines in Spark on Databricks.
Lead the offshore development team and ensure client requirements are met in a timely manner.

Sr. Data Engineer - Big Data

IBM · Lahore, Pakistan
Apr 2019 - Apr 2021 · 1 year 11 months

1 - Big Data Consultant – Allied Bank Limited Designed large-scale data pipelines with ODI, Kafka, Spark, and Cloudera CDH, including real-time fraud detection for a major Pakistani bank. Automated the creation of 28,000 ETL jobs using Groovy & ODI SDK, saving 8 months of effort, and deployed predictive models that boosted campaign response by 3x and optimized ATM operations nationwide. Led stakeholder engagements, authored technical documentation, trained 20+ colleagues, and identified $2–3M in new data-driven opportunities, earning an IBM “We Deliver” 2019 nomination. 2 - Sr. Big Data Consultant - United Bank Limited Designed Data Lake and Multiple Datamarts ETL pipelines using Informatica Big Data Suite, HDP, and IBM Watson, including a real-time fraud detection pipeline delivering end-to-end results in under 2 minutes. Built scalable solutions that overcame major product limitations, created a custom automation framework that accelerated pipeline development by 10x, and improved Teradata FSLDM integration.Served as an associate solution architect, authored key technical/architecture documents, and conducted multiple knowledge- sharing sessions on Informatica Big Data Suite, HDP, and the Hadoop ecosystem.

Sr. Data Engineer - Big Data

IBM
Apr 2019 - Apr 2021 · 2 years

Designed large-scale data pipelines with ODI, Kafka, Spark, and Cloudera CDH, including real-time fraud detection for a major Pakistani bank.
Automated the creation of 28,000 ETL jobs using Groovy and ODI SDK, saving 8 months of effort.
Deployed predictive models that boosted campaign response by 3x and optimized ATM operations nationwide.
Led stakeholder engagements, authored technical documentation, trained 20+ colleagues, and identified $2–3M in new data-driven opportunities.
Earned an IBM “We Deliver” 2019 nomination.

Data Engineer

Arbisoft
Oct 2015 - Mar 2019 · 3 years 5 months

Redesigned MapReduce/Hive ETL pipeline to PySpark/Hive, reducing costs and saving $50K annually.
Migrated legacy development platform to Docker and Kubernetes.
Developed the company's 1st hybrid mobile application using React Native.

Data Engineer

Arabisoft · Lahore, Pakistan
Sep 2015 - Feb 2019 · 3 years 4 months

Software development company Redesigned MapReduce/Hive ETL pipeline to Pyspark/Hive, reducing costs and saving $50K annuallyMigrated legacy development platform to Docker and Kubernetes Developed the company's 1st hybrid mobile application using React Native

Software Engineer

Nextbridge · Lahore, Pakistan
Dec 2014 - Oct 2015 · 9 months

Software development company Designed & developed web projects using Django, AngularJS, and ReactJSOverseeing and quality controlling backend and frontend development work Practiced agile methodologies and team collaboration

Software Engineer

Nextbridge
Jan 2015 - Oct 2015 · 9 months

Designed and developed web projects using Django, AngularJS, and ReactJS.
Overseeing and quality controlling backend and frontend development work.
Practiced agile methodologies and team collaboration.

Software Engineer

PureLogics · Lahore, Pakistan
Nov 2012 - Dec 2014 · 2 years

Software development company Developed proficiency with Laravel, Codeigniter framework, and jQuery. Conducted thorough unit testing and improved performance through automated testing.Liaised with quality assurance to ensure professional standards

Software Engineer

PureLogics
Dec 2012 - Dec 2014 · 2 years

Developed proficiency with Laravel, CodeIgniter framework, and jQuery.
Conducted thorough unit testing and improved performance through automated testing.
Liaised with quality assurance to ensure professional standards.

Sr. Big Data Consultant

United Bank Limited

Designed Data Lake and multiple datamarts ETL pipelines using Informatica Big Data Suite, HDP, and IBM Watson, including a real-time fraud detection pipeline delivering end-to-end results in under 2 minutes.
Built scalable solutions that overcame major product limitations.
Created a custom automation framework that accelerated pipeline development by 10x.
Improved Teradata FSLDM integration.
Served as an associate solution architect.
Authored key technical and architecture documents.
Conducted multiple knowledge-sharing sessions on Informatica Big Data Suite, HDP, and the Hadoop ecosystem.

Certifications

Databricks Certified Associate Developer for Apache Spark 3.0

Databrick · Lahore, Pakistan · 2024

Databricks Certified Data Engineer Professional

Databricks · Lahore, Pakistan · 2024

Oracle Data Integrator 12c Certified Implementation Specialist

Oracle · Lahore, Pakistan · 2019

CCA Spark and Hadoop Developer Cloudera - License No (100-018-963)

Cloud · Lahore, Pakistan · 2017

Skills

MySQL Oracle PHP: Hypertext Preprocessor (PHP) Python Teradata AngularJS Cassandra Data Integration Django Hadoop Hive JavaScript Laravel Scala Spark SVN Docker Groovy HBase Jenkins Kafka Kubernetes Zookeeper Solution Architecture Data Modeling Data Mapping Data Lineage Databricks Delta Live Tables Unity Catalog Delta Storage AWS EMR Athena Redshift Glue S3 RDS Kinesis Data Firehose Kinesis Data Streams PostgreSQL GCP BigQuery Google Storage
Report this Profile?