Data Engineer with 4 years of experience architecting real-time and batch ETL/ELT platforms for telecom analytics, processing using PySpark, Kafka, and Spark Structured Streaming. Owns Medallion lakehouse architecture en…
Data Engineer with 4 years of experience architecting real-time and batch ETL/ELT platforms for telecom analytics, processing using PySpark, Kafka, and Spark Structured Streaming. Owns Medallion lakehouse architecture end to end on AWS (S3, EMR) and Databricks, with Snowflake and Star Schema modeling for downstream BI and ML consumption. Led Spark cluster tuning and workload right-sizing (dynamic allocation, Spot instance migration for batch jobs) that cut compute spend by 20%. Partners with data scientists, analysts, and product owners to translate business requirements into production-grade, SLA-driven data pipelines orchestrated via Airflow.
<p>Built and operated a ˜8TB/day streaming lakehouse using Kafka (Amazon MSK) and Spark Structured Streaming, landing raw events in S3 and implementing a Bronze/Silver/Gold Medallion architecture with curated Gold tables feeding Redshift.</p><p> Designed Star Schema data models for data warehousing on Gold-layer outputs in Snowflake and Redshift, supporting downstream BI reporting and ML feature consumption while partnering with data scientists, analysts, and product owners on requirements.</p><p> Orchestrated production pipelines using Apache Airflow, building SLA-driven, dependency-managed DAGs with IAM-scoped access across S3, Glue, and Redshift, along with monitoring and alerting for pipeline health.</p><p> Led Spark cluster tuning and workload right-sizing on AWS EMR using dynamic executor allocation and Spot instance migration, reducing compute costs by ˜20%.</p><p> Reduced recurring production incidents by 60% by root-causing streaming failures related to checkpoint corruption and hardening the recovery path for reliable streaming workloads.</p><p> Engineered data pipelines supporting real-time telecom usage and charging analytics for a client-facing Vodafone engagement, translating business and rating-logic requirements into production data models.</p><p> Adopted AI-assisted engineering workflows (Claude Code, Cursor) to speed up PySpark/Airflow development, scaffold internal automation and monitoring tooling, and streamline debugging and code review, improving overall engineering velocity.</p>
<p>Worked on AWS-based DevOps workflows involving EC2, S3, IAM, and CloudWatch.<br>
Contributed to CI/CD pipeline automation and earned foundational AWS credentials (Cloud Practitioner, SysOps Administrator, and DevOps Engineer) during the tenure.</p>
<p>Queried and cross-validated data across Hive and Impala tables during the transition to Databricks, supporting legacy analytics compatibility while modernizing workloads to Delta Lake.</p><p> Processed ˜20TB/month of telecom data on Databricks using PySpark/Spark SQL for cleaning, standardization, and aggregation, leveraging Delta Lake for ACID writes and schema enforcement.</p><p> Orchestrated daily batch aggregation jobs using AWS Glue, delivering curated datasets to downstream reporting systems.</p><p> Implemented data validation and reconciliation checks at ingestion, detecting malformed and duplicate records before they reached the Gold layer.</p><p> Modeled curated Gold-layer tables using PySpark, replacing ad hoc SQL extraction scripts before loading data into Snowflake for analyst and ML consumption.</p>
<p>Built a full-stack web application with authentication and role-based access control using Node.js, React, and SQL.</p>
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
