Samarth Jain

Architecting and processing real-time airline traveller event data (Travel DNA) from Azure Event Hubs into Azure Databricks using a scalable Databricks Lakehouse and Medallion Architecture.
Designed event-driven streaming pipelines with exactly-once processing semantics and fault-tolerant architecture, ensuring highly reliable downstream analytics.
Optimised data model by analysing 30+ tables and eliminating 700+ redundant columns, reducing storage footprint by ~30% and improving query performance.
Implemented Delta Lake optimisations (partitioning, compaction, schema evolution, Z-ordering) to improve query performance and reduce compute costs.
Built and standardised incremental data processing frameworks, real-time ingestion pipelines, and streaming ETL solutions for near real-time analytics use cases.
Enabled high-volume streaming analytics using Spark Structured Streaming and Azure Event Hub integration.

Architecting and processing real-time airline traveller event data (Travel DNA) from Azure Event Hubs into Azure Databricks using a scalable Databricks Lakehouse and Medallion Architecture., Designed event-driven streaming pipelines with exactly-once processing semantics and fault-tolerant architecture, ensuring highly reliable downstream analytics., Optimised data model by analysing 30+ tables and eliminating 700+ redundant columns, reducing storage footprint by ~30% and improving query performance., Implemented Delta Lake optimisations (partitioning, compaction, evolution, schema, Z-ordering) to improve query performance and reduce compute costs., Built and standardised incremental data processing frameworks, real-time ingestion pipelines, and streaming ETL solutions for near real-time analytics use cases., Enabled high-volume streaming analytics using Spark Structured Streaming and Azure Event Hub integration.

Architected enterprise-scale Azure Data Platform (ADLS, ADF, Databricks), migrating from on-prem SQL Server and enabling cloud modernisation, resulting in 30% cost reduction and 2× scalability.
Built batch & real-time streaming pipelines (ADF, Databricks, Event Hub) processing TB-scale data into Delta Lake.
Reduced pipeline runtime 5 hrs → 20 mins (90% improvement) via Spark optimisation, partition tuning, and PySpark performance engineering.
Developed incremental ingestion frameworks (watermarking, Delta) → faster, reliable Power BI refreshes.
Standardised data quality & validation framework across projects (null, format, duplicates, numeric checks).
Delivered serverless Azure Functions solutions → 20% infra cost savings.
Led a team of 8 engineers, driving scalable architecture, enterprise data governance and delivery excellence.
Unified heterogeneous data sources into an enterprise-wide analytics and reporting platform supporting customer analytics, business intelligence and e-commerce reporting use cases.
Built scalable data pipelines enabling real-time customer behaviour analysis, sales analytics, and operational reporting.

Architected enterprise-scale Azure Data Platform (ADLS, ADF, Databricks), migrating from on-prem SQL Server and enabling cloud modernisation, resulting in 30% cost reduction and 2× scalability., Built batch & real-time streaming pipelines (ADF, Databricks, Event Hub) processing TB-scale data into Delta Lake., Reduced pipeline runtime 5 hrs → 20 mins (90% improvement) via Spark optimisation, partition tuning, and PySpark performance engineering., Developed incremental ingestion frameworks (watermarking, Delta) → faster, reliable Power BI refreshes., Standardised data quality & validation framework across projects (null, format, duplicates, numeric checks)., Delivered serverless Azure Functions solutions → 20% infra cost savings., Led a team of 8 engineers, driving scalable architecture, enterprise data governance and delivery excellence., Unified heterogeneous data sources into an enterprise-wide analytics and reporting platform supporting customer analytics, business intelligence and e-commerce reporting use cases., Built scalable data pipelines enabling real-time customer behaviour analysis, sales analytics, and operational reporting.

Analysed daily trading data (equities, derivatives) for a Banking client on risk management & financial risk identification.
Optimise Spark/Hive pipelines, improving query performance by 40%, enhancing SLA.
Automated file transfers via Shell script, reducing manual efforts by 90% (~5 hours weekly).
Automate the file/data arrival report for management, saving over 36 man-hours per month.
Implemented data masking for sensitive PII, ensuring compliance with client policies.
Built CI/CD pipelines with Jenkins, Ansible, and Bitbucket—accelerating deployments by 50%.

Analysed daily trading data (equities, derivatives) for a Banking client on risk management & financial risk identification., Optimise Spark/Hive pipelines, improving query performance by 40%, enhancing SLA., Automated file transfers via Shell script, reducing manual efforts by 90% (~5 hours weekly)., Automate the file/data arrival report for management, saving over 36 man-hours per month., Implemented data masking for sensitive PII, ensuring compliance with client policies., Built CI/CD pipelines with Jenkins, Ansible, and Bitbucket—accelerating deployments by 50%.

Implemented various data transformations on Batch data using Spark processing large-scale CDR (Call Detail Records), subscriber usage and billing reconciliation for a Telecom client.
Designed efficient Hive queries to join tables and filter data to optimise queries.

Implemented various data transformations on Batch data using Spark processing large-scale CDR (Call Detail Records), subscriber usage and billing reconciliation for a Telecom client., Designed efficient Hive queries to join tables and filter data to optimise queries.

Collaborated with stakeholders to gather requirements and translate them into technical deliverables., Resolved 50+ critical live defects in SQL production systems, ensuring high availability., Keeping the work of EPC in sync using DB Dump between multiple ref masters to close all gaps.