Aditya bajpai

Aditya bajpai

Data Engineer
India
English, Hindi
Full-time Hybrid

About Me

Data Engineer with 4 years of production experience building data platforms on Azure Databricks, Snowflake, and Delta Lake. Delivered ETL/ELT pipelines processing millions of records daily for Fortune 500 clients across …

Experience

Consultant – Data Engineering

Deloitte Touche Tohmatsu India LLP · Gurugram, India
Jul 2025 - Present · 1 year 3 months

<p>Engineered 10+ production ETL/ELT pipelines on Azure Databricks and ADF ingesting 20M+ records daily across supply chain, sales, and logistics domains — owning design, deployment, on-call support, and incident resolution.</p><p> Delivered a Bronze-Silver-Gold Medallion Architecture on Delta Lake with schema enforcement, structured checkpointing at each layer, and automated OPTIMIZE and Z-Ordering compaction jobs — providing the project its first auditable, lineage-tracked data tier.</p><p> Reduced Spark cluster compute time on recurring jobs by replacing sort-merge joins with broadcast joins, enabling Adaptive Query Execution (AQE), and applying partition pruning — changes validated by comparing Spark UI job timelines before and after.</p><p> Refactored a legacy Scala transformation codebase to Python/PySpark, cutting total lines by ~35%, centralizing exception handling and structured logging, and reducing new-engineer ramp-up from weeks to days.</p><p> Defined Star Schema dimensional models (Fact and Dimension tables) that the programmer adopted as the standard template across two additional workstreams — reducing modelling rework on each new domain.</p><p> Automated CI/CD release pipelines in Azure DevOps — replacing a manual promotion process and enabling the 3-engineer team to ship environment-validated changes to production with a complete audit trail.</p><p> Delivered config-driven Python ingestion frameworks for REST APIs, ERP flat files, and partner feeds — with centralized exception handling, retry logic on transient failures, and structured logging consumed by Azure Monitor.</p><p> Established production runbooks and monitoring hooks for all pipeline failures — enabling root-cause analysis on recurring incidents and cutting mean time to resolution across the team.</p>

Program Analyst – Data Engineering

Cognizant Technology Solutions · Noida, India
Oct 2022 - Jul 2025 · 2 years 9 months

<p>Owned end-to-end ELT pipeline development on Snowflake and DBT for Nike's global retail data warehouse — covering product catalogue, orders, inventory, and digital analytics across 6 upstream source systems.</p><p> Refactored full-refresh pipeline runs into incremental DBT models backed by Snowflake Streams and Tasks with SCD Type 2 logic — eliminating the morning reporting delay that had been a recurring business complaint.</p><p> Orchestrated legacy data migration into Snowflake using Time Travel for point-in-time validation and Zero Copy Clone for environment promotion — completing cutover with zero data loss and no business downtime.</p><p> Engineered a Snowpipe-driven, event-triggered ingestion layer for flat file and API feeds, reducing data latency from source arrival to analytics availability from hours to under 15 minutes.</p><p> Improved analytical query performance on high-volume Fact tables by applying Snowflake Clustering Keys with micro-partitioning analysis, and rewriting CTE chains using Window Functions and Stored Procedures.</p><p> Standardized DBT testing (not-null, uniqueness, referential integrity) and Jinja macro patterns across 15 data models — establishing the project's first systematic data quality framework and reducing analyst-reported data issues.</p><p> Reduced Snowflake credit consumption by tuning warehouse auto-suspend thresholds and restructuring Task scheduling to avoid overlapping compute — a cost optimization that required no changes to downstream queries.</p><p> Led code reviews, maintained Git branching conventions, and authored internal documentation on DBT patterns and Snowflake best practices — raising baseline code quality across the delivery team.</p>

Skills

Python PySpark SQL Spark SQL Scala Apache Spark Delta Lake Medallion Architecture Partitioning Broadcast Joins AQE Shuffle Reduction Caching Z-Ordering OPTIMIZE VACUUM Azure Databricks Azure Data Factory Azure Data Lake Storage (ADLS Gen2) Azure DevOps Azure Storage Azure Monitor Snowflake Snowpipe Streams Tasks Time Travel Zero Copy Clone Materialized Views Clustering Oracle Database DBT ETL/ELT CDC SCD Type 2 Incremental Loading Star Schema Fact & Dimension Tables CI/CD Git GitHub Jira YAML Pipelines Release Management Data Validation Reconciliation Data Governance Schema Enforcement Error Handling Structured Logging
Report this Profile?