shivdeep mohite

shivdeep mohite

Data Engineer
India
Marathi, English, Hindi
Full-time On-site

About Me

Data Engineer with 1+ years of experience building and migrating production data pipelines on Databricks and Microsoft SQL Server for AT&T at Amdocs. Skilled in PySpark, SQL, Delta Live Tables (DLT), Autoloader, and Meda…

Experience

Data Engineer

Amdocs · Pune, India
Apr 2025 - Present · 1 year 6 months

<p>Migrated 50+ database objects and ETL workflows from SQL Server (SSMS) to the Databricks Lakehouse with 100% data reconciliation, modernizing legacy reporting workloads.</p><p> Designed and deployed Delta Live Table (DLT) pipelines using PySpark and Spark SQL for automated, reliable ingestion and transformation, reducing manual intervention and pipeline failures.</p><p> Built a file-watcher and file-loader framework that auto-detects, validates, and loads incoming source files, eliminating manual file handling and reducing load errors.</p><p> Developed an Initial Load framework running parallel full loads of selected tables, cutting new-table onboarding time by ~40%.</p><p> Built an Incremental Load framework (last-max-date to current) for efficient delta processing, applying PySpark performance tuning and optimization to improve refresh performance and reduce compute cost.</p><p> Implemented Autoloader-based streaming ingestion from Oracle GoldenGate (OGG) sources into the Lakehouse for near real-time data availability.</p><p> Developed Gold-layer "Biller" reporting tables per business logic, applying dimensional / star-schema data modeling within the Medallion (Bronze/Silver/Gold) architecture; managed data access and governance via Unity Catalog.</p><p> Authored YAML-based job orchestration with dependency-aware scheduling (Databricks Jobs and Azure Data Factory) for automated batch pipeline runs.</p><p> Automated job execution and monitoring on Linux using shell scripts and PuTTY; collaborated in an Agile team using Git/GitHub with CI/CD (Azure DevOps) workflows for version control and deployment.</p>

Skills

Python SQL PySpark Shell Scripting Databricks Delta Live Tables (DLT) Delta Lake Autoloader Unity Catalog Spark SQL Azure Data Factory (ADF) Azure ADLS Snowflake ETL/ELT Data Warehousing Data Modeling Dimensional Modeling Star Schema Medallion Architecture Lakehouse PySpark Performance Tuning Optimization Incremental Load Frameworks Initial Load Frameworks Data Reconciliation Data Quality Checks Microsoft SQL Server SSMS Databricks Jobs Scheduler YAML CI/CD Azure DevOps Git GitHub PuTTY Linux Oracle GoldenGate (OGG) Agile Scrum
Report this Profile?