About Me
Senior Data Engineer with 6.9 years of experience in designing, developing, and optimizing scalable data pipelines across Enterprise Data Lake and Azure Cloud platforms. Strong expertise in Azure Databricks, Azure Data F…
Senior Data Engineer with 6.9 years of experience in designing, developing, and optimizing scalable data pipelines across Enterprise Data Lake and Azure Cloud platforms. Strong expertise in Azure Databricks, Azure Data Factory, PySpark, and SQL. Experienced in handling large-scale structured and semi-structured datasets, implementing CDC & SCD strategies, and building high-performance data solutions. Established track record of resolving production issues and delivering enterprise-scale data platforms.
Experience
Senior Data Engineer (Azure Cloud & Machine Learning Developer)
Designed and implemented scalable Azure Databricks pipelines processing large-scale enterprise sales datasets.
Built automated data ingestion frameworks extracting data from Oracle and file-based systems into Azure Data Lake.
Developed complex transformation logic using PySpark and SQL improving processing efficiency by ~30%.
Created Databricks notebooks and job orchestration to automate ML data preparation workflows.
Designed source-to-target mapping documents, data lineage diagrams, HLD & LLD documents.
Led a team in delivering business solutions aligned with client requirements.
Resolved high-priority production issues ensuring SLA adherence and system stability.
Collaborated with DevOps for smooth code deployment across testing and production environment.
Senior Data Engineer (Azure Cloud & Machine Learning Developer)
Designed and implemented scalable Azure Databricks pipelines processing large-scale enterprise sales datasets., Built automated data ingestion frameworks extracting data from Oracle and file-based systems into Azure Data Lake., Developed complex transformation logic using PySpark and SQL improving processing efficiency by ~30%., Created Databricks notebooks and job orchestration to automate ML data preparation workflows., Designed source-to-target mapping documents, data lineage diagrams, HLD & LLD documents., Led a team in delivering business solutions aligned with client requirements., Resolved high-priority production issues ensuring SLA adherence and system stability., Collaborated with DevOps for smooth code deployment across testing and production environment.
Data Engineer - Data Migration Project
Implemented CDC logic for incremental data extraction.
Migrated enterprise data from on-prem Enterprise Data Lake to Azure Cloud.
Built batch ingestion pipelines in Databricks for multi-source RDBMS and flat file systems.
Performed volumetric and integration analysis for multiple source systems.
Designed scalable transformation workflows using PySpark and SQL.
Supported Go-Live, Hypercare, and Production stabilization activities.
Improved migration efficiency through reusable notebook frameworks.
Identified and handled PII data ensuring compliance and governance standards.
Data Engineer - Data Migration Project
Implemented CDC logic for incremental data extraction., Migrated enterprise data from on-prem Enterprise Data Lake to Azure Cloud., Built batch ingestion pipelines in Databricks for multi-source RDBMS and flat file systems., Performed volumetric and integration analysis for multiple source systems., Designed scalable transformation workflows using PySpark and SQL., Supported Go-Live, Hypercare, and Production stabilization activities., Improved migration efficiency through reusable notebook frameworks., Identified and handled PII data ensuring compliance and governance standards.
Data Engineer – Enterprise Data Lake
Implemented SCD Type 1 & Type 2 logic to maintain historical data tracking.
Developed ETL pipelines using Sqoop, Talend, Hive, and SQL for enterprise reporting.
Built daily incremental load pipelines using CDC methodology.
Automated repetitive operational tasks using shell scripting.
Supported Sales, Partner 360, Customer 360, and KYC analytics modules.
Worked closely with Business Analysts and stakeholders to convert requirements into technical solutions.
Provided production support and resolved critical data issues within SLA.
Data Engineer – Enterprise Data Lake
Implemented SCD Type 1 & Type 2 logic to maintain historical data tracking., Developed ETL pipelines using Sqoop, Talend, Hive, and SQL for enterprise reporting., Built daily incremental load pipelines using CDC methodology., Automated repetitive operational tasks using shell scripting., Supported Sales, Partner 360, Customer 360, and KYC analytics modules., Worked closely with Business Analysts and stakeholders to convert requirements into technical solutions., Provided production support and resolved critical data issues within SLA.