siva sai

siva sai

Azure Data Engineer
India
Telugu, English, Hindi
Full-time On-site

About Me

Azure Data Engineer with 4.5+ years of experience designing, developing, and supporting scalable ETL/ELT data pipelines using Microsoft Azure, Azure Data Factory, Azure Databricks, Apache Spark/PySpark, ADLS Gen2, Azure …

Experience

Data Engineer

Starlite Infotech · Bengaluru
Sep 2024 - Jan 2026 · 1 year 3 months

<p>Designed, developed, and maintained scalable Azure Data Factory ETL/ELT pipelines for reliable ingestion and transformation of data from multiple source systems.</p><p> Developed Azure Databricks and PySpark solutions for distributed, large-scale data processing, transformation, cleansing, filtering, joins, and aggregations.</p><p> Implemented and maintained Azure DevOps CI/CD pipelines to automate build, validation, release, and deployment workflows for data applications and pipeline changes.</p><p> Supported automated deployment processes across development and higher environments, improving consistency and reducing manual deployment activities.</p><p> Ingested structured and semi-structured data including CSV, JSON, and relational datasets into ADLS Gen2 and maintained bronze, silver, and gold data layers.</p><p> Implemented incremental data loading and data-quality validation to improve processing efficiency, completeness, accuracy, consistency, and data integrity.</p><p> Developed SQL extraction, transformation, validation, reconciliation, and analysis queries using Azure SQL.</p><p> Prepared curated datasets in Azure Synapse Analytics for downstream analytics and reporting.</p><p> Monitored production data pipelines and CI/CD workflows, investigated failures, performed troubleshooting and root-cause analysis, and supported reliable data operations.</p><p> Collaborated with technical and cross-functional teams to deliver scalable Azure data solutions and support Agile release activities.</p>

Data Engineer

L&T Technology Services (LTTS) · Bengaluru
Sep 2021 - Sep 2024 · 3 years

<p>Developed and maintained Azure Data Factory pipelines for extraction, transformation, and loading from multiple source systems.</p><p> Configured ADF datasets, linked services, activities, parameters, variables, dependencies, and triggers based on business requirements.</p><p> Supported Azure DevOps CI/CD processes for automated build and deployment of data-platform components and pipeline changes.</p><p> Worked with deployment pipelines and release workflows to promote data solutions across environments and maintain consistent delivery practices.</p><p> Ingested relational and file-based data into ADLS Gen2 and organized raw and processed datasets across bronze, silver, and gold layers.</p><p> Developed Python and PySpark programs in Azure Databricks for distributed data processing and scalable data transformation.</p><p> Implemented PySpark transformations including joins, aggregations, filtering, deduplication, null handling, and data-type conversions.</p><p> Implemented incremental processing using date, timestamp, and key columns to efficiently process new and modified records.</p><p> Performed source-to-target validation, reconciliation, and data-quality checks to ensure data completeness, accuracy, consistency, and integrity.</p><p> Monitored scheduled pipelines and deployment workflows, investigated failures, and performed troubleshooting and root-cause analysis for data ingestion and transformation issues.</p><p> Supported production data pipelines, resolved data-quality issues, collaborated with cross-functional teams, and maintained data-flow, mapping, pipeline, and deployment documentation.</p>

PROJECTS

data acquiring

starlite infotech
Duration : 23-Sep-2024 - 12-Jan-2026

<p>Designed, developed, and maintained scalable Azure Data Factory ETL/ELT pipelines for reliable ingestion and transformation of data from multiple source systems. • Developed Azure Databricks and PySpark solutions for distributed, large-scale data processing, transformation, cleansing, filtering, joins, and aggregations. • Implemented and maintained Azure DevOps CI/CD pipelines to automate build, validation, release, and deployment workflows for data applications and pipeline changes. • Supported automated deployment processes across development and higher environments, improving consistency and reducing manual deployment activities. • Ingested structured and semi-structured data including CSV, JSON, and relational datasets into ADLS Gen2 and maintained bronze, silver, and gold data layers. • Implemented incremental data loading and data-quality validation to improve processing efficiency, completeness, accuracy, consistency, and data integrity. • Developed SQL extraction, transformation, validation, reconciliation, and analysis queries using Azure SQL. • Prepared curated datasets in Azure Synapse Analytics for downstream analytics and reporting. • Monitored production data pipelines and CI/CD workflows, investigated failures, performed troubleshooting and root-cause analysis, and supported reliable data operations. • Collaborated with technical and cross-functional teams to deliver scalable Azure data solutions and support Agile release activities</p>

Skills

Microsoft Azure Azure Databricks Azure Data Factory ADLS Gen2 Azure Synapse Analytics Azure SQL Azure Storage Apache Spark PySpark Spark DataFrames ETL ELT data ingestion data transformation data cleansing incremental data loading source-to-target validation reconciliation data quality data integrity Azure DevOps Azure Pipelines CI/CD pipelines automated build and deployment release management deployment automation source-control-based delivery Python SQL SQL Server data manipulation validation pipeline monitoring failure investigation logging troubleshooting root-cause analysis production support performance optimization Agile cross-functional collaboration technical documentation pipeline design scalable data processing operational excellence
Report this Profile?