Koti A

Koti A

Azure Data Engineer
India
Kannada, Tamil, Telugu, English, Hindi
Full-time Hybrid

About Me

Azure Data Engineer with 4years of experience in designing and implementing scalable ETL/ELT solutions on Microsoft Azure. Hands-on expertise in Azure Data Factory (ADF), Azure Databricks, PySpark, Azure Synapse Analytic…

Experience

Data Engineer Consultant

IBM · Bengaluru, India
Jul 2022 - Present · 4 years 4 months

<p>Developed and maintained ETL/ELT pipelines using Azure Data Factory and Azure Synapse Analytics to ingest data from SQL Server, databases, flat files and other source systems into ADLS Gen2.</p><p> Built metadata-driven pipelines to process multiple tables using configuration-driven parameters such as source table, target path, load type and watermark columns.</p><p> Implemented both full-load and incremental-load patterns using watermark columns and last processed timestamps to load only new or changed records where applicable.</p><p> Worked with Azure Synapse Analytics, Dedicated SQL Pools, Serverless SQL Pools and SQL Server for data transformation, querying and data processing.</p><p> Developed SQL queries, stored procedures, views, CTEs and window functions to support data transformation and business requirements.</p><p> Implemented Slowly Changing Dimension (SCD) Type 1 and Type 2 logic to maintain current and historical dimension data.</p><p> Worked on Star Schema data modelling, including fact and dimension tables, to organize data for analytical requirements.</p><p> Implemented a Medallion Architecture using ADLS Gen2, organizing data into Bronze, Silver and Gold layers.</p><p> Performed unit testing, source-to-target validation, record-count checks, duplicate checks, null checks and data-quality validations before promoting changes.</p><p> Followed Agile/Scrum development practices, participating in sprint planning, development, testing, defect resolution and deployment activities.</p><p> Monitored pipeline executions, investigated failures and supported troubleshooting of SQL, Synapse and data-loading issues.</p>

Skills

Microsoft Azure Azure Data Factory (ADF) Azure Databricks ADLS Gen2 Azure Synapse Analytics ETL/ELT Data Pipelines Data Integration Data Transformation Data Migration Incremental & Full Load Processing PySpark Python T-SQL Azure Devops Git Microsoft Fabric SQL Server CI/CD Agile/Scrum Data Validation Unit Testing Performance Optimization Troubleshooting Data Warehousing Lakehouse Medallion Architecture Delta Lake Unity Catalog SCD Type 1 SCD Type 2 Star Schema
Report this Profile?