Vakula

Azure Data Engineer with 10 Years of IT Industry experience and 4 Years of experience in Azure Data Factory and Databricks with proven knowledge in design, development, testing, and optimizing of data pipelines in both on-premises and Azure cloud platforms. Aiming to obtain an elevated data professional career in data and analytics.

<p>Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.<br>
Developed Batch and Near Real Time data processing using ADF and Azure Databricks.<br>
Developed python and pyspark scripts on Azure Databricks platform for processing data from various sources and variety of formats.<br>
Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different Azure Activities like Move &Transform, Copy, filter, for each, Databricks etc.<br>
Experience in building Lakehouse using Azure Databricks and open-source Delta (DataLake).<br>
Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.<br>
Used Azure Devops & Jenkins pipelines to build and deploy different resources(Code and Infrastructure) in Azure.<br>
Experienced in developing audit, balance and control framework using SQL DB audit tables to control the ingestion, transformation and load process in Azure.<br>
Implemented both ETL and ELT architectures in Azure using Data Factory, Databricks, SQL DB and SQL Data warehouse.<br>
Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.<br>
Using Azure Synapse to stage data and created external tables in azure Synapse loaded data from ADLS Location.<br>
Created High level technical design documents and Application design document as per the requirements and delivered clear, well-communicated and complete design documents.<br>
Scheduled, automated business processes and workflows using Azure Logic Apps.</p>

<p>Involved in gathering & analysis of business Requirements from End users & internal Business Analysts to Develop Strategies for ETL processes.<br>
Experience in Performance Tuning - Identified and fixed bottlenecks and tuned the complex mappings for better Performance.<br>
Worked on SQL Scripts, T-SQL Stored procedures, queries and packages to load data in SQL Server, and SQL Data warehouse.<br>
Experience in developing Spark applications using Data frames, Spark SQL, and Notebooks in Databricks for data extraction, transformation, and aggregation from multiple file formats to uncover insights.<br>
Used pandas to transform the raw data and upload to ftp sites.<br>
Build data pipelines, data processing tools and technologies in open source and proprietary products.<br>
Developed several Azure Data Factory re-usable pipelines, linked services and activities with parameterization techniques which can be used to cut down development time.<br>
Used Pandas in Python for Data Cleansing and validating the source data.<br>
Built Teradata ELT frameworks which ingests data from different sources using Teradata Legacy load utilities.<br>
Used Python and Shell scripts to Automate Teradata and SQL Server ELT activities</p>

<p>Worked as a ETL SSIS, SSRS Developer.<br>
Created SSIS packages to migrate data from different sources to data warehouse.<br>
Successfully extracted transformed and load data into data warehouse.<br>
Used various SSIS tasks such as conditional Split, Derived Column etc, which did Data Scrubbing, including data validation checks during Staging, before loading the data into the Data warehouse.<br>
Created ETL packages with different data sources (SQL Server, Flat files, Excel source files, Oracle) and loaded the data into target tables by performing different kinds of transformations using SSIS.<br>
Prepare Unit Test Plan and Integrated test Plan with interface modules.<br>
Applied various data transformations like Aggregate, Sort, Multicasting, Conditional Split, Derived column etc.<br>
Data quality done by using the Data cleansing by using ETL development<br>
Designing Spotfire dashboards / reports with complex multiples data sources (SQL, Excel, Flat files) relational design and optimizing applications.<br>
Deploying and troubleshooting the SSIS Packages.</p>

<p>Worked as a ETL Data test Engineer<br>
Created Test plans and cases to test the migration of data from different sources to data warehouse.<br>
Created a SQL Queries to Successfully extract and validate the transformed data into data warehouse.<br>
Verified reports and data ingestion pipelines for quality<br>
Created Test reports and finding the defects in data pipelines<br>
Prepare Unit Test Plan and Integrated test Plan with interface modules.</p>