About Me
Having 4.5 Years of experience in ETL/Cloud ETL like Azure Data Factory, Azure Data Bricks, Pyspark, SQL Server, Power BI Development. Experience in writing of SQL Scripting and T-SQL experience in coding Stored Procedur…
Having 4.5 Years of experience in ETL/Cloud ETL like Azure Data Factory, Azure Data Bricks, Pyspark, SQL Server, Power BI Development. Experience in writing of SQL Scripting and T-SQL experience in coding Stored Procedures, User Functions, Indexes, Views, Joins, Triggers, Temporary Tables and CTE etc. Experience of Azure services–Azure SQL (DB and DWH), Azure Storage, USQL, Data Lake and Data Factory. Expertise in creating a dynamic pipeline in Azure Data Factory v2 with Azure Data Lake Store Gen1 and BLOB Storage. Implemented Control flow aactivities: copy activity, Execute Pipeline, Get Meta data, If Condition, look up, Set Variable, Filter, For Each pipeline Activities for On-cloud ETL processing Azure Security Identity: Azure Active Directory App Permissions, Key Vaults Strongly worked on Migrating existing packages from on-premises to cloud, migrated on-premises SQL Server to Blob Storage using Azure DataFactoryv2 Experienced in working with Synapse, ADF pipelines to connect with different sources and load data to cloud environment. Experienced in writing the PySpark code using Azure Data bricks for data transformations. Having experience in Azure Data Bricks and Azure Synapse Analytics. Created notebooks in data bricks and transformed the data using Spark. Designed external tables from parquet files via poly base mechanism. Having experience in Power Query to transform the data within Power BI. Extensive use of DAX (Data Analysis Expressions) functions for the Reports and for the Tabular Models. Worked on Complex model by using both import and Direct Query mode in developing the reports. Implemented Incremental Refresh in Power BI service. Created Reports in Power BI desktop using different type of Visualizations (Pie, Tree Map, Bar, Tabular, Matrix, Card and Slicers). Have implemented Role based security and Dynamic security. Comfortable in working with filters/calculated columns/measures/relationships and transformations of edit Query section. Created New Calculated Columns and Calculated Measures using DAX Expressions. Published/shared the reports by creating Content Pack/Sharing the pbix file. Created User Groups and Implemented Row Level Security to restrict the data access to the users.
Experience
Software Engineer
Working as a Software Engineer in Mphasis Limited, Pune from jan 2020 to Till date
Data Engineer
I am writing to express my interest in the data engineer
position at your company. I have over 5 years of experience in
IT industry field, and I believe I have the skills and
qualifications that match your requirements.
In my previous role as a Data engineer role at Mphasis Limited,
I've worked with ADF, Azure Data Bricks, Power BI, Azure Data
Lake, and SQL Server.
In this role, I've contributed to the successful integration of
Oracle systems for our client, ensuring seamless data flow and
efficient reporting. I thrive in collaborative environments and
am passionate about leveraging technology to solve real-world
problems.
I have also gained valuable skills and knowledge in terms of
technology. I am eager to learn more and expand my expertise
in technology skills or areas that you want to develop
Data Engineer
Implemented activities Copy activity, Execute Pipeline, Get Meta data, If Condition, Lookup, Set Variable, Filter, For Each pipeline Activities for On-cloud ETL processing.
Developed Azure data factory Pipelines for moving data from staging to Data warehouse using Incremental data load process.
Azure Security& Identity: Azure Active Directory App Permissions, Key Vaults.
Designed Azure ADF pipelines to move data from 6 diff sources to Azure Data Lake Gen2 and then to Azure Data Warehouse.
ARM Templates creation, deployment to Infrastructure of resources, Implemented Azure Dev Ops (CI/CD).
Azure SQL to Data Lake.
Data Lake to Data Lake.
SFTP (Atoms) to Data Lake.
SFTP(CASS) to Data Lake.
On Premise SQL to Data Lake.
Oracle to Data Lake.
Explored data in a variety of ways and across multiple visualizations using Power BI.
Effectively used POWER BI joins like inner join and outer joins.
Executed and tested required queries and reports before publishing the reports.
Building, publishing customized interactive reports and dashboards, report scheduling using Power BI service.
Generating reports using DAX time intelligence functions and other kind of DAX functions.
Creating Content packs and Sharing reports and Dashboards.
Scheduling reports using Gateway.
Create a Tabular Model Project, process the Project and Deployed into SQL Server Analysis.
Connecting Live and extract data from SSAS into Power BI desktop.
Data Engineer
Created stored procedures using Common Table Expression (CTE) and various types of UDF function.
Developed Complex T-SQL queries, common table expressions (CTE), stored procedures.
Creating Data Factories, Stored Procedure and scheduled them in Azure Environment.
Worked on ADF Pipelines development with activities Copy Activity, Stored Procedure activity, Lookup activity, Web Activity etc.
Scheduled Jobs in Flows and ADF Pipelines, Scheduling Pipelines and monitoring the data movement from source to destinations, Monitoring Produced and Consumed Data Sets of ADF.
Designed Azure ADF pipeline to move data from 6 different sources to Azure Data Lake Gen2 and then to Azure Data Warehouse.
Implemented activities Copy activity, Execute Pipeline, Get Meta data, If Condition, Lookup, Set Variable, Filter, For Each pipeline Activities for On-cloud ETL processing.
Primarily involved in Data Migration using SQL, SQL Azure, Azure Data Lake, and Azure Data Factory.
Professional in creating data warehouse, design-related extraction, loading data functions, testing designs, data modelling, and ensure the smooth running of applications.
Responsible for extracting the data from OLTP and OLAP using Azure Data factory and Databricks to Data Lake.
Developed pipelines that can extract data from various sources and merge into single source datasets in Data Lake using Databricks.
Creating the linked service for source and target connectivity Based on the requirement.
Once it’s created pipelines and datasets will be triggered based on LOAD (HISTORY/DELTA) operations.
Created Mount point for Datalake and extracted Different formats of data like CSV and Parquet.
Created data frames and transformed DF’s using Pyspark.
Implemented SCD1 in delta lake to handle incremental load and write data back to the Azure SQL Server.
Always actively participate in four ceremonies: Sprint planning meeting, Daily Scrum, Sprint review meeting, and Sprint retrospective meeting.