About Me
Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise datab…
Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise databases to Azure Data Lake store using Azure Data factory. Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns. Good understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, Driver Node, Worker Node, Stages, Executors and Tasks. Experience in Database Design and development with Business Intelligence using SQL Server 2014/2016, Integration Services (SSIS), DTS Packages, SQL Server Analysis Services (SSAS), DAX, OLAP Cubes, Star Schema and Snowflake Schema. Excellent communication skills with excellent work ethics and a proactive team player with a positive attitude. Experience in Database Design and development with Business Intelligence using SQL Server 2014/2016, Integration Services (SSIS), DTS Packages, SQL Server Analysis Services (SSAS), DAX, OLAP Cubes, Star Schema and Snowflake Schema. Azure Data lake, Data factory Experience working on Azure/AWS cloud, implementing Continuous Testing and Snowflake. Involvement in design, implementation and modifying the Python code. Experience working with Synapse. Strong skills in visualization tools Power BI, Confidential Excel - formulas, Pivot Tables, Charts and DAX Commands. Expertise in various phases of project life cycles (Design, Analysis, Implementation, and testing).
Experience
Sr. Azure data Engineer
Hands on experience troubleshooting, configuring Azure services like Azure Data factory, Azure Data Lake.
Gathering the requirements from the clients about the existing applications to apply the security measures.
Understand the latest features like (Azure DevOps, OMS, NSG Rules, etc..) introduced by Microsoft Azure and utilized for existing business applications.
Creating, validating, and reviewing solutions and effort estimate of converting existing workloads from classic to ARM based Azure Cloud Environment
Developed automation system using PowerShell scripts and SON templates to remediate the Azure services.
Worked on GIT to maintain source code in Git and GitHub repositories.
Python on Django Web Framework and created User Interface (UI) using JavaScript, bootstrap, Cassar with MySQL and HTML5/CSS.
Designed and automated AZURE Infrastructure as aService (laaS) and Platform as a Service (PaaS), Saas capabilities which includes virtual machine. container services, virtual network, and cloud services.
Developed and maintained data pipelines using Snowflake's native tools and integration methods, such as Snowpipe, SnowSQL, and Snowflake tasks, ensuring timely data loading and availability.
Created and improved ETL pipelines to extract, transform, and load data from diverse sources utilizing Snowflake's built-in features and functions, such as SnowSQL and stored procedures
Working experience with Azure Resource Manager (ARM) to deploy, update, and delete multiple Azure resources, as well as migrating on-premises resources to Azure with Azure site recovery (ASR), and Azurebackups.
Used Azure DevOps services such as Azure Repos, Azure Boards, and Azure Test Plans to plan work and collaborate on code development, built and deployed application.
Developed, maintained, and provided the team with Various Azure DevOps-related tools like deployment tools, staged virtual environments, and provisioning scripts.
Used Shared Image Gallery to store the created images and built Azure pipelines in Azure DevOps to implement all these services in Azure.
Structured cluster AutoScaler for Azure Kubernetes Service (AKS) by using Terraform and worked with scheduling, deploying, and managing pods and replicas in AKS.
Terraform was used along with Packer to create custom machine images, and Ansible was used to install the software dependencies once the infrastructure was provisioned.
Developed and maintained Continuous Integration (CI) using tools in Azure DevOps (VSTS) spanning multiple environments, enabling teams to safely deploy code in Azure Kubernetes Services (AKS) using YAML scripts.
Managing the Azure Kubernetes Services (AKS).
Experience implementing Azure services such as Azure.
Active Directory (AD), Azure storage, Azure cloud services, IIS, Azure Resource Manager (ARM), Azure Blob Storage, Azure VM, SQL Database, Azure Functions, Azure Service Fabric, Azure Service Bus.
Working knowledge in deploying CI/CD system using Azure DevOps on Kubernetes container environment. and for the runtime environment of CI/CD system to build, test and Deployment we have utilized Kubernetes and Docker.
Proficient in Python, Shell Scripting, SQL, and building utilities. like open make, ANT and Cruise Control.
Developed Python and shell scripts for automation of the build and release process.
Involved in design, implementation and modifying the Python code.
Improve speed, efficiency, and scalability of the continuous integration environment, automating wherever possible using Python, Ruby, Shell and PowerShell Scripts.
Used Azure Data Factory V2 to Pause and Resume Azure SQL Data Warehouse.
Implemented Copy activity, Custom Azure Data Factory Pipeline Activities.
Developed the pipelines in Azure Data factory for various scenarios to meet business requirement using blob storages and ingesting the data into azure synapse analytics.
Processed the fixed width files using derived column activity and loaded them into ADLS/ azure synapse analytics using copy activity.
Designed SSIS Packages to transfer data from flat files, Excel SQL Server using Business Intelligence Development Studio.
Created SSIS packages to implement error/failure handling with event handlers, row redirects, and loggings.
Designed complex data intensive reports in Power BI utilizing various graph features such as gauge, funnel.
Created Azure key vault for storing connection string details, certificates and used the key vaults in Azure Data factory while creating linked services.
Developed logic apps to trigger email notification whenever the pipeline failed using web activity.
Working Experienced with Azure logic apps to schedule, automate, and orchestrate tasks.
Worked on POC’s with Apache Spark using Scala to implement spark in project.
Involved in performance tuning of pipelines in Azure Data factory.
Created data frames for reading and writing the various file formats in notebooks using data bricks.
Sr. Azure data Engineer
• Hands on experience troubleshooting, configuring Azure services like Azure Data factory, Azure Data Lake.
• Gathering the requirements from the clients about the existing applications to apply the security measures.
• Understand the latest features like (Azure DevOps, OMS, NSG Rules, etc..) introduced by Microsoft Azure and utilized for existing business applications.
• Creating, validating, and reviewing solutions and effort estimate of converting existing workloads from classic to ARM based Azure Cloud Environment
• Developed automation system using PowerShell scripts and SON templates to remediate the Azure services.
• Worked on GIT to maintain source code in Git and GitHub repositories.
• Python on Django Web Framework and created User Interface (UI) using JavaScript, bootstrap, Cassar with MySQL and HTML5/CSS.
• Designed and automated AZURE Infrastructure as aService (laaS) and Platform as a Service (PaaS), Saas capabilities which includes virtual machine. container services, virtual network, and cloud services.
• Developed and maintained data pipelines using Snowflake's native tools and integration methods, such as Snowpipe, SnowSQL, and Snowflake tasks, ensuring timely data loading and availability.
• Created and improved ETL pipelines to extract, transform, and load data from diverse sources utilizing Snowflake's built-in features and functions, such as SnowSQL and stored procedures
• Working experience with Azure Resource Manager (ARM) to deploy, update, and delete multiple Azure resources, as well as migrating on-premises resources to Azure with Azure site recovery (ASR), and Azurebackups.
• Used Azure DevOps services such as Azure Repos, Azure Boards, and Azure Test Plans to plan work and collaborate on code development, built and deployed application.
• Developed, maintained, and provided the team with Various Azure DevOps-related tools like deployment tools, staged virtual environments, and provisioning scripts.
• Used Shared Image Gallery to store the created images and built Azure pipelines in Azure DevOps to implement all these services in Azure.
• Structured cluster AutoScaler for Azure Kubernetes Service (AKS) by using Terraform and worked with scheduling, deploying, and managing pods and replicas in AKS.
• Terraform was used along with Packer to create custom machine images, and Ansible was used to install the software dependencies once the infrastructure was provisioned.
• Developed and maintained Continuous Integration (CI) using too
Azure Data Engineer
Analyze, design, and build Modern data solutions using Azure PaaS service to support visualization of data.
Understand current Production state of application and determine the impact of new implementation on existing business processes.
Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics.
Used Liquibase for database migration and JDBC for database connections.
Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.
Migration of on-premises data (SQL Server / MongoDB) to Azure Data Lake Store (ADLS) using Azure Data Factory (ADF V1/V2).
Exposures with Azure Active Directory compatibility.
Extensive experience in deployment,migration, patching and troubleshooting of windows 2008 and 2012 R2 Domain Controllers in Active Directory.
Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
To meet specific business requirements wrote UDF’s in Scala and Pyspark.
Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Sql Activity.
Hands-on experience on developing SQL Scripts for automation purpose.
Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).
Generated complex stored procedures and functions for better performance and flexibility.
Designed logical and physical database structure to facilitate analysis of data from both operational and customer perspectives.
Experience with version control tools such as GIT andBitbucket.
Comprehensive knowledge of source controller concepts including branches, tags, and mergers.
Developed build and deploy scripts using MAVEN and activated them using Jenkins to migrate from one environment to another.
Worked with Azure Monitoring tools such as Azure Log Analytics, Azure Network Watcher, and Azure Service Health to diagnose and minimize service degradation.
Experience in using monitoring tools like Azuremonitoring and Dynatrace to setup the desired alerts to avoid disruption.
Built and maintained SQL scripts, indexes, and complex queries for data analysis and extraction.
Developed the pipelines in Azure Data factory for various scenarios to meet business requirement using blob storages and ingesting the data into azure synapse analytics.
Developed the pipelines in Azure Data factory for various scenarios to meet business requirement using blob storages and ingesting the data into azure synapse analytics.
Developed automation system using PowerShell scripts and SON templates to remediate the Azure services.
Worked on GIT to maintain source code in Git and GitHub repositories.
SQL Developer
Requirements gathering from Business users to perform technical design, development and testing.
Worked extensively in writing and debugging complex stored procedures, views, triggers and user-defined functions
Worked on designing databases according to industry standards and make tuning recommendations.
Designed the Data warehouse and done the mappings from Source to the Target tables.
Created views and Stored Procedures for generating report.
Responsible for creating the reports based onthe requirements using SSRS.
Worked on several complex drilldown, parameterized reports and on reports that call sub reports using SSRS.
Performance Tuning of Stored Procedures and SQL queries using SQL Profiler and Index Tuning Wizard in SSIS.
Created SSIS Packages using Pivot Transformation, Fuzzy Lookup, Derived Columns, Condition Split, Term extraction, Aggregate, Execute SQL Task, Data Flow Task, and Execute Package Task etc.
Created SQL Server jobs and scheduled them to load data periodically using SQL server Agent.
Identified the dimension, fact tables and designed the Data-Mart using Star schema.
Created and Configured Data Source & Data Source Views, Dimensions, Cubes, Measures, Partitions, KPI’s&MDX Queries using SQL Server 2008R2 Analysis Services.
Transferred data from various OLTP data sources, such as Oracle, MS Access, MS Excel, Flat Files, CSV filesinto SQL Server2008R2.
Generated multi parameterized reports in MS SSRS 2008R2 allowing users the ability to make selections before executing reports; thus, making them userfriendly.
Report Design and Coding for Standard Tabular type reports, including Drill down and Drill through functionality and Graphical presentations such as Charts and Dashboard type metrics using SSRS
Created Ad Hoc and parameterized various Reports using SSAS and SSRS.
Assisted in handling escalating billing issues and propagating bug fixes in production environment.
Scheduled reports for daily, weekly, monthly reports for executives, Business analyst and customer representatives for various categories based on businessneeds.