unishayella

unishayella

Azure Data Engineer
United States of America

About Me

Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise datab…

Experience

Sr. Azure data Engineer

USAA, San Antonio TX
Mar 2021 - Present · 5 years 4 months

Hands on experience troubleshooting, configuring Azure services like Azure Data factory, Azure Data Lake.
Gathering the requirements from the clients about the existing applications to apply the security measures.
Understand the latest features like (Azure DevOps, OMS, NSG Rules, etc..) introduced by Microsoft Azure and utilized for existing business applications.
Creating, validating, and reviewing solutions and effort estimate of converting existing workloads from classic to ARM based Azure Cloud Environment
Developed automation system using PowerShell scripts and SON templates to remediate the Azure services.
Worked on GIT to maintain source code in Git and GitHub repositories.
Python on Django Web Framework and created User Interface (UI) using JavaScript, bootstrap, Cassar with MySQL and HTML5/CSS.
Designed and automated AZURE Infrastructure as aService (laaS) and Platform as a Service (PaaS), Saas capabilities which includes virtual machine. container services, virtual network, and cloud services.
Developed and maintained data pipelines using Snowflake's native tools and integration methods, such as Snowpipe, SnowSQL, and Snowflake tasks, ensuring timely data loading and availability.
Created and improved ETL pipelines to extract, transform, and load data from diverse sources utilizing Snowflake's built-in features and functions, such as SnowSQL and stored procedures
Working experience with Azure Resource Manager (ARM) to deploy, update, and delete multiple Azure resources, as well as migrating on-premises resources to Azure with Azure site recovery (ASR), and Azurebackups.
Used Azure DevOps services such as Azure Repos, Azure Boards, and Azure Test Plans to plan work and collaborate on code development, built and deployed application.
Developed, maintained, and provided the team with Various Azure DevOps-related tools like deployment tools, staged virtual environments, and provisioning scripts.
Used Shared Image Gallery to store the created images and built Azure pipelines in Azure DevOps to implement all these services in Azure.
Structured cluster AutoScaler for Azure Kubernetes Service (AKS) by using Terraform and worked with scheduling, deploying, and managing pods and replicas in AKS.
Terraform was used along with Packer to create custom machine images, and Ansible was used to install the software dependencies once the infrastructure was provisioned.
Developed and maintained Continuous Integration (CI) using tools in Azure DevOps (VSTS) spanning multiple environments, enabling teams to safely deploy code in Azure Kubernetes Services (AKS) using YAML scripts.
Managing the Azure Kubernetes Services (AKS).
Experience implementing Azure services such as Azure.
Active Directory (AD), Azure storage, Azure cloud services, IIS, Azure Resource Manager (ARM), Azure Blob Storage, Azure VM, SQL Database, Azure Functions, Azure Service Fabric, Azure Service Bus.
Working knowledge in deploying CI/CD system using Azure DevOps on Kubernetes container environment. and for the runtime environment of CI/CD system to build, test and Deployment we have utilized Kubernetes and Docker.
Proficient in Python, Shell Scripting, SQL, and building utilities. like open make, ANT and Cruise Control.
Developed Python and shell scripts for automation of the build and release process.
Involved in design, implementation and modifying the Python code.
Improve speed, efficiency, and scalability of the continuous integration environment, automating wherever possible using Python, Ruby, Shell and PowerShell Scripts.
Used Azure Data Factory V2 to Pause and Resume Azure SQL Data Warehouse.
Implemented Copy activity, Custom Azure Data Factory Pipeline Activities.
Developed the pipelines in Azure Data factory for various scenarios to meet business requirement using blob storages and ingesting the data into azure synapse analytics.
Processed the fixed width files using derived column activity and loaded them into ADLS/ azure synapse analytics using copy activity.
Designed SSIS Packages to transfer data from flat files, Excel SQL Server using Business Intelligence Development Studio.
Created SSIS packages to implement error/failure handling with event handlers, row redirects, and loggings.
Designed complex data intensive reports in Power BI utilizing various graph features such as gauge, funnel.
Created Azure key vault for storing connection string details, certificates and used the key vaults in Azure Data factory while creating linked services.
Developed logic apps to trigger email notification whenever the pipeline failed using web activity.
Working Experienced with Azure logic apps to schedule, automate, and orchestrate tasks.
Worked on POC’s with Apache Spark using Scala to implement spark in project.
Involved in performance tuning of pipelines in Azure Data factory.
Created data frames for reading and writing the various file formats in notebooks using data bricks.

Sr. Azure data Engineer

USAA
Mar 2021 - Present · 5 years 4 months

• Hands on experience troubleshooting, configuring Azure services like Azure Data factory, Azure Data Lake.
• Gathering the requirements from the clients about the existing applications to apply the security measures.
• Understand the latest features like (Azure DevOps, OMS, NSG Rules, etc..) introduced by Microsoft Azure and utilized for existing business applications.
• Creating, validating, and reviewing solutions and effort estimate of converting existing workloads from classic to ARM based Azure Cloud Environment
• Developed automation system using PowerShell scripts and SON templates to remediate the Azure services.
• Worked on GIT to maintain source code in Git and GitHub repositories.
• Python on Django Web Framework and created User Interface (UI) using JavaScript, bootstrap, Cassar with MySQL and HTML5/CSS.
• Designed and automated AZURE Infrastructure as aService (laaS) and Platform as a Service (PaaS), Saas capabilities which includes virtual machine. container services, virtual network, and cloud services.
• Developed and maintained data pipelines using Snowflake's native tools and integration methods, such as Snowpipe, SnowSQL, and Snowflake tasks, ensuring timely data loading and availability.
• Created and improved ETL pipelines to extract, transform, and load data from diverse sources utilizing Snowflake's built-in features and functions, such as SnowSQL and stored procedures
• Working experience with Azure Resource Manager (ARM) to deploy, update, and delete multiple Azure resources, as well as migrating on-premises resources to Azure with Azure site recovery (ASR), and Azurebackups.
• Used Azure DevOps services such as Azure Repos, Azure Boards, and Azure Test Plans to plan work and collaborate on code development, built and deployed application.
• Developed, maintained, and provided the team with Various Azure DevOps-related tools like deployment tools, staged virtual environments, and provisioning scripts.
• Used Shared Image Gallery to store the created images and built Azure pipelines in Azure DevOps to implement all these services in Azure.
• Structured cluster AutoScaler for Azure Kubernetes Service (AKS) by using Terraform and worked with scheduling, deploying, and managing pods and replicas in AKS.
• Terraform was used along with Packer to create custom machine images, and Ansible was used to install the software dependencies once the infrastructure was provisioned.
• Developed and maintained Continuous Integration (CI) using too

Azure Data Engineer

National Grid, Waltham MA
Sep 2019 - Feb 2021 · 1 year 5 months

Analyze, design, and build Modern data solutions using Azure PaaS service to support visualization of data.
Understand current Production state of application and determine the impact of new implementation on existing business processes.
Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics.
Used Liquibase for database migration and JDBC for database connections.
Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.
Migration of on-premises data (SQL Server / MongoDB) to Azure Data Lake Store (ADLS) using Azure Data Factory (ADF V1/V2).
Exposures with Azure Active Directory compatibility.
Extensive experience in deployment,migration, patching and troubleshooting of windows 2008 and 2012 R2 Domain Controllers in Active Directory.
Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
To meet specific business requirements wrote UDF’s in Scala and Pyspark.
Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Sql Activity.
Hands-on experience on developing SQL Scripts for automation purpose.
Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).
Generated complex stored procedures and functions for better performance and flexibility.
Designed logical and physical database structure to facilitate analysis of data from both operational and customer perspectives.
Experience with version control tools such as GIT andBitbucket.
Comprehensive knowledge of source controller concepts including branches, tags, and mergers.
Developed build and deploy scripts using MAVEN and activated them using Jenkins to migrate from one environment to another.
Worked with Azure Monitoring tools such as Azure Log Analytics, Azure Network Watcher, and Azure Service Health to diagnose and minimize service degradation.
Experience in using monitoring tools like Azuremonitoring and Dynatrace to setup the desired alerts to avoid disruption.
Built and maintained SQL scripts, indexes, and complex queries for data analysis and extraction.
Developed the pipelines in Azure Data factory for various scenarios to meet business requirement using blob storages and ingesting the data into azure synapse analytics.
Developed the pipelines in Azure Data factory for various scenarios to meet business requirement using blob storages and ingesting the data into azure synapse analytics.
Developed automation system using PowerShell scripts and SON templates to remediate the Azure services.
Worked on GIT to maintain source code in Git and GitHub repositories.

SQL Developer

Walmart, Bentonville, AR
May 2018 - Sep 2019 · 1 year 4 months

Requirements gathering from Business users to perform technical design, development and testing.
Worked extensively in writing and debugging complex stored procedures, views, triggers and user-defined functions
Worked on designing databases according to industry standards and make tuning recommendations.
Designed the Data warehouse and done the mappings from Source to the Target tables.
Created views and Stored Procedures for generating report.
Responsible for creating the reports based onthe requirements using SSRS.
Worked on several complex drilldown, parameterized reports and on reports that call sub reports using SSRS.
Performance Tuning of Stored Procedures and SQL queries using SQL Profiler and Index Tuning Wizard in SSIS.
Created SSIS Packages using Pivot Transformation, Fuzzy Lookup, Derived Columns, Condition Split, Term extraction, Aggregate, Execute SQL Task, Data Flow Task, and Execute Package Task etc.
Created SQL Server jobs and scheduled them to load data periodically using SQL server Agent.
Identified the dimension, fact tables and designed the Data-Mart using Star schema.
Created and Configured Data Source & Data Source Views, Dimensions, Cubes, Measures, Partitions, KPI’s&MDX Queries using SQL Server 2008R2 Analysis Services.
Transferred data from various OLTP data sources, such as Oracle, MS Access, MS Excel, Flat Files, CSV filesinto SQL Server2008R2.
Generated multi parameterized reports in MS SSRS 2008R2 allowing users the ability to make selections before executing reports; thus, making them userfriendly.
Report Design and Coding for Standard Tabular type reports, including Drill down and Drill through functionality and Graphical presentations such as Charts and Dashboard type metrics using SSRS
Created Ad Hoc and parameterized various Reports using SSAS and SSRS.
Assisted in handling escalating billing issues and propagating bug fixes in production environment.
Scheduled reports for daily, weekly, monthly reports for executives, Business analyst and customer representatives for various categories based on businessneeds.

Skills

MySQL Python SQL Database Azure Azure Data Factory Azure Data Lake Azure Data Lake Analytics Azure SQL Database Databricks Azure SQL Data Warehouse Spark SQL Spark Core Spark DataFrames Spark Streaming SQL Server 2014/2016 SSIS DTS Packages SSAS DAX OLAP Cubes Star Schema Snowflake Schema Azure DevOps OMS NSG Rules Azure Resource Manager PowerShell SON Templates Git GitHub Django JavaScript Bootstrap HTML5 CSS Infrastructure as Code IaaS PaaS SaaS Snowpipe SnowSQL Snowflake Tasks ETL Azure Site Recovery Azure Backups Azure Repos Azure Boards Azure Test Plans Azure Pipelines Terraform Packer Ansible
Report this Profile?