Shiva

Implemented reporting datamarts for Sales and Marketing teams on AWS redshift.
Handled data schema design development, ETL pipelines in Python/ MySQL Stored Procedures Automation using Jenkins.
Worked on AWS to aggregate clean files in Amazon S3 and on Amazon EC2 Clusters to deploy files into Buckets.
Designed and architected solutions to load multipart files which can't rely on a scheduled run and must be event driven, leveraging AWS SNS.
Conducted data wrangling to clean, transform, and reshape data using Python's Pandas library, ensuring data quality and consistency.
Created ETL pipelines for Oracle Data Warehouse using a combination of Python and Oracle SQL, orchestrating data movement and transformation processes.
Leveraged Oracle Exadata for scalable data processing, benefiting from its high-performance capabilities for handling large datasets efficiently.
Integrated Oracle Database with Kubernetes for comprehensive infrastructure management, ensuring seamless orchestration of database containers across multiple hosts.
Implemented Jenkins CI/CD pipelines to automate the build, test, and deployment processes of Oracle microservices, facilitating rapid and reliable software delivery.
Collaborated with Oracle NoSQL databases like Oracle NoSQL Database to retrieve and load data for real-time processing, utilizing REST APIs for seamless data interaction.
Used AWS EMR clusters for creating Hadoop and spark clusters.
These clusters are used for submitting and executing python applications in production.
Developing data pipeline with Amazon AWS to extract data from weblogs and store it in HDFS.
Migrated the data from AWS S3 to HDFS using Kafka.
Transforming and loading the large sets of structured, semi structured, and unstructured data.
Worked on designing AWS EC2 instance architecture to meet high availability application architecture and security parameters.
Created AWS S3 buckets and managed policies for S3 buckets and Utilized S3 buckets and Glacier for storage and backup.
Worked on Hadoop cluster and data querying tools to store and retrieve data from the stored databases.
Designed and deployed automated ETL workflows using AWS lambda, organized and cleansed the data in S3 buckets using AWS Glue and processed the data using Amazon Redshift.

● Conducted data wrangling to clean, transform, and reshape data using Python's Pandas library, ensuring data quality and consistency.
● Created ETL pipelines for Oracle Data Warehouse using a combination of Python and Oracle SQL, orchestrating data movement and transformation processes.
● Leveraged Oracle Exadata for scalable data processing, benefiting from its high-performance capabilities for handling large datasets efficiently.
● Integrated Oracle Database with Kubernetes for comprehensive infrastructure management, ensuring seamless orchestration of database containers across multiple hosts.
● Implemented Jenkins CI/CD pipelines to automate the build, test, and deployment processes of Oracle microservices, facilitating rapid and reliable software delivery.
● Collaborated with Oracle NoSQL databases like Oracle NoSQL Database to retrieve and load data for real-time processing, utilizing REST APIs for seamless data interaction.
● Used AWS EMR clusters for creating Hadoop and spark clusters. These clusters are used for submitting and executing python applications in production.
● Developing data pipeline with Amazon AWS to extract data from weblogs and store it in HDFS.
● Migrated the data from AWS S3 to HDFS using Kafka.
● Transforming and loading the large sets of structured, semi structured, and unstructured data.
● Worked on designing AWS EC2 instance architecture to meet high availability application architecture and security parameters.
● Created AWS S3 buckets and managed policies for S3 buckets and Utilized S3 buckets and Glacier for storage and backup.
● Worked on Hadoop cluster and data querying tools to store and retrieve data from the stored databases.
● Designed and deployed automated ETL workflows using AWS lambda, organized and cleansed the data in S3 buckets using AWS Glue and processed the data using Amazon Redshift

Developed comprehensive migration strategies and executed data extraction, transformation, and loading (ETL) processes utilizing Informatica PowerCenter and Azure Data Factory and Azure Data Bricks guaranteeing data integrity and consistency throughout the migration journey.
Collaborated closely with cross-functional teams including data architects, business analysts, and stakeholders to analyze existing data structures and design optimal solutions for migration to Azure cloud services.
Implemented data quality checks and validation procedures to ensure accuracy and completeness of migrated data, minimizing potential risks and errors during the transition phase.
Leveraged Informatica's advanced features such as mappings, workflows, and sessions to efficiently extract data from diverse sources, transform it according to business requirements, and load it into Azure data repositories.
Optimized performance and scalability of data migration processes by fine-tuning Informatica workflows, configuring Azure resources, and implementing parallel processing techniques to meet stringent project timelines and performance benchmarks.
Implemented Apache Airflow for authoring, scheduling, and monitoring Data Pipelines.
Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines.
Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
Responsible for writing Hive Queries to analyze the data in Hive warehouse using Hive Query Language (HQL).
Involved in developing Hive DDLs to create, drop and alter tables.
Extracted the data and updated it into HDFS using Sqoop Import from various sources like Oracle, Teradata, SQL server etc.
Worked on Microsoft Azure services like HDInsight Clusters, BLOB, ADLS, Data Factory and Logic Apps and also done POC on Azure Data Bricks.
Worked with various formats of files like delimited text files, click stream log files, Apache log files, Avro files, JSON files, XML Files.
Mastered in using different columnar file formats like RC, ORC and Parquet formats.
Developed custom the Unix/BASH SHELL scripts for the purpose of pre and post validations of the master and slave nodes, before and after the configuration of the name node and datanodes.
Developed job workflows in Oozie for automating the tasks of loading the data into HDFS.
Implemented compact and efficient file storage of big data by using various file formats like Avro, Parquet, JSON and using compression methods like GZip, Snappy on top of the files.
Exploring with Spark, improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame and Pair RDD's.
Used Ansible to configure and manage the infrastructure.
Worked on Docker CE and curl.
Implemented the application using Spring IOC (Inversion of Control), Django Framework and handled the security using Python Spring Security.

Creating AWS Lambda functions using python for deployment management in AWS and designed, investigated, and implemented public facing websites on Amazon Web Services and integrated it with other applications infrastructure.
Performed ETL processes to data ready for creating business analysis visuals which help the leadership team to make the right business decisions.
Designed infrastructure for AWS application and workflow using Terraform and had done implementation and continuous delivery of AWS infrastructure using Terraform.
Worked on visuals as Employee Info, project details, skills, leaves, calendar, time sheet.
Created and shared the dashboards within the organization for updating and editing the report as per business.
Developed and tested environments of different applications by provisioning Kubernetes clusters on AWS using Docker, Ansible, and Terraform.
Worked on connecting Cassandra database to the Amazon EMR File System for storing the database in S3.
Implemented usage of Amazon EMR for processing Big Data across a Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).
Deployed the project on Amazon EMR with S3 connectivity for setting a backup storage.
Well versed in using Elastic Load Balancer for Auto scaling in EC2 servers.
Used Amazon Cloud Watch to monitor AWS services and Amazon Cloud Watch logs to monitor application.
Coordinated with the SCRUM team in delivering agreed user stories on time for every sprint.