Suma

 Developed and implemented real-time Kafka data streams using PySpark for seamless event handling from Kafka topics and used AWS Lambda functions to trigger data processing tasks in response to new data events.
 Leverage PySpark's DataFrame for data manipulation, filtering, aggregation, and enrichment.
 Worked on different file formats like Text, Avro, Parquet, JSON, and Flat files using Spark.
 Developed ETL jobs on Spark clusters on EMR for large scale batch processing.
 Designed and implemented end-to-end ETL pipelines using AWS Glue, automating the ETL process from various sources to target destinations. Automated ETL pipelines with AWS Glue crawlers, catalog and jobs.
 Designed and managed AWS Redshift clusters to store and process structured data for analytical purposes.
 Loaded data from S3 into Redshift and Snowflake for structured analytics using clustering and partitioning.
 Created dashboards, reports, and alerts in Snowflake to monitor data SLAs and pipeline health for stakeholders.
 Performed ad-hoc SQL queries on S3 data using Amazon Athena.
 Implemented data transformations using both PySpark and AWS Glue ETL jobs to enrich raw data with contextual information from MongoDB and Cassandra.
 Developed ETL pipelines in Python, Scala, and SQL within Databricks notebooks for data transformation.
 Orchestrated workflows with Airflow for task scheduling and monitoring.
 Designed and managed Airflow DAGs to automate data processing, quality checks, and data movement tasks.
 Leveraged AWS IAM roles and security groups for access controls and auditing across all services.
 Automated deployments using CodePipeline CI/CD and Git version control.
 Logged bugs and issues discovered during testing and debugging of data pipelines in JIRA for tracking.
 Developed and executed comprehensive test scenarios for Python scripts using Scala in a local development environment, ensuring code quality and functionality before deployment to AWS EMR.

Developed and implemented real-time Kafka data streams using PySpark for seamless event handling from Kafka topics and used AWS Lambda functions to trigger data processing tasks in response to new data events.
Leverage PySpark's DataFrame for data manipulation, filtering, aggregation, and enrichment.
Worked on different file formats like Text, Avro, Parquet, JSON, and Flat files using Spark.
Developed ETL jobs on Spark clusters on EMR for large scale batch processing.
Designed and implemented end-to-end ETL pipelines using AWS Glue, automating the ETL process from various sources to target destinations.
Automated ETL pipelines with AWS Glue crawlers, catalog and jobs.
Designed and managed AWS Redshift clusters to store and process structured data for analytical purposes.
Loaded data from S3 into Redshift and Snowflake for structured analytics using clustering and partitioning.
Created dashboards, reports, and alerts in Snowflake to monitor data SLAs and pipeline health for stakeholders.
Performed ad-hoc SQL queries on S3 data using Amazon Athena.
Implemented data transformations using both PySpark and AWS Glue ETL jobs to enrich raw data with contextual information from MongoDB and Cassandra.
Developed ETL pipelines in Python, Scala, and SQL within Databricks notebooks for data transformation.
Orchestrated workflows with Airflow for task scheduling and monitoring.
Designed and managed Airflow DAGs to automate data processing, quality checks, and data movement tasks.
Leveraged AWS IAM roles and security groups for access controls and auditing across all services.
Automated deployments using CodePipeline CI/CD and Git version control.
Logged bugs and issues discovered during testing and debugging of data pipelines in JIRA for tracking.
Developed and executed comprehensive test scenarios for Python scripts using Scala in a local development environment, ensuring code quality and functionality before deployment to AWS EMR.

Successfully managed end-to-end migration from Informatica to Microsoft Azure, overseeing planning, execution, and post-migration validation while assessing dependencies and migration prerequisites in existing workflows and ETL processes.
Designed data migration pipelines via Azure Data Factory to seamlessly transfer data from Informatica to Azure data stores, while also ensuring data compatibility between Informatica and Azure models through data mapping and transformation activities.
Designed and implemented data pipelines to extract data from diverse sources like databases, object storage, REST APIs, IoT hubs into Azure Databricks.
Designed and implemented data ingestion pipelines using Apache Spark, Scala, and Python on Azure Databricks, facilitating seamless data movement from various sources to ADLS Gen2.
Used Spark SQL, Dataframes APIs for complex batch data transformation and enrichment.
Leveraged Scala and python functional programming capabilities to implement complex data transformations and data cleansing logic for improved data quality.
Utilized Azure Data Factory to create robust data pipelines, orchestrating data movement and transformation across various Azure services and external sources using Scala.
Built reusable SSIS packages for ETL transformations and orchestrated execution using Azure Data Factory pipelines.
Implemented Azure SQL Data Warehouse (Synapse Analytics) as the core data warehousing solution, enhancing data storage optimization, and enabling robust high-performance analytical querying through strategies like distribution keys and partition schemes for optimized tables, views, and indexes.
Utilized Azure Cosmos DB for storing and querying semi-structured and NoSQL data, enriching the analytics landscape with diverse data types.
Developed T-SQL queries, stored procedures, user defined functions to prepare and transform data for analysis in Azure Synapse.
Utilized Azure Active Directory and Azure Key Vault to implement robust security measures for data protection and compliance.
Developed data processing workflows using Azure Functions, enabling serverless data processing triggered by events and facilitating real-time data analytics.
Developed interactive data visualizations and reports using Azure Power BI, providing actionable insights to stakeholders.
Highly skilled in writing complex SQL queries and stored procedures and functions in DB2, oracle, facilitating efficient data extraction, transformation, and loading (ETL) processes.
Managed SharePoint as a central platform for data storage, collaboration, and document management.
Oversaw data ingestion and migration processes, transferring data from various sources into SharePoint while ensuring data integrity.
Worked on optimizing DB2, oracle database performance through query tuning, indexing, and database parameter adjustments.
Designed Java based distributed applications for real-time data streaming from Kafka and Spark.
Created user stories, epics, tasks, and subtasks to manage agile development processes using JIRA.
Collaborated with cross-functional teams, including data scientists and business analysts, to gather data requirements and deliver solutions aligned with business objectives.
Conducted performance tuning and optimization of Azure data solutions, enhancing data processing efficiency and reducing latency.
Provided documentation and training to team members on best practices for Azure data engineering, ensuring knowledge sharing.

Conducted data analysis on large datasets using SQL queries, Excel, and data visualization tools to extract valuable insights and support decision-making processes.
Collaborated with cross-functional teams to gather business requirements and translate them into data analysis tasks and actionable recommendations.
Developed and maintained automated data reports and dashboards, improving data accessibility, and reducing report generation time by 50%.
Performed data cleaning, data validation, and data enrichment to ensure data accuracy and reliability for analytical purposes.
Assisted in the development of machine learning models for predictive analytics and forecasting.
Conducted A/B testing and analyzed experiment results to assess the impact of business initiatives.
Presented data findings and insights to stakeholders through clear and concise data visualizations and presentations.
Created and maintained data reports and dashboards to provide key performance indicators (KPIs) and performance metrics to stakeholders.
Gained proficiency in data analysis tools and technologies, including SQL, Excel, Tableau, Power BI, and other relevant tools.
Develop reports in Snowflake with SQL queries and generate Tableau dashboards with Snowflake connection.
Manage the health of the database and identify any data quality or logic issues.
Build & maintain batch/streaming pipelines using SnowPipe.