Divya Sri Vikrayala

Developed and optimized large-scale data processing pipelines using PySpark on Databricks, ensuring efficient data transformation and analysis.
Implemented real-time data streaming solutions in Scala, enhancing data ingestion and processing capabilities for high-velocity data sources.
Utilized Python for data manipulation and ETL processes, improving data quality and reducing processing time by 30%.
Leveraged AWS services, including S3, EC2, and Lambda, to build scalable and cost-effective data storage and processing solutions.
Designed and maintained SQL queries and stored procedures to support data analysis and reporting, ensuring data accuracy and consistency.
Deployed Amazon Athena for interactive querying of large datasets stored in S3, reducing query times and enhancing data accessibility.
Designed and implemented data models in SQL to support business intelligence and reporting needs, enhancing data-driven decision-making.
Designed and implemented data pipelines in Snowflake using SnowSQL and SnowPipe to ingest large volumes of financial data from diverse sources, enhancing the data processing speed by 30%.
Configured and managed AWS infrastructure, including IAM roles, VPCs, and security groups, to ensure secure and scalable data environments.
Utilized AWS Glue for data cataloging and ETL tasks, improving data discoverability and processing efficiency.
Developed and optimized complex SQL queries, stored procedures, and views in Snowflake to support data analytics teams, reducing query execution times by 40% through partitioning and clustering techniques.
Conducted performance tuning and optimization of PySpark jobs, significantly reducing execution times and resource utilization.
Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume
Expertise in Creating, Debugging, Scheduling and Monitoring jobs using Airflow for ETL batch processing to load into Snowflake for analytical processes.
Worked in building ETL pipeline for data ingestion, data transformation, data validation on cloud service AWS, working along with data steward under data compliance.
Worked on scheduling all jobs using Airflow scripts using python added different tasks to DAG, LAMBDA.
Used Pyspark for extract, filtering and transforming the Data in data pipelines.
Worked on scheduling all jobs using Airflow scripts using python. Adding different tasks to DAG’s and dependencies between the tasks.
Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Implemented end-to-end data pipelines on Databricks, enabling efficient data integration and processing for Capital Group’s analytics platform.
Used Data Build Tool for transformations in ETL process, AWS lambda, AWS SQS
Skilled in monitoring servers using Nagios, Cloud watch and using ELK Stack Elasticsearch Kibana
Utilized Databricks’ Delta Lake for managing large-scale data lakes, ensuring data reliability and optimized performance for Capital Group.
Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark databricks cluster.
Created Unix Shell scripts to automate the data load processes to the target Data Warehouse.
Responsible for implementing monitoring solutions in Ansible, Terraform, Docker, and Jenkins.
Experience in migrating existing databases from on premise to AWS Redshift using various AWS services
Developed the Pysprk code for AWS Glue jobs and for EMR.
Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs in java and Scala for data cleaning and preprocessing
Implemented Spark using Python and Spark SQL for faster testing and processing of data.

Developed and optimized large-scale data processing pipelines using PySpark on Databricks, ensuring efficient data transformation and analysis. Implemented real-time data streaming solutions in Scala, enhancing data ingestion and processing capabilities for high-velocity data sources. Utilized Python for data manipulation and ETL processes, improving data quality and reducing processing time by 30%. Leveraged AWS services, including S3, EC2, and Lambda, to build scalable and cost-effective data storage and processing solutions. Designed and maintained SQL queries and stored procedures to support data analysis and reporting, ensuring data accuracy and consistency. Deployed Amazon Athena for interactive querying of large datasets stored in S3, reducing query times and enhancing data accessibility. Designed and implemented data models in SQL to support business intelligence and reporting needs, enhancing data-driven decision-making. Designed and implemented data pipelines in Snowflake using SnowSQL and SnowPipe to ingest large volumes of financial data from diverse sources, enhancing the data processing speed by 30%. Configured and managed AWS infrastructure, including IAM roles, VPCs, and security groups, to ensure secure and scalable data environments. Utilized AWS Glue for data cataloging and ETL tasks, improving data discoverability and processing efficiency. Developed and optimized complex SQL queries, stored procedures, and views in Snowflake to support data analytics teams, reducing query execution times by 40% through partitioning and clustering techniques. Conducted performance tuning and optimization of PySpark jobs, significantly reducing execution times and resource utilization. Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka, and Flume. Expertise in Creating, Debugging, Scheduling and Monitoring jobs using Airflow for ETL batch processing to load into Snowflake for analytical processes. Worked in building ETL pipeline for data ingestion, data transformation, data validation on cloud service AWS, working along with data steward under data compliance. Worked on scheduling all jobs using Airflow scripts using python added different tasks to DAG, LAMBDA. Used Pyspark for extract, filtering and transforming the Data in data pipelines. Worked on scheduling all jobs using Airflow scripts using python. Adding different tasks to DAG's and dependencies between the tasks. Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns. Implemented end-to-end data pipelines on Databricks, enabling efficient data integration and processing for Capital Group’s analytics platform. Used Data Build Tool for transformations in ETL process, AWS lambda, AWS SQS. Skilled in monitoring servers using Nagios, Cloud watch and using ELK Stack Elasticsearch Kibana. Utilized Databricks’ Delta Lake for managing large-scale data lakes, ensuring data reliability and optimized performance for Capital Group. Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark Databricks cluster. Created Unix Shell scripts to automate the data load processes to the target Data Warehouse. Responsible for implementing monitoring solutions in Ansible, Terraform, Docker, and Jenkins. Experience in migrating existing databases from on-premise to AWS Redshift using various AWS services. Developed the Pyspark code for AWS Glue jobs and for EMR. Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs in Java and Scala for data cleaning and preprocessing. Implemented Spark using Python and Spark SQL for faster testing and processing of data.

Configured AWS Step Functions and Apache Airflow to optimize SQL-based data ingestion from diverse sources.
Performed data cleansing, transformation, and enrichment using PySpark and SQL, ensuring data consistency across multiple data sources.
Optimized data warehouse performance by implementing Snowflake's multi-cluster warehouse setup, dynamically scaling compute resources based on workload demands.
Built and maintained a data lake using PySpark, AWS S3, and Glue, enabling efficient and scalable data storage and access for various analytics applications.
Integrated data from multiple payment processing systems into Snowflake, using secure data sharing to provide real-time access for analytics teams.
Developed and deployed machine learning models using PySpark MLlib, improving customer segmentation, fraud detection, and churn prediction.
Integrated Databricks with AMEX’s data ecosystem, enabling real-time data processing and advanced analytics for better customer insights and decision-making.
Utilized PySpark streaming to process real-time data from Kafka, generating timely and accurate insights and alerts.
Developed Spark Programs using Scala and Java API's and performed transformations and actions on RDD's.
Involved in converting Hive/SQL queries into Spark transformations using Spark RDD, Scala and Python.
Created and automated data pipelines using PySpark, Airflow, and AWS Lambda, achieving high performance and reliability.
Migrated legacy on-prem data warehouse to Snowflake, improving query speeds by 73% and enabling rapid scaling on demand.
Optimized complex queries and table structures in Snowflake reducing average runtime by 65%, enabling faster analytics.
Established secure data sharing with external partners leveraging Snowflake data exchange for seamless collaboration.
Wrote and optimized Hive SQL queries on HDFS datasets up to 50TB on Unix platform. Used different file formats like Parquet, Avro, Json for querying.
Built data pipelines with Apache Airflow on Unix for scheduling and monitoring ETL jobs. Configured dependencies using directed acyclic graphs.
Developing UDFs in java for hive and pig and worked on reading multiple data formats on HDFS using Scala.
Built efficient data ingestion pipelines in AWS Glue and AWS DMS, enhancing data with Alteryx.
Utilized Amazon QuickSight, Tableau, SQL, and Python for data analysis, visualization, and reporting, working with MySQL and Amazon RDS.
Calculated stock metrics with PySpark and Pandas, enhancing data analytics skills.
Orchestrated PySpark ETL pipelines on AWS Databricks through AWS Step Functions.
Built real-time streaming apps with PySpark, Apache Flink, Kafka, and Hive on AWS EMR clusters.

Configured AWS Step Functions and Apache Airflow to optimize SQL-based data ingestion from diverse sources. Performed data cleansing, transformation, and enrichment using PySpark and SQL, ensuring data consistency across multiple data sources. Optimized data warehouse performance by implementing Snowflake's multi-cluster warehouse setup, dynamically scaling compute resources based on workload demands. Built and maintained a data lake using PySpark, AWS S3, and Glue, enabling efficient and scalable data storage and access for various analytics applications. Integrated data from multiple payment processing systems into Snowflake, using secure data sharing to provide real-time access for analytics teams. Developed and deployed machine learning models using PySpark MLlib, improving customer segmentation, fraud detection, and churn prediction. Integrated Databricks with AMEX’s data ecosystem, enabling real-time data processing and advanced analytics for better customer insights and decision-making. Utilized PySpark streaming to process real-time data from Kafka, generating timely and accurate insights and alerts. Developed Spark Programs using Scala and Java API's and performed transformations and actions on RDD's. Involved in converting Hive/SQL queries into Spark transformations using Spark RDD, Scala and Python. Created and automated data pipelines using PySpark, Airflow, and AWS Lambda, achieving high performance and reliability. Migrated legacy on-prem data warehouse to Snowflake, improving query speeds by 73% and enabling rapid scaling on demand. Optimized complex queries and table structures in Snowflake reducing average runtime by 65%, enabling faster analytics. Established secure data sharing with external partners leveraging Snowflake data exchange for seamless collaboration. Wrote and optimized Hive SQL queries on HDFS datasets up to 50TB on Unix platform. Used different file formats like Parquet, Avro, Json for querying. Built data pipelines with Apache Airflow on Unix for scheduling and monitoring ETL jobs. Configured dependencies using directed acyclic graphs. Developing UDFs in Java for Hive and Pig and worked on reading multiple data formats on HDFS using Scala. Built efficient data ingestion pipelines in AWS Glue and AWS DMS, enhancing data with Alteryx. Utilized Amazon QuickSight, Tableau, SQL, and Python for data analysis, visualization, and reporting, working with MySQL and Amazon RDS. Calculated stock metrics with PySpark and Pandas, enhancing data analytics skills. Orchestrated PySpark ETL pipelines on AWS Databricks through AWS Step Functions. Built real-time streaming apps with PySpark, Apache Flink, Kafka, and Hive on AWS EMR clusters.