Karre Bhavani Sankar

Comprehend the business challenge and formulate data-driven solutions for specific requirements through leading standalone meetings with the customer.
Engage in daily-standup meetings, ad-hoc sessions with both the team and clients, fostering open discussions and providing updates on each project.
Load OBS structured and Unstructured data and Create Spark Dataframe on DLI.
Perform Spark Transformations and actions to preprocess and transform data.
Write UDF for complex analytical operations and customise functions.
Apply Job tuning params and optimization techniques on DLI Spark Jobs to optimise job turnaround time.
Identify and build a recovery mechanism system for failed jobs in order to avoid business impact.
Create a DataLake and in order to use ready made data for Hive Data Warehouse and other teams to query the data.
Load Multiple Elasticsearch Indices data by passing the targeted queries based on the business requirements and use Python API and create pandas dataframe.
Process the pandas dataframe using various filters, methods and derive possible fields to cover all possible scenarios for reports, visualisations.
Create Test Cases and perform various testings to ensure the DataPipeline runs smoothly as expected.
Create Daily and Weekly based Schedule jobs settings for datapipeline jobs using various cloud services.
Execute thorough data quality assessments on diverse sets of indices data in relation to the resultant index data.
Create Kibana visualisations and dashboard and add all possible patterns, trends in the visualisation to satisfy business requirements.
Create and build Docker image and push to SWR and create workload and attach image to activate CCE container.
Create Automated CI/CD pipeline for managing the entire workflow using Jenkins and Gitlab integration.
Use GITLAB version control tool and Docker build tools for maintaining the updated source code in git repo.
Establish a comprehensive wiki page containing all pertinent details for each project, ensuring its continuous upkeep with the latest information.

Understand the Business problem, check the data quality, design and propose data driven solutions by driving the standalone meetings with Customers on their site.
Create and estimate monthly cost on various AWS services using AWS calculator and optimise the cost within the budget.
Make Sprint plan, assign, track, report, create Epic, Stories/backlogs and manage all the workflow on JIRA tool.
Setup AWS Eventbridge CRON settings to trigger lambda function to create an EMR cluster and set up all the steps to run multiple job’s and terminate the cluster.
Track and Validate job status and create a SNOW Incident if the job fails to create output in the targeted location’s.
Pull the data from remote S3 bucket to customer S3 bucket using various ETL jobs and with given specifications from multiple input sources.
Create Spark Dataframe on terabytes of data using s3 bucket and db sources such as RDS postgres db, Mysql db…etc.
Perform Spark Transformations and actions for analysing huge amounts of data.
Write UDF for complex analytical operations and customise functions.
Apply Job tuning params and optimization techniques on EMR Spark Jobs to reduce job turnaround time.
Create Hourly, Daily and Weekly based CRON settings for datapipeline jobs using various AWS services.
On demand basis Scale up the cluster’s hardware infra such as worker node count, instance type and their RAM, EBS, cores to optimise production jobs to meet business requirements.
Identify and build a recovery mechanism system for failed job in order to avoid business impact.
Explore latest technologies and apply them to the current system in order to make the system robust, stabilise and manage with ease.
Make necessary test cases and perform each of them in Dev/QA and UAT env before deploying into PROD to justify problem statements.
Attend daily scrum calls, Sprint reviews, ORR to make all necessary release activities and collaborate with the Devops team to deploy in the PROD.
Use Apache Airflow workflow management tool to maintain orchestration of data pipelines.
Seasonally derive new patterns by analysing multiple datasets based on Data science team requirements and create necessary features and data preparations to feed their ML models.
Create a data lake and ATHENA analysis base platform in order to use ready made data for Data Warehouse and other BA teams to query on large datasets.
Check Data lake size and manage during processing 250 ~ TB+ data and in order to use cluster resources effic

Understand the Business problem, check the data quality, design and propose data driven solutions by driving the standalone meetings with Customers on their site.
Create and estimate monthly cost on various AWS services using AWS calculator and optimise the cost within the budget.
Make Sprint plan, assign, track, report, create Epic, Stories/backlogs and manage all the workflow on JIRA tool.
Setup AWS Eventbridge CRON settings to trigger lambda function to create an EMR cluster and set up all the steps to run multiple job’s and terminate the cluster.
Track and Validate job status and create a SNOW Incident if the job fails to create output in the targeted location’s.
Pull the data from remote S3 bucket to customer S3 bucket using various ETL jobs and with given specifications from multiple input sources.
Create Spark Dataframe on terabytes of data using s3 bucket and db sources such as RDS postgres db, Mysql db…etc.
Perform Spark Transformations and actions for analysing huge amounts of data.
Write UDF for complex analytical operations and customise functions.
Apply Job tuning params and optimization techniques on EMR Spark Jobs to reduce job turnaround time.
Create Hourly, Daily and Weekly based CRON settings for datapipeline jobs using various AWS services.
On demand basis Scale up the cluster’s hardware infra such as worker node count, instance type and their RAM, EBS, cores to optimise production jobs to meet business requirements.
Identify and build a recovery mechanism system for failed job in order to avoid business impact.
Explore latest technologies and apply them to the current system in order to make the system robust, stabilise and manage with ease.
Make necessary test cases and perform each of them in Dev/QA and UAT env before deploying into PROD to justify problem statements.
Attend daily scrum calls, Sprint reviews, ORR to make all necessary release activities and collaborate with the Devops team to deploy in the PROD.
Use Apache Airflow workflow management tool to maintain orchestration of data pipelines.
Seasonally derive new patterns by analysing multiple datasets based on Data science team requirements and create necessary features and data preparations to feed their ML models.
Create a data lake and ATHENA analysis base platform in order to use ready made data for Data Warehouse and other BA teams to query on large datasets.
Check Data lake size and manage during processing 250 ~ TB+ data and in order to use cluster resources effic

Create spark sql dataframe using mysql source and get user id,service provider ID using mysql source Based on the problem type.
Extract service providers and hit sensors and fetch data for each and every user by passing date , time using bigquery.
Filter the data using threshold values based on each appliance id by using spark transformations and actions.
Create analysis patterns like daily, weekly, midnight usage, unusual flag usage, short and long power utilisation over the week and week over week.
Write Spark UDF's to perform various time modules, mathematical, statistical operations and data structure operations on data to justify the problem statement.
Write all analysis results into the Nosql database called BigTable and make it available to the other warehouse teams.

Use HDFS for Data storage and access it using Ambari.
Load the data from mysql db and HDFS, create spark sql dataframes.
Data preprocessing, data cleaning and data cleaning using spark sql
Build Recommendation Engine model using Spark Mllib with Collaborative Filtering Algorithm with ALS method.
Train the model with train data and evaluate the model with test data.
Tune the model and cross validate it by adding more features and from above steps till Model Optimization is satisfied by the customer.
Build a runnable jar file using SBT build tool and execute end to end system on Yarn and analyse job metrics and optimise job runtime if necessary.

Formulated data-driven solutions for client-specific requirements, addressing business challenges.
Load structured and unstructured data onto DLI, creating Spark Dataframes and performing transformations.
Create UDF for complex analytical operations, customized functions, and optimized Spark Jobs on DLI.
Load ES Indices data using Python API, creating pandas dataframes with comprehensive processing.
Create Kibana visualizations and dashboards, capturing patterns and trends to meet business requirements.
Build & deploy Docker images via SWR, activating CCE containers and automate CI/CD pipelines using Jenkins and GitLab integration.
Establish and maintain a comprehensive wiki page for each project, ensuring continuous updates with the latest information.

Analyze business problems, ensure data quality, and propose data-driven solutions through standalone meetings with clients on-site.
Configure AWS Eventbridge CRON settings for Lambda-trigger, EMR cluster creation, managing job execution and termination.
Create Spark dataframe on 250 TB data from S3 and various db sources.
Perform Spark transformations and actions for in-depth analysis of extensive datasets.
Create UDF for intricate analytical tasks, refine and optimize Spark Jobs, and significantly reduce job turnaround time, resulting in a 20% monthly cost reduction.
Collaborate with the Data Team on a data migration project, overseeing the seamless transfer of data from MsSQL to the Redshift data warehouse.
Perform data quality checks and provide comprehensive reports to business units for diverse business solutions.

Understand the Business problem by driving the standalone meetings with Customers on their site.
Propose the best solution to make the difference and add value using big data technologies.
Provide daily updates on task progress to the customer and uphold transparent communication throughout the project.
Leverage AWS cloud and essential cloud services tailored to software and hardware requirements in alignment with business agreements.
Collect and ingest IoT devices data and create Datapipeline ELT/ETL jobs using Hadoop and Spark Frameworks.
Establish an analytics base platform and discern and communicate data requirements to the customer, tailored to specific business use cases.
Automate pipeline jobs for analytical business operations using Pig Latin scripts and Hive QL.
Provide Knowledge Transfer KT to customer data science teams and collaborate with them to create new features for feeding ML models.
Establish an e-Confluence page, incorporating high-level information for each project and ensuring the continuous update of the latest details.

Explore latest trends in big data and AI algorithms for research.
Collect real time tweets using spark streaming with Kafka.
Identify trends, patterns, and insights within Twitter data through the application of diverse complex analytical operations to derive business conclusions.
Use NLP techniques to transform, tokenize, and analyze each word, computing TF-IDF to develop tweet sentiment analysis.
Create Classification and regression ML/DL models on available public datasets.
Build a recommendation engine using the ALS matrix factorization model to suggest top N shows for each user.
Create ready to use data sets and share Data Marketplace platform that facilitates offering data to consumers, enabling them to research, sample, compare, and purchase the third-party data.