Honeyshmitha

<p>Orchestrated Data Lake setup, merging diverse Teradata and multiple source data through Hadoop stack (SQOOP, Hive/HQL).<br>
Developed a data ingestion tool using PySpark capable of gathering data from diverse sources such as APIs, RDBMS, files, and Kafka streams.<br>
Addressed challenges including updating hive table structures to accommodate incoming data for incremental loads, performing data type conversions, and identifying error records during file ingestions.<br>
Enhanced ingestion performance by 20 times compared to existing tools like Stream Sets, leveraging multi-core processing tailored to the data pipeline's workload.<br>
Optimized SQL queries for reading tables as partitions to achieve better performance through parallel processing.<br>
Handled importing of data from various data sources, performed transformations using spark and storing into Hive.<br>
Scheduled Autosys jobs to execute multiple Spark tasks via shell scripts, ensuring independent execution based on time, seamless data loading and resource availability.<br>
Designed robust big data ingestion pipelines in Spark, ensuring data quality checks, transformations, and efficient storage.<br>
Conducted comprehensive Data Wrangling on diverse datasets using PySpark across various sources.<br>
Solved performance issues in Pyspark with understanding of groups, joins and aggregation functions and Scheduled Pyspark jobs in Cluster by using Spark UI.<br>
Leveraged Scala scripts in Spark for aggregation, querying, and HDFS data writing via data frames/SQL and RDD/MapReduce.<br>
Developed SQL and Hadoop artifacts: stored procedures, views, UDFs, and CTEs for streamlined data operations.<br>
Created UNIX scripts for data reconciliation and engineered Spark jobs for user data analysis.<br>
Demonstrated adeptness in RDBMS data extraction and transformation for ingestion, optimizing computational logic with efficient Spark code using Scala and spark-SQL.</p>

• Orchestrated Data Lake setup, merging diverse Teradata and multiple source data through Hadoop
stack (SQOOP, Hive/HQL).
• Developed a data ingestion tool using PySpark capable of gathering data from diverse sources such as
APIs, RDBMS, files, and Kafka streams.
• Addressed challenges including updating hive table structures to accommodate incoming data for
incremental loads, performing data type conversions, and identifying error records during file
ingestions.
• Enhanced ingestion performance by 20 times compared to existing tools like Stream Sets, leveraging
multi-core processing tailored to the data pipeline's workload.
• Optimized SQL queries for reading tables as partitions to achieve better performance through parallel
processing.
• Handled importing of data from various data sources, performed transformations using spark and
storing into Hive.
• Scheduled Autosys jobs to execute multiple Spark tasks via shell scripts, ensuring independent
execution based on time, seamless data loading and resource availability.
• Designed robust big data ingestion pipelines in Spark, ensuring data quality checks, transformations,
and efficient storage.
• Conducted comprehensive Data Wrangling on diverse datasets using PySpark across various sources.
• Solved performance issues in Pyspark with understanding of groups, joins and aggregation functions
and Scheduled Pyspark jobs in Cluster by using Spark UI.
• Leveraged Scala scripts in Spark for aggregation, querying, and HDFS data writing via data frames/SQL
and RDD/MapReduce.
• Developed SQL and Hadoop artifacts: stored procedures, views, UDFs, and CTEs for streamlined data
operations.
• Created UNIX scripts for data reconciliation and engineered Spark jobs for user data analysis.
• Demonstrated adeptness in RDBMS data extraction and transformation for ingestion, optimizing
computational logic with efficient Spark code using Scala and spark-SQL.

<p>Developed PySpark tasks to handle data processing, including reading from external sources, merging, enriching, and loading into target destinations.<br>
Created efficient Hive tables with appropriate static and dynamic partitions to meet specific requirements.<br>
Managed data importation from diverse sources, performed transformations using Spark, and stored results in Hive and S3 buckets.<br>
Imported RDBMS tables into Hive using Sqoop and created data visualizations in Tableau.<br>
Scheduled Apache Airflow DAGs to execute multiple Hive and Spark jobs independently based on time and data availability.<br>
Migrated an on-premises application to AWS, utilizing EMR and Glue jobs for processing and storing small datasets.<br>
Created on-demand tables on S3 files using Lambda Functions and AWS Glue with Python and Spark.<br>
Developed and optimized complex SQL queries to extract data from SQL server databases, tuning ETL processes and SQL Queries for improved performance.<br>
Maintained technical documentation for executing Hive queries, and actively participated in code reviews and bug fixing to enhance performance.</p>

<p>Engaged in all project stages including analysis, design, development, and testing.<br>
Proficient in understanding Business Requirement Documents and crafting detailed Low-Level Design Documents.<br>
Managed Database Objects, Tables, and Views effectively.<br>
Developed APIs for new clients and devised corresponding business logic for scheduled and immediate transfers.<br>
Designed projects utilizing Python Modules and Packages.<br>
Created comprehensive unit test cases and maintained a release tracker for each software release.<br>
Designed and developed webpages using CSS, HTML, and JavaScript, while also implementing automation resources through Jenkins.<br>
Utilized various Python modules and controls to rapidly build applications, leveraging Python data structures and advanced coding techniques like list comprehensions, generators, lambda functions, and built-in functions such as map and filter.<br>
Addressed performance issues through efficient coding and optimized SQL queries for improved response time, while providing development support by coordinating with production teams.</p>