supriya chandru

Align architecture with business requirements
Develop, construct, test and maintain architectures
Performing analysis on General Ledger Inventory data
Creating Fact tables using Azure databricks
Creating Routine logic to apply business rules
Notebook generates GL-fact table with all business logic
Creation of Notebooks to Validate data after ingestion
Copying data from SAP HANA to Azure Data Lake
Cleaning raw data using mapping Data-flow features
Linking to ADLS using Linked services

Working on AWS glue for running ETL jobs, Monthly load of FCA contains different stages and each stages with different pipeline to run sequentially.
Data present in PostgreSQL of RDS instance, creating Glue crawler for importing the database and tables to AWS glue and create glue workflow to run the etl jobs and storing the data to s3 and from s3 to snowflake.
Code was return in java initially and while running monthly jobs it was taking much longer than expected.
In-order to improve the performance we re-writing the code in spark and Scala and running the job in AWS glue and store.
Read the data from tables , creates object and pass record by record to Pipeline for next process.
Helping my team members in resolving technical issues assisting them with project design and involving discussion with clients for solution architect of project and how better we can improve the performance of ETL jobs.

The project is to implement and support investigative application for financial crime detection, anti- money laundering and risk assessment.
It provides a comprehensive view of the client by collecting data from different source systems and by scoring all the entities and networks.
The application runs on Spark framework with Cloudera and MapR Distributions.
Transformations were applied on the data using Spark with Scala.
Developed different Spark-jobs to Convert the CSV files into parquet files, using Dataset API and then providing single view of class by giving them hierarchical structure, using Scala implicit, Case and typed classes etc.
Developed different Spark-jobs to create compounds (joining different columns) to uniquely identify an entity (real world object), using Optics in Scala, traversals, lens, extractor etc. and then dumping the data into Elasticsearch, later which will be used in Quantexa UI for investigation.
Developed different Spark-jobs to create UI scores according to the business logic’s.
Testing our scores using,JUNIT test cases, FlatSpec and Matches in Scala.
Developed Hive queries for applying Business logic and transformation as per the requirement and automated the same using OOZIE scripts in Hadoop ecosystem.
Developed OOZIE coordinators for scheduling the jobs as per the business requirement (frequency and time).

Collection and obfuscation of all the HBO Email address.
Instantiate a process for collecting and obfuscating past, present and potentials HBO emails coming from 3 different sources which is present in S3 bucket.
Scala code to get particular columns from 3 different sources present in S3 buckets and converting the parquet file to CSV files.
Obfuscation must be replicate and granular enough to provide unique hashes for potential growth of HBO’s customer databases.
Push obfuscated email securely to statsocial s3 bucket via GNUPG using Airfow DAG’s.
StatSocial Enriched Data is typically received as JSON file.
StatSocial enrich data file is typically formatted as .tsv.gpg, need to pull the encrypted data using GNUPG.
Once JSON file is ready from StatSocial team we start transformation based on JSON files based on the requirements.
Replace Hashed Md5 obfuscated Email address with HBO universal ID.
Creating Athena tables to query the data which is present in S3 as per the requirements.
Github to deploy the code into PROD environment via Jenkins.
Airflow to schedule and run the DAG,s.

Good understanding in analyzing big datasets and finding patterns and insights with in unstructured and structured data.
Developing Spark API’s to compare the performance of spark with hive and SQL.
Implemented Spark using scala and Spark-SQL for faster testing and processing of data.
Involved in converting Hive/SQL queries into spark transformations using spark RDD.
Imported the required tables from RDBMS to HDFS using Sqoop and also used kafka to get real-time streaming of data into datalake.
The overview of the project is to ingest the data from different sources to hadoop Data Lake.
Data flow from source server into STAGE AND FROM stage TO RAW and ARCHIVE FOLDER.
Data ingestion framework used for ingesting the data into data lake.
Supporting commonly formatted data feed ( JDBC, CSV, JSON).
Data Lake provides QA, UAT environments for source system to perform DEV testing, make sure the proper testing has been done & provide signoff to Data Lake team before requesting production promotion.
Using Spark to ingesting the data for tables more than 15GB data
Creating spark framework for historical and Normal load using scala.
Creating metadata & views after data ingestion for different sources like (Sql Server, Teradata and Oracle).
Developed Scala and SQL code to extract the data from different Databases.
Performed various business transformations as per the Business requirements using data frames. and datasets.
Used AVRO, parquet and ORC data formats to store in to HDFS.
Develop and Transform the Raw data into useful information and load it into Kafka Queue (further loaded to HDFS)and databases for UI team to display it using web Application.
Uploaded the Data to HIVE and combined new tables with existing tables.
Creating Metadata for the Tables present in the respective sources.
Github to check in the code and to the deployment from UAT to PRODUCTION.

Oracle was already installed on Linux server, adding of connection string in ORACLE ORA file to access ORACLE DB from Toad or any DB clients.
Creation of DB tables with the help of WAMS components.
Evaluated and reported enterprise hardware and operating systems.
Implemented suitable application frameworks and participated in client discussions.

Provided Production Environment Support for Eagle Enterprise Application on Solaris, Aix and Windows servers.
Effectively Coordinated
Version Control of Store Procedure and Files done by Harvest Tool a CA application.
Tickets Handled
Incident Management tickets handled by GEIRS Portal and Client Tools.
Change Management tickets handled by Remedy and Service center Tools.
Database Migration was taken care in all the Environments.

The Project is to create data model for different reports, designing the fact and dimension model and create Azure databricks Notebooks for Fact and dimension tables using Scala and spark SQL.
Creating Routine logic to apply business rules while creating the fact tables.
As a Data Engineer Align architecture with business requirements
Develop, construct, test and maintain architectures.
Performing analysis on General Ledger Inventory data coming from source SAP_ECC and create data model for fact and Dimensions.
Creating Fact tables using Azure databricks using business key and calculate key figures and creating surrogate key mappings.
Notebook generates GL-fact table with all business logic and surrogate keys based on incremental processing using timestamp of last successful run, write parquet output files for actual’s to use by SQL Data Warehouse.
Creation of Notebooks to Validate data after ingestion from Source to Landing to Curated to processed Database.
Azure Synapse studio to ingest, explore , Analyse and Visualize data.
Copying data from SAP HANA(source data) to Azure Data Lake using copy data tool.
Cleaning raw data using mapping Data-flow features and load to synapse and schedule job to run using interface Apache spark cluster.
Linking to ADLS using Linked services.

Retrieving data (dealer, warranty, vehicle, campaign) from RDBMS and storing data into HDFS using Sqoop.
implementing Spark RDD transformations, actions to implement business analysis..
Implemented partitioning, dynamic partitions and buckets in HIVE
Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Scala.
Worked on transformation modules Dealer, Vehicle, campaign, and warranty.

Create, validate and maintain scripts to load data using Sqoop manually.
Worked on reading multiple data formats on HDFS using Qubole.
Creation EMR Cluster for different Hadoop components.
Creating Redshift Cluster, creating tables and loading the data into Redshift cluster from S3, EMR cluster and other local files.
Monitoring the different jobs running on Qubole.
Develop, validate and maintain HiveQL queries.
Running reports in Hive Queries.
Analyzing data with Hive
Designed Hive tables to load data to and from external tables.

Creating Ingestion Scripts, This script would ingest the ORC generated file (i.e. .csv file) and the original scanned images into the Azure blob storage
Using Azure PowerShell scripts to ingest flat files from as server location using SFTP GET.
Parsing the ingested ORC generated files (i.e. .csv) accessed from the Azure blob storage.
Working on Ambari and Jypyter to create Hive tables and Pyspark algorithm for parsing
Creating Hindsight Spark, Hbase cluster.
Creating external and internal tables in HIVE for the ORC files.

Migrating On-premise Applications to AWS (Amazon) and Colo cloud.
Migration of almost 500 servers which includes 8 BU’s like Zynx, HMG, etc. for both and non- prod application
VPC creation Hearst will have complete control over the virtual networking environment, including selection of IP address ranges, creation of subnets, configuration of route tables and network gateways Customization of the network configuration is easily achieved.
Working on Cloud formation script for creating VPC, Subnets, route tables and NACLS
Importing the server with the help of tools like VM import and Racemi.
Planning and consideration needs to be taken in this regard when designing and creating SGs.