Data Engineer with 6+ Years of experience in Data Engineering Domain. Experienced in working with big query transformations, cloud services like cloud composer, cloud run, cloud function, cloud SQL. Skilled in using tool…
Data Engineer with 6+ Years of experience in Data Engineering Domain. Experienced in working with big query transformations, cloud services like cloud composer, cloud run, cloud function, cloud SQL. Skilled in using tools like Spark, AWS Glue, Databricks, Sagemaker, Airflow, and more.
Designed and implemented end-to-end data flow using stored procedures in big query, worked on nested structures, table optimizations, ingesting data from mainframe, setting up Google's DVT and mainframe transcoder.
Designed, developed, and implemented end-to-end ETL processes using AWS.
Glue to extract, transform, and load data from diverse sources to destinations.
Optimized Glue jobs and worked extensively on Glue version upgrade from version 2 to version 4.
Worked on Glue job to perform SCD type 1 and 2.
Designed and maintained release and versioning strategies.
Developed Glue jobs to pull reference data into team specific data zones.
Designed Data flows, DLDs, HLDs and performed code reviews.
Orchestrated jobs using schedule based and event-based triggers.
Worked on existing jobs to debug, optimize and provide support till the jobs were released and stable in production.
Worked on Glue notebooks to debug and develop Pyspark code.
Worked on testing end-end upstream and downstream jobs workflow.
Used Jira for agile process and to track issues and Change Management.
Designed and implemented ETL processes, optimized Glue jobs, worked on Glue version upgrade, performed SCD type 1 and 2, orchestrated jobs, debugged and optimized code for production.
Designed and Developed ETL Processes with Pyspark in AWS Glue to migrate data from S3 to generate Reports.
Worked on generating and automating the Teradata report generation process from glue and databricks.
Deployed databricks jobs using Jenkins and terraforms.
Used Sagemaker as dev endpoint for the glue development to build and develop business logic.
Authored Spark Jobs for data filtering and data transforming through Pyspark data frames both in AWS glue and databricks.
Involved in writing and Scheduling the Databricks jobs Using Airflow.
Used AWS glue catalog with Athena to get the data from S3 and perform SQL query operations.
Wrote various data normalization jobs for new data ingested to s3.
Created Airflow master DAGs to schedule the jobs on daily, weekly, monthly schedules.
Designed and Developed ETL Processes with Pyspark in AWS Glue to migrate data from external sources and S3 Files.
Involved in converting SQL queries into Spark transformations using Spark data frames and Python Pyspark.
Involved in writing and Scheduling the Glue jobs, Building data catalog and mapping from S3.
Authored Spark Jobs for data filtering and data transforming through Pyspark data frames.
Used RALLY and JIRA to track issues and Change Management.
Developed ETL processes, generated reports, migrated data, deployed jobs, authored Spark jobs, scheduled and orchestrated jobs using Airflow. mainly worked on designing and developing data pipelines using aws glue, databricks, kafka, lambda.
Developed AWS Lambdas to handle different types of feed files from API gateway.
Designed and Developed ETL Processes with Pyspark in AWS Glue to migrate data from external sources and S3 Files into AWS Redshift.
Involved in writing and Scheduling the Glue jobs, Building data catalog and mapping from S3 to Redshift.
Created AWS Lambda functions and assigned IAM roles to schedule python scripts using CloudWatch Triggers to support the infrastructure needs that needed extraction of xml tags.
Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions.
Implemented usage of Amazon EMR for processing Big Data across a Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).
Created AWS Lambda functions and assigned IAM roles to schedule python scripts using CloudWatch Triggers to support the infrastructure needs.
Modelled script to hit REST API’s and extract data to AWS S3.
Authored Spark Jobs for data filtering and data transforming through Pyspark data frames.
Involved in connecting Redshift to Tableau for creating dynamic dashboard for analytics team.
Developed AWS Lambdas, designed and implemented ETL processes, worked on data migration, lambda functions, data cleansing, data transformations, and data analytics.
Designed, developed, and implemented end-to-end data flow using stored procedures in big query.
Worked on nested structures in big query and on optimizations in tables.
Worked on setting up process to ingest data from mainframe to big query using cloud composer.
Wrote the DAGs to ingest the data from sources to big query.
Worked on inserting audit logs to cloud SQL.
Worked on designing the architecture.
Worked on setting up Google’s DVT and mainframe transcoder in GCP environment.
Designed and maintained release and versioning strategies.
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
