Vivek Raval

Vivek Raval

Big Data Engineer
Canada
Full-time Work from Home

About Me

Big Data Engineer with 4+ years of experience designing, building, and optimizing scalable cloud ETL pipelines and distributed storage architectures. Strong hands-on expertise across AWS (EMR, Glue, S3), PySpark, and the…

Experience

Big Data Engineer

Programmr Technologies Pvt. Ltd. · Bangalore, India
Nov 2021 - Jul 2026 · 4 years 7 months

<p>Designed and maintained automated PySpark and AWS Glue ETL pipelines processing 3–4 TB of daily telemetry data, achieving 90% automation and cutting monthly AWS cloud costs by 20% through cluster tuning and Parquet compression.</p><p> Optimized 28-node AWS EMR clusters (R5.8xlarge), reducing processing time for 5–6 TB datasets from 7 hours to 4.5 hours, a 35% performance gain.</p><p> Orchestrated complex workflow DAGs in Apache Airflow with dynamic schema validation, maintaining 99% pipeline uptime and cutting failure-recovery time to under 30 minutes.</p><p> Streamlined real-time IoT data processing with Apache Kafka and PySpark, reducing telemetry ingestion latency by 40%.</p><p> Executed zero-downtime migration of 10+ TB of enterprise workloads from legacy SQL Server to Snowflake and Hadoop using Sqoop and PySpark, improving downstream query speeds by 25%.</p><p> Architected Avro schemas and managed dynamic schema evolution across Hive and Delta Lake tables, preventing downstream breaking changes during field additions and data-type updates.</p><p> Integrated AWS S3 data lakes with Delta Lake governance to enable ACID transactions, optimized partitioning, and scalable business analytics.</p><p> Combining PySpark tasks with AWS S3, HDFS, and EC2 to handle data lakes and guarantee dependable storage and access of intermediate and final results.</p><p> Minimizing serialization costs and enhancing data compression by choosing the best formats and codecs suited to specific workload types.</p><p> Facilitating data-driven decision-making by providing quicker access to high-quality data across platforms integrated in the cloud and distributed systems.</p>

PROJECTS

Ola Electric

Automotive
Duration : 01-Nov-2021 - 01-Jul-2026

<p>Designed and maintained automated PySpark and AWS Glue ETL pipelines processing 3–4 TB of daily telemetry data, achieving 90% automation and cutting monthly AWS cloud costs by 20% through cluster tuning and Parquet compression.</p><p> Optimized 28-node AWS EMR clusters (R5.8xlarge), reducing processing time for 5–6 TB datasets from 7 hours to 4.5 hours, a 35% performance gain.</p><p> Orchestrated complex workflow DAGs in Apache Airflow with dynamic schema validation, maintaining 99% pipeline uptime and cutting failure-recovery time to under 30 minutes.</p><p> Streamlined real-time IoT data processing with Apache Kafka and PySpark, reducing telemetry ingestion latency by 40%.</p><p> Executed zero-downtime migration of 10+ TB of enterprise workloads from legacy SQL Server to Snowflake and Hadoop using Sqoop and PySpark, improving downstream query speeds by 25%.</p>

Certifications

Python for Data Science

IBM · Anand , Gujarat , India · 2024

Skills

Python Scala Java SQL HiveQL PySpark RDD DataFrame Spark SQL Apache Spark Hadoop HDFS Apache Hive Apache Kafka EMR Glue S3 EC2 Lambda Redshift Athena Step Functions Apache Airflow Sqoop Docker Kubernetes Snowflake SQL Server MySQL Parquet Avro JSON Delta Lake
Report this Profile?