Data Engineer with 3.5 years building and operating production data platforms on AWS. Specializes in distributed ETL with PySpark and real-time ingestion via Kinesis/Firehose, processing ~650M telemetry records/day from …
Data Engineer with 3.5 years building and operating production data platforms on AWS. Specializes in distributed ETL with PySpark and real-time ingestion via Kinesis/Firehose, processing ~650M telemetry records/day from 6.8M connected devices. Deep PostgreSQL expertise across schema-per-tenant isolation, transaction-safe migrations, and SCD Type 2 modeling. Experienced in Kafka-based data-lake delivery, orchestration (Step Functions, Airflow), lakehouse patterns (Delta Lake), and significant AWS cost optimization.
الخبرة
T
Data Engineer
Tata Elxsi Limited, Trivandrum·Adur , Kerala , India
Nov 2022 - حتى الآن · 3 سنوات 11 أشهر
<p>Built and operate production PySpark/AWS Glue ETL processing ~650M telemetry records/day from 6.8M connected STB devices reporting every 15 minutes, transforming raw multi-tenant telemetry into analytics-ready datasets.</p><p> Engineered real-time ingestion with Amazon Kinesis and Firehose; diagnosed a hot-shard bottleneck and scaled the stream 28 → 32 shards with enhanced shard-level metrics, eliminating write throttling under peak load.</p><p> Delivered processed telemetry to client-side data lakes via Apache Kafka and Kafka streaming, enabling reliable downstream consumption across analytics teams.</p><p> Optimized AWS Glue jobs by diagnosing data skew, tuning Adaptive Query Execution and caching (persist MEMORY_AND_DISK), and right-sizing workers (G.1X → G.2X) with autoscaling — substantially reducing job runtime and compute spend.</p><p> Drove cloud cost optimization across the pipeline — Athena partition pruning and predicate pushdown, RDS storage migration gp2 → gp3 (higher IOPS at lower cost), and Glue worker right-sizing — drastically reducing monthly AWS spend.</p><p> Built transaction-safe PostgreSQL patterns: atomic staging-table swaps (delete/insert), SCD Type 2 MERGE with surrogate keys and SHA-256 row hashing, and partial indexing on current records — enabling lossless retroactive reloads of historical data.</p><p> Orchestrated multi-job pipelines with AWS Step Functions and EventBridge using idempotent retry/catch logic and event-driven scheduling for reliable, hands-off runs.</p><p> Owned CI/CD deployment of production Glue pipelines via Jenkins with dev → prod environment promotion and CloudFormation-managed infrastructure, enabling safe, repeatable releases.</p><p> Managed production schema evolution via blue/green column migration and Glue Schema Registry compatibility modes with versioned migration scripts for backward-compatible reprocessing.</p><p> Built a data-quality framework — invariant-based validation (conservation, bounds, rollup consistency) and cross-database reconciliation via S3 Parquet exports and Athena full outer joins — surfacing missing/divergent records at scale.</p>
المشاريع
I
iCX
TATA ELXSI
المدة : 15-May-2019 - 30-Jun-2026
<p>RMS – Multi-Tenant STB Telemetry Data Platform</p><p>Tata Elxsi Limited, Trivandrum | Data Engineer (~3.5 years)</p><p></p><p>What it does: A production, multi-tenant AWS data platform that ingests and processes set-top box (STB) telemetry data at roughly 650 million records/day across nine tenants, powering downstream analytics and reporting.</p><p></p><p>Tech stack:</p><p></p><p> • Ingestion/Streaming: AWS Kinesis, Firehose</p><p> • Processing: PySpark on AWS Glue</p><p> • Storage: S3 (Parquet)</p><p> • Query: Athena</p><p> • Database: PostgreSQL (schema-per-tenant multi-tenancy</p><p></p><p> • Orchestration: Step Functions, Lambda</p><p></p><p>Key contributions / talking points:</p><p></p><p> • Diagnosed and resolved skew-bound vs. capacity-bound Glue job performance issues</p><p> • Refactored a precompute_latest job from row_number-based logic to hash aggregation for null-safe column resolution</p><p> • Debugged Glue Python shell timeout issues tied to scheduled trigger configuration</p><p> • Investigated and resolved Kinesis throttling on a data stream, scaling shards (28→32)</p><p> • Analyzed RDS PostgreSQL ReadIOPS alarms and recommended a gp2→gp3 migration</p><p> • Implemented SCD Type 2 merge logic using SHA-256 row hashing for idempotent, auditable updates</p><p> • Designed multi-tenant schema-per-tenant PostgreSQL architecture</p><p> • Built technical documentation for QA on metric trend comparison and weekly metrics consolidation jobs</p><p></p><p>GenAI-adjacent work (differentiator):</p><p>Contributed to CCAssist, an internal RAG-based support tool using AWS Strands Agents, Bedrock, Aurora PostgreSQL with pgvector, and Langfuse for observability</p><p></p>
الشهادات
BTECH
Sree Chitra Thirunal Institute of Medical Sciences and Technology, India
· Kazhakkoottam , Kerala , India
· 2023
