خبراء الذكاء الاصطناعي

Mugil Muraleedharan

Data Engineer

India

English

نبذة عني

Data Engineer with hands-on experience designing and deploying production-grade ELT pipelines, real-time streaming systems, and cloud data platforms. Proven expertise in Airflow (DAG orchestration), dbt (dimensional mode…

الخبرة

Fully Automated ELT Platform

E-Commerce Intelligence Analytics system

Architected fully automated ELT pipeline processing 50K+ daily transactions from AWS S3 to Snowflake, reducing manual processing by 90%
Designed 3-layer dbt transformation architecture (staging->intermediate->marts) with 25+ models implementing star schema for analytics
Orchestrated workflows using Apache Airflow with 12+ DAGs, implementing task dependencies, retry logic, and SLA monitoring

Kafka + FastAPI + PostgreSQL

Live Air Traffic Monitoring ETL Pipeline

Engineered high-throughput Kafka pipeline processing 10K+ flight telemetry records per minute with sub-100ms latency
Designed Kafka topic architecture with 8-partition strategy achieving parallel processing and improving throughput by 300%
Implemented real-time data enrichment (geocoding, airline mapping) using Python consumer groups, transforming raw data to analytics-ready

FastAPI, PostgreSQL, SQLAlchemy, Docker, WebSockets, Cloud Storage (R2)

Socionn — Social Platform Data Infrastructure

Architected production data infrastructure supporting 1000+ concurrent users with comprehensive 20+ table relational schema
Implemented database optimization: connection pooling (PgBouncer), strategic indexing (B-tree, GIN), query optimization - reducing API response times from 2s to 200ms
Engineered cascade deletion workflows across 20+ foreign key relationships ensuring referential integrity and GDPR-compliant data removal

OCR + LLM – Data Extraction Pipeline

Land Deed Extraction Pipeline

Designed multi-stage automated pipeline (ingestion->OCR->cleaning->LLM parsing->validation) processing 500+ documents daily, reducing manual entry by 85%
Leveraged LLM APIs (GPT-4) with engineered prompts to extract structured fields from unstructured OCR text with 92% field-level accuracy
Standardized extracted data into normalized PostgreSQL schemas, enabling downstream analytics on property trends and ownership patterns

Data Engineer

E-Commerce Intelligence Analytics system (Fully Automated ELT Platform)

Architected fully automated ELT pipeline processing 50K+ daily transactions from AWS S3 to Snowflake, reducing manual processing by 90%, Designed 3-layer dbt transformation architecture (staging->intermediate->marts) with 25+ models implementing star schema for analytics, Orchestrated workflows using Apache Airflow with 12+ DAGs, implementing task dependencies, retry logic, and SLA monitoring

Data Engineer

Land Deed Extraction Pipeline (OCR + LLM – Data Extraction Pipeline)

Designed multi-stage automated pipeline (ingestion->OCR->cleaning->LLM parsing->validation) processing 500+ documents daily, reducing manual entry by 85%, Leveraged LLM APIs (GPT-4) with engineered prompts to extract structured fields from unstructured OCR text with 92% field-level accuracy, Standardized extracted data into normalized PostgreSQL schemas, enabling downstream analytics on property trends and ownership patterns

Data Engineer

Socionn — Social Platform Data Infrastructure

Architected production data infrastructure supporting 1000+ concurrent users with comprehensive 20+ table relational schema, Implemented database optimization: connection pooling (PgBouncer), strategic indexing (B-tree, GIN), query optimization - reducing API response times from 2s to 200ms, Engineered cascade deletion workflows across 20+ foreign key relationships ensuring referential integrity and GDPR-compliant data removal

Data Engineer

Live Air Traffic Monitoring ETL Pipeline (Kafka + FastAPI + PostgreSQL)

Engineered high-throughput Kafka pipeline processing 10K+ flight telemetry records per minute with sub-100ms latency, Designed Kafka topic architecture with 8-partition strategy achieving parallel processing and improving throughput by 300%, Implemented real-time data enrichment (geocoding, airline mapping) using Python consumer groups, transforming raw data to analytics-ready

المهارات

بوستجريس كيو إل بايثون (لغة برمجة) لغة الاستعلامات الهيكلية (SQL) نمذجة البيانات نومباي (مكتبة لغة Python) خدمات أمازون ويب (AWS) دوكر باور بي آي إس 3 (خدمة تخزين البيانات على الإنترنت) التحليل الإحصائي تيرافورم Apache Airflow DAG design scheduling monitoring task dependencies dbt dimensional modeling incremental loads data quality tests Snowflake virtual warehouses COPY commands query optimization Apache Kafka producers consumers topics partitions Real-Time Streaming Pipelines star schema snowflake schema Data Warehousing Pandas SQLAlchemy FastAPI complex joins CTEs window functions JavaScript/TypeScript Glue EC2 IAM Lambda containerization multi-stage builds Infrastructure as Code CI/CD pipelines DAX data modeling dashboards Matplotlib Seaborn Feature Engineering indexing connection pooling REST APIs Git OCR/LLM Pipelines Tesseract GPT APIs Data Engineering & ETL/ELT Programming & Scripting Cloud & Infrastructure Data Visualization & Analytics Additional Tools JavaScript TypeScript CI/CD OCR LLM ETL ELT Real-Time Streaming Query Optimization Dimensional Modeling Star Schema Snowflake Schema Incremental Loads Data Quality Testing Connection Pooling WebSockets Cloud Storage B-tree Indexing GIN Indexing