Sneha M

Sneha M

sr. Data engineer
India
Tamil, English, Malayalam, French, Hindi
Full-time Remote

About Me

Data Engineering Specialist: ~5 years of experience architecting, streamlining, and maintaining enterprise ETL/ELT pipelines, distributed data systems, and cloud data lakes across AWS and Azure environments.

Distributed…

Experience

Sr. Data Engineer

iLink Systems and Digital Solutions · Chennai , Tamil Nadu , India
Jul 2024 - Present · 2 years 4 months

<p>Architected serverless and hybrid cloud ingestion pipelines utilizing AWS Glue, AWS Lambda, Amazon S3, Azure Data Factory (ADF), and ADLS Gen2 to automate daily financial asset records and valuation ingestions.</p><p> Engineered distributed transformation workflows in PySpark and Azure Databricks, enforcing schema compliance, partitioning strategies, automated duplicate detection, and data reconciliation routines.</p><p> Implemented Generative AI-enabled data pipelines, curating structured financial knowledge bases and contextual datasets for automated document summarization and LLM search queries.</p><p> Automated multi-tier ELT processes using AWS Step Functions, Azure Data Factory triggers, and Apache Airflow DAGs, ensuring fault-tolerant task coordination and proactive incident alerts within an Agile framework.</p><p> Designed analytical layers across Snowflake and Azure Synapse Analytics, building performant dimensional tables, views, and SQL scripts to accelerate quantitative deal analyses and compliance reporting.</p><p> Prototyped data preparation pipelines in Google Colab and Pandas, implementing feature engineering routines aligned with future machine learning model deployments.</p>

Data Engineer

Sutherland Global Services · Chennai , Tamil Nadu , India
Dec 2022 - Oct 2023 · 10 months

<p>Engineered end-to-end multi-cloud ingestion pipelines, aggregating multi-region order transactions, clickstream events, and inventory feeds into centralized Amazon S3 and Azure ADLS Gen2 storage layers.</p><p> Constructed scalable PySpark, Python, and Pandas jobs in Azure Databricks to compute critical business KPIs such as cart abandonment rates, SKU turnover, and customer session metrics.</p><p> Orchestrated workflow schedules via Apache Airflow, deploying dynamic DAGs with built-in sensors, automated failure retries, and data profiling checks to ensure uninterrupted data refreshes.</p><p> Optimized data delivery into Snowflake via staging areas and Snowpipe automated loads, significantly decreasing data latency for cross-functional business intelligence teams.</p><p> Enforced robust data validation, duplicate detection, and data governance standards across distributed workflows to guarantee schema consistency and regulatory compliance.</p><p> Conducted exploratory data analysis in Google Colab, preparing clean structured datasets to support upcoming predictive merchandising and forecasting models.</p>

Associate Data Engineer

Evodigital Technologies · Chennai , Tamil Nadu , India
Jul 2021 - Aug 2022 · 1 year 1 month

<p>Developed enterprise data integration mappings utilizing Informatica PowerCenter and complex SQL to extract, transform, and clean clinical trial patient records.</p><p> Managed high-volume distributed batch jobs using Hadoop and Python, adhering to stringent healthcare privacy, HIPAA compliance, and data governance mandates.</p><p> Executed structured database migrations from legacy source systems into scalable relational databases and cloud staging stores, ensuring strict data reconciliation and zero data discrepancy.</p><p> Implemented automated data quality checks, data profiling scripts, and custom lookup logic to identify, isolate, and remediate anomalous metrics prior to warehouse loading.</p><p> Collaborated with clinical analytics teams in Agile/Scrum sprints, drafting clear source-to-target mapping documentation, pipeline lineage, and operational runbooks.</p>

PROJECTS

Financial records

Ilink systems
Duration : 01-Jul-2024 - 01-Sep-2026

<p></p><p>• Architected serverless ingestion pipelines using AWS Glue, AWS Lambda, and Amazon S3 to automate daily financial asset records and valuation data ingestion. • Built distributed data transformation workflows in PySpark and Apache Spark, implementing schema enforcement, partitioning strategies, and deduplication routines. • Automated multi-tier ELT processes with AWS Step Functions and Apache Airflow to ensure fault-tolerant task coordination and proactive incident alerting. • Designed analytical layers in Snowflake, developing high-performance dimensional tables, views, and SQL scripts to support quantitative deal analysis and compliance reporting. • Built Generative AI-enabled data pipelines to curate structured financial knowledge bases and contextual datasets for automated document search and summarization. • Prototyped data prep pipelines in Google Colab, implementing feature engineering routines aligned with machine learning model deployments.</p>

Migration and ingestion

evodigital
Duration : 01-Dec-2022 - 01-Oct-2023

<p>• Engineered end-to-end data ingestion pipelines aggregating multi-region order transactions, clickstream events, and inventory feeds into Amazon S3. • Developed scalable PySpark and Python jobs to calculate key business metrics, including cart abandonment rates, SKU turnover, and session analytics. • Orchestrated workflow schedules using Apache Airflow, deploying dynamic DAGs with automated sensors and retry logic for continuous data delivery. • Optimized automated data loading into Snowflake using staging areas and Snowpipe to reduce data latency for analytics teams. • Conducted exploratory data analysis in Google Colab, preparing structured datasets for predictive merchandising and forecasting models. • Implemented data validation checks across distributed pipelines to enforce schema consistency and regulatory audit standards.</p>

EHR

Sutherland Global
Duration : 01-Jul-2021 - 01-Aug-2022

<p>• Developed enterprise data integration mappings using Informatica PowerCenter and complex SQL queries to clean and load clinical trial patient records. • Managed high-volume distributed batch data processing using Hadoop and Python in compliance with healthcare data regulations. • Executed structured database migrations from legacy source systems into relational data stores while verifying end-to-end data integrity. • Implemented automated data quality checks and validation routines to isolate anomalous records prior to warehouse loading. • Collaborated with analytics teams to author source-to-target mapping documentation, Azure, data lineage, and operational runbooks.</p>

Skills

AWS Glue AWS Lambda Amazon S3 AWS Step Functions CloudWatch Azure Databricks ADLS Gen2 Azure Data Factory Azure Synapse Analytics Python SQL Snowflake SQL PostgreSQL Oracle Unix Shell Scripting Apache Spark PySpark Hadoop Distributed Data Processing Snowflake Snowpipe Streams Tasks Time Travel Dimensional Modeling Star Schema Snowflake Schema Pandas Data Ingestion Data Transformation Pipeline Automation ETL ELT Data Reconciliation Duplicate Detection Data Profiling Data Governance Apache Airflow DAGs Custom Operators Informatica PowerCenter Agile Scrum Git GitHub JIRA Linux Jupyter Notebooks Google Colab Generative AI
Report this Profile?