Data Engineer with 13 years of experience designing, developing, and optimizing scalable data solutions using Python, PySpark, SQL, AWS, Databricks, Snowflake, Pandas, and Apache Airflow. Strong experience developing and…
Data Engineer with 13 years of experience designing, developing, and optimizing scalable data solutions using Python, PySpark, SQL, AWS, Databricks, Snowflake, Pandas, and Apache Airflow. Strong experience developing and maintaining ETL/ELT pipelines for large-scale data ingestion, transformation, cleansing, validation, and processing. Hands-on experience with AWS services including S3, Glue, Lambda, Athena, Redshift, CloudWatch, SNS, SSM, and EventBridge for cloud data lake and warehouse solutions. Proficient in Databricks and Apache Spark distributed processing, including performance tuning, partitioning, caching, handling data skew, and scalable pipeline development. Strong Snowflake experience in cloud data warehousing, SQL transformations, data loading, warehouse management, Streams, Tasks, Stored Procedures, and CDC processing. Experienced in dbt and Data Vault 2.0, including staging, Hub, Link, and Satellite models, incremental loads, reusable macros, historical tracking, and data quality tests. Skilled in SQL optimization, complex joins, aggregations, window functions, data validation, error handling, and source-to-target reconciliation; experienced in Airflow orchestration and production troubleshooting.
Experience
Data Engineer
Infosys Limitied·Chennai , Tamil Nadu , India
Apr 2025 - Present · 1 year 6 months
<p>DataEngineer with 13 years of experience designing, developing, and optimizing scalabledata solutions using Python,PySpark, SQL, AWS, Databricks, Snowflake, Pandas and Apache Airflow.experience in ETL/ELT pipelines usingPython and PySparkfor large-scale data ingingestion, transformation,cleansing,validation and processing.</p><p>Experience in creating External tables, Streams, Stored Procedures, Functions and Tasks in Snowflake. AWS s3 was used as a storage integration of snowflake. Daily incoming files were placed directly in s3 landing buckets and that was used as an external stage.</p>
PROJECTS
S
STATE STREET CORPORATION
Infosys Limitied
Duration : 01-Apr-2025 - 30-Sep-2026
Description:<p>Designed and developedscalable data pipelinesfor insurance analytics, processing claims, policy, customer, coverage, and premiumdata. Built scalableETL/ELT</p><p></p><p></p><p></p><p></p><p></p><p></p><p> workflows using AWS Glue, Databricks, PySpark, SQL, and Airflow. Processed and curated data in Amazon S3 and loaded it into Snowflake for analyticsand reporting. Implemented Databricks andSnowflake optimization techniques, data qualitychecks for reliable,secure, and governeddata pipelines.</p><p></p>Responsibilities:<p></p><ul><li>Designed and developed scalable data pipelines to process insurance claims, policy, customer, coverage, and premium data for analytics and reporting.</li><li>Built robust ETL/ELT workflows using AWS Glue, Databricks, PySpark, SQL, and Apache Airflow.</li><li>Developed data ingestion and transformation processes to extract data from multiple sources and curate it in Amazon S3.</li><li>Implemented scalable data loading pipelines to integrate curated datasets into Snowflake for enterprise analytics and reporting.</li><li>Applied Databricks optimization techniques, including Spark performance tuning, partitioning, caching, and efficient transformations.</li><li>Optimized Snowflake workloads through query tuning, data modeling, clustering, and warehouse optimization.</li><li>Implemented data quality checks, validation frameworks, error handling, and reconciliation processes to ensure data accuracy and reliability.</li><li>Followed data security, governance, and operational best practices while monitoring and supporting production data pipelines.</li></ul><p></p>
