Harini Abburi

Harini Abburi

Senior Data Engineer
Canada
Contract Hybrid

About Me

Results-driven Senior Data Engineer with 7+ years of experience designing, building, and operating enterprise-scale data pipelines, cloud data platforms, and real-time streaming systems across Canadian financial services…

Experience

Senior Data Engineer

Foxquilt · Toronto, Ontario, Canada
Apr 2022 - Present · 4 years 7 months

<p>Architected enterprise-scale commercial insurance data ingestion pipelines using AWS and Snowflake, consolidating policyholder, quote, and claims data from 15+ source systems to enable unified underwriting analytics.</p><p> Designed and implemented Medallion Architecture (Bronze/Silver/Gold) in Snowflake and S3, organizing raw policy event data, curated risk features, and business-ready underwriting metrics, reducing time-to-insight for the actuarial and product teams by 60%.</p><p> Built event-driven streaming ingestion pipelines using Kafka and AWS Lambda to capture real-time policy bind, quote, and cancellation events from Foxden microservices, enabling live rate adjustment signals and reducing data latency from hours to under 3 minutes.</p><p> Developed 20+ analytics dashboards surfacing key insurtech KPIs, empowering underwriting, product, and executive teams to make data-driven pricing decisions.</p><p> Implemented CI/CD pipelines for data workflows using GitHub Actions and AWS CodePipeline, automating testing and deployment of Snowflake DBT models and Lambda-based ingestion jobs, reducing release cycle time by 65% and eliminating manual deployment errors.</p><p> Led data platform build-out to support Foxquilt’s US market expansion and Markel partnership, designing cross-border data schemas to unify Canadian and US policy data under a single Snowflake warehouse while maintaining regulatory separation between jurisdictions.</p><p> Developed DBT transformation models to standardize risk signal features used by Foxquilt’s ML underwriting algorithms, ensuring consistent feature definitions across the actuarial rating engine and analytics layer, and reducing data discrepancies in pricing decisions by 35%.</p><p> Designed and deployed Airflow DAGs on Astronomer to orchestrate nightly ingestion, DBT transformation, and data quality workflows across Foxden API sources, third-party partner feeds (Munich Re, Markel), and internal Snowflake layers.</p><p> Optimized AWS Lambda concurrency, S3 storage tiering, and Snowflake virtual warehouse auto-suspend settings, reducing cloud infrastructure spend by 30% while supporting a 3x increase in daily policy event volume driven by new partner onboarding.</p><p> Collaborated with Legal and Compliance teams to embed PIPEDA data privacy controls and FSRA regulatory requirements directly into pipeline design, ensuring policyholder PII was masked at ingestion, audit-logged at transformation, and retained per Ontario insurance data governance standards.</p>

Senior Data Engineer

National Bank · Toronto, Ontario
Jan 2021 - Mar 2022 · 1 year 1 month

<p>Developed end-to-end data ingestion pipelines on Azure HDInsight Spark using Azure Data Factory, processing 500GB+ daily from heterogeneous sources (MySQL, Cassandra, Blob Storage) to Azure Synapse Analytics.</p><p> Migrated 10+ legacy data sets to Databricks Spark platform, reducing processing time from 6 hours to 45 minutes through distributed computing optimization and efficient data partitioning strategies.</p><p> Engineered real-time streaming analytics solution using Spark Streaming and Azure Data Lake Gen2, enabling fraud detection with

Data Engineer

Pineapple Financials Inc · Toronto, Canada
Oct 2019 - Dec 2020 · 1 year 2 months

<p>Built distributed data processing jobs using Apache Spark and SQL for large-scale transformations across multi-source ingestion pipelines feeding Snowflake and cloud storage layers.</p><p> Designed ETL pipelines integrating relational (SQL Server) and cloud data sources, implementing Medallion Architecture (Bronze/Silver/Gold) for clean separation of raw, curated, and business-ready data.</p><p> Developed Kafka-based streaming ingestion pipelines to capture CDC (Change Data Capture) events, enabling near-real-time analytics for operational dashboards.</p><p> Created Apache Airflow DAGs to orchestrate multi-step ingestion, transformation, and data quality workflows, replacing manual scheduling with fully automated dependency-driven pipelines.</p><p> Developed Flask REST APIs to expose aggregated data assets to downstream BI tools and internal data consumers, incorporating token-based authentication and rate limiting.</p><p> Containerized pipeline services with Docker; contributed to Kubernetes deployment manifests and CI/CD pipeline configurations in Jenkins for consistent environment delivery.</p><p> Wrote Python automation scripts and shell scripts for operational tasks; managed deployments and monitoring on Linux/Unix environments.</p><p> Optimized complex SQL queries and Snowflake warehouse sizing, improving performance of large-table analytical queries used in executive reporting.</p><p> Maintained PyTest-based unit and integration test suites; integrated test runs into Jenkins build pipelines to enforce quality gates before production deployments.</p><p> Collaborated with cross-functional teams to translate business requirements into backend data models and scalable pipeline architectures within an Agile delivery model.</p>

PROJECTS

Cloud-Based Insurance Data Engineering and Analytics Platform

Duration : 02-May-2022 - 06-Nov-2026

<p>Worked on developing and maintaining scalable data pipelines to integrate insurance data from multiple source systems into a centralized analytics platform. Used Python, SQL, AWS, Snowflake, Kafka, dbt, and Airflow to support data ingestion, transformation, and processing. Responsibilities included building and maintaining ETL/ELT workflows, integrating data from different sources, implementing data quality checks, troubleshooting pipeline failures, and optimizing performance. Contributed to improving data reliability, accessibility, and availability for downstream reporting and business analytics.</p>

Certifications

AWS Certified Data Engineer

AWS · Toronto , Ontario , Canada · 2026

Skills

Python SQL Apache Spark PySpark Spark Streaming Azure Event Hubs Kafka Snowflake PostgreSQL SQL Server MongoDB Apache Airflow Astronomer DBT AWS S3 AWS Lambda AWS Glue AWS Kinesis AWS CodePipeline DynamoDB CloudWatch IAM EMR Athena Flask FastAPI REST API design Docker Kubernetes EKS GitHub Actions Jenkins Delta Lake Parquet ORC Avro JSON XML CSV Medallion Architecture Lambda Architecture Data Lakehouse Dimensional Modeling Agile Scrum TDD PyTest CI/CD Terraform Azure Data Factory
Report this Profile?