Aiden Diao

Aiden Diao

Senior Data Engineer
Australia
English, Chinese
Full-time Remote

About Me

data team leader with strong experience designing, building and operating cloud data platforms across AWS, Snowflake, Airflow, dbt, Python, SQL and Terraform. Background spans data lake and data warehouse engineering, or…

Experience

Senior Data Engineer

Archistar.ai, data engineering, NSW · Sydney , New South Wales , Australia
Nov 2020 - Present · 5 years 11 months

<p>Designed and built production ETL/ELT pipelines ingesting data from multiple source systems into AWS-based data lake and Snowflake data warehouse environments using AWS Batch, S3, Lambda, Athena, SQS, ECS, Airflow, DynamoDB and RDS.</p><p> Built and maintained workflows using DBT and Snowflake, including SQL models, stored procedures and recurring data processing jobs.</p><p> Re-designed a large-scale workflow from AWS Lambda to AWS Batch, using manifest-driven parallelism and controlled concurrency to resolve throttling/timeouts, reduce run cost by ~96%, and cut processing time from ~2 hours to 48 minutes.</p><p> Investigated and resolved complex data quality and pipeline issues across PostgreSQL, DBT, Snowflake and AWS services.</p><p> Provisioned and configured cloud/container infrastructure using Terraform, ECS and Docker.</p><p> Implemented data validation, schema controls and monitoring using CloudWatch and Vanta to improve data quality, governance and pipeline reliability.</p><p> Mentored and provided technical leadership to a team of 3, supporting planning, delivery coordination, code review and technical problem solving.</p>

Data Lead

Downer EDI, Technology & Innovation, NSW · Sydney , New South Wales , Australia
Jun 2019 - Nov 2020 · 1 year 5 months

<p>Designed and delivered data integration pipelines across Dynamics CRM, REST APIs, Azure Logic Apps, Power Automate, Azure Blob Storage and PostgreSQL.</p><p> Built Python tools to retrieve and process geocoded imagery data from MSSQL and Azure Blob Storage.</p><p> Led technical delivery across data, GIS and field-worker application initiatives, including sprint planning, issue resolution and stakeholder communication.</p><p> Guided data analysts through technical decisions while maintaining feature delivery and operational support.</p><p> Designed and optimised PostgreSQL data warehouse.</p>

Data Lead

Downer EDI, Utilities, QLD · Bowen Hills , Queensland , Australia
Jun 2018 - Jun 2019 · 1 year

<p>Conducted data analysis using Tableau.</p><p> Built various data pipelines using Python and SQL.</p><p> Managed internal and external design teams and stakeholders.</p>

Design Coordinator

Downer EDI, Utilities, NSW · Newcastle , New South Wales , Australia
Sep 2016 - May 2018 · 1 year 8 months

<p>Built various python pipelines for productivity and BOM auditing.</p><p>Liaised with stakeholders on project deliverables.</p><p>Completed in field design variations.</p>

Structural Engineer

Harbin Design Institute · Harbin , Heilongjiang , China
Nov 2014 - Sep 2016 · 1 year 10 months

<p>Performed design and drafting support throughout the project.</p><p>Modelled buildings using FEM method for structural analysis.</p><p>Maintainted design and construction documents.</p><p></p>

Skills

Python SQL PySpark Snowflake DBT Airflow AWS Terraform Docker Git PostgreSQL REST API Databricks JSON GeoJSON Jira Confluence Linux Tableau Azure Logic Apps Power Automate Azure Blob Storage MSSQL CloudWatch Vanta ECS S3 Lambda Athena SQS DynamoDB RDS AWS Batch Delta Lake pgvector
Report this Profile?