نبذة عني
Data Engineer with 15+ years of experience across data engineering, databases, data analysis and software development, including five years in dedicated Data Engineering roles. Hands-on experience designing and optimisin…
Data Engineer with 15+ years of experience across data engineering, databases, data analysis and software development, including five years in dedicated Data Engineering roles. Hands-on experience designing and optimising scalable AWS data pipelines and ETL/ELT processes using AWS Glue, Python, PySpark, SQL, Amazon S3 and Athena.
Experienced in distributed processing and the ingestion and transformation of large volumes of structured and semi-structured data. Skilled in building reliable, scalable data workflows, optimising Spark performance, and implementing data-quality, validation and reconciliation controls across S3-based data lake and lakehouse architectures. Comfortable working collaboratively in Agile environments, with experience of Azure DevOps, Git, code review and deployment processes.
الخبرة
Data Engineer
<p>Designed, developed and maintained production ETL/ELT pipelines using AWS Glue and Python/PySpark, processing 1bn+ attendance records weekly for 9,000+ schools.<br>
Led the expansion of the data engineering platform from 30–40 to 150+ Glue jobs/scripts across multiple workflows, supporting rapidly increasing data volumes and evolving business requirements.<br>
Optimised Spark-based pipelines through code review, Spark UI, performance analysis and targeted changes to partitioning, caching/persistence, broadcast joins and predicate/projection pushdown, reducing workflow runtime by 15–20% and eliminating recurring process failures.<br>
Introduced Apache Iceberg tables to facilitate and simplify student record updates, moving away from labour-intensive manual CSV uploads, thus improving efficiency<br>
Created automated Spark SQL data-quality and QA workflows to validate pipeline integrity and production outputs, identifying issues in third-party source data including UUID mismatches across slowly evolving datasets.<br>
Developed PySpark solutions using boto3, AWS Wrangler and Pandas, supporting data integration, S3 I/O and data-processing requirements across the platform.<br>
Supported application and data teams with PostgreSQL and SQL Server development, including stored procedures, database functions, data modelling and curated Athena/SQL Server analytical datasets for data warehousing, analysts and downstream reporting.<br>
Worked within Azure DevOps for source control, pull requests, deployment processes, database schema versioning using EF Core, and Agile software development.<br>
Provided code review and technical support to a junior Data Engineer.</p>
Data Manager
<p>Data lead on the national Reception Baseline Assessment pilot, processing 350,000+ pupil records; responsible for refining the data specification, processing, cleaning and delivery for statistical analysis<br>
Cleaned, transformed, and validated large educational datasets (SPSS, SSMS), producing analysis-ready outputs to strict stakeholder specifications<br>
Designed, built, and maintained complex online surveys (Questback, Qualtrics) for commissioned research projects, managing the full lifecycle from development through to data delivery</p>
Data and IT Systems Developer
<p>Developed and maintained the company's flagship education data analysis platform using VB.NET, ASP.NET, HTML and CSS, providing full-stack development across the application.<br>
Administered and optimised SQL Server databases across development and production environments, including database migrations and management of education and client datasets.<br>
Developed bespoke SQL queries and reports against sensitive pupil-level datasets, maintaining strict data security and integrity controls.<br>
Led the company's successful Cyber Essentials accreditation, developing security policies and supporting alignment with ISO 27001 requirements.</p>
Scheme Data Analyst
<p>Migrated private pension scheme data to in-house administration platforms, developing bespoke SQL queries for data mapping and transformation using SQL Server and MS Access.<br>
Conducted data audits, validation and QA checks, including control totals, to ensure integrity throughout multi-stage migration processes.<br>
Produced handover documentation and provided post-conversion support to administration teams and clients.</p>