Data Engineering Consultant

Role Overview

A leading technology consulting organization is seeking an experienced Data Engineering Consultant with deep expertise in the Microsoft Azure this role you will be responsible for architecting developing and sustaining scalable data frameworks that support enterprise-wide data needs. You will work across the full data lifecycle from ingestion and transformation through to governance and performance optimization enabling clients to derive value from their data assets reliably and efficiently.

Key Responsibilities

• Design build and maintain scalable end-to-end data solutions on Microsoft Azure tailored to business data requirements.
• Develop and manage data pipelines for ingesting transforming and integrating structured and unstructured data from diverse sources using Apache Spark.
• Implement robust data security governance and privacy controls across Azure data platforms in line with organizational standards.
• Monitor troubleshoot and continuously optimize data workflows to ensure high reliability performance and cost efficiency.
• Apply Azure best practices to optimize data storage and processing solutions across the platform.
• Collaborate with stakeholders to understand data requirements and translate them into technically sound scalable architectures.

Mandatory Technical & Functional Skills

• Bachelors degree in Computer Science a related field or equivalent practical experience with a minimum of 4 years in the domain.
• At least 3 years of consulting or client-facing service delivery experience in Azure-based Microsoft data engineering.
• Minimum 1 year of hands-on experience building data ingestion processing and analytics pipelines using SQL Server Azure Synapse Analytics Azure Databricks and Microsoft Fabric.
• Practical experience with ETL and data processing using Azure Data Factory Azure Functions Azure Data Lake Storage (ADLS) OneLake and Microsoft Fabric services.
• At least 1 year working with Azure and Big Data technologies including Microsoft Fabric Databricks Python SQL ADLS/Blob Storage Spark and Spark SQL.
• Minimum 1 year of hands-on experience with RDBMS technologies.
• Familiarity with big data file formats and compression techniques.
• Proficiency with developer tools including Azure DevOps Git and Visual Studio Team Services (VSTS).

Preferred Technical & Functional Skills

• Experience designing and deploying ETL solutions on Azure Cloud using Azure Data Factory Synapse Analytics Azure Functions and Notebooks.
• Hands-on experience with Microsoft Fabric including Medallion Architecture metadata-driven ingestion frameworks data quality frameworks and SCD Type 1 & Type 2 transformations via Fabric Notebooks or Azure Databricks.
• Exposure to Microsoft Copilot Generative AI fundamentals and industry-relevant AI use cases is an advantage.
• Relevant Microsoft role-based certifications are preferred (e.g. DP-600 DP-700 DP-203 DP-900 AI-102 AI-900).
• Working knowledge of Microsoft Power BI for developing reports dashboards and delivering actionable business insights.
• Understanding of Azure RBAC and IAM including access control principles and cloud security concepts.
• Genuine interest in Microsofts technology vision and roadmap with a drive to adopt emerging tools and capabilities.

Educational Qualifications

• Graduate or postgraduate degree in Computer Science Engineering or a related specialization.

Work Experience

• Minimum 4 years of hands-on experience in software development with a significant focus on Azure Data Engineering.