Microsoft Fabric Data Engineer

Role Overview:

The Microsoft Fabric Data Engineer is responsible for designing building optimizing and operating enterprise-scale data platforms on Microsoft Fabric. This role focuses on data ingestion transformation storage governance automation and platform reliability using OneLake Lakehouse Data Warehouse Data Pipelines and Spark technologies.

The position is heavily focused on Data Engineering and Platform Engineering rather than reporting and dashboard development. The successful candidate will build scalable governed and high-performance data solutions that support analytics AI operational reporting and business intelligence initiatives across the organization.

This role is primarily focused on Data Engineering Data Platform Development and Microsoft Fabric architecture. Candidates whose experience is primarily centered around Power BI report development dashboard creation or data visualization without substantial Data Engineering experience may not be a fit for this position.

Key Responsibilities

1. Data Engineering & Platform Development

• Design and implement enterprise-scale Lakehouse architectures using OneLake and Delta Lake.
• Build and maintain robust batch incremental CDC and near real-time data ingestion pipelines.
• Develop scalable ETL/ELT solutions using Fabric Data Pipelines Dataflows Gen2 PySpark and SQL.
• Implement and manage Medallion Architecture (Bronze Silver Gold).
• Develop reusable and metadata-driven ingestion and transformation frameworks.
• Integrate data from ERP systems SAP REST APIs SQL Server Dataverse and other enterprise applications.
• Design and maintain enterprise data models supporting analytical and operational workloads.

2. Fabric Engineering & Optimization

• Develop and optimize Fabric Lakehouses and Data Warehouses.
• Build advanced Notebook-based transformations utilizing PySpark and Spark SQL.
• Implement Delta Lake capabilities including:
  • Merge/Upsert
  • Change Data Feed (CDF)
  • Time Travel
  • Schema Evolution
  • Optimize
  • Vacuum
• Design scalable storage partitioning and file management strategies within OneLake.
• Optimize Spark workloads and Data Warehouse performance.

3. Performance Reliability & Monitoring

• Tune large-scale Spark and SQL workloads.
• Implement monitoring alerting and operational dashboards using Fabric Monitoring Hub and Metrics.
• Conduct performance testing and scalability assessments.
• Troubleshoot pipeline failures and platform performance bottlenecks.
• Define and monitor SLAs for critical data assets and pipelines.
• Ensure high availability and operational excellence across data workloads.

4. DevOps & Platform Automation

• Implement CI/CD using Fabric Git Integration and Deployment Pipelines.
• Build automated deployment processes across Development Test and Production environments.
• Develop automated testing frameworks for data quality schema validation and regression testing.
• Manage environment configurations secrets and deployment parameters.
• Utilize Azure DevOps or GitHub Actions to support release automation and governance.
• Define rollback release management and change control processes.

5. Data Quality Governance & Security

• Implement automated data quality validation frameworks.
• Develop reconciliation completeness and consistency checks.
• Manage data lineage metadata and documentation standards.
• Implement sensitivity labels access controls auditing and compliance requirements.
• Ensure adherence to organizational standards for governance security retention and PII handling.
• Manage workspace permissions Managed Identities RLS and CLS where required.

6. Technical Leadership & Collaboration

• Translate business requirements into scalable technical solutions.
• Collaborate with Data Architects Data Analysts Integration Engineers and business stakeholders.
• Lead engineering best practices and platform standards.
• Mentor junior engineers and support knowledge-sharing initiatives.
• Produce architecture documentation and technical design specifications.

Drive continuous improvement of the data platform and engineering practices.

Qualifications :

Qualifications and Experience:

• Bachelors degree in Computer Science Data Engineering Information Systems or a related field.
• Minimum 5 years of experience in Data Engineering.
• Minimum 2 years of hands-on experience with Microsoft Fabric Azure Data Engineering Databricks or modern Lakehouse platforms.
• Proven experience designing and building enterprise-grade data platforms.

Required Technical Skills

Microsoft Fabric

• OneLake
• Lakehouse
• Delta Lake
• Fabric Data Warehouse
• Data Pipelines
• Dataflows Gen2
• Notebooks
• Monitoring Hub
• Deployment Pipelines
• Git Integration

Data Engineering

• PySpark
• Spark SQL
• SQL / T-SQL
• ETL / ELT Development
• Data Modeling
• Incremental Processing
• Change Data Capture (CDC)
• Data Quality Frameworks
• Medallion Architecture

Performance Optimization

• Delta Lake Optimization
• Partitioning Strategies
• File Management
• Query Optimization
• Spark Performance Tuning
• Workload Management

Integration

• REST APIs
• SQL Server
• SAP
• Azure Storage
• Dataverse
• Enterprise Application Integration

DevOps

• Git
• Azure DevOps
• GitHub Actions
• YAML Pipelines
• CI/CD
• Automated Testing

Security & Governance

• Row-Level Security (RLS)
• Column-Level Security (CLS)
• Sensitivity Labels
• Managed Identities
• Data Lineage
• Audit and Compliance Controls

Additional Information :

Preferred Qualifications

• Microsoft Fabric DP-600 Certification.
• DP-203 Azure Data Engineer Associate Certification.
• Experience with Microsoft Purview.
• Experience with Eventstream and Real-Time Intelligence.
• Experience with Azure Databricks.
• Exposure to AI and Machine Learning data engineering workloads.
• Experience implementing enterprise data governance frameworks.
• Familiarity with Infrastructure as Code (Terraform/Bicep).

Remote Work :

No

Employment Type :

Full-time