MS Fabric Data Engineer

Deadline for Applications: 22 April 2025 18:00 (CET)

As part of a broader Digital Transformation Accelerator Program CGIAR System Organization is transitioning its data infrastructure to Microsoft Fabric. This includes migrating to industrystandard data engineering practices in MS Fabric including modern architecture based on Medallion architecture deployment pipeline GIT integration and others.

CGIAR has already set up a standard and solid architecture in MS Fabric and built a solution to collect same data domain (personnel) records from different federated data sources to curate them and stack them in a data Warehouse. This solution requires ongoing improvements and requires us to integrate additional data sources to ensure it is in a functional format.

CGIAR seeks a Data Engineer consultant with handson experience in MS Fabric to lead the improvements of the existing implementation but also should help design implementation and migrate additional data assets and data solutions to this new architecture.

Scope of Work

Enhancements and fixes to existing data solutions based on dataflows and notebooks to collect curate correct stack and assign global unique ID to stacked records.

Drive the adoption and implementation of standardized data engineering practices within Microsoft Fabric.

Migrate existing data assets (Power BI dataflows datasets warehouses dashboards) into the new MS Fabric based architecture.

Ensure clear separation between data backend and frontend layers using the DEV TEST and Production workspaces and medallion architecture (Bronze Silver Gold layers).

Establish robust deployment pipelines for Fabric artifacts (e.g. Lakehouses Dataflows Datasets Dashboards).

Support capacity monitoring diagnostics and tuning by leveraging MS Fabric capacity metrics and best practices.

Qualifications and Experience

Essential:

Proven experience in data engineering within Microsoft Fabric ecosystem including Python Notebooks Warehouses Lakehouses Data Pipeline Deployment Pipeline and others.

Deep understanding of data quality concepts and how to apply them in MS Fabric using Python / Pandasbased Notebooks.

Strong understanding and implementation experience of the Medallion architecture pattern.

Demonstrated ability to migrate and refactor Power BI assets (dataflows datasets dashboards).

Proficient in DAX SQL and Power Query (M).

Familiarity with source control (e.g. Azure DevOps GitHub) and structured deployment practices.

Proficiency in English Language.

Desirable:

Experience with MS Fabric capacity planning and monitoring.

Familiarity with Fabric Admin roles and workspace governance.

Understanding of Microsoft Purview or other data governance tools.

DP600 certification or similar.

Experience in capacity building / training to bring other technical members to the same level of expertise.

Deliverables

1. Enhanced and fixed issues on the personnel database solution including the generation of global unique ID and the inclusion of 6 additional data sources

2. All existing workspaces are migrated to new architecture (dataflows dashboards and warehouses) from legacy workspaces (based on PowerBI capacity) to new workspaces (based on Fabric Capacity) including dataflows datasets and dashboards.

3. Implemented deployment pipelines and Git integration for versioning.

4. Recommendation on improvements on Fabric utilization based on capacity monitoring dashboard/report and usage analysis.

5. User guides technical documentation and knowledge transfer materials and delivery (training).

Performance Measurement

1. Key Performance Indicators (KPIs)

Documentation quality: Thorough and clear documentation.

Training satisfaction: Satsifaction rating from the training delivered to end users.

Working solution: Solution is working automatically and free of bugs.

Timeliness: The consultant will be evaluated on their ability to meet deadlines and milestones as outlined in the timeline.

2. Performance Review

Weekly meetings with the consultant will be carried out to review project plan obstacles and adjustments to the timeline. During the meetings feedback will be provided to steer performance and prioritize objectives if needed.

Evaluation Criteria

Certification in MS Fabric (e.g. DP600

Years of experience with MS Fabric data engineering

Years of experience with data science and Python

Years of experience with data management and data quality controls using MS Fabric notebooks

Confidentiality

As a condition of engagement with CGIAR the Data Engineer (the Consultant) acknowledges and agrees to the following confidentiality obligations relating to the access use handling and protection of personal data and other sensitive information:

1. Confidential Information
   The Consultant understands that during the course of engagement they may have access to confidential proprietary and sensitive information including but not limited to:
• Personal data as defined under applicable data protection laws (e.g. GDPR DPA 2018;
• Technical and business information relating to systems software datasets infrastructure and security;
• Any information documentation or datasets marked or reasonably understood to be confidential.
2. Use and Disclosure
   The Consultant agrees to:
• Use confidential information solely for the purpose of fulfilling their duties and responsibilities;
• Not disclose share copy transmit or otherwise make available any confidential information to unauthorized persons within or outside the organization either during or after the period of engagement;
• Handle personal data strictly in accordance with internal policies contractual obligations and applicable data protection laws and regulations.
3. Data Protection Compliance
   The Consultant commits to:
• Adhere to the organizations Data Protection Policy Acceptable Use Policy and Information Security Protocols;
• Complete mandatory data protection and information security training as required;
• Promptly report any actual or suspected data breaches unauthorized access or misuse of personal data.
4. Return or Destruction of Information
   Upon termination of egagement or upon request by the organization the Consultant shall return or securely destroy all confidential and personal data in their possession in accordance with organizational policies and procedures.
5. Survival of Obligation
   These confidentiality obligations shall survive the termination of the Consultants engagement for any reason and remain in effect until such confidential information becomes publicly available through no fault of the Consultant.
6. Breach and Remedies
   Any breach of this confidentiality obligation may result in disciplinary action including termination of consultancy and may lead to civil or criminal liability. The organization reserves the right to pursue legal remedies to protect its interests.

All applications must be submitted online by clicking the Apply button below. If you require assistance or face challenges in submitting your application please email with the position title in the subject line. Please note we will not accept applications through this email.

Please ensure that your resume and cover letter are in English and do not contain your marital status age or photograph. Documents provided in a language other than English will not be considered.

CGIAR is committed to fair safe and inclusive workplaces. We believe that diversity powers our innovation contributes to our excellence and is critical for our mission. We offer a multicultural multicolor multigenerational and multidisciplinary collegial working environment. We consciously create an inclusive organization that reflects our global character and commitment to gender equity. We therefore encourage applicants from all cultures races ethnicities religions sexes national or regional origins ages disability status sexual orientations and gender identities.

All received applications will be acknowledged; however only shortlisted applicants will be contacted.

We look forward to hearing from you!