Data Engineer – Azure Databricks & Fabric

About the team:

Capcos Data Team helps our clients transform every aspect of their business. We are highly skilled at formulating data strategy defining business and technology initiatives across the data management lifecycle and aligning multi-year strategic roadmaps with clients business goals. As digital technologies advance and regulations tighten todays consumers and therefore todays businesses are becoming more aware of the importance of good quality data. We work to establish holistic ways to effectively manage data through the modern data supply chain and facilitate consumption through analytics modelling AI machine learning dashboarding and reporting.

About the Job:

The Data Engineer will serve as the lead technical specialist for designing and implementing data science and advanced analytics capabilities on Microsoft Azure Fabric and Databricks. This role focuses on data processing identity resolution entity linking and data warehouse development that enable organizations to unify fragmented data across multiple systems into a trusted governed and analytics-ready model. The ideal candidate combines deep hands-on expertise in Databricks engineering data modeling and applied data science with the ability to build scalable production-grade data solutions in collaboration with business engineering and analytics teams.

What Youll Get to Do:

Data Platform & Warehouse Development

• Design and develop data lakehouse and warehouse structures within Azure Databricks and Fabric environments.

• Build ETL and ELT pipelines to extract cleanse normalize and enrich data from CRM ERP LMS and financial systems .

• Develop reusable data transformation and validation frameworks leveraging PySpark SQL and Delta Live Tables.

• Support the operationalization of the central data warehouse using Azure SQL and Fabric Data Warehouse.

Identity Resolution & Data Linking

• Implement entity resolution models to unify customer member or participant records across systems using deterministic and probabilistic matching techniques.

• Design and deploy matching algorithms utilizing Databricks MLflow PySpark and Azure Machine Learning for cross-system deduplication and linkage.

• Collaborate with architects to define unique identifiers external keys and golden record frameworks for enterprise data integration.

• Monitor and continuously refine data matching accuracy precision and recall metrics.

Data Processing & Automation

• Develop and schedule data ingestion pipelines in Azure Fabric and Databricks for recurring Excel CSV and structured PDF sources using Power Automate Form Recognizer and Fabric Dataflows.

• Apply data quality and validation rules to flag incomplete inconsistent or stale records.

• Build and automate data lineage change tracking (CDC) and error-handling workflows.

• Support performance tuning and scalability for high-volume processing environments.

Analytics & Modeling Support

• Provide curated and feature-engineered datasets for Power BI dashboards and machine learning use cases.

• Partner with data analysts to define KPIs and enable cross-system reporting and predictive insights.

• Develop scripts and notebooks to support exploratory data analysis (EDA) and visualization in Databricks.

What Youll Bring with You:

• BA in Data Science Computer Science Applied Mathematics or related discipline.

• 5 years of experience in data engineering and applied data science on Azure platforms.

• 3 years building and managing pipelines in Azure Databricks (PySpark Delta Lake MLflow).

• 2 years hands-on experience with Microsoft Fabric (Data Factory Dataflow Gen2 Data Warehouse).

• Power BI integration and data modeling

• Entity resolution and master data management (MDM) methods

• Statistical modeling clustering and record linkage algorithms

• Data governance lineage tracking and compliance (PII HIPAA etc)

• Proven track record implementing identity resolution and entity linking frameworks.

• Strong background in SQL Python and large-scale data processing for analytics.

Preferred Certifications:

• Microsoft Certified: Fabric Analytics Engineer Associate

• Microsoft Certified: Azure Data Scientist Associate

• Databricks Certified Machine Learning Professional

• Azure Data Engineer Associate

Why Capco

A career at Capco is a chance to help reshape the competitive landscape in financial services. We launch new banks transform existing ones and help our clients navigate complex change. As consultants we work on the front-end business design all the way through to technology implementation.

We are the largest Financial Services focused consultancy in the world serving everyone from global banks to emerging FinTechs from strategy through digital transformation design business consulting data and analytics cyber cloud technology architecture and engineering.

Capco is a young and growing firm. We maintain an entrepreneurial spirit and growth mindset and have minimal bureaucracy. We have no internal silos that get in the way of your career opportunities or ability to focus on our clients and make a difference to the business. We offer the opportunity for everyone to learn rapidly take on tough challenges and get promoted quickly. We take pride in our creative collaborative diverse and inclusive culture where everyone can #BYAW.

We offer highly competitive benefits including medical dental and vision insurance a 401(k) plan tuition reimbursement and a work culture focused on innovation and creation of lasting value for our clients and employees.

Ready to take the Next Step

If this sounds like you we would love to hear from you. This is an opportunity to make a difference and contribute to a highly successful company with a significant growth trajectory.