Senior Data Engineer

The Senior Data Engineer will own the data side of our client engagement in Doha. Joining the founding team in Doha and working day-to-day with the Senior Solutions Architect the successful candidate will bring the Client’s own data and new sector-specific sources into the platform extend the data model and ontology to cover them and ensure the data is reliable enough for AI workflows to reason upon.

The Senior Data Engineer will build and operate pipelines using Databricks and Azure Data Factory on an Azure-hosted platform while upholding agile engineering practices including unit testing automation and continuous integration and deployment through CI/CD.

The successful candidate will be someone who takes initiative stays curious and has an unrelenting drive to push the boundaries of AI and data science. They will be adaptive and introspective willing to learn guide lead and follow with a low-ego mindset where outcomes matter more than individual recognition.

This is an opportunity to play a key role in advancing AI capabilities within the financial technology sector working with highly experienced professionals as one of the first engineers in the Doha office and helping shape how the Client delivers solutions across the region.

As part of a fast-paced high-growth environment the successful candidate will have a direct impact on business growth and strategic direction with significant opportunities for personal and professional advancement while contributing to AI-driven financial solutions.

• Design develop and implement robust efficient and scalable data pipelines into the Client's data platform.

• Ingest structured data feeds APIs and unstructured documents.

• Build batch pipelines for pre-populated datasets.

• Build on-demand enrichment pipelines triggered by user requests and business requirements.

• Build and operate pipelines using Databricks and Azure Data Factory.

• Ensure all pipelines include unit testing automation and CI/CD.

• Develop production-grade pipelines that are tested automated repeatable and scalable from the outset.

• Onboard the Client's proprietary and licensed datasets into the platform.

• Establish data contracts schema mapping secure connectors and permission-based access controls.

• Work within agreed data residency security and governance requirements.

• Ensure client data is securely integrated and managed according to agreed access and residency rules.

• Extend the data ontology to accommodate each new data source.

• Map new sources across entities such as organizations people transactions funds and their relationships.

• Work with the Head of Data to determine whether new concepts should be represented as entities or attributes.

• Register new entities relationships and attributes within the field catalogue to ensure exposure through the platform API.

• Build the data layer for sector-specific ontology packs including new source types such as patents scientific literature regulatory records and project registries.

• Develop new entity types based on the Client's priority use cases.

• Ensure new data is consistently modelled and accessible through the platform API.

• Extend record-linkage and entity-resolution capabilities to new data sources.

• Ensure client and third-party records are accurately matched to the correct organizations and entities.

• Apply techniques such as fuzzy matching and blocking where appropriate.

• Measure matching precision and recall to validate data quality and resolution accuracy.

• Define and enforce data-quality checks across every pipeline.

• Monitor completeness freshness duplication and schema conformance.

• Implement alerting and dashboards to identify data issues proactively.

• Ensure issues are identified and addressed before reaching client-facing workflows.

• Evaluate the economic impact of every new data source.

• Estimate acquisition processing storage and refresh costs.

• Recommend whether sources should be pre-populated through batch processing or retrieved on demand.

• Ensure each data source has a clear cost and maintenance strategy.

• Develop comprehensive documentation for pipelines data contracts and operational runbooks.

• Ensure solutions can be operated and maintained by the wider data team and future hires in Doha.

• Build sustainable systems that do not depend on a single individual.

This role requires a combination of hands-on data engineering at scale strong data-quality practices and the ability to work directly with Client data. We are looking for someone who:

• Has a Master's degree at minimum in Computer Science Engineering or a related field.

• Has 7 years of significant Data Engineering experience excluding internships.

• Has excellent verbal and written English communication skills.

• Has outstanding programming skills in PySpark SQL and Python.

• Has proven experience processing data at terabyte scale in production environments.

• Is comfortable working with Git and modern version-control practices.

• Is familiar with document databases such as Cosmos DB and MongoDB graph databases such as Neo4j and ELK technologies.

• Has hands-on experience with Azure Cloud services particularly Databricks Azure Data Factory and Azure DevOps.

• Has a strong understanding of SQL and NoSQL database internals query languages and distributed computing fundamentals.

• Has experience building or operating entity-resolution and record-linkage solutions across messy and complex data sources.

• Has experience with fuzzy matching blocking and measuring match quality through precision and recall.

• Has designed schemas or ontologies for knowledge graphs or linked-data environments including entity-relationship modelling.

• Has delivered data integration for external clients or partners including data contracts secure data transfer and access controls.

• Learns quickly and demonstrates a continuous-improvement mindset.

• Is highly collaborative and team-oriented.

• Is proactive in identifying data issues and trade-offs and presenting recommendations.

• Is comfortable working directly with Client teams and explaining complex data issues to non-technical stakeholders.

• Can work autonomously within a small team while maintaining close collaboration with wider technical teams.

• Experience working in regulated or sovereign environments with data-residency requirements would be advantageous.

• Experience with financial services private markets energy or sustainability data would be valuable; however a proven track record of delivering reliable production-grade data engineering solutions is more important than experience in any specific domain.