Lead Azure Databricks Engineer

Anticipated Contract End Date/Length: December 31 2026
Work set up: Hybrid

Our client in the Information Technology and Services industry is looking for a Lead Azure Databricks Engineer to drive the design build optimisation and governance of enterprise data platforms within a highly regulated insurance environment. This is a 100% hands-on role requiring deep technical expertise combined with strong leadership capability.

The successful candidate will operate within complex fast-paced settings and bring proven delivery experience aligned to Lloyds of London regulatory and operational requirements. This role will play a pivotal part in shaping cloud data strategy enabling advanced analytics and delivering secure scalable and high-performing solutions across the organisation.

What you will do:

• Lead the development and optimisation of large-scale data platforms using Azure services including ADF Data Lake Key Vault and Azure Functions alongside Databricks technologies such as Delta Lake PySpark and Unity Catalog
• Collaborate with underwriting actuarial delegated authority bordereaux exposure management reinsurance finance Solvency II and risk teams to deliver compliant and auditable data solutions
• Translate business and regulatory requirements into scalable secure cloud data architectures
• Own and resolve complex technical challenges with clarity speed and accountability
• Champion best-practice design patterns Medallion architecture Lakehouse principles and data lifecycle management
• Implement CI/CD pipelines using Azure DevOps including branching and merging strategies for data solutions
• Drive automation and cloud-native engineering principles across the platform
• Provide technical leadership and mentorship setting engineering standards and uplifting team capability
• Identify opportunities to improve performance reduce cost and enhance resilience through continuous improvement initiatives
• Oversee end-to-end data engineering delivery ensuring reliability observability data quality and performance optimisation

Qualifications :

• Expert-level engineering experience across Azure Data Services and Databricks in enterprise-scale environments
• Deep understanding of Delta Lake Medallion architecture distributed compute Lakehouse patterns data modelling and optimisation
• Strong proficiency in Python PySpark and Spark SQL
• Experience implementing CI/CD for data engineering solutions using Azure DevOps
• Solid understanding of data governance lineage access controls FinOps and secure cloud engineering practices
• Proven experience delivering within the Lloyds of London market or similarly regulated insurance environments
• Excellent communication skills with the ability to engage senior stakeholders and lead cross-functional discussions
• Demonstrated ownership accountability and consistent delivery in high-expectation fast-paced environments

Additional Information :

All your information will be kept confidential according to EEO guidelines.

Candidates must be legally authorized to live and work in the country where the position is based without requiring employer sponsorship.

HelloKindred is committed to fair transparent and inclusive hiring practices. We assess candidates based on skills experience and role-related requirements.

We appreciate your interest in this opportunity. While we review every application carefully only candidates selected for an interview will be contacted.

HelloKindred is an equal opportunity employer. We welcome applicants of all backgrounds and do not discriminate on the basis of race colour religion sex gender identity or expression sexual orientation age national origin disability veteran status or any other protected characteristic under applicable law.

Remote Work :

No

Employment Type :

Contract