Team Lead Databricks Engineer

Team Lead - Data Engineer Databricks

About Citco:

Since the 1940s Citco has provided specialist financial services to alternative investment funds investors multinationals and private clients worldwide. Our data engineering initiatives represent one of our most critical technical focus areas driving innovation across our global operations.

About the Role:

As a Team Lead you willlead a technical teamof data engineers ensuring best practices in performance security and scalability while working on Citcos enterprisewide Centralized Data Platform (CDP) built on Databricks.

This role requires adeep hands-on understanding of Databricksinternals and a track record ofdelivering large-scale data platformsin a cloud environment.

Youll be part of a team driving technical excellence and innovation within Citcos data engineering practice.

Key Responsibilities:

• Lead and mentor a team of data engineers conducting code reviews design reviews and knowledge-sharing sessions across multiple locations

• Drive the Agile/Scrum SDLC process and collaborate with team members
• Design and develop Databricks solutions leveraging Lakehouse architecture for enterprise data processing and analytics
• Develop and optimize ETL/ELT pipelines
• Create and manage structured streaming pipelines for real-time data processing
• Configure and optimize Databricks clusters and Spark jobs for optimal performance
• Utilize Delta Live Tables fordata ingestion and transformations
• Apply Unity Catalog features and IAM best practices for security governance and access control
• Support infrastructure and resource management using Terraform
• Implement monitoring solutions for pipeline performance and data quality
• Contribute to code reviews and knowledge-sharing sessions

Required Qualifications:

8 years of experience in data engineering

3 years of hands-on experience with Databricks platform

Proven experience leading a team

Strong expertise in Python and Spark programming

Demonstrable experience in using AI in development

Proven experience with AWS or other similar cloud services

Deep understanding of data modeling and SQL

Experience with Delta Lake and Lakehouse architecture

Strong knowledge of ETL/ELT principles and patterns

Experience with version control systems (Git)

Demonstrated ability to optimize data pipelines

Strong problem-solving and analytical skills

Excellent communication and collaboration abilities

Nice to Have:

Financial services industry experience

Experience with multiple cloud providers

Knowledge of AI/ML implementation patterns

API development experience

Experience with real-time data processing

Data governance framework experience

What Sets You Apart:

Active interest in emerging technologies and industry trends

Experience implementing complex data solutions

Track record of continuous learning and skill development

Strong technical documentation abilities

Experience using AI tools to enhance productivity

Ability to mentor junior team members

Technical Environment:

Primary Platform: Databricks

Cloud Platform: AWS (S3 Glue Lambda)

Languages: Python SQL

Tools: Delta Lake Unity Catalog Git

Additional: Real-time processing API integrations

Our Benefits:

Your well-being is of paramount importance to us and central to our success. We provide a range of benefits training and education support and flexible working arrangements to help you achieve success in your career while balancing personal needs. Ask us about specific benefits in your location.

We embrace diversity prioritizing the hiring of people from diverse backgrounds. Our inclusive culture is a source of pride and strength fostering innovation and mutual respect.

Citco welcomes and encourages applications from people with disabilities. Accommodation is available upon request for candidates taking part in all aspects of the selection.

A