Lead Data Engineer(pyspark)

ABOUT TIDE

At Tide we are building a business management platform designed to save small businesses time and money. We provide our members with business accounts and related banking services but also a comprehensive set of connected administrative solutions from invoicing to accounting.

Launched in 2017 Tide is now used by over 1 million small businesses across the world and is available to UK Indian and German SMEs. Headquartered in central London with offices in Sofia Hyderabad Delhi Berlin and Belgrade Tide employs over 2000 employees.

Tide is rapidly growing expanding into new products and markets and always looking for passionate and driven people. Join us in our mission to empower small businesses and help them save time and money.

ABOUT THE ROLE

We are seeking a highly skilled and experienced Senior Data Engineer with a deep expertise in PySpark to join our ML/Data engineering team. This team is responsible for feature development data quality checks deploying and integrating ML models with backend services and the overall Tide platform.
In this role you will be instrumental in designing developing and optimizing our next-generation data pipelines and data platforms. You will work with large-scale datasets solve complex data challenges and contribute to building robust scalable and efficient data solutions that drive business value.
This is an exciting opportunity for someone passionate about big data technologies performance optimization and building resilient data infrastructure.

As a Data Engineer youll be:

• Performance Optimization: Identify diagnose and resolve complex performance bottlenecks in PySpark jobs and Spark clusters leveraging Spark UI query plans and advanced optimization techniques (e.g. partitioning caching broadcasting AQE UDF optimization).
• Design & Development: Lead the design and implementation of highly scalable fault-tolerant and optimized ETL/ELT pipelines using PySpark for batch and potentially real-time data processing.
• Data Modeling: Collaborate with data scientists analysts and product teams to understand data requirements and design efficient data models (e.g. star/snowflake schemas SCDs) for analytical and operational use cases.
• Data Quality & Governance: Implement robust data quality checks monitoring and alerting mechanisms to ensure the accuracy consistency and reliability of our data assets.
• Architectural Contributions: Contribute to the overall data architecture strategy evaluating new technologies and best practices to enhance our data platforms capabilities and efficiency.
• Code Review & Best Practices: Promote and enforce engineering best practices including code quality testing documentation and version control (Git). Participate actively in code reviews.
• Mentorship & Leadership: Mentor junior data engineers share knowledge and contribute to a culture of continuous learning and improvement within the team.
• Collaboration: Work closely with cross-functional teams including software engineers data scientists product managers and business stakeholders to deliver impactful data solutions.

WHAT ARE WE LOOKING FOR

• 8 years of professional experience in data engineering with at least 4 years specifically focused on PySpark development and optimization in a production environment.
• Expert-level proficiency in PySpark including Spark SQL DataFrames RDDs and understanding of Sparks architecture (Driver Executors Cluster Manager DAG).
• Strong hands-on experience with optimizing PySpark performance on large datasets debugging slow jobs using Spark UI and addressing common issues like data skew shuffles and memory management.
• Excellent programming skills in Python with a focus on writing clean efficient and maintainable code.
• Proficiency in SQL for complex data manipulation aggregation and querying.
• Basic understanding of data warehousing concepts (dimensional modeling ETL/ELT processes data lakes data marts).
• Experience with distributed data storage solutions such as Delta Lake Apache Parquet etc.
• Familiarity with version control systems (Git).
• Strong problem-solving abilities analytical skills and attention to detail.
• Excellent communication and interpersonal skills with the ability to explain complex technical concepts to both technical and non-technical audiences.
• Bachelors or Masters degree in Computer Science Engineering or a related quantitative field.

WHAT YOU WILL GET IN RETURN

Make work work for you! We are embracing new ways of working and support flexible working arrangements. With our Working Out of Office (WOO) policy our colleagues can work remotely from home or anywhere in their assigned Indian state. Additionally you can work from a different country or Indian state for 90 days of the year. Plus youll get:

• Competitive salary
• Self & Family Health Insurance
• Term & Life Insurance
• OPD Benefits
• Mental wellbeing through Plumm
• Learning & Development Budget
• WFH Setup allowance
• 15 days of Privilege leaves
• 12 days of Casual leaves
• 12 days of Sick leaves
• 3 paid days off for volunteering or L&D activities
• Stock Options

TIDEAN WAYS OF WORKING

At Tide we champion a flexible workplace model that supports both in-person and remote work to cater to the specific needs of our different teams.

While remote work is supported we believe in the power of face-to-face interactions to foster team spirit and collaboration. Our offices are designed as hubs for innovation and team-building where we encourage regular in-person gatherings to foster a strong sense of community.

TIDE IS A PLACE FOR EVERYONE

At Tide we believe that we can only succeed if we let our differences enrich our culture. Our Tideans come from a variety of backgrounds and experience levels. We consider everyone irrespective of their ethnicity religion sexual orientation gender identity family or parental status national origin veteran neurodiversity or differently-abled status. We celebrate diversity in our workforce as a cornerstone of our success. Our commitment to a broad spectrum of ideas and backgrounds is what enables us to build products that resonate with our members diverse needs and lives.

We are One Team and foster a transparent and inclusive environment where everyones voice is heard.

At Tide we thrive on diversity embracing various backgrounds and experiences. We welcome all individuals regardless of ethnicity religion sexual orientation gender identity or disability. Our inclusive culture is key to our success helping us build products that meet our members diverse needs. We are One Team committed to transparency and ensuring everyones voice is heard.

• Tide doesnotcharge any fees at any stage of the recruitment process.
• All official Tide job opportunities are listed exclusively on ourCareers Pageand applications should be submitted through this channel.
• Communication from Tide will only come from an official@tide.coemail address.
• Tide does not work with agencies or recruiters without prior formal engagement and we do not authorize third parties to make job offers on our behalf.

If you are contacted by anyone misrepresenting Tide or requesting payment please treat it as fraudulent and report it to us immediately at
Your safety and trust are important to us and we are committed to ensuring a fair and transparent recruitment process.

You personal data will be processed by Tide for recruitment purposes and in accordance withTides Recruitment Privacy Notice.