Senior Data Engineer Python and Terraform skills

Shift is the leading AI platform for insurance. Shift combines generative agentic and predictive AI to transform underwriting claims and fraud and risk - driving operational efficiency exceptional customer experiences and measurable business impact. Trusted by the worlds leading insurers Shift delivers AI when and where it matters most at scale and with proven results.

Our culture is built on innovation trust and a drive to transform the insurance industry through our SaaS platform. We come from more than 50 different countries and cultures and together we are creating the future of insurance.

Our Engineering team lies at the core of the value we offer to our customers. We solve complex problems not only within our squads but also by working collaboratively with other teams across the organization.

Were reshaping the way data drives decision-making by building powerful scalable software and data platforms. Whether youre passionate about distributed systems building APIs or enabling data-driven insights youll find exciting opportunities here.

Our tech stack Core (C#) Python MS SQL Server Spark Azure Data Factory DatabricksGoodData terraform and more. We are also progressing towards distributed data systems on cloud platforms with exciting challenges in both batch and real-time processing.

WHAT YOULL DO

• Drive end-to-end data architecture design for scalable and resilient systems.
• Design implement and maintain robust ETL/ELT pipelines.
• Own and optimize Apache Spark-based pipelines (debugging hardening tuning performance optimization).
• Translate business requirements into scalable data models and warehouse schemas.
• Improve pipeline performance reliability scalability and cost efficiency.
• Lead or contribute to POCs and new technology adoption initiatives.
• Ensure best practices in data governance quality observability and monitoring.
• Work closely with cross-functional teams (Analytics ML Product Platform).
• Mentor and support junior engineers providing technical guidance and code reviews.
• Take ownership of infrastructure provisioning and unblock yourself when needed.
• Actively Document and Diagram the technical design and architecture.

WHAT YOU BRING

• 8 years experience Tech Lead experience is a plus.
• Drive software architecture understand the data model/ETL mapping contribute to new tech adoption (POCs) and mentor peers.
• Apache Spark a must with experience debugging hardening and optimising pipelines
• 3 years of SQL database CosmosDB
• 3 years experience in custom ETL design implementation and maintenance
• 3 years experience with Data Modeling
• Strong understanding of:
• Distributed data systems
• Data warehousing concepts
• Batch and event-driven processing
• Performance and scalability troubleshooting

• Hands-on experience with:
• Databricks
• DBT
• Infrastructure as Code (Terraform ARM Bicep etc.)
• Azure data services (Event Hub Service Bus ADLS Synapse etc.)
• Experience provisioning infrastructure and setting up data pipelines independently.
• Bonus Experience with stores: DBT Analytical tools feature store vector database
• Nice to have: OLAP db GraphDB / SQL mesh (Niche)

HIRING PROCESS

• Talent Acquisition interview - 30 minutes
• Hiring Manager interview - 45 minutes
• Technical Screening with an Architect - 45 to 60 minutes
• Second Technical Screening with a Tech Lead or Data Engineer - 45 to 60 minutes
• Final Round with Hiring Manager - 60 minutes

#LI-RH1 #LI-HYBRID

To support our permanent full time employees at every stage of their careers and lives we provide a competitive total rewards and benefits package. Here are the global benefits wed like to highlight:

• Flexible remote and hybrid working options
• Competitive Salary and a variable component tied to personal and company performance
• Multiple Learning and Development opportunities including Focus Fridays a half-day each month to focus on learning and personal growth
• Generous PTO and paid holidays
• Mental health benefits
• 2 MAD Days per year (Make A Difference Days for paid volunteering)

Additional benefits may be offered by country based on your eligibility - ask your recruiter for more information. Intern and Apprentice positions may receive some of these benefits - ask your recruiter for more details.

At Shift we strive to be a diverse and inclusive workforce. We welcome applications from and hire people who will contribute to the diversity of our companywithout regard to race color religion marital status age national or ethnic origin physical or mental disability medical condition pregnancy genetic information gender identity or expression sexual orientation or other non-merit criteria.

Shift Technology is committed to providing reasonable accommodations for qualified individuals with disabilities in our application and employment process. Should you require accommodation please email and we will work with you to meet your accessibility needs.

Please be aware of scammers and only trust correspondence that comes from emails ending in . We will never do initial outreach to you via Whatsapp/Text/SMS never ask for banking information or personal identification numbers (ex. Social Security Number) as part of our recruitment process.

Shift Technology does not accept unsolicited CVs from recruiters or employment agencies in response to the Shift Technology Careers page or a Shift Technology social media post. Any unsolicited CVs including those submitted directly to hiring managers are deemed to be the property of Shift Technology.