Software Engineer Senior Consultant II (Hybrid)

At Allstate great things happen when our people work together to protect families and their belongings from lifes uncertainties. And for more than 90 years our innovative drive has kept us a step ahead of our customers evolving needs. From advocating for seat belts air bags and graduated driving laws to being an industry leader in pricing sophistication telematics and more recently device and identity protection.

Your role in the team

Allstates Data & Analytics Technology organization is seeking a highly skilled and hands-on Senior Data Engineer to design build optimize and support scalable reliable and high-performing data platforms and pipelines. This role is focused on developing enterprise data solutions that enable analytics reporting AI/ML and data-driven decision making across the organization.

The ideal candidate possesses strong expertise in modern data engineering practices Apache Spark cloud-based data platforms and lakehouse architectures. As a senior individual contributor the candidate will provide technical leadership mentor junior engineers drive engineering best practices and contribute directly to the design and implementation of enterprise-scale data solutions.

The role requires close collaboration with Analytics Engineers Data Scientists Product Managers Platform Engineers Security teams and business stakeholders to deliver trusted scalable and analytics-ready data products.

Key responsibilities:

• Design develop and maintain scalable batch and streaming data pipelines using Apache Spark cloud-native technologies and modern data platforms.

• Build and optimize ETL/ELT frameworks to ingest transform validate and curate data from diverse enterprise source systems.

• Develop and implement data solutions leveraging Microsoft Fabric OneLake Lakehouse Data Warehouse and Semantic Model capabilities.

• Design data models storage strategies and data processing architectures that support enterprise analytics and reporting requirements.

• Partner with Analytics Engineers BI teams Data Scientists and business stakeholders to deliver trusted reusable and analytics-ready datasets.

• Drive adoption of engineering best practices including code quality peer reviews automated testing CI/CD and documentation standards.

• Ensure data quality governance lineage metadata management security privacy and compliance requirements are incorporated into data solutions.

• Monitor and optimize performance scalability reliability and cost efficiency of distributed data processing workloads.

• Develop reusable frameworks libraries templates and engineering patterns that improve development productivity and consistency.

• Implement and support CI/CD pipelines Infrastructure-as-Code principles and deployment automation for data engineering solutions.

• Troubleshoot complex data integration performance and operational issues across platforms and pipelines.

• Collaborate with platform infrastructure architecture and security teams to build resilient scalable and secure enterprise data solutions.

• Create and maintain technical documentation architecture diagrams operational runbooks and implementation guides.

• Participate in technical design sessions architecture reviews and engineering discussions while providing recommendations and solutions.

• Mentor and guide junior and mid-level data engineers through technical coaching code reviews design reviews and knowledge sharing.

• Evaluate and adopt emerging technologies engineering practices and AI-assisted development capabilities to improve delivery velocity and quality.

• Support production operations through monitoring root cause analysis incident resolution and continuous improvement initiatives.

• Contribute to Agile delivery processes including sprint planning estimation backlog refinement and execution of engineering commitments.

• Promote DataOps practices automation observability and engineering excellence across development teams.

• Serve as a technical subject matter expert for modern data platforms Spark processing Microsoft Fabric and lakehouse architectures.

Essential Skills:

• All applicants must demonstrate they have a legal right to work in the UK for employment at Allstate. Allstate is not providing sponsorship for this vacancy.
• A minimum of 2 years of experience in Data Engineering and Big Data technologies.
• Deep expertise in Apache Spark (PySpark/Spark SQL).
• Experience with Microsoft Fabric OneLake Lakehouse Architecture Data Warehouse and building Semantic Models.
• Strong SQL and data modeling skills.
• Experience building and optimizing ETL/ELT pipelines.

Desirable Skills:

• Strong proficiency in Java and experience building distributed applications and services.
• Experience with Kafka or similar event-streaming platforms.
• Understanding of Data Governance Data Quality Security and Privacy controls.
• Ability to mentor junior engineers and provide technical leadership without direct people management responsibilities.
• Knowledge of streaming and batch data processing patterns.
• Experience with observability solutions including logging monitoring tracing and performance analysis.
• Experience working in cloud-based analytics and data platform environments.
• Experience with CI/CD Git-based development testing and deployment automation.
• Experience working in Agile delivery environments.

Job Posting End Date: Friday 11th September

#LI-TR3

Skills

Software Development

Why join us
Allstate NI is proud to be Allstates European Digital Centre of Excellence a hub for innovation and engineering excellence. Were recent winners of Best Place to Work in IT (100 employees) and Best Use of Cloud Services at the Belfast Telegraph IT Awards and weve been recognised for our community and sustainability impact with Platinum in the Northern Ireland Environmental Benchmarking Survey.

Were a product-driven cloud-first organisation delivering real outcomes through modern technology a digital product-centric talent model and a culture rooted in engineering excellence. Our teams work in cross-functional structures guided by an outcome-based delivery approach that accelerates speed agility and value.

We also invest in you. At Allstate NI your career growth matters. Youll have access to our Continuous Learning Hub designed to support skills development and professional advancement through tailored learning paths certifications and mentoring opportunities. Whether youre deepening technical expertise or exploring leadership roles we provide the tools and support to help you thrive.

As well as receiving a competitive annual salary our reward package includes:

• Corporate bonus scheme
• Pension scheme
• Annual performance-related pay reviews
• Life assurance and income protection
• Flexible working options
• Hybrid working
• Private medical and dental insurance
• Access to an employee assistance programme
• Discounted gym membership
• Two paid volunteering days each year
• Cycle to work scheme

Be part of a high-performing socially responsible organisation where your work has purpose and your growth is supported every step of the way.