Lead – DataOps

About us:

We are proudly part of a privately owned global group of insurance and online comparison related businesses with approximately 50 offices and 10000 employees. In Australia the Group includes trusted and well-known brands Budget Direct Ozicare Chasing Cars iSelect and Compare the Market. In late 2024 iSelect and Compare the Market formed a new Aggregation business within the group and now support millions of Australians to compare and buy their personal finance and household products such as insurance energy and loans.

What you do:

The Lead - DataOps is responsible for designing implementing and maintaining highly available scalable and secure data pipelines and platforms. This role ensures efficient CI/CD processes infrastructure automation and monitoring to support the organizations data ecosystem.

The role requires a strong background in DataOps DevOps cloud data platforms and infrastructure automation. They will work closely with Data Engineering Security IT Operations and Analytics teams to streamline data workflows enhance system reliability and optimize cloud-based data environments.

The primary focus areas include infrastructure as code (IaC) CI/CD for data pipelines monitoring & observability cost optimization and cloud security.

Data Platform & Pipeline Automation

• Automate data pipeline deployments monitoring and scaling to ensure reliability and performance.
• Design implement and maintain CI/CD pipelines for data engineering workflows.
• Ownership of the platforms data operations specifically end-to-end troubleshooting of failed pipelines adding more observability/robustness/scalability there whether related to CI/CD OR data workflows.
• Develop and enforce infrastructure-as-code (IaC) practices using tools like Terraform AWS CloudFormation or Azure Bicep.
• Work with Data Engineers and IT teams to optimize ETL/ELT processes and cloud data platforms.

Cloud Infrastructure & Security

• Oversee data infrastructure in cloud environments (AWS Azure GCP) ensuring high availability and security.
• Implement role-based access control (RBAC) encryption and network security best practices within data environments.
• Collaborate with Security and Compliance teams to ensure regulatory compliance
• Optimize cloud storage compute and networking costs while maintaining performance

Monitoring Observability & Incident Response

• Implement monitoring solutions for real-time alerting anomaly detection and performance optimization.
• Utilize logging and observability tools such as Prometheus Grafana Datadog Splunk or CloudWatch.
• Establish incident response processes including automated rollback strategies and disaster recovery plans.
• Work with Site Reliability Engineering (SRE) teams to ensure high system availability and minimal downtime.

Data Ops & DevOps Best Practices

• Advocate for DataOps methodologies to improve collaboration between Data Engineering IT and Analytics teams.
• Implement version control automated testing and release management for data pipelines.
• Continuously improve CI/CD workflows to enhance deployment frequency and reduce failures.
• Enforce best practices for data governance lineage tracking and auditability.

Collaboration & Stakeholder Engagement

• Work closely with Data Engineers Analytics Security and IT Operations to streamline data infrastructure.
• Act as the subject matter expert (SME) on Data DevOps providing training and mentorship to team members.
• Engage with business leaders and data users to align infrastructure improvements with business goals.

What you need:

• Bachelors or Masters degree in Computer Science Data Engineering Cloud Computing or a related field/relevant experience.
• Experience & Skills:
• 3-5 years of experience in DataOps Data Engineering or Cloud Infrastructure roles.
• Hands-on experience with CI/CD tools (e.g. GitHub Actions Jenkins AWS).
• Proficiency in containerization and orchestration (Docker Kubernetes ECS EKS AKS) as plus.
• Experience implementing infrastructure as code (IaC) using Terraform
• Expertise in cloud data platforms (AWS Azure GCP) with services like AWS Lambda S3 RDS Redshift Glue EMR Databricks BigQuery etc.
• Strong understanding of cloud networking security policies and identity management (IAM VPC KMS Secrets Manager etc.).
• Experience setting up monitoring and alerting using tools like Prometheus Grafana CloudWatch Datadog etc.
• Ability to troubleshoot performance issues and optimize system reliability.
• Strong experience in Python Bash or PowerShell for automation and scripting.
• Ability to write automation scripts for infrastructure provisioning deployments and data pipeline management.
• Ability to lead a small team and mentor junior engineers and promote DataOps best practices across teams.
• Strong communication skills to work with both technical and non-technical stakeholders

Whats in it for you:

• Vibrant and social community who have annual celebrations family fun days and regular events.
• Enjoy flexible work arrangements including the option of one day working from home each week or a 9-day fortnight.
• Enjoy additional leave days - ME leave and Volunteer Day leave.
• Employee discounts on Car Home Travel insurance.
• Income protection insurance provided to support you in the event of non-work-related illness or injury.

If you are someone who fosters team spirit has resilience and aspires to continue learning and developing their skills to drive commercial outcomes you will have tremendous success in helping us achieve our progressive growth agendas. High energy levels and a love of having fun are also important to working in this energetic business.

InterestedWed love to hear from you!

At Compare the Market and iSelect and were proud to be an Equal Opportunity Employer and we embrace diversity.

This role is available to candidates with unlimited Australian working rights only. Sponsorship is not available.

Compare the Market and iSelect are committed to protecting your privacy. All applications and personal information are kept confidential and securely stored in line with Australian privacy laws used only for recruitment purposes and not shared with third parties outside the Auto & General Holdings Group without your consent.