Principal Data Engineer

Job Description

Lead AI & Data Platform Engineer

• Location: Melbourne (Hybrid Work Model)
• Salary: $180000 $210000 Super
• Industry: Financial Services
• Contract Type: Permanent / Full-Time

About the Role

We are seeking a high-caliber Lead AI & Data Platform Engineer to spearhead the evolution of our next-generation data this role you will bridge the gap between advanced Data Engineering and production-grade MLOps designing resilient petabyte-scale systems within a highly regulated Financial Services environment.

You will be our internal Databricks Expert driving the strategy and deployment of AI Agentic workloads advanced analytics pipelines and secure cloud infrastructure.

Key Responsibilities

1. Databricks Platform Excellence

• Architect and maintain our enterprise Lakehouse infrastructure utilising Delta Lake and Medallion topology (Bronze/Silver/Gold layers).
• Enforce robust data governance and secure discovery across the enterprise using Unity Catalog.
• Build monitor and scale production pipelines via Delta Live Tables.

2. Production AI & MLOps Infrastructure

• Deploy monitor and scale production-grade ML and AI Agentic workloads.
• Operationalise LLM and Generative AI pipelines utilizing Databricks Mosaic AI MLflow Langchain and Vector Databases.
• Manage corporate feature stores and optimize model serving endpoints.

3. AWS Cloud Architecture & Infrastructure as Code (IaC)

• Design secure high-performance cloud environments using Terraform.
• Manage intricate AWS configurations including S3 optimization VPC peering AWS PrivateLink and granular IAM roles.
• Integrate cloud-native AI services like AWS Bedrock seamlessly into our data ecosystem.

4. Performance Tuning & Streaming

• Conduct advanced Spark optimization to handle high-concurrency petabyte-scale data pipelines.
• Tune GPU clusters specifically optimized for heavy AI/ML training and inference workloads.
• Architect real-time streaming pipelines leveraging Kinesis or Kafka.

Your Profile

To be successful in this role you will bring a blend of elite engineering skills and a deep understanding of modern data-ops.

Technical Requirements:

• Core Languages: Mastery of PySpark Python SQL and Scala.
• AI Frameworks: Strong familiarity with PyTorch TensorFlow or Hugging Face.
• DataOps & CI/CD: Proven experience embedding automated testing and CI/CD pipelines into data deployments.
• Industry Experience: Prior experience within Financial Services or a similarly regulated environment is highly desirable (understanding of data privacy compliance and risk frameworks).

Whats on Offer

• Competitive Compensation: $180K $210K base salary depending on experience.
• Hybrid Flexibility: A healthy balance of remote work and collaborative time in our modern Melbourne CBD office.
• Cutting-Edge Tech Stack: Total backing to utilize the absolute latest in Generative AI Mosaic AI and cloud-native tools.