Cloud Engineer

Responsibilities

Platform Architecture

• Design and implement scalable cloud infrastructure for data lakes and analytics platforms using Infrastructure as Code (Terraform)
• Define reusable modular infrastructure patterns for repeatable deployments across environments

Security & Governance

• Implement identity and access management (IAM / Microsoft Entra ID) with least-privilege principles
• Design secure network architectures using VPC/VNet Private Link and endpoint-based access
• Enforce data governance standards using tools such as Unity Catalog

Automation & Delivery

• Build and maintain CI/CD pipelines for infrastructure and data platform components
• Automate provisioning of compute workspaces and platform services
• Enable policy-as-code and automated compliance checks within deployment workflows

Observability & Cost Management

• Implement monitoring and alerting for performance reliability and cost efficiency of data workloads
• Establish dashboards and controls for usage optimisation and capacity planning

Data Platform Enablement

• Create self-service infrastructure capabilities for Data Engineering teams
• Enable seamless deployment of pipelines without direct infrastructure dependency
• Support standardised patterns for scalable data product delivery

Requirements

• Minimum 5 years of professional cloud engineering experience
• Strong exposure to data engineering environments and platforms (Databricks highly desirable)

Skills & Experience

Core Competencies

• Advanced expertise in Terraform (modules state management workspaces)
• Exposure to CloudFormation Bicep or Crossplane is beneficial

• Strong experience with Docker and Kubernetes (EKS AKS or equivalent)

• Advanced Git practices
• Experience with Azure DevOps and GitHub Actions for infrastructure automation and policy enforcement

• Strong proficiency in Python and Bash
• Experience with AWS CLI Azure CLI DBX CLI and REST APIs

• Role-based access control (RBAC) least privilege design
• Encryption in transit and at rest
• Secrets management (Key Vault / Secrets Manager)

• Design and implementation of VPC/VNet architectures
• Private connectivity (Private Link/Endpoints) DNS firewalls and segmentation strategies

• Understanding of Medallion architecture Delta Lake and Data Vault concepts

• Cost monitoring logging (CloudWatch / Azure Monitor) and alerting frameworks

• Familiarity with Spark PySpark SQL dbt and streaming workloads (e.g. Spark Structured Streaming)

Platform Expertise

AWS

• Storage: Advanced S3 configuration (lifecycle policies access points intelligent tiering)
• Identity: IAM role design cross-account access federation
• Networking: VPC architecture Transit Gateways Route53 Gateway Endpoints
• Serverless: Lambda and Step Functions automation
• Data Services: Glue Kinesis EMR

Azure

• Storage: ADLS Gen2 (ACLs hierarchical namespace) Blob Storage
• Identity: Microsoft Entra ID service principals managed identities
• Networking: VNets NSGs Private Link architectures
• Data Services: Azure Data Factory Synapse integration

Databricks Platform Engineering

• Workspace provisioning using Terraform provider
• Unity Catalog setup (metastores catalogs external locations)
• Cluster policies instance profiles and cost controls
• Secure connectivity (Private Link VNet injection IP access controls)
• Platform support for Jobs Delta Live Tables (DLT) and orchestration tools
• Infrastructure enablement for MLflow Mosaic AI and Vector Search

Certifications (Nice to Have)

• AWS Certified Solutions Architect Associate / Professional
• Microsoft Certified: Azure Solutions Architect Expert
• HashiCorp Certified: Terraform Associate
• Certified Kubernetes Administrator (CKA)

• Databricks Certified Data Engineer Professional (highly desirable)
• AWS Certified Data Engineer Associate
• Microsoft Azure Data Engineer Associate (DP-203)