Cloud Engineer

We are looking for a specialised Cloud Engineer to build secure and maintain the underlying infrastructure for modern data platforms. Unlike traditional cloud engineering roles focused on general application hosting this position is dedicated to the unique requirements of data workloads including large-scale storage high-throughput networking and specialised compute clusters for Spark and SQL.

You will bridge the gap between DevOps and Data Engineering ensuring data scientists and analysts have a robust secure and automated platform for building pipelines and models. The role focuses on the core infrastructure of the data ecosystem primarily within Databricks AWS and Azure environments.

Responsibilties

Platform Architecture:

• Design and deploy scalable cloud infrastructure for data lakes and analytics platforms using Infrastructure as Code (Terraform)
  

Security & Governance:

• Implement identity management (IAM / Entra ID) network security (Private Link / VPC) and governance controls (Unity Catalog)
  

• Build CI/CD pipelines for infrastructure and data products.
  
• Automate provisioning of compute resources and workspaces
  

Observability:

• Monitor cost performance and reliability of data workloads
  

Enablement:

• Create self-service infrastructure patterns for Data Engineers
  

Requirements

Requirements

• Minimum of 5 years professional cloud engineering experience
  
• Experience in data engineering and Databricks is highly desirable
  

Core Skills Tools & Frameworks

• Terraform (modules state workspaces)
  
• CloudFormation Bicep or Crossplane advantageous
  

Containerisation & Orchestration:

• Docker and Kubernetes (EKS AKS or self-managed)
  

CI/CD & Version Control:

• Git Azure DevOps GitHub Actions
  

Scripting & Automation:

• Python and Bash
  
• DBX CLI AWS CLI Azure CLI REST APIs
  

Cloud Security:

• Least Privilege RBAC encryption secrets management
  

Cloud Networking:

• VNET/VPC design Private Link DNS Firewalls
  

Data Platform Architecture:

• Medallion architecture Delta Lake Data Vault
  

Observability:

• Cost management logging alerting
  

Data Workload Understanding:

• PySpark SQL dbt Spark Structured Streaming
  

Azure Platform Skills:

Databricks Platform & Infrastructure:

• Workspace automation via Terraform
  
• Unity Catalog configuration
  
• Cluster policies and instance profiles
  
• Private Link VNet Injection IP Access Lists
  
• Workflows DLT Airflow integrations
  
• MLflow Mosaic AI Vector Search infrastructure
  

General Cloud & DevOps:

• AWS Solutions Architect (Associate/Professional)
  
• Azure Solutions Architect Expert
  
• Terraform Associate
  
• Certified Kubernetes Administrator
  

Data & Platform:

• Databricks Data Engineer Professional
  
• AWS Data Engineer Associate
  
• Azure Data Engineer Associate (DP-203)