Data Platform Engineer

We are seeking a Data Platform Engineer with strong quality engineering and automation expertise to support the development validation and reliability of large-scale data platforms. This role will focus on Databricks Lakehouse PySpark SQL distributed data pipelines event-driven architectures and cloud-based data ecosystems while also supporting automated testing data validation CI/CD quality gates and platform resilience.

The successful candidate will play a key role in ensuring end-to-end data integrity across complex systems validating data flows between applications and data platforms and supporting performance scalability and reliability initiatives. Working closely with cross-functional teams the individual will leverage strong Databricks Python and SQL expertise to investigate data quality issues validate platform behaviour and contribute to the continuous improvement of automation and engineering practices.

This is a hands-on technical role suited to a data engineering professional who can operate independently collaborate across multiple teams and apply quality engineering principles within a modern data platform environment. While testing and automation experience are important the primary focus is deep expertise in Databricks Python SQL and data platform engineering.

Responsibilities

• Design develop and maintain automated test frameworks and test suites.
• Build and enhance quality gates within GitLab CI/CD pipelines.
• Perform end-to-end testing of complex distributed data flows.
• Validate data integrity across RabbitMQ Databricks Lakehouse Lakebase and Elasticsearch platforms.
• Conduct functional integration regression and data validation testing.
• Collaborate with developers platform engineers and fellow SDETs to ensure seamless system integration.
• Coordinate cross-team testing activities across multiple application teams.
• Design and execute load performance and scalability testing strategies.
• Perform chaos testing and resilience validation of distributed systems.
• Investigate and troubleshoot data discrepancies across multiple storage systems.
• Contribute to continuous improvement of test automation practices and quality engineering standards.

Qualifications :

• Proven experience as a Senior Data Engineer Data Platform Engineer SDET Data Quality Engineer or similar technical role.
• Strong hands-on experience with Python PySpark and SQL in enterprise-scale environments.
• Demonstrated experience working with Databricks Lakehouse and large-scale distributed data processing platforms.
• Experience validating and testing complex data pipelines data integrations and distributed systems.
• Strong understanding of data quality data validation and automated testing methodologies.
• Experience with GitLab CI/CD and implementing automated quality gates within deployment pipelines.
• Experience working with Elasticsearch event-driven architectures and message brokers such as RabbitMQ.
• Knowledge of Infrastructure as Code (IaC) tools preferably Terraform.
• Strong analytical problem-solving and troubleshooting skills.
• Excellent communication and stakeholder management skills with the ability to collaborate across multiple teams.

• Desirable Qualifications & Experience

• Experience with performance load scalability resilience and chaos testing.
• Experience working within cloud-native and modern data platform environments.
• Exposure to DevOps and Site Reliability Engineering (SRE) practices.
• Knowledge of C# technologies.
• Experience working in Agile delivery environments.
• Relevant certifications in Databricks Cloud Technologies or Data Engineering would be advantageous.

Additional Information :

Why NCS

This is a place for people who like to get stuck in bring ideas to life and make things happen. Youll work alongside experienced people who care about what they do contribute to meaningful work and have the support to keep learning and grow your career.  Whether you want to deepen your expertise explore something new or take your career in a different direction theres room to make it your own.  We value Adventure Excellence Integrity Ownership and Unity - bringing curiosity collaboration and accountability to the way we work with our clients and each other.

Ready to make extraordinary happen

Wed love to hear from you.

We celebrate diversity and inclusion

We value different perspectives experiences and strengths our people bring.  Were committed to an inclusive workplace where everyone has the opportunity to contribute grow and succeed and to providing equal employment opportunities and reasonable adjustments throughout the recruitment process.

Important information

Applicants will need valid Australian work rights and may be required to undergo relevant background probity and police checks.

Agencies: NCS accepts candidate submissions only from agencies on our preferred supplier panel through the NCS Agency Portal.

Remote Work :

No

Employment Type :

Contract