Lead Databricks Engineer

As a Lead Databricks Engineer you will play a critical role in delivering large-scale data transformation programs. You will lead the design and implementation of modern Lakehouse platforms on Databricks working closely with client stakeholders architects and cross-functional teams. This is a client-facing leadership role requiring strong consulting capabilities technical depth and delivery ownership.

The Lead Databricks Engineer is responsible for designing and implementing scalable data platforms using Databricks Apache Spark and Delta Lake including building robust batch and real-time data pipelines and developing ingestion frameworks for both structured and unstructured data while optimising performance cost and reliability. The role involves leading and mentoring a team of data engineers driving best practices across coding CI/CD and DevOps and collaborating with cross-functional stakeholders to translate business needs into effective data solutions while owning end-to-end project delivery. Additionally the engineer will implement data solutions across cloud platformsprimarily Azure but also AWS or GCPintegrating with services such as Azure Data Factory AWS Glue and modern data warehouses like Snowflake or Synapse while managing infrastructure through Infrastructure-as-Code tools. A strong focus is placed on data governance including implementing quality checks monitoring and observability frameworks ensuring compliance with security and privacy standards and managing metadata and lineage using tools like Unity Catalog. The role also supports advanced analytics by enabling data science workflows building ML-ready datasets using Databricks capabilities and supporting real-time analytics through technologies such as Structured Streaming and Delta Live Tables.

Qualifications :

• Strong experience in data engineering (enterprise-scale environments preferred).
• Deep expertise in Databricks Apache Spark (PySpark preferred) and Delta Lake.
• Proven experience designing and implementing Lakehouse architectures (Medallion model).
• Advanced proficiency in SQL and Python.
• Strong experience with cloud platforms Azure (ADLS ADF Synapse) or AWS (S3 Glue Redshift).
• Experience building batch and real-time/streaming data pipelines.
• Hands-on experience with data orchestration tools (ADF Airflow Databricks Workflows).
• Experience implementing CI/CD pipelines and DevOps practices (Azure DevOps GitHub or similar).
• Working knowledge of Infrastructure-as-Code (Terraform or similar tools).
• Strong understanding of data modelling and data warehousing concepts.
• Experience working in client-facing consulting or system integrator environments.
• Proven ability to lead and mentor engineering teams including onshore/offshore models.
• Strong stakeholder management communication and presentation skills.
• Experience delivering solutions for enterprise clients in regulated industries such as Banking Insurance or Government.

Additional Information :

Why join us: 

NCS Australia is where you can feel at home nurturing your talents and skills as we make tomorrow together one day at a time. Our benefits include paid parental leave initiatives focused on your well-being and discounted health insurance. You will also enjoy discounts on various products and services and be regularly recognised and rewarded for high performance. We are committed to your career development through our Capability Fingerprint industry and partner training programs special interest groups and an AI-driven learning platform. No matter where you are in your career we offer meaningful work and opportunities for growth.

NCS Australia is an equal-opportunity employer and we take pride in our commitment to valuing and supporting our people and the communities we are dedicated to attracting retaining and developing our people regardless of gender identity ethnicity sexual orientation disability and age. Applications are encouraged from all sectors of the community and we strongly encourage applications from the Veterans Aboriginal and/or Torres Strait Islander community.

At NCS Australia we are committed to supporting adjustments throughout the recruitment and selection process as well as during employment. We actively support and encourage people with disability to apply. 

Agencies:

Weve got this. We request that you do not contact NCS employees outside of the Talent Acquisition team. NCS exclusively accepts resumes from agencies on our preferred supplier panel through the NCS Agency Portal. Agencies that submit resumes must have a valid fee agreement and be assigned to the particular requisition by the Talent Acquisition team. Any resumes that are submitted outside of this process will become the sole property of NCS. If a candidate is hired outside of this process no fee or payment will be given.

Work rights and background checks: 

To be eligible for a position with us applicants will need to have valid work rights for Australia and be willing to undergo a comprehensive background checking process including probity and police checks

Remote Work :

No

Employment Type :

Full-time