Lead Data Engineer
Lead Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
At Endava we believe that harnessing Data & AI is essential for navigating todays rapidly evolving technology landscape. With a global team of over 500 data and AI professionals our Data & AI Discipline focuses on real-world impact delivering sustainable solutions underpinned by proven frameworks and methodologies.
As we expand our capability in Australia were looking for a Lead Data Engineer to help build and scale our local offering. This role is key to orchestrating modern data pipelines architectures and governance frameworks that enable our clients to unlock actionable insights improve decision-making and drive continuous innovation.
The Lead Data Engineer designs builds and optimises robust and scalable data infrastructure. This role ensures that data is not only collected but also strategically transformed modelled governed and surfaced to enable impactful business decisions. Collaborating with analysts data scientists and technical business analysts is central to delivering high-quality solutions aligned with our clients goals.
Key Responsibilities
Data Pipeline Development
- Design build and maintain real-time and batch data pipelines that enable high-performance analytics.
- Develop ETL and ELT processes to transform cleanse and enrich data from diverse sources.
- Optimise pipelines for scalability reusability and cost efficiency using modern frameworks and cloud-native tools.
Data Infrastructure & Architecture
- Design and implement data architectures that support secure reliable and high-availability storage and access.
- Ensure data infrastructure supports seamless integration across reporting analytics and ML workloads.
Data Integration Transformation & Modelling
- Collaborate with analysts and data scientists to understand source-to-target mappings and design appropriate models.
- Perform data wrangling modelling and transformation to enable accurate and timely insights.
- Maintain high standards of data quality lineage and documentation.
Collaboration
- Work closely with Data Scientists Data Analysts and Technical Business Analysts to ensure business requirements are met.
- Support junior engineers and foster a culture of technical excellence and collaboration.
Governance Security & Compliance
- Implement and uphold enterprise-grade security and privacy practices including RBAC encryption and compliance (e.g. APP).
- Promote best practices in data stewardship documentation and ownership.
Performance & Innovation
- Continuously refine data platforms for reliability performance and cost optimisation.
- Stay current with technology trends recommending new tools or approaches where beneficial.
Qualifications :
Technical Skills
- Programming: Python SQL
- Data Orchestration & Transformation: DBT Airflow Dagster
- Big Data Processing: Apache Spark Databricks Dask
- Cloud Platforms: AWS (Glue Redshift) Azure (Synapse Fabric Data Factory) GCP (BigQuery Dataflow)
- Storage & Warehousing: Relational (PostgreSQL SQL Server Snowflake) NoSQL (MongoDB Cassandra) Graph (Neo4j Amazon Neptune)
- DevOps & Automation: Docker Kubernetes Terraform CI/CD for data workflows
- Monitoring & Maintenance: Tools for data observability lineage tracking and pipeline health
Architectural Competencies
- Data Modelling: Dimensional relational and hierarchical design
- Scalability & Performance: Building high-availability data architectures
- Governance & Compliance: Security models audit trails access control
Soft Skills
- Problem Solving: Ability to debug complex data challenges quickly and effectively
- Collaboration: Strong cross-functional communicator and team player
- Leadership: Supports and mentors engineers promotes engineering best practices
Preferred Industry Experience
- Experience working in regulated industries such as banking financial services insurance or healthcare is highly valued.
Additional Information :
Discover some of the global benefits that empower our people to become the best version of themselves:
- Finance: Competitive salary package share plan company performance bonuses value-based recognition awards referral bonus;
- Career Development: Career coaching global career opportunities non-linear career paths internal development programmes for management and technical leadership;
- Learning Opportunities: Complex projects rotations internal tech communities training certifications coaching online learning platforms subscriptions pass-it-on sessions workshops conferences;
- Work-Life Balance: Hybrid work and flexible working hours employee assistance programme;
- Health: Global internal wellbeing programme access to wellbeing apps;
- Community: Global internal tech communities hobby clubs and interest groups inclusion and diversity programmes events and celebrations.
At Endava were committed to creating an open inclusive and respectful environment where everyone feels safe valued and empowered to be their best. We welcome applications from people of all backgrounds experiences and perspectivesbecause we know that inclusive teams help us deliver smarter more innovative solutions for our customers. Hiring decisions are based on merit skills qualifications and potential. If you need adjustments or support during the recruitment process please let us know.
Remote Work :
No
Employment Type :
Full-time
Employment Type
Full-time
