Data Engineering Consultant

Data Engineering Consultant designs implements and optimizes scalable data pipelines and architectures. This role bridges raw data and actionable insights ensuring robustness performance and data governance. Collaboration with analysts and scientists is central to delivering high-quality solutions aligned with business objectives.

Key Responsibilities:

1. Data Pipeline Development

• Architect implement and maintain real-time and batch data pipelines to handle large datasets efficiently.
• Employ frameworks such as Apache Spark Databricks Snowflake or Airflow to automate ingestion transformation and delivery.
• Design and implement fault-tolerant self-healing data pipelines to ensure continuous data flow.
• Optimize data pipeline performance through parallel processing and distributed computing techniques.

1. Data Integration & Transformation

• Work with Data Analysts to understand source-to-target mappings and quality requirements.
• Build ETL/ELT workflows validation checks and cleaning steps for data reliability.
• Develop and maintain data dictionaries and metadata repositories for improved data understanding.
• Implement data quality monitoring tools and processes to ensure data integrity throughout the pipeline.

1. Automation & Process Optimization

• Automate data reconciliation metadata management and error-handling procedures.
• Continuously refine pipeline performance scalability and cost-efficiency.
• Implement monitoring and alerting systems for early detection of pipeline issues.
• Develop and maintain documentation for all automated processes and workflows.

1. Collaboration & Leadership

• Coordinate with Data Scientists Data Architects and Analysts to ensure alignment with business goals.
• Mentor junior engineers and enforce best practices (version control CI/CD for data pipelines).
• Participate in technical presales activities and client engagement initiatives.
• Lead cross-functional team meetings to discuss data engineering challenges and solutions.
• Contribute to the development of data engineering standards and best practices within the organization.

1. Governance & Compliance

• Apply robust security measures (RBAC encryption) and ensure regulatory compliance (GDPR).
• Document data lineage and recommend improvements for data ownership and stewardship.
• Implement data masking and anonymization techniques for sensitive information.
• Collaborate with legal and compliance teams to ensure adherence to data protection regulations.

1. Performance Tuning & Optimization

• Conduct regular performance audits of data pipelines and storage systems.
• Implement caching strategies and query optimization techniques to improve data access speeds.
• Utilize partitioning and indexing strategies to enhance query performance on large datasets.
• Optimize resource allocation and utilization in cloud environments to manage costs effectively.

1. Emerging Technologies & Innovation

• Stay abreast of emerging data engineering technologies and methodologies.
• Evaluate and recommend new tools and frameworks that can enhance data engineering capabilities.
• Conduct proof-of-concept projects to test innovative data engineering solutions.
• Contribute to the companys thought leadership through blog posts whitepapers or conference presentations on data engineering topics.

   

Qualifications :

Qualifications:

Required:

• Bachelors degree in Computer Science Information Technology or a related field; Masters degree preferred
• 5-10 years of experience in data engineering with a strong focus on big data technologies
• Expert-level proficiency in Python/PySpark
• Extensive experience with cloud platforms particularly AWS-or- Azure-or- GCP
• Advanced knowledge of data warehousing solutions such as Databricks -or- Snowflake-or- BigQuery -or- Redshift
• Proven track record in designing implementing and optimizing large-scale data pipelines
• Strong understanding of data modeling ETL/ELT processes and data governance principles
• Proficiency in SQL and NoSQL databases
• Familiarity with data visualization tools and techniques

Preferred:

• Relevant certifications such as AWS Certified Data Analytics - Specialty Azure Data Engineer Associate or Google Cloud Professional Data Engineer
• Experience with real-time data streaming technologies (e.g. Kafka Apache Flink)
• Knowledge of machine learning concepts and their application in data engineering
• Familiarity with data lake architectures and associated technologies
• Experience with CI/CD practices for data pipelines
• Understanding of data security and compliance requirements (e.g. GDPR)
• Strong problem-solving skills and ability to work independently
• Excellent communication skills both written and verbal
• Experience mentoring junior engineers and collaborating with cross-functional teams

Additional Information :

At Endava were committed to creating an open inclusive and respectful environment where everyone feels safe valued and empowered to be their best. We welcome applications from people of all backgrounds experiences and perspectivesbecause we know that inclusive teams help us deliver smarter more innovative solutions for our customers. Hiring decisions are based on merit skills qualifications and potential. If you need adjustments or support during the recruitment process please let us know.

Remote Work :

No

Employment Type :

Full-time