Senior Data Engineer

• Lead onshore technical engineers 
• Act as the lead engineer for each domain 
• Collaborate with Solution Architects to ensure understanding needed for engineering level guidance to nearshore teams 
• Translate architecture design to engineering level guidance 
• Own specifying architectural designs into sprint-ready engineering tasks for assigned domain 
• Ensure delivery to the solution architecture within domain 
• Answer design questions for sprint teams 
• Participate in backlog refinement sessions to align team with the architecture 
• Act as the internal technical expert supporting Product design and Engineering teams on demonstrations and technical discussions. 
• Create/Update documentation such as solution design patterns and engineering standards. 
• Lead technical discussions around APIs integrations data flow compliance and security. 
• Serve as a trusted advisor on best practices technical implementation and scalability. 
• Design and implement highly performant and scalable ETL/ELT data pipelines using Databricks (Spark) and orchestration tools. 
• Develop data ingestion transformation and enrichment workflows using Scala Python and advanced SQL. 
• Support real-time and batch processing needs with optimized workflows adhering to best practices for scalability and maintainability. 
• Leverage Databricks for developing and managing Spark-based data pipelines and analytics workloads. 
• Implement Unity Catalog for centralized data governance access control lineage and auditability. 
• Manage Delta Lake schemas and optimize data lake performance using Z-ordering compaction and partitioning strategies. 
• Knowledge about testing and validation frameworks into pipelines using tools such as Great Expectations or custom frameworks. 
• Monitor data flows for anomalies failures and integrity issues and proactively troubleshoot and resolve them. 
• Knowledge about automated deployment pipelines for data workflows using Azure DevOps GitHub Actions or similar. 
• Apply version control and collaborative development practices for notebooks pipelines and data models. 
• Partner with Data Scientists BI teams and Product stakeholders to understand data needs and translate them into scalable solutions. 
• Mentor junior engineers and help standardize practices across the data engineering team. 

Qualifications :

Required

• 5 years of professional experience in data engineering roles.
• Experience with Azure Cloud and some hands-on experience with cloud services (such as ADF Data Lake Key Vault etc.) 
• Hands-on expertise with: 
  • Databricks (including Spark tuning and cluster management) 
  • Scala Python and SQL 
  • Azure SQL Cosmos DB PostgreSQL 
• Knowledge of databases (SQL and/or NoSQL) and data modeling. 
• Proven experience building and managing data pipelines in production. 
• Strong understanding of Data Lakehouse Delta Lake and medallion architecture. 
• Experience with data governance access control and lineage using Unity Catalog. 
• Must be able to work onsite 2-3 days a week

Desired:

• Pharmacy Business domain experience
• Exposure to DevOps CI/CD pipelines and containerization (Docker Kubernetes
• Familiarity with: 
  • dbt for transformations and documentation. 
  • Azure Event Hubs or Kafka for streaming data ingestion. 
  • Airflow Synapse or Power BI for orchestration and reporting integration. 
  • Experience with Infrastructure-as-Code tools like Terraform or Bicep for provisioning Azure data resources. 
  • Exposure to data observability platforms (e.g. Monte Carlo OpenLineage). 
  • Knowledge of data mesh and distributed data ownership models. 

Additional Information :

Discover some of the global benefits that empower our people to become the best version of themselves:

• Finance: Competitive salary package share plan company performance bonuses value-based recognition awards referral bonus;  
• Career Development: Career coaching global career opportunities non-linear career paths internal development programmes for management and technical leadership;
• Learning Opportunities: Complex projects rotations internal tech communities training certifications coaching online learning platforms subscriptions pass-it-on sessions workshops conferences;
• Work-Life Balance: Hybrid work and flexible working hours employee assistance programme;
• Health: Global internal wellbeing programme access to wellbeing apps;
• Community: Global internal tech communities hobby clubs and interest groups inclusion and diversity programmes events and celebrations.

Additional Employee Requirements  

• Participation in both internal meetings and external meetings via video calls as necessary. 
• Ability to go into corporate or client offices to work onsite as necessary. 
• Prolonged periods of remaining stationary at a desk and working on a computer as necessary. 
• Ability to bend kneel crouch and reach overhead as necessary. 
• Hand-eye coordination necessary to operate computers and various pieces of office equipment as necessary. 
• Vision abilities including close vision toleration of fluorescent lighting and adjusting focus as necessary. 
• For positions that require business travel and/or event attendance ability to lift 25 lbs as necessary. 
• For positions that require business travel and/or event attendance a valid drivers license and acceptable driving record are required as driving is an essential job function.

*If requested reasonable accommodations will be made to enable employees requiring accommodations to perform the essential functions of their jobs absent undue hardship.   

USA Benefits (Full time roles only does not apply to contractor positions)

• Robust healthcare and benefits including Medical Dental vision Disability coverage and various other benefit options  
• Flexible Spending Accounts (Medical Transit and Dependent Care)  
• Employer Paid Life Insurance and AD&D Coverages  
• Health Savings account paired with our low-cost High Deductible Medical Plan  
• 401(k) Safe Harbor Retirement plan with employer match with immediately vest   

At Endava were committed to creating an open inclusive and respectful environment where everyone feels safe valued and empowered to be their best. We welcome applications from people of all backgrounds experiences and perspectivesbecause we know that inclusive teams help us deliver smarter more innovative solutions for our customers. Hiring decisions are based on merit skills qualifications and potential. If you need adjustments or support during the recruitment process please let us know.

Remote Work :

No

Employment Type :

Full-time