Data Engineer

Role Summary

We are looking for a Data Engineer responsible for operating monitoring and enhancing data ingestion and processing pipelines within a Google Cloudbased data platform. The role combines day-to-day platform reliability data validation and development activities supporting continuous platform evolution new data onboarding and automation initiatives. The ideal candidate is hands-on detail-oriented and comfortable working across production and pre-production environments.

Key Responsibilities

A) Data Ingestion Operations & Monitoring

• Monitor and ensure successful daily and monthly data ingestion across multiple upstream systems.
• Oversee scheduled jobs and queries validating successful execution and investigating failures or delays.
• Maintain and enhance data ingestion monitoring checks including:
  • Updating checks when new datasets or tables are introduced.
  • Adapting monitoring logic based on frequency country or business dimensions.
• Track and investigate unprocessed or delayed files in cloud storage performing root cause analysis and coordinating resolution with relevant stakeholders.
• Manage and follow up on incident or service requests related to missing or delayed data until fully resolved.

B) Automation & Platform Reliability

• Support and troubleshoot automation workflows used to capture ingestion exceptions and feed monitoring logic.
• Ensure automation processes run reliably and intervene quickly in case of failures.
• Handle ad-hoc automation requests to reduce manual effort and improve operational efficiency.
• Proactively identify recurring issues and implement improvements to strengthen pipeline stability.

C) Data Validation & Reporting

• Perform data validation and reconciliation activities by comparing datasets across environments or systems when required (typically on a recurring or monthly basis).
• Develop maintain and execute Python-based validation scripts to identify discrepancies and data quality issues.
• Prepare recurring operational and reconciliation reports in line with agreed schedules.

D) Development & Enhancements

• Support environment setup and migration activities within Google Cloud including configuration changes between production and pre-production environments.
• Create test and deploy new tables and datasets following platform standards.
• Implement schema changes and enhancements to existing tables while ensuring backward compatibility and data integrity.
• Enable onboarding of new countries business units or data domains into existing ingestion and monitoring frameworks.
• Continuously refine ingestion and validation logic to align with evolving business requirements.

Technologies & Tools

• Google Cloud Platform (GCP)
• SQL
• Python
• ODBC
• Postman
• Workflow and automation tools

Issue tracking tools (e.g. ticketing systems)

Qualifications :

Required Skills & Competencies

2-6 years of experience

Technical Skills

• Experience with data engineering operations: ingestion monitoring troubleshooting and reliability ownership.
• Hands-on exposure to GCP administration (environment handling scheduled workloads monitoring).
• Strong SQL skills for investigation validation and operational queries.
• Proficiency in Python for scripting automation and data validation.
• Ability to work confidently across pre-production and production environments.
• Solid understanding of system architecture and end-to-end data flows.

Professional Skills

• Strong analytical and problem-solving capabilities with a structured approach to root cause analysis.
• Effective communication and stakeholder management skills including coordination with cross-functional teams.
• Ability to balance BAU operations with development and enhancement work.
• High level of ownership accountability and attention to detail.

Nice to Have

• Experience with cloud data storage patterns and file-based ingestion.
• Familiarity with incident management and operational SLAs.
• Interest in improving observability monitoring and data quality frameworks.

Remote Work :

No

Employment Type :

Full-time