Senior Data Engineer

Company : Willware Technologies

Job Title: Senior Data Engineer

Location: Chennai / Bangalore

Experience: 8-15 Years

Job Overview

We are looking for a Senior Data Engineer with strong expertise in GCP SQL and ETL pipelines. The ideal candidate will have hands-on experience in building scalable data solutions working with complex data architectures and managing end-to-end data pipelines.

Key Responsibilities

• Design develop and maintain scalable ETL/ELT pipelines
• Develop and optimize SQL queries and stored procedures (delta-load ingestion)
• Work with GCP BigQuery Cloud Storage and Composer/Airflow
• Design and implement data ingestion patterns and architectures
• Build and manage workflows using Airflow/Composer (complex DAGs)
• Deploy and manage infrastructure using Terraform
• Collaborate with stakeholders for requirements gathering and delivery
• Ensure data quality security and governance
• Monitor troubleshoot and optimize data pipelines

Required Experience

• 8 years of experience in Data Engineering
• 6 years of strong SQL experience (mandatory)
• 4 years of GCP BigQuery (mandatory)
• 2 years of GCP Cloud Storage (mandatory)
• 4 years of Airflow / GCP Composer (mandatory)
• 4 years in Data Architecture & ingestion patterns
• 6 years working across full SDLC
• 2 years of Terraform (GCP deployments)

Technical Expertise

• Strong knowledge of distributed data processing (Spark Beam)
• Experience with cloud platforms (GCP preferred AWS/Azure plus)
• Hands-on with Python for data engineering tasks
• Experience in batch & streaming data pipelines
• Understanding of data quality governance and lifecycle management
• Experience with Git and infrastructure-as-code tools

Nice to Have

• Apache NiFi
• Confluent Kafka ecosystem (Connect Schema Registry ksqlDB)
• GCP Dataflow Pub/Sub
• Docker Ansible
• Monitoring tools: Grafana Prometheus Loki