Lead Data Engineer
Share
Job Location:
Monthly Salary:
Experience Required:
Posted on:
2 hours ago
Vacancies:
1 Vacancy
Job Summary
Lead Data Engineer - Job Description
Overview
We are seeking a highly skilled Lead Data Engineer to architect build and scale enterprise-grade data platforms and pipelines. The ideal candidate has deep expertise in modern data engineering cloud-native services real-time data ingestion & CDC frameworks. This role involves leading complex data initiatives guiding engineering teams and collaborating with cross-functional stakeholders to deliver robust scalable and secure data solutions.
Key Responsibilities
Data Architecture & Pipeline Development
Design and build scalable ETL/ELT pipelines using Python / PySpark.
Architect and implement batch and real-time data pipelines using Kafka Kafka Connect and Debezium.
Develop and optimize data ingestion workflows across multiple databases (MySQL Postgres MongoDB).
Build and manage Apache Airflow DAGs for end-to-end pipeline orchestration.
Cloud Data Engineering
Lead data platform development on GCP (BigQuery Data Proc Data Stream ).
Implement data ingestion orchestration and transformation at scale using Pub/Sub Cloud Functions/Cloud Run Dataflow and GKE.
Own data warehouse design and optimization across Snowflake and BigQuery.
Requirements
Required Skills & Experience
Core Technical Skills
Strong proficiency in Python SQL PySpark.
Hands-on expertise with Kafka Kafka Connect Debezium Airflow Databricks.
Deep experience with BigQuery Snowflake MySQL Postgres MongoDB.
Solid understanding of vector data stores and search indexing.
Knowledge of GCP services like Big Query Cloud Functions Cloud Run Data Flow Data Proc Data Stream etc..
Good to have
Certifications:
GCP Professional Data Engineer
Elastic Certified Engineer
AI
Gemini Enterprise
Vertex AI Agent Builder
ADK
Non-Technical & Leadership Skills
Communication: Exceptional verbal and written communication skills with the ability to articulate complex technical concepts to both technical and non-technical audiences.
Mentorship & Coaching: Proven experience in mentoring junior and mid-level engineers fostering a culture of continuous learning and growth.
Problem-Solving: Strong analytical and debugging skills with a proactive approach to identifying and resolving technical roadblocks.
Ownership & Accountability: Demonstrates a high level of responsibility for project outcomes system reliability and code quality.
Agile Proficiency: Deep understanding and practical experience with Agile methodologies (Scrum/Kanban).
Stakeholder Management: Ability to effectively manage expectations and build consensus across different teams.
Qualifications
Bachelors or Masters degree in Computer Science Engineering or a related field (or equivalent practical experience).
Required Skills:
Required Skills & Experience Core Technical Skills Strong proficiency in Python SQL PySpark. Hands-on expertise with Kafka Kafka Connect Debezium Airflow Databricks. Deep experience with BigQuery Snowflake MySQL Postgres MongoDB. Solid understanding of vector data stores and search indexing. Knowledge of GCP services like Big Query Cloud Functions Cloud Run Data Flow Data Proc Data Stream etc.. Good to have Certifications: GCP Professional Data Engineer Elastic Certified Engineer AI Gemini Enterprise Vertex AI Agent Builder ADK Non-Technical & Leadership Skills Communication: Exceptional verbal and written communication skills with the ability to articulate complex technical concepts to both technical and non-technical audiences. Mentorship & Coaching: Proven experience in mentoring junior and mid-level engineers fostering a culture of continuous learning and growth. Problem-Solving: Strong analytical and debugging skills with a proactive approach to identifying and resolving technical roadblocks. Ownership & Accountability: Demonstrates a high level of responsibility for project outcomes system reliability and code quality. Agile Proficiency: Deep understanding and practical experience with Agile methodologies (Scrum/Kanban). Stakeholder Management: Ability to effectively manage expectations and build consensus across different teams. Qualifications Bachelors or Masters degree in Computer Science Engineering or a related field (or equivalent practical experience). Typically 7 years of progressive experience in data engineering with 2 years in a technical leadership or lead engineer role.
Company Industry
IT Services and IT Consulting
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
