Hiring || GCP Bigquery Professionals (Hyderabad)
Job Summary
- Design develop and maintain end-to-end batch and streaming data pipelines using BigQuery Cloud Composer/Airflow Dataflow and Dataproc.
- Implement robust data warehousing solutions using BigQuery with advanced techniques such as SCD partitioning and clustering.
- Build and manage event-driven data workflows using Pub/Sub and integrate with Kafka for data ingestion or migration scenarios.
- Write efficient scalable SQL and Python code to process and transform large-scale datasets.
- Leverage Apache Beam for pipeline development and ensure consistency across Dataflow-based implementations.
- Collaborate with data scientists and analysts to support advanced analytics and ML initiatives using BigQuery ML and Vertex AI.
- Ensure data reliability performance security and observability across all data systems.
- Participate in architecture discussions code reviews and continuous improvement of data infrastructure.
- 6-11 years of professional experience in data engineering with a strong focus on GCP technologies.
- Expertise in BigQuery Cloud Composer/Airflow Dataflow Dataproc and Apache Beam (explicit or demonstrated via Dataflow pipelines).
- Proven experience building end-to-end batch and streaming data pipelines.
- Deep understanding of data warehouse concepts including SCD partitioning and clustering.
- Strong proficiency in Python and SQL for data transformation and pipeline development.
- Hands-on experience with Pub/Sub for event-driven data processing.
- Experience with Kafka including integration or migration use cases.
- Basic exposure to BigQuery ML and Vertex AI for predictive analytics and machine learning workflows.
- Familiarity with cloud-native best practices CI/CD monitoring and infrastructure as code.
Required Skills:
Summary: We are seeking a skilled Data Engineer with expertise in Google Cloud Platform (GCP) to design build and maintain scalable data pipelines and data warehousing solutions. This role is critical in enabling data-driven decision-making across the organization by ensuring reliable efficient and secure data processing across batch and streaming workloads. The ideal candidate will leverage tools such as BigQuery Cloud Composer/Airflow Dataflow Dataproc and Apache Beam to deliver end-to-end data solutions while applying strong data modeling principles including SCD partitioning and clustering. Experience with event-driven architectures using Pub/Sub and Kafka integration or migration is highly valued. Additionally familiarity with BigQuery ML and Vertex AI enables advanced analytics and machine learning capabilities. This position is based in Hyderabad and requires 6-11 years of hands-on experience in cloud-based data engineering. Location: Hyderabad Responsibilities: Design develop and maintain end-to-end batch and streaming data pipelines using BigQuery Cloud Composer/Airflow Dataflow and Dataproc. Implement robust data warehousing solutions using BigQuery with advanced techniques such as SCD partitioning and clustering. Build and manage event-driven data workflows using Pub/Sub and integrate with Kafka for data ingestion or migration scenarios. Write efficient scalable SQL and Python code to process and transform large-scale datasets. Leverage Apache Beam for pipeline development and ensure consistency across Dataflow-based implementations. Collaborate with data scientists and analysts to support advanced analytics and ML initiatives using BigQuery ML and Vertex AI. Ensure data reliability performance security and observability across all data systems. Participate in architecture discussions code reviews and continuous improvement of data infrastructure. Requirements 6-11 years of professional experience in data engineering with a strong focus on GCP technologies. Expertise in BigQuery Cloud Composer/Airflow Dataflow Dataproc and Apache Beam (explicit or demonstrated via Dataflow pipelines). Proven experience building end-to-end batch and streaming data pipelines. Deep understanding of data warehouse concepts including SCD partitioning and clustering. Strong proficiency in Python and SQL for data transformation and pipeline development. Hands-on experience with Pub/Sub for event-driven data processing. Experience with Kafka including integration or migration use cases. Basic exposure to BigQuery ML and Vertex AI for predictive analytics and machine learning workflows. Familiarity with cloud-native best practices CI/CD monitoring and infrastructure as code.
Required Education:
Graduate