Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
30+ days ago
Vacancies:
1 Vacancy
Job Summary
Design implement and manage Change Data Capture (CDC) and bulk data replication tasks using Qlik Replicate from diverse sources (e.g. Oracle SQL Server SAP) to GCP targets.
Configure and optimize Qlik Replicate endpoints and settings to ensure high-speed low-latency and reliable data movement into Cloud Storage and BigQuery.
Monitor and troubleshoot replication latency errors and performance of Qlik Replicate tasks ensuring data integrity and consistency between source and target systems.
Develop and implement strategies for initial data loads resumption of replication and recovery from source/target outages with minimal data loss.
GCP Data Pipeline Development
Develop construct and optimize robust ETL/ELT pipelines using core GCP services such as BigQuery Cloud Dataflow (Apache Beam) Cloud Composer (Apache Airflow) and Cloud Pub/Sub.
Design and implement scalable data warehouse solutions in BigQuery following dimensional modeling (Star/Snowflake schema) and data vault best practices.
Utilize Python and SQL to perform complex data transformations quality checks cleansing and enrichment within the GCP environment.
Architecture Monitoring and Operations
Collaborate with Data Architects and Analytics teams to define the data model and schema design for analytics and business intelligence.
Implement automated monitoring alerting and logging for all data pipelines and replication tasks using Cloud Monitoring and other GCP native tools.
Configure and optimize Qlik Replicate endpoints and settings to ensure high-speed low-latency and reliable data movement into Cloud Storage and BigQuery.
Monitor and troubleshoot replication latency errors and performance of Qlik Replicate tasks ensuring data integrity and consistency between source and target systems.
Develop and implement strategies for initial data loads resumption of replication and recovery from source/target outages with minimal data loss.
GCP Data Pipeline Development
Develop construct and optimize robust ETL/ELT pipelines using core GCP services such as BigQuery Cloud Dataflow (Apache Beam) Cloud Composer (Apache Airflow) and Cloud Pub/Sub.
Design and implement scalable data warehouse solutions in BigQuery following dimensional modeling (Star/Snowflake schema) and data vault best practices.
Utilize Python and SQL to perform complex data transformations quality checks cleansing and enrichment within the GCP environment.
Architecture Monitoring and Operations
Collaborate with Data Architects and Analytics teams to define the data model and schema design for analytics and business intelligence.
Implement automated monitoring alerting and logging for all data pipelines and replication tasks using Cloud Monitoring and other GCP native tools.
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
