Data Engineer
Job Summary
Direct Client Requirement
Job Title - Data Engineer
Location - Remote
Employment Type - Full- time
Experience-
6 years in data engineering ETL/ELT analytics engineering data warehouse development or big data pipelines; minimum 4 years on GCP cloud data platforms preferred.
Common Job description
- Strong SQL and Python.
- Data modelling warehousing concepts partitions clustering indexing optimisation.
- ETL/ELT pipeline design and production support.
- Batch and streaming pipeline fundamentals.
- Orchestration using Airflow Composer Control-M ADF dbt or equivalent.
- Big data processing using Spark PySpark Beam Dataflow Dataproc Databricks EMR or equivalent.
- Cloud storage and data lake concepts.
- Data quality reconciliation schema validation lineage monitoring and alerting.
- CI/CD for data pipelines and code versioning.
- Ability to explain scale scenarios not just tool definitions.
Generic Skills (Must Have)
- Python and advanced SQL.
- Data quality checks reconciliation alerting retries and backfills.
- Production pipeline monitoring and troubleshooting.
- Mandatory : BigQuery: datasets tables views partitioning clustering SQL optimisation cost awareness.
Option 2 : Dataproc / PySpark or strong Spark experience.
- Pub/Sub or equivalent event streaming.
- Cloud Composer / Airflow DAG development.
- Dataform dbt CI/CD for data.
- BigQuery ML.
- Dataflow templates.
- Looker / LookML basics.
- Terraform for data infrastructure.
Required Experience:
IC
About Company
IT Management Quisque placerat vitae lacus ut scelerisque fusce luctus odio ac nibh luctu. Cloud Services Quisque placerat vitae lacus ut scelerisque fusce luctus odio ac nibh luctu. Data Security Quisque placerat vitae lacus ut scelerisque fusce luctus odio ac nibh luctu. Machine Lea ... View more