Enter a job title or keyword
Senior Data Engineer
Job Location:
Monthly Salary:
Posted:
12 November 2026 (20 hours ago)
Posted:
Application Deadline:
9 February 2027
Application Deadline:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
About us
Factori is a rapidly growing location and consumer intelligence company built for an AI-first world. Each day we ingest billions of privacy-compliant signals and maintain the worlds most comprehensive consumer mobility and point-of-Interest database.
Our AI-driven insights help Fortune 500 brands global media agencies and technology innovators decide where to invest build and grow. Headquartered in New York City with hubs across APAC EMEA and the Americas we operate with the ownership mindset of a startup and the global footprint of an enterprise data leader.
About the Role:
We are hiring a Senior Data Engineer to own and scale a petabyte-scale data platform on Google Cloud Platform (GCP) processing TB scale of data every day with Apache Spark. You will lead a small team of data engineers drive cloud cost optimisation across our data workloads and build AI-driven workflows that change how we support and operate the platform.
This is a product engineering role not a services or consulting engagement. You will own systems end to end live with your design decisions in production and help define data engineering culture in India.
Key Responsibilities:
Build and scale the platform
- Design build and operate highly available fault-tolerant batch and streaming pipelines on GCP that ingest hundreds of terabytes a day into a petabyte-scale data lake and warehouse.
- Write and tune large Spark workloads (PySpark Spark SQL Scala/Java) on Dataproc or Spark on Kubernetes (GKE): partitioning shuffle and join optimization skew handling memory and executor sizing.
- Build real-time and near-real-time pipelines with Pub/Sub or Kafka and Spark Structured Streaming or Dataflow (Apache Beam) with clear guarantees on latency ordering and exactly-once or idempotent processing.
- Model data in BigQuery and the lakehouse (Parquet Avro Delta Lake or Apache Iceberg) into clean simplified datasets that teams can self-serve.
Own data quality
- Build data cleaning validation and reconciliation pipelines that stay correct when source data is late duplicated malformed or missing.
- Put data contracts schema evolution rules quality checks and alerting in place and own data mapping business logic and transformations end to end.
Lead cloud cost optimization
- Own the cost of our GCP data workloads: measure cost per pipeline and per customer then reduce it through right-sizing autoscaling preemptible/Spot VMs storage tiering BigQuery slot and query optimization and job consolidation.
- Set up cost visibility (billing export labels dashboards) and FinOps practices so cost is tracked and owned not discovered after the fact.
Lead and mentor
- Lead and mentor 24 junior and mid-level data engineers: plan and break down work review code and designs and raise the teams engineering bar.
- Lead system design and architecture discussions write design docs and make trade-offs on scalability reliability latency and cost.
Innovate with AI
- Design AI-driven workflows for technical support and operations: LLM-assisted triage of pipeline failures and data incidents automated root-cause analysis runbook automation and agents that answer data questions from internal teams.
- Turn repeated support work into automated process-driven workflows and measure the time and tickets they save.
Run it in production
- Own production reliability: on-call monitoring and observability incident response postmortems and low-level debugging and performance tuning on large clusters.
- Orchestrate pipelines with Airflow / Cloud Composer and ship through CI/CD and infrastructure as code (Terraform).
What we are looking for :
Experience:
- 6 plus years in data engineering or backend/software engineering with at least 4 years building big data pipelines in production.
- Minimum 12 years as a tech lead or team lead mentoring junior engineers and owning delivery for a team.
Product background:
- Most of your experience at product or SaaS companies owning platforms you built and ran in production rather than consulting or IT services delivery.
Apache Spark at scale:
- Deep hands-on PySpark and Spark SQL; Spark internals (DAG stages shuffle catalyst AQE) performance tuning and handling data skew on multi-terabyte jobs. Scala or Java Spark is a strong plus.
Google Cloud Platform:
- Production experience with BigQuery Dataproc Cloud Storage (GCS) Pub/Sub and Cloud Composer or Airflow. Dataflow GKE and Bigtable are a plus.
Batch and stream processing:
- Designed and run both with a solid grasp of windowing watermarks late data checkpointing backfills and idempotency. Kafka or Pub/Sub plus Spark Structured Streaming Flink or Dataflow.
SQL:
- Expert-level including window functions complex joins and query optimization on large analytical warehouses.
Programming and OOP:
- Strong Python and Java with object-oriented design design patterns and clean testable reusable code.
Data quality:
- Built data cleaning validation deduplication and reconciliation pipelines with automated quality checks and alerting.
System design:
- Can design a scalable fault-tolerant distributed data system and defend the trade-offs in consistency latency cost and failure handling.
Cloud cost optimization:
- A track record of measurably reducing cloud or data platform costs (state the numbers on your CV).
Databases and storage:
- Parquet Avro and ORC; partitioning and clustering; production experience with PostgreSQL Elasticsearch and Bigtable (or comparable relational search and NoSQL stores).
Good to have :
- Google Cloud Professional Data Engineer or Professional Cloud Architect certification.
- Lakehouse table formats: Delta Lake Apache Iceberg or Apache Hudi. dbt Trino/Presto or ClickHouse.
- Terraform Docker and Kubernetes (GKE) for data infrastructure.
- Data observability and lineage tools such as Monte Carlo OpenLineage or DataHub.
- Building with LLM APIs RAG or agent frameworks (Vertex AI Gemini OpenAI Claude LangChain).
- Domain experience in ad tech martech marketing attribution or measurement.
Who thrives here :
- You own outcomes. When a pipeline breaks at 2 a.m. or a number looks wrong you treat it as yours until it is fixed and the root cause is gone.
- You are accountable. You make commitments you can keep flag risks early and close the loop without being chased.
- You think like a product engineer. You care about the customer impact of data quality latency and cost not just whether the job finished.
- You raise the people around you. You give clear reviews share context and help junior engineers grow into owners.
- You simplify. You prefer fewer well-understood systems and automate repeated work increasingly with AI.
How to apply :
If this sounds like you apply now at Wed love to hear from you! For more information visit - Experience:
Senior IC
