Data Engineering Manager
Job Location:
Monthly Salary:
Posted:
Application Deadline:
Vacancies:
Job Summary
We are looking for a Data Engineering Manager (or a senior Tech Lead ready to step into management) to lead the teams building and running our data lake platform for enterprise clients. This is a player-coach role: you will manage 1015 data engineers while staying deep enough in the stack to review pipeline designs debug the ugly production incident and set the engineering standards the platform is held to.
The work spans two distinct modes and you will own both:
- Implementation projects greenfield and migration builds: standing up data lake environments for new clients building ingestion and transformation pipelines and taking them through UAT to production
- Maintenance and support engagements running live SLA-bound production environments: pipeline reliability data quality batch-window adherence incident response and continuous enhancement
Hands-on Scala and production data pipeline experience are mandatory. This is a non-negotiable screening bar our core processing layer is Scala/Spark and you cannot lead this team review its work or earn its respect from behind a status report.
Manage 1015 data engineers across build and run workstreams. Own hiring onboarding one-on-ones performance management and growth paths. Rotate engineers deliberately between implementation and maintenance so the team builds full-lifecycle judgment engineers who build pipelines theyll never have to operate build worse pipelines. Grow senior engineers into tech leads who can own a client workstream independently.
Own the technical quality of the data platform:
- Ingestion pipelines batch and file-based ingestion from heterogeneous client source systems handling late arrivals reprocessing and upstream format drift
- Transformation layers Scala/Spark processing jobs: design reviews performance tuning (partitioning shuffle behavior skew handling memory management) and code quality standards
- Orchestration dependency-aware scheduling retry and recovery semantics idempotent job design and batch-window management
- Data modeling lake zoning (raw/curated/consumption) file formats and layout strategy schema evolution and downstream consumption contracts
- Data quality engineering validation frameworks reconciliation checks anomaly detection and quality gates that stop bad data before clients see it
Own the engineering side of new client implementations: estimation technical planning environment setup pipeline build data migration and historical loads UAT support and cutover to production. Work with delivery and program management on committed timelines and flag technical risk early with options rather than at the deadline.
Own live environments for maintenance clients:
- Batch SLA adherence pipelines complete inside committed windows and misses are caught and communicated before the clients business day starts
- Incident management triage root cause permanent fixes and post-incident reviews that reduce recurrence
- Monitoring and observability pipeline health data freshness quality metrics and alerting that catches issues before downstream consumers do
- Capacity and performance management as client data volumes grow
- Disciplined change management in production environments releases hotfixes and reprocessing runs that dont create new incidents
Keep the platform coherent across clients: config-over-fork as the default client-specific customization contained and documented and a core codebase that improvements flow back into. Resist the entropy of every client engagement drifting into its own snowflake the economics of the practice depend on it.
Define and enforce the engineering standards for the data practice:
- Scala coding standards review discipline and shared libraries
- Pipeline design patterns templates for ingestion transformation and reconciliation that new workstreams start from
- Testing expectations for data pipelines: unit tests on transformations data validation suites regression testing on reprocessing
- CI/CD for data workloads versioned promotable rollback-capable deployments
- Documentation standards: pipeline runbooks data dictionaries and operational handover packs
Standards are measured in the codebase and the incident trends not in the wiki.
Be the senior technical face of the data team to client counterparts: design walkthroughs technical governance calls incident explanations and enhancement scoping. Give delivery leadership honest engineering status. Translate client asks into technical plans and push back with alternatives when asks threaten platform coherence or committed SLAs.
Must-Haves
- 812 years in data engineering with 24 years leading data engineering teams as a Tech Lead or Engineering Manager
- Mandatory: deep hands-on Scala expertise production Scala/Spark development you have written reviewed and tuned yourself not overseen from a distance
- Mandatory: production data pipeline experience at scale you have built operated and debugged batch pipelines processing large volumes under committed time windows
- Demonstrated experience leading 8 data engineers including performance management and technical mentoring
- Strong command of the Spark internals that matter in production: partitioning strategy shuffle and skew behavior memory tuning join strategies and job troubleshooting from the Spark UI down
- Data lake / lakehouse implementation experience lake zoning file format and layout decisions (Parquet/ORC Delta/Iceberg/Hudi or equivalent) schema evolution and metadata management
- Experience with orchestration frameworks (Airflow or equivalent) and designing dependency-aware restartable idempotent job flows
- Experience running SLA-bound production data environments batch windows incident response reconciliation and the operational discipline live clients demand
- Strong SQL and data modeling fundamentals across relational and lake paradigms
- Experience with cloud data platforms (at least one of the major hyperscalers data stacks; Databricks Snowflake or BigQuery exposure valued) and comfort with on-premise or hybrid deployments
- Ability to review a pull request in the morning and walk a client through an incident root cause in the afternoon credible in both rooms
Strong-to-Have
- Experience in BFSI ideally Mutual Fund / Asset Management familiarity with NAV cycles T1 processing pressure regulatory reporting and reconciliation-heavy data flows
- Experience in multi-tenant or multi-client platform environments where one codebase serves many deployments
- Background spanning both implementation projects and long-running support engagements
- Hands-on cloud experience on AWS and/or GCP with working knowledge of their core data engineering services e.g. S3/GCS Glue/Dataproc EMR/Dataflow Redshift/BigQuery IAM and networking basics for data workloads and comfort designing pipelines that run natively on cloud-managed infrastructure rather than only on self-managed clusters
- Exposure to modern lakehouse and transformation tooling Snowflake dbt and/or Databricks including how these fit alongside or replace hand-rolled Scala/Spark transformation layers for parts of the platform
- Experience with real-time streaming architectures Kafka and/or Flink for use cases needing continuous ingestion or low-latency processing beyond the batch-dominant core of this platform
- Experience with data governance tooling lineage cataloging access control in regulated environments
- Infrastructure-as-code and containerization familiarity (Terraform Kubernetes) for environment provisioning
- Player-coach reviews Scala tunes the stubborn Spark job joins the 6 a.m. incident bridge when it matters but measures success by what the team delivers without you
- Operationally paranoid in a good way assumes pipelines fail designs for restart and reconciliation and treats monitoring as a first-class deliverable
- Standards with teeth enforces quality through review gates templates and CI not reminders
- Direct and low-drama straight feedback to engineers straight status to leadership and clients
- Pragmatic over purist knows when the elegant refactor is worth it and when the clients batch window says otherwise
Required Experience:
Senior IC
About Company
KFintech is the largest RTA and a market leader in the financial sector. We specialize in Mutual Funds, Corporate Registry, NPS, and many more.
