Enter a job title or keyword
Data Engineer — ML Training Data Pipeline
Job Location:
Monthly Salary:
Not provided by the employer
Monthly Salary:
Experience Required:
5years
Experience Required:
Posted:
9 September 2026 (5 hours ago)
Posted:
Application Deadline:
7 December 2026
Application Deadline:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Job Title:Data Engineer - ML Training Data Pipeline
Notice period: 0-30 Days
Experience : 5 Years
Location: Hyderabad OR Pune
We are looking for Data Engineer - ML Training Data Pipeline who can Build and maintain the data pipeline that transforms raw production traces into high-quality training datasets for LLM fine-tuning-ingestion deduplication format conversion quality filtering and train/test splitting at scale on AWS.
What We Expect:
- Build end-to-end data pipelines: raw trace ingestion dedup format conversion quality gating training-ready datasets
- Process large-scale JSONL data on AWS S3 (tens of thousands of traces per batch)
- Convert between chat-completion formats (e.g. OpenAI Llama 3.1 tool-calling format)
- Implement smart deduplication and sampling to balance training distribution
- Design identity-aware train/test splits that measure true generalization
- Build data validation gates to detect schema drift and format anomalies
- Create a continuous pipeline that auto-processes new production traces for retraining
Requirements
- Experience: 6 years data engineering focused on ML data pipelines
- Python: Strong pandas pyarrow JSONL processing at scale
- ML Data Libraries: HuggingFace Datasets Arrow-based storage
- Data Formats: Multi-turn conversation/chat data structures and tokenizer-specific formatting
- Deduplication: Content hashing identity-based grouping strategies
- AWS: S3 EC2 batch processing workflows
Preferred (Not Required): LLM training data prep (chat templates tool-calling schemas); Axolotl or similar dataset formats; data versioning (DVC LakeFS); browser-automation trace data or Playwright.
Benefits
- Comprehensive Medical Coverage:
Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members) ensuring complete peace of mind. - Robust Protection Plans:
Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones. - Retirement Benefits:
PF and Gratuity provided as per standard government regulations. - Flexible Work Options:
Enjoy hybrid work arrangements & flexible working hours. - Generous Leave Policy:
21 days of annual leave in addition to 10 company-declared holidays. - Employee Well-being Spaces:
Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
Required Skills:
LLM AWS. S3 EC2 Huggingface datasets Fine tuning
Required Education:
Btech/M tech
