Data ML Platform Engineer (Python)
Share
Job Location:
Monthly Salary:
Posted on:
3 hours ago
Vacancies:
1 Vacancy
Job Summary
Data / ML Platform Engineer (Python LLM Pipelines Batch Processing)
Job Summary
We are looking for a Data / ML Platform Engineer to design and manage Python-based batch pipelines for evaluating machine learning and LLM models.
The role involves building scalable pipelines that process large datasets interact with LLM APIs and store structured outputs for analytics and benchmarking.
Key Responsibilities
ML / Data Pipeline Development
- Build and maintain Python batch pipelines for:
- Model evaluation
- Dataset processing
- Design resumable and fault-tolerant workflows
LLM Integration
- Integrate with:
- GPT / Claude / other LLM APIs
- Execute large-scale evaluation workflows
Data Engineering
- Load and process data from:
- Cloud storage (S3 GCS Azure Blob)
- Transform and structure data for analytics
Database & Storage
- Store results in:
- MongoDB (structured outputs)
- Ensure efficient data retrieval and storage
Performance & Scalability
- Optimize:
- Pipeline execution
- Cost and latency
- Handle high-volume batch jobs
Cloud & DevOps
- Deploy pipelines on:
- AWS / GCP / Azure
- Use:
- CI/CD pipelines
- Version control (Git)
Monitoring & Reliability
- Implement:
- Logging and monitoring
- Error handling & retries
- Ensure pipeline stability
Required Skills
Core Skills
- Strong Python development
- Experience with:
- Batch pipelines / ETL
- Data processing
AI / ML
- Hands-on with:
- LLM APIs (GPT Claude)
- Prompt engineering
- Experience with:
- RAG pipelines (preferred)
Data & Storage
- MongoDB / NoSQL databases
- Data structuring and transformation
Cloud
- AWS / GCP / Azure
- Cloud storage services
DevOps
- Git / GitHub
- CI/CD pipelines
Experience Required
- 5-8 years total experience
- 2 years in ML / data pipelines / LLM-based systems
