Benchmark Engineer – AI Infrastructure Vector Search
Share
Job Location:
New York City, NY - USA
Monthly Salary:
Not Disclosed
Posted on:
Yesterday
Vacancies:
1 Vacancy
Job Summary
Benchmark Engineer AI Infrastructure / Vector Search
Location: Remote
Employment Type: Full-time
We are recruiting on behalf of a fast-growing AI infrastructure company that develops a high-performance open-source vector database used in semantic search recommendation systems and RAG pipelines.
We are seeking a Benchmark Engineer to design build and maintain performance benchmarks that help validate and improve large-scale AI and database systems.
This role sits at the intersection of engineering performance optimization and developer experience and offers an opportunity to work on core infrastructure used in modern AI applications.
About the Role
The Benchmark Engineer will own the benchmarking strategy ensuring performance testing reflects real-world workloads and produces reliable reproducible insights.
You will collaborate closely with engineering teams to identify bottlenecks validate improvements and communicate performance results clearly to both technical and non-technical audiences.
Key Responsibilities
- Design and maintain reproducible benchmarks for:
- Vector search workloads
- Indexing and filtering performance
- Distributed system scenarios
- Measure and analyze:
- Latency
- Throughput
- Recall and accuracy
- Memory usage and infrastructure cost
- Build and maintain benchmarking tools datasets and automation pipelines
- Compare performance against alternative solutions using fair and transparent methodologies
- Identify regressions bottlenecks and optimization opportunities
- Translate benchmark results into clear reports documentation or technical content
- Ensure testing reflects realistic production workloads
Requirements
- Strong software engineering background (Python Rust Go or similar)
- Understanding of:
- Databases or distributed systems
- Search engines or data infrastructure
- Experience with:
- Performance testing and benchmarking
- Profiling tools and performance analysis
- Automation pipelines and large datasets
- Ability to evaluate technical trade-offs such as:
- Speed vs accuracy
- Memory vs latency
- Strong communication skills and ability to present technical findings
Preferred Qualifications
- Experience with vector search or ANN algorithms
- Familiarity with machine learning infrastructure or AI systems
- Experience with cloud environments and containerized workloads
- Open-source project experience
- Knowledge of observability and monitoring tools
Whats Offered
- Opportunity to work on cutting-edge AI infrastructure
- Fully remote work environment
- Collaborative engineering-driven culture
- High technical ownership and impact
- Professional development opportunities
Interested
If this role sounds like a good fit please apply with your resume.
Full client details will be shared with qualified candidates before submission.
Key Skills
- Jenkins
- Ruby
- Python
- Active Directory
- Cloud
- PowerShell
- Windows
- AWS
- Linux
- SAN
- Java
- Troubleshoot
- Backup
- Puppet
- hardware
