Lead AI Engineer

Job Title: Lead AI Engineer
Location: Austin Texas (Hybrid)

Duration: Longterm Contract

Lead AI Engineer (Search Modernization)

Mandatory Skills: Elastic Search OpenSearch Python LLM GenAI Semantic Search Re-Ranking AWS Search Engineer

Job Description:

We are looking for an AI Engineer to modernize and enhance our existing regex/keyword-based Elastic Search system by integrating state-of-the-art semantic search dense retrieval and LLM-powered ranking techniques.
This role will drive the transformation of traditional search into an intelligent context-aware personalized and high-precision search experience.

The ideal candidate has hands-on experience with Elastic Search internals information retrieval (IR) embedding-based search BM25 re-ranking LLM-based retrieval pipelines and AWS cloud deployment.

Roles & Responsibilities

Modernizing the Search Platform

• Analyze limitations in current regex & keyword-only search implementation on ElasticSearch.
• Enhance search relevance using:
  • BM25 tuning
  • Synonyms analyzers custom tokenizers
  • Boosting strategies and scoring optimization
• Introduce semantic / vector-based search using dense embeddings.

2. LLM-Driven Search & RAG Integration

• Implement LLM-powered search workflows including:
  • Query rewriting and expansion
  • Embedding generation (OpenAI Cohere Sentence Transformers etc.)
  • Hybrid retrieval (BM25 vector search)
  • Re-ranking using cross-encoders or LLM evaluators
• Build RAG (Retrieval Augmented Generation) flows using ElasticSearch vectors OpenSearch or AWS-native tools.

3. Search Infrastructure Engineering

• Build and optimize search APIs for latency relevance and throughput.
• Design scalable pipelines for:
  • Indexing structured and unstructured text
  • Maintaining embedding stores
  • Real-time incremental updates
• Implement caching failover and search monitoring dashboards.

4. AWS Cloud Delivery

• Deploy and operate solutions on AWS leveraging:
  • OpenSearch Service or EC2-managed ElasticSearch
  • Lambda ECS/EKS API Gateway SQS/SNS
  • SageMaker for embedding generation or re-ranking models
• Implement CI/CD for search models and pipelines.

5. Evaluation & Continuous Improvement

• Develop search evaluation metrics (nDCG MRR recall).
• Conduct A/B experiments to measure improvements.
• Tune ranking functions and hybrid search scoring.
• Partner with product teams to refine search behaviors with real usage patterns.

Required Skills & Qualifications

• 510 years of experience in AI/ML NLP or IR systems with hands-on search engineering.
• Strong expertise in ElasticSearch/OpenSearch: analyzers mappings scoring BM25 aggregations vectors.
• Experience with semantic search:
  • Embeddings (BERT SBERT Llama GPT-based Cohere)
  • Vector databases or ES vector fields
  • Approximate nearest neighbor (ANN) techniques
• Working knowledge of LLM-based retrieval and RAG architectures.
• Proficient in Python; familiarity with Java/Scala is a plus.
• Hands-on AWS experience (OpenSearch SageMaker Lambda ECS/EKS EC2 S3 IAM).
• Experience building and deploying APIs using FastAPI/Flask and containerizing with Docker.
• Familiar with typical IR metrics and search evaluation frameworks.

Preferred Skills

• Knowledge of cross-encoder and bi-encoder architectures for re-ranking.
• Experience with query understanding spell correction autocorrect and autocomplete features.
• Exposure to LLMOps / MLOps in search use cases.
• Understanding of multi-modal search (text images) is a plus.
• Experience with knowledge graphs or metadata-aware search.