Mohsin Sheikhani

Mohsin Sheikhani

Senior AI Engineer
Pakistan
Open to Offers
Full-time Work from Home

About Me

Senior AI Engineer with 6+ years of backend and cloud engineering on AWS, the last 2+ shipping LLM agents, RAG and multi-agent systems to production in Python. Specializes in evaluation and reliability: eval harnesses, c…

Experience

AI Engineer

VenianAI · Dubai (Remote)
Aug 2025 - Present · 1 year 3 months

<p>Build a maintenance-triage agent in Python (LangGraph, FastAPI, Postgres, MCP) with guardrails and human-in-the-loop clarification, resolving 60-85% of requests end to end and cutting median human touch time from 4 hours to 12 minutes across 1,400 tickets.</p><p> Design the agent's evaluation harness from error analysis of 150 Langfuse traces, grouping 86 silent failures into 8 failure modes, then build 14 graders (11 code-based, 3 LLM-as-judge) with judges calibrated to 95% TPR / 93% TNR against human labels.</p><p> Gate every prompt change in CI on the eval suite, blocking merges on pass-rate regressions, and score sampled production traffic with online LLM judges, with label-based rollback.</p><p> Route all model calls through a LiteLLM AI gateway with provider fallback and cost budgets; benchmark 5 models there on the same eval set, picking one at 15x lower cost per correct outcome, and catch an unbounded tool loop inflating spend 100x.</p><p> Build a RAG pipeline (Haystack, Qdrant) with structure-aware chunking, hybrid BM25 + dense search and cross-encoder reranking, raising MRR@5 from 0.31 to 0.73 and Hit Rate@5 from 18% to 87% over a dense-only baseline.</p><p> Cut P95 time-to-first-token on repeat queries from 3.2s to 1.3s (N=120) with Redis semantic caching on a streaming FastAPI service.</p>

Cloud Software Architect

McCarthy Tire Services · US (Remote, Contract)
Jan 2025 - Jul 2025 · 5 months

<p>Served product recommendations at 2.3s P95 from a three-zone S3 data lake (42k interactions, 6k users) using Firehose, Glue ETL, Amazon Personalize and Lambda, with nightly demand forecasts generated on Amazon Bedrock.</p><p> Cut new AWS environment setup from 12 days to 1 by standardizing account structure, IAM and networking for 9 environments across 6 teams as reusable Terraform modules.</p><p> Turned requirements from product, sales and client success teams into cloud system designs, working with each team directly.</p>

Cloud Software Engineer

SYDT OÜ · Tallinn, Estonia (Remote, Contract)
Jan 2024 - Jan 2025 · 1 year

<p>Cut manual VAT filing effort 95% with a GDPR-compliant document pipeline on Step Functions and Lambda that extracts invoice data with OCR and enriches it with OpenAI LLMs, secured with KMS encryption and automated virus scanning.</p><p> Processed tens of thousands of transactions a month through event-driven microservices (Lambda, ECS, SQS, EventBridge, PostgreSQL) with retries and failure isolation.</p><p> Cut MTTR 50% with X-Ray tracing, CloudWatch dashboards and alarms wired to auto-remediation Lambdas.</p>

Cloud Solutions Engineer

Zapdas Technologies · Karachi, Pakistan
Mar 2020 - Dec 2023 · 3 years 9 months

<p>Held 99.99% uptime for a 24-microservice ERP platform on ECS Fargate (Docker) with autoscaling and blue-green releases.</p><p> Cut idle compute cost 70% and removed manual handoffs by moving ERP data sync and reporting to event-driven serverless workflows (Lambda, API Gateway, EventBridge, SQS).</p>

PROJECTS

Customer Support Automation

Duration : 11-Sep-2025 - 30-Sep-2025

<p>LangGraph-powered customer support automation with multi-component agentic workflow, AWS CDK infrastructure and MCP server, RAG with Pinecone. AI agents parse emails, fetch tracking data, apply policies, update tickets and generate responses automatically.</p>

Certifications

AWS Agentic AI Demonstrated

Amazon Web Services · Karachi , Sindh , Pakistan · 2025

Skills

Evaluation harnesses Offline evals Online evals Multi-turn evals LLM-as-judge RAGAS DeepEval Golden datasets Synthetic test data Langfuse OpenTelemetry AWS X-Ray CloudWatch LLM applications RAG Embeddings Chunking Hybrid search Reranking Prompt engineering Tool calling MCP Agentic workflows Multi-agent orchestration A2A Guardrails Human-in-the-loop OCR Document extraction LangGraph LangChain OpenAI Agents SDK Google ADK Strands Agents Haystack Temporal Amazon Bedrock Amazon Bedrock AgentCore GPT Claude Llama Qwen Mistral Ollama LiteLLM Prompt versioning CI eval gates Semantic caching Qdrant pgvector
Report this Profile?