Backend Engineer AI Gateway

We are building an AI Gateway a high-performance secure intermediary layer that routes inspects and governs all traffic between enterprise clients and LLM/AI service providers. This role requires deep expertise in network programming async Python and cloud infrastructure to design and operate a latency-sensitive policy-driven gateway that handles massive concurrent streaming connections at scale.

What we are looking for

• Async Python & Network Programming (3  yrs) Strong handson experience with asyncio aiohttp/anyio lowlevel TCP/UDP sockets WebSocket & SSE servers TLS handshake connectionpooling keepalive and zerocopy buffering.
• Streaming & LargePayload Handling Ability to manage chunked transfer bidirectional streaming backpressure and highthroughput data paths for realtime LLM inference.
• Protocol Knowledge Working familiarity with HTTP/2 & HTTP/3 (multiplexing HPACK) gRPC and occasional custom protocol parsers.
• FastAPIBased Backend (3  yrs) Design develop and test productiongrade REST/Streaming APIs; pragmatic use of Pydantic dependency injection and OpenAPI docs.
• Containerization & Kubernetes Docker AKS/EKS/GKE deployment experience; HPA/VPA scaling rolling updates and servicemesh (e.g. Istio/Envoy) basics.
• Async Messaging & EventDriven Design Production use of Redis Streams RabbitMQ or Kafka for decoupled task queues retries and backoff logic.
• PostgreSQL & Vault Advanced query tuning partitioning PgBouncer pooling plus HashiCorp Vault (dynamic secrets transit encryption) for key management.
• Cloud Networking & Security L4/L7 load balancing TLS termination reverseproxy (NGINX/Envoy) mTLS between services and automated cert lifecycle (ACME/Lets Encrypt).
• IaC & CI/CD Terraform modules for multienv infra plus GitHub Actions/GitLab CI pipelines with container image scanning and canary/bluegreen deployments.

• Availability to work onsite in New Jersey for the initial assignment and relocate to Dallas by October 2026

Qualifications :

• Enterprise Security Gateways Integration with CASB/DLP/WSS or similar edgesecurity platforms; centralized logging & audit trails.
• IAM & SSO Configuring Azure AD/Entra ID Okta SAML 2.0 or OIDC SSO flows; managing Conditional Access & SCIM provisioning.
• AI/LLM Pipelines Building promptengineering workflows PII anonymisation and monitoring Azure OpenAI (or comparable) usage rate limits & cost.
• APIGateway Operations Custom webhook/REST integrations circuitbreaker patterns ratelimiting autoretry and healthchecking.
• Observability & FinOps Endtoend monitoring with Datadog Prometheus/Grafana or Azure Monitor; proactive costoptimisation and 24/7 incident response (SLA/SLO).

   

Additional Information :

All your information will be kept confidential according to EEO guidelines.

*****NO C2C******

Remote Work :

No

Employment Type :

Contract