Open Source Summit Korea 2026

Standard load balancers route LLM requests without awareness of KV-cache state or GPU queue depth — causing inflated Time-To-First-Token and wasted accelerator capacity.

loxilb closes this gap with an eBPF-native AI gateway. The L4 layer uses XDP/TC and kernel sockmap for zero-copy forwarding. The L7 layer adds API-key validation, token-quota enforcement, and accelerator-aware routing via Consistent Hashing with Bounded Loads (CHWBL).

This talk focuses on KV-exact routing for P/D disaggregated deployments. When requests share a prompt prefix, prefill GPUs recompute identical KV tensors repeatedly — wasting up to 80% of GPU cycles. Tier 1.5 eliminates this: loxilb tokenizes the prompt in-process (HuggingFace Rust tokenizer via CGO), computes block hashes matching vLLM's internal format, and routes to the exact GPU already holding those KV blocks — via a live inventory fed by vLLM's native ZMQ event stream.

Unlike serving-layer schedulers (llm-d, Dynamo), this runs in the eBPF data-plane hot path with no Kubernetes dependency — works on bare metal, VMs, and BlueField DPUs.

We'll trace a live request through the P/D testbed and share lessons from building GPU-state-aware routing