Load balancing vector queries
Load balancing spreads incoming vector query traffic across multiple nodes (coordinators or shards) so that no single node is overwhelmed and throughput and latency stay within targets. It is essential when you have multiple replicas or coordinator instances behind a single endpoint.
Summary
- Load balancing spreads query traffic across multiple nodes (coordinators or shards) so no single node is overwhelmed; keeps throughput and latency within targets. Essential with multiple replicas or coordinator instances.
- Strategies: round-robin, least connections, latency-based, consistent hashing. For vector search, stateless coordinators make round-robin or least-connections effective. Can sit in front of coordinators (L4/L7, Envoy) or inside the VDB client/gateway.
- Works with replication (read traffic across replicas) and hot shard mitigation; failing nodes taken out of the pool. Pipeline: request arrives, LB selects node (round-robin/least-conn/latency), forward to node. Practical tip: use least-connections for variable-cost vector queries; health-check coordinators and remove failed nodes.
Load balancing strategies
Common strategies: Round-robin—rotate requests across a list of nodes; simple but ignores current load. Least connections—send to the node with the fewest active connections; better for long-running or variable-cost queries. Latency-based—prefer the node with lowest observed latency or health. Consistent hashing—stick a client or request key to a node for cache affinity when applicable. For vector search, stateless coordinators make round-robin or least-connections effective; stateful sessions may need sticky routing.
Placement and integration
Load balancing can sit in front of coordinators (L4/L7 load balancer, or a proxy like Envoy) or be implemented inside the VDB client or gateway. It should work with replication (read traffic across replicas) and hot shard mitigation so that traffic is distributed fairly and failing nodes are taken out of the pool. Pipeline: request arrives at LB, LB selects node (round-robin, least-connections, or latency-based), forwards to node. Practical tip: use least-connections for variable-cost vector queries; health-check coordinators and remove failed nodes from the pool.
Frequently Asked Questions
Why load balance vector queries?
To avoid overwhelming a single node and keep throughput and latency within targets. Essential when you have multiple replicas or coordinator instances behind one endpoint.
What strategies are common?
Round-robin (rotate across nodes), least connections (send to node with fewest active connections), latency-based (prefer lowest latency/health), consistent hashing (stick client to node for cache affinity). Stateless coordinators make round-robin or least-connections effective. See coordinator role.
Where does load balancing sit?
In front of coordinators (L4/L7 load balancer, or proxy like Envoy) or inside the VDB client or gateway. Should work with replication and hot shard mitigation so traffic is distributed fairly.
How does it interact with hot shards?
Load balancing distributes traffic across replicas; if one shard is hot, adding replicas and load balancing across them can mitigate. Failing nodes should be taken out of the pool. See throttling and rate limiting.