We scaled pricing from two Pods to ten, but two Pods still do the work. In this illustrative chart, p1 and p2 handle about five hundred requests per second each, while p3 through p10 see only a handful. The new Pods are running and healthy. Why aren't they getting traffic?

The ClusterIP is a virtual address, not a real Pod. When an order Pod opens a connection to it, kube-proxy rules match that address and pick a backend, here pricing-1. That choice happens once, at connect time. After that, the connection stays pinned to pricing-1.

Follow one order Pod. It opens a connection, kube-proxy routes it to pricing-1, and that's the only decision made. gRPC runs on HTTP/2, so every GetPrice call becomes a stream on that same connection. The replies come back, the next requests go out, and they all follow the original pick. Meanwhile, pricing-7 is never chosen.

On the left, balancing happens per connection. One decision is made when the client connects, then one thick pipe carries every request to a single Pod while the others sit empty. On the right, balancing happens per request, so each call can land on a different Pod. Only that second model helps immediately when you scale.

A ClusterIP Service deals out connections, not requests. And an HTTP/2 connection stays open, so every request rides the same Pod it first landed on.

Watch one order Pod. Its traffic keeps flowing to pricing-1 over the old connection. Nothing sends work to pricing-7 until that Pod opens a new connection, say after a restart or reconnect. Then kube-proxy picks pricing-7, and the new GetPrice call lands there. Until reconnects happen, load evens out slowly, if ever.

This skew isn't unique to gRPC. HTTP/1.1 keep-alive pools hold several connections, so the skew is milder. WebSockets pin like gRPC does. External traffic through an L7 proxy is often balanced per request, so internal service-to-service calls suffer most.

The first fix moves balancing into the client. A headless Service skips the single virtual IP, so DNS returns every pricing Pod IP. The order client connects to each one and spreads requests, for example round robin. The catch is that it has to re-resolve DNS, or it won't notice new Pods.

Here, the Order Pod keeps one connection, but it terminates at a sidecar proxy. Because the proxy understands HTTP/2 streams, it picks a Pod for each request, so pricing-1 and pricing-7 both get traffic. The client doesn't change. The cost is more components, more configuration, and more operational complexity.

The third option is a server-side maximum connection age. When pricing-1 reaches that age, it sends GOAWAY, which drains in-flight calls. The order Pod reconnects through kube-proxy, which may now pick pricing-7, and new GetPrice calls flow there. The trade-off is periodic handshake overhead, and rebalancing only happens gradually, as connections expire.

Diagnose in order. First, compare per-Pod request rates; skew toward pricing-1 and pricing-2 is the symptom. Then count connections each order Pod holds; one or a few confirms pinning. Finally, check for an L7 proxy in the path, since that would rule this out.

One caveat: kube-proxy modes and replacement dataplanes differ in the details, so check how connections actually behave in your own cluster instead of assuming.

A Service deals out connections; HTTP/2 rides one connection forever. So for gRPC between services, plan request-level balancing before you scale your Pods.