Why does a healthy Kubernetes Pod get 403 Forbidden from the API?

The Pod is running, probes are green, and the API still says 403. Probes only test the app itself, not its permissions. But look at the error. It already names the user, the verb, list, the resource, pods, and the namespace, shop.

The Pod sends its list request with the service account token. The API server accepts that token, so authentication passes. If it hadn't, you'd see a 401. Next, the server asks RBAC whether this account may list pods. RBAC finds no matching rule, and the default is deny. The server returns 403 Forbidden.

The Pod spec names inventory-sa in serviceAccountName. The kubelet mounts that account's token into the Pod, and the app sends it as a Bearer header. The API server then authenticates it as system:serviceaccount:shop:inventory-sa. If you omit the name, the Pod uses the default service account.

RBAC has two pieces. The Role pod-reader holds the rules: which resources, and which verbs, like list on pods. But a Role alone grants nothing. A RoleBinding has to connect it to a subject. In our case, no binding names inventory-sa, so nothing allows the call.

Don't guess, ask the API server. The as flag impersonates the exact service account, so the answer reflects its real permissions. It prints no, which confirms authorization is denying us. Then can-i with list shows current grants, and get rolebindings shows whether any binding names inventory-sa.

The Role grants only get and list on pods. The empty string in apiGroups means the core group, where pods live. The RoleBinding ties it to inventory-sa, and name and namespace must match exactly. Rerun can-i, and it should now print yes, only inside shop.

Now the chain is complete. The inventory Pod authenticates as inventory-sa. The RoleBinding matches that subject and points through its roleRef to Role pod-reader, which allows listing pods in shop. The request ends in 200 OK. The token proves who you are. The binding decides what you may do.

The binding kind decides scope. A RoleBinding grants permissions only inside its own namespace, even when it points at a ClusterRole. A ClusterRoleBinding grants them across every namespace in the cluster. Our inventory app only needs shop, so a RoleBinding is the narrowest fit.

A RoleBinding in namespace shop references a ClusterRole that allows listing pods. Where can inventory-sa list pods?

Only in the shop namespace. The binding, not the role, sets the scope. A RoleBinding grants a ClusterRole's rules only inside its own namespace, which is a common way to reuse rule sets. Cluster-wide access would require a ClusterRoleBinding.

The fix should stay small. Grant only get and list, bind inside the namespace, and give each workload its own service account. Don't reach for cluster-admin or wildcards, because if this app is compromised, that's the blast radius.

Authentication says who you are; authorization says what you may do. A healthy Pod can still be forbidden. On a 403, read the user in the error, run can-i with --as, then add the narrowest binding that fixes it.