Why does kubectl apply get Forbidden in production but not staging?

The same deploy job configures web in staging, then gets Forbidden in production. Same ServiceAccount, same manifest. But read the error closely. It already tells you who asked, which verb, which resource, and which namespace. That's everything you need to find what's missing.

Every request hits two gates inside the API server. First, authentication asks who you are. Here, it answers ci:deployer, so that gate passes. Then authorization, RBAC, asks whether this identity may patch in production. It finds no matching grant, and the API server returns 403 Forbidden. The token was never the problem. The failure is gate two.

A 401 means the server couldn't establish who you are, usually because the token is missing, expired, or invalid. A 403 means it knows exactly who you are and still says no. Our error is a 403, so the credentials are fine.

Here's the wiring. The ClusterRole deploy-edit lists what's allowed: verbs on deployments. By itself it grants nothing. A RoleBinding connects it to a subject. In staging, a RoleBinding references that role and grants it to the ServiceAccount deployer from the ci namespace. In production, there's no binding at all. Same identity, same role, but nothing connects them there.

The ClusterRole lists the verbs allowed on deployments, including patch. The RoleBinding's roleRef picks that role, and subjects names the deployer ServiceAccount from the ci namespace. The binding's namespace, staging, is what limits where those permissions apply. I've left out apiVersion here for brevity.

A ClusterRole is only a set of permissions. It doesn't say where they apply. The binding decides that. Bind deploy-edit with a RoleBinding, and the grant exists in one namespace, staging. Bind it with a ClusterRoleBinding, and it would apply in all namespaces. That's why production stays closed.

RBAC has no deny rules to hunt for. There are only grants. If no binding exists in a namespace, there's no access there.

Take the identity, verb, resource, and namespace from the error and feed them to kubectl auth can-i with --as. Staging says yes, production says no. That confirms the gap. Then create a RoleBinding in production to deploy-edit, not cluster-admin. One caveat: impersonating with --as needs its own permission.

RBAC isn't the only gate. Every request passes authentication, then authorization, which can include other modes like webhooks, and then admission before anything is persisted. So if can-i says yes but the apply still fails, don't keep editing bindings. Read the error text, because it usually names the component that rejected you.

RBAC is wiring: no binding in a namespace, no access there. So a 403 means the identity was recognized but unbound. Check it with kubectl auth can-i, then add a scoped RoleBinding in that namespace. You don't need cluster-admin, just the missing wire.