What really happens when you run kubectl

Every kubectl command you have ever typed, every controller reconciling state, every kubelet heartbeat goes through one process: kube-apiserver. It is the only component that talks to etcd, the gatekeeper that decides who may do what, and the event bus every controller listens to.

I used it for years without really looking inside it. Then I took one apart on a plain kubeadm cluster, request by request, and a lot of "Kubernetes magic" turned into ordinary, inspectable mechanics. This post walks through what I found:

• The API server is just an HTTPS API, and you can drive it with curl.
• Authentication turns a credential into an identity, and Kubernetes has no User objects at all.
• Authorization decides on request attributes, and RBAC can only ever add permissions.
• Admission looks inside the object, and has a classic hole that most people miss.
• Audit logging tells you who did what, if you write the policy in the right order.
• etcd holds everything, and your Secrets are plaintext there until you change that.

Every command here is real, and I ran each one against a Kubernetes v1.37 cluster. At the end there's a link to a hands-on lab where you can do all of it yourself, with automatic checks and hints.

It's just HTTP#

kubectl is a convenient REST client, nothing more. Turn up its verbosity and you can watch the real calls go out:

kubectl get pods -n kube-system -v=6 2>&1 | grep -E 'GET|Response'

Every object lives at a predictable URL. Core resources sit under /api/v1, everything else under /apis/{group}/{version}, and subresources hang off the object's own path:

/api/v1/namespaces/{ns}/{resource}/{name}
/apis/{group}/{version}/namespaces/{ns}/{resource}/{name}
.../{name}/{subresource}        # status, scale, log, exec, eviction...

The server describes itself through discovery endpoints, which is how kubectl api-resources works. Try kubectl get --raw /apis/apps/v1 and notice the deployments/scale entry. We'll come back to it.

The first surprise came from sending a request with no credentials at all:

curl -sk https://cplane-01:6443/api/v1/namespaces

The answer is 403 Forbidden, not 401. The API server didn't reject me as unknown. It identified me as system:anonymous and then authorization said no. A 401 only happens when the credentials you send are invalid. That distinction is the whole pipeline in miniature: first who are you, then may you.

With the client certificate from my kubeconfig, curl can do anything kubectl does. Creating an object is a POST to the collection URL:

curl -s --cacert ca.crt --cert admin.crt --key admin.key \
  -X POST -H 'Content-Type: application/json' \
  https://cplane-01:6443/api/v1/namespaces/default/configmaps \
  -d '{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"raw-api"},"data":{"created-by":"curl"}}'

Changing it shows why PATCH exists. A PUT replaces the whole object, so any key you leave out is gone. A PATCH with Content-Type: application/merge-patch+json sends only the change. Afterwards, kubectl get cm raw-api -o yaml --show-managed-fields lists curl as the field manager, taken from my User-Agent. The API server records who owns every field.

One more thing: controllers don't poll. They open a watch, a long-lived request the API server streams changes into:

kubectl get --raw '/api/v1/namespaces/default/configmaps?watch=1'

Each ADDED, MODIFIED or DELETED event carries a growing resourceVersion. Where that number comes from turned out to be one of my favourite findings, further down.

Authentication: "who are you?"#

Kubernetes has no User objects. There is no kubectl create user. Instead, a chain of authenticators inspects each request, and the first one that recognises the credentials outputs an identity: a username, a list of groups, and some optional extras. Client certificates, ServiceAccount tokens, bootstrap tokens, OIDC and authenticating proxies all end up producing the same thing.

For X.509 client certificates the rule is simple: the certificate's CN is the username and each O is a group. My admin certificate said:

subject=O = kubeadm:cluster-admins, CN = kubernetes-admin

That O is the only reason I'm an admin. A ClusterRoleBinding grants the kubeadm:cluster-admins group the cluster-admin role. Nothing about the certificate itself is special.

So creating a new user is just getting a certificate signed by the cluster CA, and you don't even need the CA's private key for it. The CertificateSigningRequest API lets an admin approve a request, and the controller manager signs it:

openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev-team"
# wrap jane.csr in a CertificateSigningRequest with signer
# kubernetes.io/kube-apiserver-client, then:
kubectl certificate approve jane
kubectl get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt

With a kubeconfig built around that certificate, kubectl auth whoami reports user jane in groups dev-team and system:authenticated. It also shows a credential-id extra, the certificate's SHA-256 fingerprint. That fingerprint later shows up in audit logs, so you can tell which credential was used. Then kubectl get pods as Jane returns Forbidden: authenticated, but not authorized.

Workloads use ServiceAccount tokens instead. These are short-lived JWTs the API server signs itself. The payload is signed, not encrypted, so anyone holding a token can read it:

TOKEN=$(kubectl -n dev create token ci-bot --duration=1h)
echo "$TOKEN" | jq -R 'split(".")[1] | @base64d | fromjson'

The sub claim is system:serviceaccount:dev:ci-bot, and that is the username the API server uses for the token. ServiceAccounts also land in the groups system:serviceaccounts and system:serviceaccounts:dev, which are handy targets for RBAC.

Authorization: "may you do this?"#

Once a request has an identity, the API server reduces it to a handful of request attributes: user, groups, verb, API group, resource, subresource, namespace and name. It then asks each authorizer in --authorization-mode in turn, and each one can allow, deny, or have no opinion. On a kubeadm cluster that flag is Node,RBAC: the Node authorizer handles kubelets, and RBAC handles everyone else.

RBAC is purely additive. There are no deny rules, only permissions you grant. A Role lists permissions in one namespace, and a RoleBinding grants them to users, groups or ServiceAccounts.

The habit I'd recommend most is binding to groups, not people. Giving Jane's team read-only access to Pods in dev takes two commands:

kubectl -n dev create role pod-reader --verb=get,list,watch --resource=pods
kubectl -n dev create rolebinding dev-team-pod-reader --role=pod-reader --group=dev-team

Every future member of dev-team now gets the same access with no RBAC changes, because group membership travels in their certificate.

You never have to guess whether a permission works. Impersonation answers "what if I were Jane?":

kubectl auth can-i list pods -n dev --as jane --as-group dev-team     # yes
kubectl auth can-i delete pods -n dev --as jane --as-group dev-team   # no

And a SubjectAccessReview asks the authorizer the exact question it answers for real requests. Its reason even names the binding that allowed it: RBAC: allowed by RoleBinding "dev-team-pod-reader/dev" of Role "pod-reader" to Group "dev-team".

Admission: "is this object acceptable?"#

Authorization can say whether Jane may create a Deployment, but it never looks inside the object. Admission control does: it sees the full object and can change it (mutating admission) or reject it (validating admission). Reads skip admission entirely; it only runs for create, update, patch, delete and connect.

Custom rules used to mean running an admission webhook, a separate HTTPS service. Now you can write them in CEL as a ValidatingAdmissionPolicy, evaluated inside the API server. Here's one that caps Deployments at 3 replicas:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: max-replicas
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  validations:
  - expression: "object.spec.replicas <= 3"
    message: "Deployments in this namespace may not have more than 3 replicas."

A policy does nothing on its own. A ValidatingAdmissionPolicyBinding decides where it applies (I used a namespaceSelector on the dev namespace) and what happens on failure (Deny). With both in place, creating a 5-replica Deployment in dev is rejected, and 2 replicas go through.

Then I ran this:

kubectl -n dev scale deployment web --replicas=10

It worked. Ten replicas, policy or not.

The reason is in the URL. kubectl scale doesn't write to deployments; it writes to the deployments/scale subresource, a separate endpoint with its own object type (autoscaling/v1 Scale). My policy only matched deployments, so admission never even ran it. The fix is one word:

      resources: ["deployments", "deployments/scale"]

The same CEL expression works for both, because a Scale object also has spec.replicas. The lesson goes well beyond this policy: RBAC rules and webhooks have exactly the same trap. Whenever you restrict a resource, ask which subresources write to it too.

Audit logging: "what happened?"#

Audit logging records each request the API server handles: who made it, what it touched, when, from where, and whether it was allowed. It's off by default. Turning it on means editing the API server itself, which on kubeadm is a static Pod: a manifest in /etc/kubernetes/manifests/ that the kubelet runs directly, no API server required.

You need two flags, --audit-policy-file and --audit-log-path. Both paths are read inside the container, so the host directories also need a volumeMount and a hostPath volume each. Forgetting the mounts is the most common reason the API server doesn't come back.

Two things I learned the hard way. Keep your backup of the manifest outside /etc/kubernetes/manifests/, because the kubelet runs every file in that directory, backups included. And when kubectl stops answering for 30 to 60 seconds after you save, that's just the static Pod restarting. If it never comes back, sudo crictl ps -a --name kube-apiserver and journalctl -u kubelet tell you why.

The policy is where it gets interesting. For each request, the first matching rule sets the level: None, Metadata, Request or RequestResponse. My first attempt logged everything Jane did in full:

rules:
- level: RequestResponse
  users: ["jane"]
- level: Metadata

It worked: Jane's failed delete pod showed up with its full request, its 403, and authorization.k8s.io/decision: forbid. But it has a nasty side effect. If Jane ever reads a Secret, its contents land in the log in plain text, and audit logs get shipped to systems far more people can read than your Secrets.

The fix is a Secrets rule, and it has to come first:

rules:
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]
- level: RequestResponse
  users: ["jane"]
- level: Metadata

Put it after the Jane rule and it never fires for her, because her rule matches first. Order is the whole policy. Also note that the API server reads the policy file only at startup, so every change needs a restart.

Storage: what actually lands in etcd#

The API server is etcd's only client. Everything you kubectl get lives under the /registry/ prefix, in keys like /registry/configmaps/default/raw-api. etcdctl ships inside the etcd container, so with etcd's own client certificates you can look directly:

alias etcdctl='kubectl -n kube-system exec etcd-cplane-01 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key'

etcdctl get /registry/namespaces --prefix --keys-only

The values aren't JSON. Built-in types are stored as protobuf behind a k8s\0 magic prefix.

Now the finding I promised. etcd keeps one cluster-wide revision counter that grows with every write, and each key remembers the revision of its last change, mod_revision:

etcdctl get /registry/configmaps/default/raw-api -w json | jq '.kvs[0].mod_revision'
kubectl get cm raw-api -o jsonpath='{.metadata.resourceVersion}'

Same number. resourceVersion is the etcd revision of the object's last write. That's what makes optimistic concurrency work (an update must carry the current resourceVersion or it fails with a conflict), and it's how a watch can resume "from revision N".

Secrets are not secret by default#

In the API, Secret values are base64, which is encoding, not encryption. In etcd it's worse:

etcdctl get /registry/secrets/default/db-pass --print-value-only | grep -a -A1 password

That printed the password in plain text. Anyone with access to etcd, an etcd backup, or the disk can read every Secret in the cluster.

The API server can encrypt chosen resources on their way into etcd. You give it an EncryptionConfiguration through --encryption-provider-config:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets"]
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <32 random bytes, base64>
  - identity: {}

The first provider encrypts new writes, and all of them are tried on reads. Keeping identity: {} last is what lets the old plaintext data stay readable while you switch over.

After the restart, a new Secret's etcd value starts with k8s:enc:aescbc:v1:key1:. But db-pass was still plaintext. Encryption applies when an object is written, and nothing had rewritten it. The fix is to read every Secret and write it back unchanged:

kubectl get secrets -A -o json | kubectl replace -f -

In production, prefer the kms (v2) provider so the key-encryption key lives outside the node. A key in a file next to the cluster is better than plaintext, but only just.

What I took away#

• It's an HTTP API. Predictable URLs, discovery documents, and watches that stream changes. kubectl is optional.
• Authentication produces an identity, nothing more. Username plus groups, from a certificate's CN and O, a token, OIDC or a proxy. There are no User objects.
• Authorization works on request attributes. RBAC only adds; bind to groups; test with can-i, --as and SubjectAccessReview instead of guessing.
• Admission is where you look inside objects, and subresources like deployments/scale are separate doors you have to cover too.
• Audit policies are first-match-wins. Secrets need their own rule, placed first.
• etcd is the source of truth. resourceVersion is an etcd revision, and Secrets are plaintext until you enable encryption at rest and rewrite them.