Kubernetes cgroup v1 Deprecation: Fix a Kubelet That Won't Start and Migrate Nodes to cgroup v2

You upgrade a node pool to Kubernetes 1.35 or later, the node never goes Ready, and journalctl -u kubelet shows the kubelet exiting on every restart with a line containing kubelet is configured to not run on a host using cgroup v1. Nothing in your manifests changed. What changed is a default: since Kubernetes v1.35 the kubelet setting failCgroupV1 is true, so a node that still boots with cgroup v1 is refused outright. On October 6, 2026 the Kubernetes project published a long explainer on this shift. It is a good sign that the next step is coming: actual removal of cgroup v1 code is tracked in KEP-5573 and is planned for no earlier than v1.38.

This guide covers how to find the affected nodes before they break, the temporary override (and why it is only temporary), the real fix, and the application-level surprises that show up after the move.

What changed, version by version

For kubeadm clusters the failure comes earlier and louder. With a v1.35+ kubelet, the SystemVerification preflight check returns an error during kubeadm init, kubeadm join and kubeadm upgrade when it detects cgroup v1. With an older kubelet it is still only a warning, which is why people miss it on the way up.

Why your nodes are still on cgroup v1

The cgroup version is decided by the OS at boot, not by Kubernetes. If a node is on v1 today, it is almost always one of these:

• An older distribution. Ubuntu 20.04 and earlier, RHEL/Rocky/Alma 8 and earlier, and Amazon Linux 2 boot with cgroup v1 by default.
• A kernel argument forcing v1. Someone added systemd.unified_cgroup_hierarchy=0 years ago because Docker or a monitoring agent did not support v2 yet, and it is still in the image.
• A long-lived custom AMI or golden image built before the base distro switched.

Distributions that default to cgroup v2, per the Kubernetes documentation: Ubuntu 21.10+ (22.04+ recommended), Debian 11+, RHEL and RHEL-like 9+, Fedora 31+, Container-Optimized OS M97+, Arch Linux since April 2021. On Amazon EKS, AL2023 and Bottlerocket use cgroup v2, and EKS stopped publishing Amazon Linux 2 AMIs after Kubernetes 1.32, so managed EKS node groups on current versions are already fine. The risk on EKS is self-built AL2-based AMIs.

Step 1: Find cgroup v1 nodes before you upgrade

Start with a quick triage from the API. The node object does not report the cgroup version, but the OS image and kernel narrow it down fast:

1kubectl get nodes -o custom-columns=\
2NAME:.metadata.name,\
3OS:.status.nodeInfo.osImage,\
4KERNEL:.status.nodeInfo.kernelVersion,\
5RUNTIME:.status.nodeInfo.containerRuntimeVersion,\
6KUBELET:.status.nodeInfo.kubeletVersion

Anything showing Ubuntu 20.04, RHEL 8, Amazon Linux 2 or a kernel older than 5.8 goes on the list. Then confirm on the node itself. The canonical check is:

1stat -fc %T /sys/fs/cgroup/
2# cgroup2fs  -> cgroup v2
3# tmpfs      -> cgroup v1

If you do not have SSH, kubectl debug node/<node> -it --image=debian:bookworm-slim gives you a shell with the host filesystem at /host. Use a Debian or Ubuntu image there, not BusyBox: when I ran the same stat command on a cgroup v2 host, debian:bookworm-slim printed cgroup2fs while alpine:3.20 printed UNKNOWN, because BusyBox stat does not know the filesystem name. A check that works in any image is:

1test -f /host/sys/fs/cgroup/cgroup.controllers && echo v2 || echo v1

cgroup.controllers only exists at the root of a cgroup v2 hierarchy. Remember to delete the node-debugger-* Pod afterwards.

Step 2: If a node is already down, the temporary override

If production nodes are crash-looping right now, you can buy time by setting this in the kubelet configuration file (commonly /var/lib/kubelet/config.yaml):

1apiVersion: kubelet.config.k8s.io/v1beta1
2kind: KubeletConfiguration
3failCgroupV1: false

Then sudo systemctl restart kubelet. Treat this as a ticket with a deadline, not a fix. cgroup v1 is deprecated, gets no new features, and the code path is scheduled for removal no earlier than v1.38. Several newer features simply do not work on v1 at all (see the table in Step 5).

Step 3: The real fix, move the node to cgroup v2

The best option is to replace the node with one built from a distribution that already defaults to v2: Ubuntu 22.04/24.04, RHEL 9, Debian 12, AL2023 or Bottlerocket. Node pools are cattle; rebuilding is usually less risky than editing kernel arguments in place.

If you must convert an existing image, check the prerequisites first:

• Linux kernel 5.8 or later (5.9+ if you plan to use Memory QoS)
• containerd v1.4+ (v2.0+ if you want automatic cgroup driver detection) or CRI-O v1.20+ (v1.28+ for auto-detection)
• The kubelet and the runtime using the same cgroup driver, ideally systemd

Then switch the boot argument. On GRUB-based Debian/Ubuntu systems:

1# /etc/default/grub
2GRUB_CMDLINE_LINUX="... systemd.unified_cgroup_hierarchy=1"
3
4sudo update-grub

On RHEL-family systems:

1sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1"

Also remove any systemd.unified_cgroup_hierarchy=0 that is already present. Then drain, reboot, and verify:

1kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
2# reboot the node
3stat -fc %T /sys/fs/cgroup/        # expect cgroup2fs
4kubectl uncordon <node>

For the cgroup driver, match the runtime. With containerd that is SystemdCgroup = true under the runc options in /etc/containerd/config.toml, and on the kubelet side:

1apiVersion: kubelet.config.k8s.io/v1beta1
2kind: KubeletConfiguration
3cgroupDriver: systemd

On containerd v2.0+ or CRI-O v1.28+ the kubelet asks the runtime which driver it uses (KEP-4033, stable since v1.34), and the runtime's answer wins over cgroupDriver.

Step 4: Check the workloads, not just the nodes

The kubelet adapts to cgroup v2 automatically. Your applications might not. Anything that reads the cgroup filesystem directly to discover its own limits needs a v2-aware version. The minimums from the Kubernetes docs:

The Node.js row is the one that bites. I ran both versions on a cgroup v2 Docker host with a 512 MiB memory limit (docker run --memory=512m ...) and printed V8's heap limit:

1node:18-alpine  v18.20.8  heap_size_limit_MB= 2096  constrainedMemory_MB= NaN
2node:22-alpine  v22.23.3  heap_size_limit_MB= 259   constrainedMemory_MB= 512

Node 18 sized its heap at about 2 GiB inside a 512 MiB container. It will not garbage-collect hard until it is far past the limit, so the container gets OOM-killed instead. Node 22 saw the limit and sized the heap to roughly half of it. If you cannot upgrade a Node 18 service yet, set --max-old-space-size explicitly. Java 21 behaved correctly in the same test: with --memory=512m, MaxHeapSize came out at 134217728 bytes (128 MiB), the default 25% of the container limit.

Also update monitoring and security agents (node exporters, eBPF agents, older Datadog/Falco/Sysdig builds) that hard-code v1 paths like /sys/fs/cgroup/memory/. On v2 there is one unified tree, so those paths no longer exist.

Step 5: Know what behaves differently on cgroup v2

A few changes are visible after the migration even when everything is healthy.

OOM kills take the whole container. On cgroup v2 the kubelet defaults singleProcessOOMKill to false and sets memory.oom.group on each container's cgroup, so an OOM kills every process in that container together. On v1 the kernel could kill one worker and leave a half-alive container behind. Set singleProcessOOMKill: true only if you depend on the old behavior.

CPU weights look different. cgroup v1 used cpu.shares; v2 uses cpu.weight. The kubelet still sends v1-style shares, and the OCI runtime converts them. Newer runtimes (crun v1.23, runc v1.3.2 and later) use a non-linear formula that gives small CPU requests more usable weight. On a host with runc 1.3.5, I read cpu.weight from containers started with different --cpu-shares values:

1shares=2    -> cpu.weight 1
2shares=102  -> cpu.weight 17     (a 100m Kubernetes CPU request)
3shares=512  -> cpu.weight 59
4shares=1024 -> cpu.weight 100
5shares=2048 -> cpu.weight 174

For comparison, the older linear formula (1 + ((shares - 2) * 9999) / 262142) works out to 4 for 102 shares and 39 for 1024 shares. I calculated those two from the formula rather than observing them on an old runtime. Scheduling still works either way. What breaks is a dashboard or policy tool that predicts exact cpu.weight values.

Features that require cgroup v2:

Common mistakes to avoid

• Setting failCgroupV1: false and closing the ticket. It is a bridge to v1.38 at the latest, not a fix.
• Switching the kernel argument without matching cgroup drivers. A systemd runtime with a cgroupfs kubelet (or the reverse) gives you a node that starts and then misbehaves under pressure.
• Rolling the whole pool at once. Convert one node, run real workloads on it for a day, then roll out the rest.
• Forgetting old runtimes in the image. Kernel 5.8+ is the documented minimum; a v2 boot argument on an old kernel or containerd 1.3 gets you a different set of errors.
• Trusting BusyBox for the check. As shown above, it prints UNKNOWN, which looks like neither answer.

Troubleshooting

Node stays NotReady after the reboot. Check journalctl -u kubelet -b and journalctl -u containerd -b. A cgroup driver mismatch usually shows up as an error mentioning the driver right at startup.

Pod limits don't seem to apply. Follow a container down to the kernel. On the node, with the systemd driver:

1CONTAINER_ID=$(crictl ps --label io.kubernetes.pod.name=<pod> --name <container> -q | head -n1)
2PID=$(crictl inspect "$CONTAINER_ID" | jq -r '.info.pid')
3CGROUP="/sys/fs/cgroup$(awk -F: '$1=="0"{print $3}' /proc/$PID/cgroup)"
4cat "$CGROUP/memory.max" "$CGROUP/cpu.max" "$CGROUP/cpu.weight"

cpu.max holds the CPU limit as quota period (for example 50000 100000 for 500m), memory.max holds the memory limit in bytes, and cpu.weight reflects the request, not a limit. I checked the same files inside a Docker container started with --memory=256m --cpus=0.5 and got 268435456, 50000 100000 as expected.

Java or Node services get OOM-killed right after migration. Check the runtime version against the table in Step 4 before raising memory limits.

FAQ

Will Kubernetes 1.35+ run on a cgroup v1 node at all? Only if you set failCgroupV1: false in the kubelet config. kubeadm-managed clusters also fail preflight on v1 with a v1.35+ kubelet.

When will cgroup v1 support be removed completely? KEP-5573 tracks removal and says it will happen no earlier than v1.38, in line with the Kubernetes deprecation policy.

Do I need to change any manifests? No. Resource requests and limits work the same way. The changes are on the node, plus any application that reads cgroup files directly.

Is this a problem on managed services like EKS, GKE or AKS? Mostly not, if you use the provider's current node images, which default to cgroup v2. Custom images and older OS families are where the risk is. On EKS that means self-built Amazon Linux 2 AMIs.

Does docker info tell me the cgroup version? Yes, docker info --format '{{.CgroupVersion}}' prints 1 or 2 on hosts running Docker. For Kubernetes nodes running containerd only, use the stat check instead.

What I tested and what I didn't

The container-level results above (the stat output difference, cpu.weight values, Node.js and Java heap sizing, memory.max and cpu.max) were run on Docker Engine 29.5.2 with runc 1.3.5 and containerd 2.2.4 on a 6.12 kernel, which uses cgroup v2. I did not run a kubelet on a cgroup v1 host, run kubeadm preflight, or change GRUB arguments on a real node. The kubelet error text, the kubeadm behavior, version requirements and the removal timeline come from the Kubernetes blog post of October 6, 2026, the Kubernetes cgroup v2 documentation, KEP-5573, and published field reports. Test the boot-argument change on one non-critical node first.

Key takeaways

Further Reading

• Amazon EKS 1.37 Upgrade Guide: SELinuxMount, Kubelet Changes and the Auto Mode Default That Flips
• Docker vs containerd: Understanding the Shift in Kubernetes 1.24+
• Kubernetes 1.37 HPA Scale-to-Zero: Setup Guide and Cold-Start Gotchas
• Kubernetes blog: The Shift to cgroup v2 in Kubernetes: What You Need to Know
• Kubernetes docs: About cgroup v2