profile
viewpoint

Ask questionsKubelet Unable To Apply Reserved Cgroup Limits because Cgroup does not exist

I have been trying to reserve compute resources following https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/

However, when I deploy an instance group with kops that instance group comes up as NotReady and has the following errors:

4m 4m 1 kubelet, ip-172-20-180-67.ec2.internal Normal Starting Starting kubelet. 4m 4m 1 kubelet, ip-172-20-180-67.ec2.internal Warning FailedNodeAllocatableEnforcement Failed to enforce System Reserved Cgroup Limits on "system.slice": "system.slice" cgroup does not exist 4m 4m 1 kubelet, ip-172-20-180-67.ec2.internal Warning KubeletSetupFailed Failed to start ContainerManager Failed to enforce System Reserved Cgroup Limits on "system.slice": "system.slice" cgroup does not exist 4m 4m 1 kubelet, ip-172-20-180-67.ec2.internal Normal NodeAllocatableEnforced Updated Node Allocatable limit across pods 4m 1m 18 kubelet, ip-172-20-180-67.ec2.internal Normal NodeHasSufficientDisk Node ip-172-20-180-67.ec2.internal status is now: NodeHasSufficientDisk 4m 1m 18 kubelet, ip-172-20-180-67.ec2.internal Normal NodeHasSufficientMemory Node ip-172-20-180-67.ec2.internal status is now: NodeHasSufficientMemory 4m 1m 18 kubelet, ip-172-20-180-67.ec2.internal Normal NodeHasNoDiskPressure Node ip-172-20-180-67.ec2.internal status is now: NodeHasNoDiskPressure

I can't seem to figure out how to point to the right cgroups.

Here is my instance group spec:

apiVersion: kops/v1alpha2 kind: InstanceGroup metadata: labels: kops.k8s.io/cluster: xxxxxxxxx name: nodes spec: kubelet: kubeReserved: cpu: "200m" memory: "200Mi" storage: "2Gi" kubeReservedCgroup: "/kube.slice" systemReserved: cpu: "200m" memory: "200Mi" storage: "2Gi" systemReservedCgroup: "/system.slice" enforceNodeAllocatable: "pods,system-reserved,kube-reserved" image: coreos.com/CoreOS-stable-1409.9.0-hvm machineType: m3.large maxSize: 3 minSize: 3 role: Node subnets:

  • us-east-1a
  • us-east-1b
  • us-east-1c
  • us-east-1e
  • us-east-1d
  • us-east-1f

I've tried many different values and different operating systems. I feel like I'm missing something.

kubernetes/kops

Answer questions alex88

In my case I had this problem after upgrading from 1.11.9 to 1.13.10 and I with the kubeleted reserved resources kubelet had this error:

kubelet_node_status.go:338] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=eu-west-1a
kubelet_node_status.go:342] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=eu-west-1
setters.go:144] replacing cloudprovider-reported hostname of ip-10-0-1-80.eu-west-1.compute.internal with overridden hostname of ip-10-0-1-80.eu-west-1.compute.internal
kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ip-10-0-1-80.eu-west-1.compute.internal
kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ip-10-0-1-80.eu-west-1.compute.internal
kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ip-10-0-1-80.eu-west-1.compute.internal
cpu_manager.go:155] [cpumanager] starting with none policy
cpu_manager.go:156] [cpumanager] reconciling every 10s
policy_none.go:42] [cpumanager] none policy: Start
node_container_manager_linux.go:106] Enforcing System reserved on cgroup "" with limits: map[]
kubelet.go:1384] Failed to start ContainerManager Failed to enforce System Reserved Cgroup Limits on "": [] cgroup is not config properly

and this config:

  kubelet:
    enforceNodeAllocatable: pods,system-reserved,kube-reserved
    kubeReserved:
      cpu: 100m
      ephemeral-storage: 1Gi
      memory: 100Mi
    kubeReservedCgroup: /kube-reserved

what I can tell is that the argument --kubelet-cgroups="" was empty so it was trying to set resources limits on an empty cgroup. Removing that config fixed the problem

useful!

Related questions

Unable to use a local filesystem state store hot 2
Kops 1.12-beta.2 won't/can't bring up etcd server, manager or kube-api hot 1
kube controller manager refuses to connect after upgrading from 1.10.6 to 1.11.7 hot 1
Missing kops controller support for cloudproviders hot 1
InstanceGroup not found (for etcd ap-southeast-2a/main): "ap-southeast-2a" hot 1
Rolling-update fails due to calico-node with 1.12.0-beta.2 hot 1
etcd3 and kube-apiserver fail on terraform apply after terraform destroying w/ kops generated config hot 1
Upgrade from Kops 1.11 to 1.12 has failed. hot 1
Couldn't find key etcd_endpoints in ConfigMap kube-system/calico-config hot 1
Protokube has sustained cpu usage above 100% hot 1
Allow just one instance type in mixedInstancesPolicy hot 1
kubectl command: Unable to connect to the server: EOF hot 1
DNS record for public API address not updated hot 1
etcd3 and kube-apiserver fail on terraform apply after terraform destroying w/ kops generated config hot 1
Issues encountered deploying to OpenStack hot 1
source:https://uonfu.com/
Github User Rank List