Ask questions`kind` should be able to start the control plane when `/dev/kmsg` is not available

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->

What happened:

kind is not able to start the control plane (with the provided node images) when /dev/kmsg is not available in the node container.

kind then fails with:

 ✗ Starting control-plane 🕹️
Error: failed to create cluster: failed to init node with kubeadm: exit status 1

Checking on the node, I can see the following error:

# docker exec -ti kind-control-plane sh -c 'journalctl -u kubelet | grep -m1 -C3 kmsg'
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.522248  375565 kuberuntime_manager.go:205] Container runtime containerd initialized, version: 1.2.6-0ubuntu1, apiVersion: v1alpha2
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.525786  375565 server.go:1083] Started kubelet
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.526254  375565 server.go:144] Starting to listen on
Jun 26 08:18:59 kind-control-plane kubelet[375565]: F0626 08:18:59.532066  375565 kubelet.go:1405] Failed to start OOM watcher open /dev/kmsg: no such file or directory
Jun 26 08:18:59 kind-control-plane kubelet[375565]: I0626 08:18:59.535478  375565 server.go:350] Adding debug handlers to kubelet server.
Jun 26 08:18:59 kind-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Jun 26 08:18:59 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.

For the node images I bake myself, I have the following dirty hack in place:

  • I create the image with kind build node-image ...
  • I add another layer which adds a systemd file which sets a ExecStartPre for the kubelet which just does something like [ -e /dev/kmsg ] || ln -s /dev/console /dev/kmsg

After that kind create cluster ... successfully starts the control plane.

What you expected to happen:

kind should be able to bring up the control plane, at least when /dev/console is available.

How to reproduce it (as minimally and precisely as possible):

Run kind on an environment without /dev/kmsg, eg. concourse (via kind-on-c for example) and with the 1.15 node image provided by kind.

Anything else we need to know?:

I currently (need to dig deeper) have no idea what the downsides are when linking /dev/kmsg to /dev/console or what to do when /dev/console is not available either (when would that be the case). My tests seem to suggest that the cluster works just fine when doing this linking. Would the kind maintainers be open for a PR which sets such a ExecStartPre (or makes /dev/kmsg available via some other means for that matter)?


  • kind version: (use kind version):

    kind version v0.4.0
  • Kubernetes version: (use kubectl version):

    v1.15.0 (the version that is baked into kindest/node:v1.15.0)

  • Docker version: (use docker info): ``

    Containers: 1
     Running: 1
     Paused: 0
     Stopped: 0
    Images: 1
    Server Version: 18.09.6
    Storage Driver: overlay2
     Backing Filesystem: extfs
     Supports d_type: true
     Native Overlay Diff: true
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    WARNING: bridge-nf-call-iptables is disabled
    WARNING: bridge-nf-call-ip6tables is disabled
     Volume: local
     Network: bridge host macvlan null overlay
     Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
    Swarm: inactive
    Runtimes: runc
    Default Runtime: runc
    Init Binary: docker-init
    containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
    runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
    init version: fec3683
    Security Options:
      Profile: default
    Kernel Version: 4.15.0-50-generic
    Operating System: Alpine Linux v3.10 (containerized)
    OSType: linux
    Architecture: x86_64
    CPUs: 4
    Total Memory: 15.66GiB
    Name: dd95e405-f2d2-4d82-54b3-36f4d1287a53
    Docker Root Dir: /scratch/docker
    Debug Mode (client): false
    Debug Mode (server): false
    Experimental: false
    Insecure Registries:
    Live Restore Enabled: false
    Product License: Community Engine
  • OS (e.g. from /etc/os-release):

    NAME="Alpine Linux"
    PRETTY_NAME="Alpine Linux v3.10"

Answer questions hoegaarden

maybe related:


Related questions

Cluster not starting with DIND setup hot 1
Appears when using ceph of roook: map failed: (30) Read-only file system hot 1
Errors compile: version "go1.12.1" does not match go tool version "go1.10.2" hot 1
Installing Istio fails when trying to match resources hot 1
Helm: rabbitmq-ha deployment fails to discover k8s nodes hot 1
WSLv1: No DNS between pods or to the internet hot 1
Github User Rank List