profile
viewpoint

Ask questionsstart container failed with "failed to umount /var/lib/docker/containers/.../shm: no such file or directory"

$docker version Client: Version: 1.9.0-dev API version: 1.21 Go version: go1.4.3 Git commit: 1e51652-dirty Built: Sat Oct 10 03:18:00 UTC 2015 OS/Arch: linux/amd64

Server: Version: 1.9.0-dev API version: 1.21 Go version: go1.4.3 Git commit: 1e51652-dirty Built: Sat Oct 10 03:18:00 UTC 2015 OS/Arch: linux/amd64 $docker info Containers: 21 Images: 87 Engine Version: 1.9.0-dev Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 130 Dirperm1 Supported: false Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.13.0-65-generic Operating System: Ubuntu 14.04.3 LTS CPUs: 4 Total Memory: 3.854 GiB Name: ubuntu ID: KEN4:OLGJ:7WKG:TOAP:CLA3:KJ32:RK5Q:AWCG:QKYN:B6TD:UMHQ:4UGM Username: guohui Registry: https://index.docker.io/v1/ WARNING: No swap limit support $uname -a Linux ubuntu 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker logs is: INFO[0060] POST /containers/9d4d911b57853471a37a4458488cfc6f12a58b67d2323d61eb8f407779e0b742/start ERRO[0060] failed to umount /var/lib/docker/containers/9d4d911b57853471a37a4458488cfc6f12a58b67d2323d61eb8f407779e0b742/shm: no such file or directory ERRO[0060] failed to umount /var/lib/docker/containers/9d4d911b57853471a37a4458488cfc6f12a58b67d2323d61eb8f407779e0b742/mqueue: no such file or directory ERRO[0060] 9d4d911b57853471a37a4458488cfc6f12a58b67d2323d61eb8f407779e0b742: Failed to umount ipc filesystems: failed to cleanup ipc mounts: no such file or directory no such file or directory INFO[0060] GET /containers/json ERRO[0060] Handler for POST /containers/9d4d911b57853471a37a4458488cfc6f12a58b67d2323d61eb8f407779e0b742/start returned error: Cannot start container 9d4d911b57853471a37a4458488cfc6f12a58b67d2323d61eb8f407779e0b742: invalid name: k8s_POD.6a43f87c_kube-ui-v2-7lyan_kube-system_31af4581-6f09-11e5-a71e-000c297950a1_7bad0127 ERRO[0060] HTTP Error err=Cannot start container 9d4d911b57853471a37a4458488cfc6f12a58b67d2323d61eb8f407779e0b742: invalid name: k8s_POD.6a43f87c_kube-ui-v2-7lyan_kube-system_31af4581-6f09-11e5-a71e-000c297950a1_7bad0127 statusCode=500

This problem is not exist with docker-1.8.2

moby/moby

Answer questions dextarr

i see this in kubernetes too, with busybox image and 'sleep 128' as command, it happens pretty often, maybe easier to debug what causes it:

    Jul  5 13:32:57 ip-172-31-7-125 containerd[2808]: time="2019-07-05T15:32:57.498594195+02:00" level=info msg="shim reaped" id=51d1d1bda80e2ec6415585db4e7cb0e7145944381c1a11801ea78ab1781041ec
    Jul  5 13:32:57 ip-172-31-7-125 dockerd[4144]: time="2019-07-05T15:32:57.508769724+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
    Jul  5 13:32:57 ip-172-31-7-125 dockerd[4144]: time="2019-07-05T15:32:57.508877419+02:00" level=warning msg="51d1d1bda80e2ec6415585db4e7cb0e7145944381c1a11801ea78ab1781041ec cleanup: failed to unmount IPC: umount /var/lib/docker/containers/51d1d1bda80e2ec6415585db4e7cb0e7145944381c1a11801ea78ab1781041ec/mounts/shm, flags: 0x2: no such file or directory"
    Jul  5 13:32:58 ip-172-31-7-125 kubelet[9551]: E0705 15:32:58.138709    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:33:10 ip-172-31-7-125 kubelet[9551]: E0705 15:33:10.181770    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:33:24 ip-172-31-7-125 kubelet[9551]: E0705 15:33:24.181765    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:33:39 ip-172-31-7-125 kubelet[9551]: E0705 15:33:39.181495    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:33:50 ip-172-31-7-125 kubelet[9551]: E0705 15:33:50.181608    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:34:01 ip-172-31-7-125 kubelet[9551]: E0705 15:34:01.181557    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:34:16 ip-172-31-7-125 kubelet[9551]: E0705 15:34:16.181818    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:34:31 ip-172-31-7-125 kubelet[9551]: E0705 15:34:31.181595    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:34:43 ip-172-31-7-125 kubelet[9551]: E0705 15:34:43.181476    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:34:57 ip-172-31-7-125 kubelet[9551]: E0705 15:34:57.181492    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:35:08 ip-172-31-7-125 kubelet[9551]: E0705 15:35:08.181642    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:35:23 ip-172-31-7-125 kubelet[9551]: E0705 15:35:23.181395    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"
    Jul  5 13:35:34 ip-172-31-7-125 kubelet[9551]: E0705 15:35:34.181765    9551 pod_workers.go:190] Error syncing pod 843a396b-9f26-11e9-aac7-06bf1c079da2 ("ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"), skipping: failed to "StartContainer" for "app1" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=app1 pod=ds-app1-9456n_app1(843a396b-9f26-11e9-aac7-06bf1c079da2)"

any hint what should i look for or maybe i can paste the whole deamonset-app for easier infra setup?

Related questions

upgrade docker-18.09.2-ce , shim.sock: bind: address already in use: unknown hot 27
runc regression - EPERM running containers from selinux hot 16
Windows Server 2019 publish ports in swarm not working hot 14
Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded hot 13
"docker stack deploy">"rpc error: code = 3 desc = name must be valid as a DNS name component" hot 12
Swarm restarts all containers hot 10
write unix /var/run/docker.sock->@: write: broken pipe hot 10
hcsshim::PrepareLayer failed in Win32: The parameter is incorrect hot 10
OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown hot 9
integration: "error reading the kernel parameter" errors during CI hot 9
Docker 18.09.1 doesn't work with iptables v1.8.2 hot 9
dockerd stopped responding to API requests; no installed keys could decrypt message hot 9
manifest invalid error when pushing image to quay.io hot 8
Containers on overlay network cannot reach other containers hot 7
no matching manifest for linux/arm64/unknown hot 6
Github User Rank List