profile
viewpoint

Ask questionsUnable to remove a stopped container: `device or resource busy`

Apologies if this is a duplicate issue, there seems to be several outstanding issues around a very similar error message but under different conditions. I initially added a comment on #21969 and was told to open a separate ticket, so here it is!


BUG REPORT INFORMATION

Output of docker version:

Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:34:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:34:23 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 51
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 81
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-74-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.676 GiB
Name: ip-10-1-49-110
ID: 5GAP:SPRQ:UZS2:L5FP:Y4EL:RR54:R43L:JSST:ZGKB:6PBH:RQPO:PMQ5
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Running on Ubuntu 14.04.3 LTS HVM in AWS on an m3.medium instance with an EBS root volume.

Steps to reproduce the issue:

  1. $ docker run --restart on-failure --log-driver syslog --log-opt syslog-address=udp://localhost:514 -d -p 80:80 -e SOME_APP_ENV_VAR myimage
  2. Container keeps shutting down and restarting due to a bug in the runtime and exiting with an error
  3. Manually running docker stop container
  4. Container is successfully stopped
  5. Trying to rm container then throws the error: Error response from daemon: Driver aufs failed to remove root filesystem 88189a16be60761a2c04a455206650048e784d750533ce2858bcabe2f528c92e: rename /var/lib/docker/aufs/diff/a48629f102d282572bb5df964eeec7951057b50f21df7abe162f8de386e76dc0 /var/lib/docker/aufs/diff/a48629f102d282572bb5df964eeec7951057b50f21df7abe162f8de386e76dc0-removing: device or resource busy
  6. Restart docker engine: $ sudo service docker restart
  7. $ docker ps -a shows that the container no longer exists.
moby/moby

Answer questions ahuigo

If you get such error:

Unable to remove filesystem: /var/lib/docker/container/11667ef16239.../

The solution here(No need to execute service docker restart to restart docker):

# 1. find which process(pid) occupy the fs system
$ find /proc/*/mounts  |xargs -n1 grep -l -E '^shm.*/docker/.*/11667ef16239' | cut -d"/" -f3
1302   # /proc/1302/mounts

# 2. kill this process
$ sudo kill -9 1302

Related questions

start container failed with "failed to umount /var/lib/docker/containers/.../shm: no such file or directory" hot 47
upgrade docker-18.09.2-ce , shim.sock: bind: address already in use: unknown hot 24
start container failed with "failed to umount /var/lib/docker/containers/.../shm: no such file or directory" hot 17
runc regression - EPERM running containers from selinux hot 11
Windows Server 2019 publish ports in swarm not working hot 11
"docker stack deploy">"rpc error: code = 3 desc = name must be valid as a DNS name component" hot 9
OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown hot 8
Swarm restarts all containers hot 8
integration: "error reading the kernel parameter" errors during CI hot 8
hcsshim::PrepareLayer failed in Win32: The parameter is incorrect hot 8
Docker 18.09.1 doesn't work with iptables v1.8.2 hot 7
Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded hot 6
Containers on overlay network cannot reach other containers hot 6
write unix /var/run/docker.sock->@: write: broken pipe hot 6
feature request: docker cat command hot 6
Github User Rank List