profile
viewpoint

Ask questions--cache-from and Multi Stage: Pre-Stages are not cached

Description

If you want to use a Multi Stage Build together with --cache-from it's very hard and complicated to load the Cache of Pre Stages, as --cache-from disables the lookup in the local cache (see https://github.com/moby/moby/issues/32612). The only way is to tag the pre-stage images as well and add them to the --cache-from, which is very complicated.

Steps to reproduce the issue:

  1. Assuming we have a Multistage Dockerfile like:
FROM busybox as builder
RUN echo "hello" > test

FROM busybox
COPY --from=builder test test
RUN echo test
  1. Building it the first time:
$ docker build -t test:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Running in b5e871ebd251
 ---> 6889762613a0
Removing intermediate container b5e871ebd251
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> f9ee9cc534a7
Removing intermediate container 8d76fd7eb6be
Step 5/5 : RUN echo test
 ---> Running in 5b768ed39212
test
 ---> b4a81a0e7c96
Removing intermediate container 5b768ed39212
Successfully built b4a81a0e7c96
Successfully tagged test:latest

So far all good. 3. Now running it a second time, see how all layers are fully cached:

$ docker build -t test:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Using cache
 ---> 6889762613a0
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> Using cache
 ---> f9ee9cc534a7
Step 5/5 : RUN echo test
 ---> Using cache
 ---> b4a81a0e7c96
Successfully built b4a81a0e7c96
Successfully tagged test:latest
  1. Now running it with --cache-from test:latest:
$ docker build -t test:latest --cache-from test:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Running in 89d43713b017
 ---> 18e01d7690cb
Removing intermediate container 89d43713b017
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> Using cache
 ---> f9ee9cc534a7
Step 5/5 : RUN echo test
 ---> Using cache
 ---> b4a81a0e7c96
Successfully built b4a81a0e7c96
Successfully tagged test:latest

See how Step 2/5 : RUN echo "hello" > test is not using any cache. Interestingly Step 4 is using the cache again, as it finds that cache within the test:latest image. So it actually builds the first stage image but never uses it. A lot of time the first stages are very heavy computations, like installing packages, building stuff etc. So we almost loose the niceness of Multi Stage Build.

There is a way to fix this, with tagging the first stage image via --target builder:

$ docker build -t test-builder:latest --target builder .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Using cache
 ---> 18e01d7690cb
Successfully built 18e01d7690cb
Successfully tagged test-builder:latest

and then using both images for --cache-from:

$ docker build -t test:latest --cache-from test:latest --cache-from test-builder:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Using cache
 ---> 18e01d7690cb
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> Using cache
 ---> f9ee9cc534a7
Step 5/5 : RUN echo test
 ---> Using cache
 ---> b4a81a0e7c96
Successfully built b4a81a0e7c96
Successfully tagged test:latest

but IMHO that is super complicated and confusing.

I'm not 100% sure how we could fix this. Implementing --cache-from to also use the local cache as a secondary cache lookup would solve the problem (see https://github.com/moby/moby/issues/32612)

Output of docker version:

$ docker version
Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:53:38 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:54:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1059
Server Version: 17.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.41-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: FHCJ:CF22:VRF6:Y4HR:BM3W:ATJ3:3QGW:AGO5:OTKL:W2ES:OM6Q:WZ5Y
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 31
 System Time: 2017-09-03T19:59:13.529192672Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
moby/moby

Answer questions glensc

@tonistiigi --build-arg? isn't build arg supposed for Dockerfile use? it should be env like $DOCKER_BUILDKIT, looks like a huge hack. any links to issues/merge-requests that implemented it? maybe get a backstory...

Related questions

start container failed with "failed to umount /var/lib/docker/containers/.../shm: no such file or directory" hot 47
upgrade docker-18.09.2-ce , shim.sock: bind: address already in use: unknown hot 24
start container failed with "failed to umount /var/lib/docker/containers/.../shm: no such file or directory" hot 17
runc regression - EPERM running containers from selinux hot 11
Windows Server 2019 publish ports in swarm not working hot 11
"docker stack deploy">"rpc error: code = 3 desc = name must be valid as a DNS name component" hot 9
OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown hot 8
Swarm restarts all containers hot 8
integration: "error reading the kernel parameter" errors during CI hot 8
hcsshim::PrepareLayer failed in Win32: The parameter is incorrect hot 8
Docker 18.09.1 doesn't work with iptables v1.8.2 hot 7
Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded hot 6
Containers on overlay network cannot reach other containers hot 6
write unix /var/run/docker.sock->@: write: broken pipe hot 6
feature request: docker cat command hot 6
Github User Rank List