profile
viewpoint

Ask questionsAttempt to mount an overlay layer that is already in-use

The problem happens when I build an LLB graph with parallel operations and mounts with selector.

The buildctl output is basically:

#2 mkdir /nested
#2 DONE 0.0s

#1 docker-image://docker.io/library/alpine:latest
#1 resolve docker.io/library/alpine:latest
#1 resolve docker.io/library/alpine:latest 0.6s done
#1 DONE 0.6s

#1 docker-image://docker.io/library/alpine:latest
#1 sha256:0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa 2.79MB / 2.79MB done
#1 sha256:b7b28af77ffec6054d13378df4fdf02725830086c7444d9c278af25312aa39b9 1.51kB / 1.51kB done
#1 sha256:6a92cd1fcdc8d8cdec60f33dda4db2cb1fcdcacf3410a8e05b3741f44a9b5998 1.64kB / 1.64kB done
#1 sha256:57334c50959f26ce1ee025d08f136c2292c128f84e7b229d1b0da5dac89e9866 528B / 528B done
#1 unpacking docker.io/library/alpine:latest 0.0s done
#1 DONE 0.1s

#3 /bin/sh -c echo 'test' > /out/nested/file.out
#3 DONE 0.1s

#5 /bin/sh -c sed 's/test/modified 1/' < /in/file.in > /out/file.out
#5 ERROR: executor failed running [/bin/sh -c sed 's/test/modified 1/' < /in/file.in > /out/file.out]: failed to mount /tmp/buildkit-mount088927200: [{Type:overlay Source:overlay Options:[workdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/8196/work upperdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/8196/fs lowerdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/8194/fs ro]}]: device or resource busy

#4 /bin/sh -c sed 's/test/modified 0/' < /in/file.in > /out/file.out
#4 DONE 0.1s
------
 > /bin/sh -c sed 's/test/modified 1/' < /in/file.in > /out/file.out:
------
error: failed to solve: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c sed 's/test/modified 1/' < /in/file.in > /out/file.out]: failed to mount /tmp/buildkit-mount088927200: [{Type:overlay Source:overlay Options:[workdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/8196/work upperdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/8196/fs lowerdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/8194/fs ro]}]: device or resource busy

dmesg:

[...] overlayfs: upperdir is in-use by another mount, mount with '-o index=off' to override exclusive upperdir protection.

After some digging and debugging, I've figured out a minimal reproducible example!

buildctl debug dump-llb --dot: debug

buildctl debug dump-llb | jq .:

{
  "Op": {
    "Op": {
      "Source": {
        "identifier": "docker-image://docker.io/library/alpine:latest"
      }
    }
  },
  "Digest": "sha256:0e6b31ceed3e6dc542018f35a53a0e857e6a188453d32a2a5bbe7aa2971c1220",
  "OpMetadata": {
    "ignore_cache": true,
    "caps": null
  }
}
{
  "Op": {
    "Op": {
      "File": {
        "actions": [
          {
            "input": -1,
            "secondaryInput": -1,
            "output": 0,
            "Action": {
              "Mkdir": {
                "path": "/nested",
                "mode": -1,
                "timestamp": -1
              }
            }
          }
        ]
      }
    }
  },
  "Digest": "sha256:4a517527affba2505c56ac535fa9769bb1c128d61f4678bc503772cdefc80761",
  "OpMetadata": {
    "ignore_cache": true,
    "caps": {
      "file.base": true
    }
  }
}
{
  "Op": {
    "inputs": [
      {
        "digest": "sha256:0e6b31ceed3e6dc542018f35a53a0e857e6a188453d32a2a5bbe7aa2971c1220",
        "index": 0
      },
      {
        "digest": "sha256:4a517527affba2505c56ac535fa9769bb1c128d61f4678bc503772cdefc80761",
        "index": 0
      }
    ],
    "Op": {
      "Exec": {
        "meta": {
          "args": [
            "/bin/sh",
            "-c",
            "echo 'test' > /out/nested/file.out"
          ],
          "cwd": "/",
          "user": "root"
        },
        "mounts": [
          {
            "input": 0,
            "dest": "/",
            "output": -1,
            "readonly": true
          },
          {
            "input": 1,
            "dest": "/out",
            "output": 0
          }
        ]
      }
    }
  },
  "Digest": "sha256:731b1bd20ce6f1e844c4dc5fd35859d5f5c501ca2af6f4af3c459d6ce981db8d",
  "OpMetadata": {
    "ignore_cache": true,
    "caps": {
      "exec.mount.bind": true
    }
  }
}
{
  "Op": {
    "inputs": [
      {
        "digest": "sha256:0e6b31ceed3e6dc542018f35a53a0e857e6a188453d32a2a5bbe7aa2971c1220",
        "index": 0
      },
      {
        "digest": "sha256:731b1bd20ce6f1e844c4dc5fd35859d5f5c501ca2af6f4af3c459d6ce981db8d",
        "index": 0
      }
    ],
    "Op": {
      "Exec": {
        "meta": {
          "args": [
            "/bin/sh",
            "-c",
            "sed 's/test/modified 0/' < /in/file.in > /out/file.out"
          ],
          "cwd": "/",
          "user": "root"
        },
        "mounts": [
          {
            "input": 0,
            "dest": "/",
            "output": -1,
            "readonly": true
          },
          {
            "input": -1,
            "dest": "/out",
            "output": 0
          },
          {
            "input": 1,
            "selector": "/nested/file.out",
            "dest": "/in/file.in",
            "output": -1,
            "readonly": true
          }
        ]
      }
    }
  },
  "Digest": "sha256:0b2b503e42b6cec429e763ab4b58b23403b2802521ad7ee0f6804e7bca83e507",
  "OpMetadata": {
    "ignore_cache": true,
    "caps": {
      "exec.mount.bind": true,
      "exec.mount.selector": true
    }
  }
}
{
  "Op": {
    "inputs": [
      {
        "digest": "sha256:0e6b31ceed3e6dc542018f35a53a0e857e6a188453d32a2a5bbe7aa2971c1220",
        "index": 0
      },
      {
        "digest": "sha256:731b1bd20ce6f1e844c4dc5fd35859d5f5c501ca2af6f4af3c459d6ce981db8d",
        "index": 0
      }
    ],
    "Op": {
      "Exec": {
        "meta": {
          "args": [
            "/bin/sh",
            "-c",
            "sed 's/test/modified 1/' < /in/file.in > /out/file.out"
          ],
          "cwd": "/",
          "user": "root"
        },
        "mounts": [
          {
            "input": 0,
            "dest": "/",
            "output": -1,
            "readonly": true
          },
          {
            "input": -1,
            "dest": "/out",
            "output": 0
          },
          {
            "input": 1,
            "selector": "/nested/file.out",
            "dest": "/in/file.in",
            "output": -1,
            "readonly": true
          }
        ]
      }
    }
  },
  "Digest": "sha256:200b8369455d22e991bc92ecf4fd2a6e46aa47ca4d1d813f46561dddec48f9e4",
  "OpMetadata": {
    "ignore_cache": true,
    "caps": {
      "exec.mount.bind": true,
      "exec.mount.selector": true
    }
  }
}
{
  "Op": {
    "inputs": [
      {
        "digest": "sha256:0b2b503e42b6cec429e763ab4b58b23403b2802521ad7ee0f6804e7bca83e507",
        "index": 0
      },
      {
        "digest": "sha256:200b8369455d22e991bc92ecf4fd2a6e46aa47ca4d1d813f46561dddec48f9e4",
        "index": 0
      }
    ],
    "Op": {
      "File": {
        "actions": [
          {
            "input": -1,
            "secondaryInput": 0,
            "output": 0,
            "Action": {
              "Copy": {
                "src": "file.out",
                "dest": "file-0.out",
                "mode": -1,
                "timestamp": -1
              }
            }
          },
          {
            "input": 2,
            "secondaryInput": 1,
            "output": 1,
            "Action": {
              "Copy": {
                "src": "file.out",
                "dest": "file-1.out",
                "mode": -1,
                "timestamp": -1
              }
            }
          }
        ]
      }
    }
  },
  "Digest": "sha256:a6930999cbe06191c3d80dd3f4eef48a66c3c15badb8f186fbca8ad647f26096",
  "OpMetadata": {
    "caps": {
      "file.base": true
    }
  }
}
{
  "Op": {
    "inputs": [
      {
        "digest": "sha256:a6930999cbe06191c3d80dd3f4eef48a66c3c15badb8f186fbca8ad647f26096",
        "index": 1
      }
    ],
    "Op": null
  },
  "Digest": "sha256:20d51af5ebd157a019f8452f1557ec65422fb2d2784667dbcb76c0fcdd776602",
  "OpMetadata": {
    "caps": null
  }
}

Unfortunately, I can't provide a golang example code, because I'm still trying to express the graph in it, and don't have much luck with this yet (originally, an experimental rustlang library was used to produce the LLB) :disappointed:


Here is my understanding of the problem: 731b1bd20ce6f1e844c4dc5fd35859d5f5c501ca2af6f4af3c459d6ce981db8d produces an overlay layer at output 0 with nested/file.out and then concurrent sed ... operations are trying to simultaneously mount the overlay. In this case, probably a bind mount directly to upperdir can be used instead of an overlay for the sed ... operations because the mounts are read-only.

moby/buildkit

Answer questions denzp

Oh, that's interesting. I was able to reproduce the behavior on the following kernels:

  • 5.2.3 (my primary),
  • 4.20.6 (just a random older kernel),
  • 4.19.61 (LTS kernel in my linux distro).

It might have been introduced in 4.13, but the actual problem could be unrelated to the kernel change because the latter was quite a while ago.

useful!

Related questions

Documentation claims that --mount works with 18.06 hot 1
rootless image didn't work on OKD (OpenShift) 3.11 hot 1
php build fails when cache is enabled hot 1
Documentation claims that --mount works with 18.06 hot 1
Cannot build from local image with buildctl (OCI Worker) hot 1
Image id shown as missing with docker history hot 1
Image id shown as missing with docker history hot 1
Cannot build from local image with buildctl (OCI Worker) hot 1
Image id shown as missing with docker history hot 1
RUN --mount=type=cache causes the whole build context to be loaded in hot 1
rootless image didn't work on OKD (OpenShift) 3.11 hot 1
Cannot build from local image with buildctl (OCI Worker) hot 1
ssh agent only forwarded from keys on Windows hot 1
Documentation claims that --mount works with 18.06 hot 1
Github User Rank List