profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/tonistiigi/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Tõnis Tiigi tonistiigi Docker San Francisco

moby/buildkit 4068

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit

tonistiigi/audiosprite 592

Jukebox/Howler/CreateJS compatible audio sprite generator

aacebedo/dnsdock 563

DNS service discovery for Docker containers

docker/github-actions 162

:warning: This repository is deprecated and has been replaced by docker/build-push-action@v2

tonistiigi/buildkit-pack 98

buildkit frontend for buildpacks

carlosedp/riscv-bringup 81

Risc-V journey thru containers and new projects

containerd/fifo 72

fifo pkg for Go

PullRequestReviewEvent

Pull request review commentmoby/buildkit

Fix flightcontrol test and usage bugs

 func (sr *immutableRef) computeBlobChain(ctx context.Context, createIfNeeded boo func computeBlobChain(ctx context.Context, sr *immutableRef, createIfNeeded bool, compressionType compression.Type, forceCompression bool, s session.Group) error { 	baseCtx := ctx 	eg, ctx := errgroup.WithContext(ctx)-	var currentDescr ocispec.Descriptor 	if sr.parent != nil { 		eg.Go(func() error { 			return computeBlobChain(ctx, sr.parent, createIfNeeded, compressionType, forceCompression, s) 		}) 	}++	var descr ocispec.Descriptor 	eg.Go(func() error {-		dp, err := g.Do(ctx, sr.ID(), func(ctx context.Context) (interface{}, error) {-			refInfo := sr.Info()-			if refInfo.Blob != "" {-				if forceCompression {-					desc, err := sr.ociDesc()-					if err != nil {-						return nil, err-					}-					if err := ensureCompression(ctx, sr, desc, compressionType, s); err != nil {-						return nil, err-					}+		refInfo := sr.Info()

This doesn't look quite equivalent. Previously if differ was in progress, !createIfNeeded would still wait for completion. And !forceCompression should not call differ twice, even in mediatype is different.

sipsma

comment created time in 9 days

Pull request review commentmoby/buildkit

Fix flightcontrol test and usage bugs

 func (s *sharedOp) CacheMap(ctx context.Context, index int) (resp *cacheMapResp, 	if err != nil { 		return nil, err 	}-	res, err := s.g.Do(ctx, "cachemap", func(ctx context.Context) (ret interface{}, retErr error) {+	res, err := s.g.Do(ctx, fmt.Sprintf("cachemap-%d", index), func(ctx context.Context) (ret interface{}, retErr error) {

That looks pretty bad. I wonder if it might have caused some cache misses. We should pick this commit to v0.9/

sipsma

comment created time in 9 days

PullRequestReviewEvent

Pull request review commentmoby/buildkit

cache: Refactor out cacheRecord equal* fields

 func (sr *mutableRef) Mount(ctx context.Context, readonly bool, s session.Group) 	return sr.mount(ctx, readonly) } +// must be called holding cacheRecord mu+func (sr *mutableRef) mount(ctx context.Context, readonly bool) (snapshot.Mountable, error) {+	m, err := sr.cm.Snapshotter.Mounts(ctx, getSnapshotID(sr.md))+	if err != nil {+		return nil, errors.Wrapf(err, "failed to mount %s", sr.ID())+	}+	if readonly {+		m = setReadonly(m)+	}+	return m, nil+}+ func (sr *mutableRef) Commit(ctx context.Context) (ImmutableRef, error) { 	sr.cm.mu.Lock() 	defer sr.cm.mu.Unlock()  	sr.mu.Lock() 	defer sr.mu.Unlock() -	return sr.commit(ctx)+	rec := sr.cacheRecord++	ir := &immutableRef{

As far as the LLB API is concerned caching via sessionid is a valid feature as it is the only way for the client to confirm that the files did not change without retransfer. Possibly buildx bake will start to rely on this and for more complex cases this is definitely useful.

Even if you make a case that there is little use for affected callers atm, implementing the interface in a way that methods can only be called in a certain order limits future development and increases chances for this to unexpectedly break. In that case, the interface should be changed so callers would use some other, more reliable method for equality and persistence use-cases.

sipsma

comment created time in 9 days

PullRequestReviewEvent

issue commentdocker/buildx

Copy from previous stage fails

@crazy-max If you're looking into this, the question is why https://github.com/moby/buildkit/pull/1498/files doesn't work properly in this case and fall back gracefully(assuming blob is legit and there is no other corruption). I guess it might be because Load returns a lazy remote reference and actual data is pulled later. In that case, we should verify the blob during load already, and ReaderAt only invokes the download.

jojomatik

comment created time in 11 days

PullRequestReviewEvent

Pull request review commentmoby/buildkit

cache: Refactor out cacheRecord equal* fields

 func (sr *mutableRef) Mount(ctx context.Context, readonly bool, s session.Group) 	return sr.mount(ctx, readonly) } +// must be called holding cacheRecord mu+func (sr *mutableRef) mount(ctx context.Context, readonly bool) (snapshot.Mountable, error) {+	m, err := sr.cm.Snapshotter.Mounts(ctx, getSnapshotID(sr.md))+	if err != nil {+		return nil, errors.Wrapf(err, "failed to mount %s", sr.ID())+	}+	if readonly {+		m = setReadonly(m)+	}+	return m, nil+}+ func (sr *mutableRef) Commit(ctx context.Context) (ImmutableRef, error) { 	sr.cm.mu.Lock() 	defer sr.cm.mu.Unlock()  	sr.mu.Lock() 	defer sr.mu.Unlock() -	return sr.commit(ctx)+	rec := sr.cacheRecord++	ir := &immutableRef{

If you look at https://github.com/moby/buildkit/commit/eac79f7c7e936453a58508761638e16ffb255085 then previous implementation was based on Freeze logic. The ID not changing seems to be one of the main differences.

sipsma

comment created time in 11 days

Pull request review commentmoby/buildkit

cache: Refactor out cacheRecord equal* fields

 func (sr *mutableRef) Mount(ctx context.Context, readonly bool, s session.Group) 	return sr.mount(ctx, readonly) } +// must be called holding cacheRecord mu+func (sr *mutableRef) mount(ctx context.Context, readonly bool) (snapshot.Mountable, error) {+	m, err := sr.cm.Snapshotter.Mounts(ctx, getSnapshotID(sr.md))+	if err != nil {+		return nil, errors.Wrapf(err, "failed to mount %s", sr.ID())+	}+	if readonly {+		m = setReadonly(m)+	}+	return m, nil+}+ func (sr *mutableRef) Commit(ctx context.Context) (ImmutableRef, error) { 	sr.cm.mu.Lock() 	defer sr.cm.mu.Unlock()  	sr.mu.Lock() 	defer sr.mu.Unlock() -	return sr.commit(ctx)+	rec := sr.cacheRecord++	ir := &immutableRef{

I actually thought such cases already existed before this change and I also can't think of any reason why we would want to prevent them or make this be a requirement of the interface.

I don't think this problem existed in the previous implementation cause I actually remember hitting these cases when developing this. It might have even been a reason for the equal* design.

For such records, what specific problems could be caused by the underlying data changing when it gets re-opened as mutable (provided the rules about simultaneous opening of immutable and mutable refs are honored)? I thought that was the whole point of the lazy-commit optimization;

Yes, the point of lazy commit is that data can become mutable again. But once it does, the immutable should become invalid. The ID() is how other components(like the build cache system, or incremental context send) query references and store them for persistence. This is what they use to understand if two records are equal.

sipsma

comment created time in 11 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentmoby/buildkit

Compute diff from the upper directory of overlayfs-based snapshotter

+// +build linux++package cache++import (+	"bytes"+	"context"+	"fmt"+	"io"+	"io/ioutil"+	"os"+	"path/filepath"+	"strings"+	"syscall"++	"github.com/containerd/containerd/archive"+	ctdcompression "github.com/containerd/containerd/archive/compression"+	"github.com/containerd/containerd/content"+	"github.com/containerd/containerd/errdefs"+	"github.com/containerd/containerd/mount"+	"github.com/containerd/continuity/devices"+	"github.com/containerd/continuity/fs"+	"github.com/containerd/continuity/sysx"+	digest "github.com/opencontainers/go-digest"+	ocispec "github.com/opencontainers/image-spec/specs-go/v1"+	"github.com/pkg/errors"+	"golang.org/x/sys/unix"+)++var emptyDesc = ocispec.Descriptor{}++// computeOverlayBlob provides overlayfs-specialized method to compute+// diff between lower and upper snapshot. If the passed mounts cannot+// be computed (e.g. because the mounts aren't overlayfs), it returns+// an error.+func (sr *immutableRef) computeOverlayBlob(ctx context.Context, lower, upper []mount.Mount, mediaType string, ref string) (_ ocispec.Descriptor, err error) {+	upperdir, err := getOverlayUpperdir(lower, upper)+	if err != nil {+		return emptyDesc, err+	}++	var isCompressed bool+	switch mediaType {+	case ocispec.MediaTypeImageLayer:+	case ocispec.MediaTypeImageLayerGzip:+		isCompressed = true+	default:+		return emptyDesc, fmt.Errorf("unsupported diff media type: %v", mediaType)+	}++	cw, err := sr.cm.ContentStore.Writer(ctx,+		content.WithRef(ref),+		content.WithDescriptor(ocispec.Descriptor{+			MediaType: mediaType, // most contentstore implementations just ignore this+		}))+	if err != nil {+		return emptyDesc, errors.Wrap(err, "failed to open writer")+	}+	defer func() {+		if err != nil {+			cw.Close()+		}+	}()++	var labels map[string]string+	if isCompressed {+		dgstr := digest.SHA256.Digester()+		compressed, err := ctdcompression.CompressStream(cw, ctdcompression.Gzip)

follow up: we should combine this somehow with #2246 logic to avoid duplication

ktock

comment created time in 11 days

Pull request review commentmoby/buildkit

Compute diff from the upper directory of overlayfs-based snapshotter

+// +build linux++package cache++import (+	"bytes"+	"context"+	"fmt"+	"io"+	"io/ioutil"+	"os"+	"path/filepath"+	"strings"+	"syscall"++	"github.com/containerd/containerd/archive"+	ctdcompression "github.com/containerd/containerd/archive/compression"+	"github.com/containerd/containerd/content"+	"github.com/containerd/containerd/errdefs"+	"github.com/containerd/containerd/mount"+	"github.com/containerd/continuity/devices"+	"github.com/containerd/continuity/fs"+	"github.com/containerd/continuity/sysx"+	digest "github.com/opencontainers/go-digest"+	ocispec "github.com/opencontainers/image-spec/specs-go/v1"+	"github.com/pkg/errors"+	"golang.org/x/sys/unix"+)++var emptyDesc = ocispec.Descriptor{}++// computeOverlayBlob provides overlayfs-specialized method to compute+// diff between lower and upper snapshot. If the passed mounts cannot+// be computed (e.g. because the mounts aren't overlayfs), it returns+// an error.+func (sr *immutableRef) computeOverlayBlob(ctx context.Context, lower, upper []mount.Mount, mediaType string, ref string) (_ ocispec.Descriptor, err error) {+	upperdir, err := getOverlayUpperdir(lower, upper)+	if err != nil {+		return emptyDesc, err+	}++	var isCompressed bool+	switch mediaType {+	case ocispec.MediaTypeImageLayer:+	case ocispec.MediaTypeImageLayerGzip:+		isCompressed = true+	default:+		return emptyDesc, fmt.Errorf("unsupported diff media type: %v", mediaType)+	}++	cw, err := sr.cm.ContentStore.Writer(ctx,+		content.WithRef(ref),+		content.WithDescriptor(ocispec.Descriptor{+			MediaType: mediaType, // most contentstore implementations just ignore this+		}))+	if err != nil {+		return emptyDesc, errors.Wrap(err, "failed to open writer")+	}+	defer func() {+		if err != nil {+			cw.Close()+		}+	}()++	var labels map[string]string+	if isCompressed {+		dgstr := digest.SHA256.Digester()+		compressed, err := ctdcompression.CompressStream(cw, ctdcompression.Gzip)+		if err != nil {+			return emptyDesc, errors.Wrap(err, "failed to get compressed stream")+		}+		err = writeOverlayUpperdir(ctx, io.MultiWriter(compressed, dgstr.Hash()), upperdir, lower)+		compressed.Close()+		if err != nil {+			return emptyDesc, errors.Wrap(err, "failed to write compressed diff")+		}+		if labels == nil {+			labels = map[string]string{}+		}+		labels[containerdUncompressed] = dgstr.Digest().String()+	} else {+		if err = writeOverlayUpperdir(ctx, cw, upperdir, lower); err != nil {+			return emptyDesc, errors.Wrap(err, "failed to write diff")+		}+	}++	var commitopts []content.Opt+	if labels != nil {+		commitopts = append(commitopts, content.WithLabels(labels))+	}+	dgst := cw.Digest()+	if err := cw.Commit(ctx, 0, dgst, commitopts...); err != nil {+		if !errdefs.IsAlreadyExists(err) {+			return emptyDesc, errors.Wrap(err, "failed to commit")+		}+	}+	cinfo, err := sr.cm.ContentStore.Info(ctx, dgst)+	if err != nil {+		return emptyDesc, errors.Wrap(err, "failed to get info from content store")+	}+	if cinfo.Labels == nil {+		cinfo.Labels = make(map[string]string)+	}+	// Set uncompressed label if digest already existed without label+	if _, ok := cinfo.Labels[containerdUncompressed]; !ok {+		cinfo.Labels[containerdUncompressed] = labels[containerdUncompressed]+		if _, err := sr.cm.ContentStore.Update(ctx, cinfo, "labels."+containerdUncompressed); err != nil {+			return emptyDesc, errors.Wrap(err, "error setting uncompressed label")+		}+	}++	return ocispec.Descriptor{+		MediaType: mediaType,+		Size:      cinfo.Size,+		Digest:    cinfo.Digest,+	}, nil+}++// getOverlayUpperdir parses the passed mounts and identifies the directory+// that contains diff between upper and lower.+func getOverlayUpperdir(lower, upper []mount.Mount) (string, error) {+	var upperdir string+	if len(lower) == 1 && len(upper) == 1 {+		// We only support single mount configuration as of now.+		upperM, lowerM := upper[0], lower[0]++		// Get layer directories of lower snapshot+		var lowerlayers []string+		switch lowerM.Type {+		case "bind":+			// lower snapshot is a bind mount of one layer+			lowerlayers = []string{lowerM.Source}+		case "overlay":+			// lower snapshot is an overlay mount of multiple layers+			lowerlayers = getOverlayLayers(lowerM)+		default:+			return "", fmt.Errorf("cannot get layer information from mount option (type = %q)", lowerM.Type)+		}++		// Get layer directories of upper snapshot+		if upperM.Type != "overlay" {+			return "", fmt.Errorf("upper snapshot isn't overlay mounted (type = %q)", upperM.Type)+		}+		upperlayers := getOverlayLayers(upperM)++		// Check if the diff directory can be determined+		if len(upperlayers) != len(lowerlayers)+1 {+			return "", fmt.Errorf("cannot determine diff of more than one upper directories")+		}+		for i := 0; i < len(lowerlayers); i++ {+			if upperlayers[i] != lowerlayers[i] {+				return "", fmt.Errorf("layer %d must be common between upper and lower snapshots", i)+			}+		}+		upperdir = upperlayers[len(upperlayers)-1] // get the topmost layer that indicates diff+	}+	if upperdir == "" {+		return "", fmt.Errorf("cannot determine upperdir from mount option")+	}+	return upperdir, nil+}++// getOverlayLayers returns all layer directories of an overlayfs mount.+func getOverlayLayers(m mount.Mount) []string {+	var u string+	var uFound bool+	var l []string // l[0] = bottommost+	for _, o := range m.Options {+		if strings.HasPrefix(o, "upperdir=") {+			u, uFound = strings.TrimPrefix(o, "upperdir="), true+		} else if strings.HasPrefix(o, "lowerdir=") {+			l = strings.Split(strings.TrimPrefix(o, "lowerdir="), ":")+			for i, j := 0, len(l)-1; i < j; i, j = i+1, j-1 {+				l[i], l[j] = l[j], l[i] // make l[0] = bottommost+			}+		}+	}+	if uFound {+		return append(l, u)+	}+	return l+}++// writeOverlayUpperdir writes a layer tar archive into the specified writer, based on+// the diff information stored in the upperdir.+func writeOverlayUpperdir(ctx context.Context, w io.Writer, upperdir string, lower []mount.Mount) error {+	emptyLower, err := ioutil.TempDir("", "buildkit") // empty directory used for the lower of diff view+	if err != nil {+		return errors.Wrapf(err, "failed to create temp dir")+	}+	defer os.Remove(emptyLower)+	upperView := []mount.Mount{+		{+			Type:    "overlay",+			Source:  "overlay",+			Options: []string{fmt.Sprintf("lowerdir=%s", strings.Join([]string{upperdir, emptyLower}, ":"))},+		},+	}+	return mount.WithTempMount(ctx, lower, func(lowerRoot string) error {+		return mount.WithTempMount(ctx, upperView, func(upperViewRoot string) error {+			cw := archive.NewChangeWriter(w, upperViewRoot)+			if err := overlayChanges(ctx, cw.HandleChange, upperdir, upperViewRoot, lowerRoot); err != nil {+				if err2 := cw.Close(); err2 != nil {+					return errors.Wrapf(err, "failed torecord upperdir changes (close error: %v)", err2)+				}+				return errors.Wrapf(err, "failed torecord upperdir changes")+			}+			return cw.Close()+		})+	})+}++// overlayChanges is continuty's `fs.Change`-like method but leverages overlayfs's+// "upperdir" for computing the diff. "upperdirView" is overlayfs mounted view of+// the upperdir that doesn't contain whiteouts. This is used for computing+// changes under opaque directories.+func overlayChanges(ctx context.Context, changeFn fs.ChangeFunc, upperdir, upperdirView, base string) error {+	return filepath.Walk(upperdir, func(path string, f os.FileInfo, err error) error {+		if err != nil {+			return err+		}++		// Rebase path+		path, err = filepath.Rel(upperdir, path)+		if err != nil {+			return err+		}+		path = filepath.Join(string(os.PathSeparator), path)++		// Skip root+		if path == string(os.PathSeparator) {+			return nil+		}++		// Check if this is a deleted entry+		isDelete, skip, err := checkDelete(upperdir, path, base, f)+		if err != nil {+			return err+		} else if skip {+			return nil+		}++		var kind fs.ChangeKind+		var skipRecord bool+		if isDelete {+			// This is a deleted entry.+			kind = fs.ChangeKindDelete+			f = nil+		} else if baseF, err := os.Lstat(filepath.Join(base, path)); err == nil {+			// File exists in the base layer. Thus this is modified.+			kind = fs.ChangeKindModify+			// Avoid including directory that hasn't been modified. If /foo/bar/baz is modified,+			// then /foo will apper here even if it's not been modified because it's the parent of bar.+			if same, err := sameDir(baseF, f, filepath.Join(base, path), filepath.Join(upperdirView, path)); same {+				skipRecord = true // Both are the same, don't record the change+			} else if err != nil {+				return err+			}+		} else if os.IsNotExist(err) {+			// File doesn't exist in the base layer. Thus this is added.+			kind = fs.ChangeKindAdd+		} else if err != nil {+			return err+		}++		if !skipRecord {+			if err := changeFn(kind, path, f, nil); err != nil {+				return err+			}+		}++		if f != nil {+			if isOpaque, err := checkOpaque(upperdir, path, base, f); err != nil {+				return err+			} else if isOpaque {+				// This is an opaque directory. Start a new walking differ to get adds/deletes of+				// this directory. We use "upperdirView" directory which doesn't contain whiteouts.+				if err := fs.Changes(ctx, filepath.Join(base, path), filepath.Join(upperdirView, path),+					func(k fs.ChangeKind, p string, f os.FileInfo, err error) error {+						return changeFn(k, filepath.Join(path, p), f, err) // rebase path to be based on the opaque dir+					},+				); err != nil {+					return err+				}+				return filepath.SkipDir // We completed this directory. Do not walk files under this directory anymore.+			}+		}+		return nil+	})+}++// checkDelete checks if the specified file is a whiteout+func checkDelete(upperdir string, path string, base string, f os.FileInfo) (delete, skip bool, _ error) {+	if f.Mode()&os.ModeCharDevice != 0 {+		if _, ok := f.Sys().(*syscall.Stat_t); ok {+			maj, min, err := devices.DeviceInfo(f)+			if err != nil {+				return false, false, errors.Wrapf(err, "failed to get device info")+			}+			if maj == 0 && min == 0 {+				// This file is a whiteout (char 0/0) that indicates this is deleted from the base+				if _, err := os.Lstat(filepath.Join(base, path)); err != nil {+					if !os.IsNotExist(err) {+						return false, false, errors.Wrapf(err, "failed to lstat")+					}+					// This file doesn't exist even in the base dir.+					// We don't need whiteout. Just skip this file.+					return false, true, nil+				}+				return true, false, nil+			}+		}+	}+	return false, false, nil+}++// checkDelete checks if the specified file is an opaque directory+func checkOpaque(upperdir string, path string, base string, f os.FileInfo) (isOpaque bool, _ error) {+	if f.IsDir() {+		for _, oKey := range []string{"trusted.overlay.opaque", "user.overlay.opaque"} {+			opaque, err := sysx.LGetxattr(filepath.Join(upperdir, path), oKey)+			if err != nil && err != unix.ENODATA {+				return false, errors.Wrapf(err, "failed to retrieve %s attr", oKey)+			} else if len(opaque) == 1 && opaque[0] == 'y' {+				// This is an opaque whiteout directory.+				if _, err := os.Lstat(filepath.Join(base, path)); err != nil {+					if !os.IsNotExist(err) {+						return false, errors.Wrapf(err, "failed to lstat")+					}+					// This file doesn't exist even in the base dir. We don't need treat this as an opaque.+					return false, nil+				}+				return true, nil+			}+		}+	}+	return false, nil+}++// sameDir performs continity-compatible comparison of directories.+// https://github.com/containerd/continuity/blob/v0.1.0/fs/path.go#L91-L133

Can the code that is already in continuity be deduplicated later?

ktock

comment created time in 11 days

PullRequestReviewEvent

Pull request review commentmoby/buildkit

Compute diff from the upper directory of overlayfs-based snapshotter

+// +build linux++package cache++import (+	"bytes"+	"context"+	"fmt"+	"io"+	"io/ioutil"+	"os"+	"path/filepath"+	"strings"+	"syscall"++	"github.com/containerd/containerd/archive"+	ctdcompression "github.com/containerd/containerd/archive/compression"+	"github.com/containerd/containerd/content"+	"github.com/containerd/containerd/errdefs"+	"github.com/containerd/containerd/mount"+	"github.com/containerd/continuity/devices"+	"github.com/containerd/continuity/fs"+	"github.com/containerd/continuity/sysx"+	digest "github.com/opencontainers/go-digest"+	ocispec "github.com/opencontainers/image-spec/specs-go/v1"+	"github.com/pkg/errors"+	"golang.org/x/sys/unix"+)++var emptyDesc = ocispec.Descriptor{}++// computeOverlayBlob provides overlayfs-specialized method to compute+// diff between lower and upper snapshot. If the passed mounts cannot+// be computed (e.g. because the mounts aren't overlayfs), it returns+// an error.+func (sr *immutableRef) computeOverlayBlob(ctx context.Context, lower, upper []mount.Mount, mediaType string, ref string) (_ ocispec.Descriptor, err error) {+	upperdir, err := getOverlayUpperdir(lower, upper)+	if err != nil {+		return emptyDesc, err+	}++	var isCompressed bool+	switch mediaType {+	case ocispec.MediaTypeImageLayer:+	case ocispec.MediaTypeImageLayerGzip:+		isCompressed = true+	default:+		return emptyDesc, fmt.Errorf("unsupported diff media type: %v", mediaType)+	}++	cw, err := sr.cm.ContentStore.Writer(ctx,+		content.WithRef(ref),+		content.WithDescriptor(ocispec.Descriptor{+			MediaType: mediaType, // most contentstore implementations just ignore this+		}))+	if err != nil {+		return emptyDesc, errors.Wrap(err, "failed to open writer")+	}+	defer func() {+		if err != nil {+			cw.Close()+		}+	}()++	var labels map[string]string+	if isCompressed {+		dgstr := digest.SHA256.Digester()+		compressed, err := ctdcompression.CompressStream(cw, ctdcompression.Gzip)+		if err != nil {+			return emptyDesc, errors.Wrap(err, "failed to get compressed stream")+		}+		err = writeOverlayUpperdir(ctx, io.MultiWriter(compressed, dgstr.Hash()), upperdir, lower)+		compressed.Close()+		if err != nil {+			return emptyDesc, errors.Wrap(err, "failed to write compressed diff")+		}+		if labels == nil {+			labels = map[string]string{}+		}+		labels[containerdUncompressed] = dgstr.Digest().String()+	} else {+		if err = writeOverlayUpperdir(ctx, cw, upperdir, lower); err != nil {+			return emptyDesc, errors.Wrap(err, "failed to write diff")+		}+	}++	var commitopts []content.Opt+	if labels != nil {+		commitopts = append(commitopts, content.WithLabels(labels))+	}+	dgst := cw.Digest()+	if err := cw.Commit(ctx, 0, dgst, commitopts...); err != nil {+		if !errdefs.IsAlreadyExists(err) {+			return emptyDesc, errors.Wrap(err, "failed to commit")+		}+	}+	cinfo, err := sr.cm.ContentStore.Info(ctx, dgst)+	if err != nil {+		return emptyDesc, errors.Wrap(err, "failed to get info from content store")+	}+	if cinfo.Labels == nil {+		cinfo.Labels = make(map[string]string)+	}+	// Set uncompressed label if digest already existed without label+	if _, ok := cinfo.Labels[containerdUncompressed]; !ok {+		cinfo.Labels[containerdUncompressed] = labels[containerdUncompressed]+		if _, err := sr.cm.ContentStore.Update(ctx, cinfo, "labels."+containerdUncompressed); err != nil {+			return emptyDesc, errors.Wrap(err, "error setting uncompressed label")+		}+	}++	return ocispec.Descriptor{+		MediaType: mediaType,+		Size:      cinfo.Size,+		Digest:    cinfo.Digest,+	}, nil+}++// getOverlayUpperdir parses the passed mounts and identifies the directory+// that contains diff between upper and lower.+func getOverlayUpperdir(lower, upper []mount.Mount) (string, error) {+	var upperdir string+	if len(lower) == 1 && len(upper) == 1 {+		// We only support single mount configuration as of now.+		upperM, lowerM := upper[0], lower[0]++		// Get layer directories of lower snapshot+		var lowerlayers []string+		switch lowerM.Type {+		case "bind":+			// lower snapshot is a bind mount of one layer+			lowerlayers = []string{lowerM.Source}+		case "overlay":+			// lower snapshot is an overlay mount of multiple layers+			lowerlayers = getOverlayLayers(lowerM)+		default:+			return "", fmt.Errorf("cannot get layer information from mount option (type = %q)", lowerM.Type)+		}++		// Get layer directories of upper snapshot+		if upperM.Type != "overlay" {+			return "", fmt.Errorf("upper snapshot isn't overlay mounted (type = %q)", upperM.Type)+		}+		upperlayers := getOverlayLayers(upperM)++		// Check if the diff directory can be determined+		if len(upperlayers) != len(lowerlayers)+1 {+			return "", fmt.Errorf("cannot determine diff of more than one upper directories")+		}+		for i := 0; i < len(lowerlayers); i++ {+			if upperlayers[i] != lowerlayers[i] {+				return "", fmt.Errorf("layer %d must be common between upper and lower snapshots", i)+			}+		}+		upperdir = upperlayers[len(upperlayers)-1] // get the topmost layer that indicates diff+	}+	if upperdir == "" {+		return "", fmt.Errorf("cannot determine upperdir from mount option")+	}+	return upperdir, nil+}++// getOverlayLayers returns all layer directories of an overlayfs mount.+func getOverlayLayers(m mount.Mount) []string {+	var u string+	var uFound bool+	var l []string // l[0] = bottommost+	for _, o := range m.Options {+		if strings.HasPrefix(o, "upperdir=") {+			u, uFound = strings.TrimPrefix(o, "upperdir="), true+		} else if strings.HasPrefix(o, "lowerdir=") {+			l = strings.Split(strings.TrimPrefix(o, "lowerdir="), ":")+			for i, j := 0, len(l)-1; i < j; i, j = i+1, j-1 {+				l[i], l[j] = l[j], l[i] // make l[0] = bottommost+			}+		}+	}+	if uFound {+		return append(l, u)+	}+	return l+}++// writeOverlayUpperdir writes a layer tar archive into the specified writer, based on+// the diff information stored in the upperdir.+func writeOverlayUpperdir(ctx context.Context, w io.Writer, upperdir string, lower []mount.Mount) error {+	emptyLower, err := ioutil.TempDir("", "buildkit") // empty directory used for the lower of diff view+	if err != nil {+		return errors.Wrapf(err, "failed to create temp dir")+	}+	defer os.Remove(emptyLower)+	upperView := []mount.Mount{+		{+			Type:    "overlay",+			Source:  "overlay",+			Options: []string{fmt.Sprintf("lowerdir=%s", strings.Join([]string{upperdir, emptyLower}, ":"))},+		},+	}+	return mount.WithTempMount(ctx, lower, func(lowerRoot string) error {+		return mount.WithTempMount(ctx, upperView, func(upperViewRoot string) error {+			cw := archive.NewChangeWriter(w, upperViewRoot)+			if err := overlayChanges(ctx, cw.HandleChange, upperdir, upperViewRoot, lowerRoot); err != nil {+				if err2 := cw.Close(); err2 != nil {+					return errors.Wrapf(err, "failed torecord upperdir changes (close error: %v)", err2)+				}+				return errors.Wrapf(err, "failed torecord upperdir changes")+			}+			return cw.Close()+		})+	})+}++// overlayChanges is continuty's `fs.Change`-like method but leverages overlayfs's+// "upperdir" for computing the diff. "upperdirView" is overlayfs mounted view of+// the upperdir that doesn't contain whiteouts. This is used for computing+// changes under opaque directories.+func overlayChanges(ctx context.Context, changeFn fs.ChangeFunc, upperdir, upperdirView, base string) error {+	return filepath.Walk(upperdir, func(path string, f os.FileInfo, err error) error {+		if err != nil {+			return err+		}++		// Rebase path+		path, err = filepath.Rel(upperdir, path)+		if err != nil {+			return err+		}+		path = filepath.Join(string(os.PathSeparator), path)++		// Skip root+		if path == string(os.PathSeparator) {+			return nil+		}++		// Check if this is a deleted entry+		isDelete, skip, err := checkDelete(upperdir, path, base, f)+		if err != nil {+			return err+		} else if skip {+			return nil+		}++		var kind fs.ChangeKind+		var skipRecord bool+		if isDelete {+			// This is a deleted entry.+			kind = fs.ChangeKindDelete+			f = nil+		} else if baseF, err := os.Lstat(filepath.Join(base, path)); err == nil {+			// File exists in the base layer. Thus this is modified.+			kind = fs.ChangeKindModify+			// Avoid including directory that hasn't been modified. If /foo/bar/baz is modified,+			// then /foo will apper here even if it's not been modified because it's the parent of bar.+			if same, err := sameDir(baseF, f, filepath.Join(base, path), filepath.Join(upperdirView, path)); same {+				skipRecord = true // Both are the same, don't record the change+			} else if err != nil {+				return err+			}+		} else if os.IsNotExist(err) {+			// File doesn't exist in the base layer. Thus this is added.+			kind = fs.ChangeKindAdd+		} else if err != nil {+			return err+		}++		if !skipRecord {+			if err := changeFn(kind, path, f, nil); err != nil {+				return err+			}+		}++		if f != nil {+			if isOpaque, err := checkOpaque(upperdir, path, base, f); err != nil {+				return err+			} else if isOpaque {+				// This is an opaque directory. Start a new walking differ to get adds/deletes of+				// this directory. We use "upperdirView" directory which doesn't contain whiteouts.+				if err := fs.Changes(ctx, filepath.Join(base, path), filepath.Join(upperdirView, path),+					func(k fs.ChangeKind, p string, f os.FileInfo, err error) error {+						return changeFn(k, filepath.Join(path, p), f, err) // rebase path to be based on the opaque dir+					},+				); err != nil {+					return err+				}+				return filepath.SkipDir // We completed this directory. Do not walk files under this directory anymore.+			}+		}+		return nil+	})+}++// checkDelete checks if the specified file is a whiteout+func checkDelete(upperdir string, path string, base string, f os.FileInfo) (delete, skip bool, _ error) {+	if f.Mode()&os.ModeCharDevice != 0 {+		if _, ok := f.Sys().(*syscall.Stat_t); ok {+			maj, min, err := devices.DeviceInfo(f)+			if err != nil {+				return false, false, errors.Wrapf(err, "failed to get device info")+			}+			if maj == 0 && min == 0 {+				// This file is a whiteout (char 0/0) that indicates this is deleted from the base+				if _, err := os.Lstat(filepath.Join(base, path)); err != nil {+					if !os.IsNotExist(err) {+						return false, false, errors.Wrapf(err, "failed to lstat")+					}+					// This file doesn't exist even in the base dir.+					// We don't need whiteout. Just skip this file.+					return false, true, nil+				}+				return true, false, nil+			}+		}+	}+	return false, false, nil+}++// checkDelete checks if the specified file is an opaque directory+func checkOpaque(upperdir string, path string, base string, f os.FileInfo) (isOpaque bool, _ error) {+	if f.IsDir() {+		for _, oKey := range []string{"trusted.overlay.opaque", "user.overlay.opaque"} {+			opaque, err := sysx.LGetxattr(filepath.Join(upperdir, path), oKey)+			if err != nil && err != unix.ENODATA {+				return false, errors.Wrapf(err, "failed to retrieve %s attr", oKey)+			} else if len(opaque) == 1 && opaque[0] == 'y' {+				// This is an opaque whiteout directory.+				if _, err := os.Lstat(filepath.Join(base, path)); err != nil {+					if !os.IsNotExist(err) {+						return false, errors.Wrapf(err, "failed to lstat")+					}+					// This file doesn't exist even in the base dir. We don't need treat this as an opaque.+					return false, nil+				}+				return true, nil+			}+		}+	}+	return false, nil+}++// sameDir performs continity-compatible comparison of directories.+// https://github.com/containerd/continuity/blob/v0.1.0/fs/path.go#L91-L133+// This doesn't compare files because it requires to compare their contents.+// This is what we want to avoid by this overlayfs-specialized differ.+func sameDir(f1, f2 os.FileInfo, f1fullPath, f2fullPath string) (bool, error) {+	if !f1.IsDir() || !f2.IsDir() {+		return false, nil+	}++	if os.SameFile(f1, f2) {+		return true, nil+	}++	equalStat, err := compareSysStat(f1.Sys(), f2.Sys())+	if err != nil || !equalStat {+		return equalStat, err+	}++	if eq, err := compareCapabilities(f1fullPath, f2fullPath); err != nil || !eq {

I don't think this is exactly what we think of equal xattrs. We should also keep everything in user scope. Can be follow-up.

ktock

comment created time in 11 days

PullRequestReviewEvent

pull request commentdocker/buildx

Support --config to mount buildkit.toml and -driver-opt=withqemu=true,qemuimage=tonistiigi/binfmt:latest for qemu installing

Not sure why I have to updates vendor and go{.mod,.sum}

Because of the version change probably.

morlay

comment created time in 11 days

Pull request review commentmoby/buildkit

cache: Refactor out cacheRecord equal* fields

 func (sr *mutableRef) Mount(ctx context.Context, readonly bool, s session.Group) 	return sr.mount(ctx, readonly) } +// must be called holding cacheRecord mu+func (sr *mutableRef) mount(ctx context.Context, readonly bool) (snapshot.Mountable, error) {+	m, err := sr.cm.Snapshotter.Mounts(ctx, getSnapshotID(sr.md))+	if err != nil {+		return nil, errors.Wrapf(err, "failed to mount %s", sr.ID())+	}+	if readonly {+		m = setReadonly(m)+	}+	return m, nil+}+ func (sr *mutableRef) Commit(ctx context.Context) (ImmutableRef, error) { 	sr.cm.mu.Lock() 	defer sr.cm.mu.Unlock()  	sr.mu.Lock() 	defer sr.mu.Unlock() -	return sr.commit(ctx)+	rec := sr.cacheRecord++	ir := &immutableRef{

All of the current tests pass too of course, so I don't believe there's any cases where wrong records get returned.

I'm not sure if the cases are very practical (there are not a lot of places that call GetMutable but I think at least local source should be affected by it, maybe also exec mounts), but if there exists a case where there can be ImmutableRef who has ID and then later there is another ImmutableRef that has the same ID but a different content then this does not satisfy the interface requirements.

Am I correct that we can't do that today?

Yes, you can't run an older version of the daemon for tests atm. to test migrations. Not against this possibility but it is likely also quite messy to add.

sipsma

comment created time in 11 days

PullRequestReviewEvent

pull request commentmoby/buildkit

[v0.8] contenthash: use SeekLowerBound to seek radix tree

Why are we doing code refactoring in an old release branch?

thaJeztah

comment created time in 11 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentcrazy-max/buildkit

moby/buildkit#2269

 func (s *sourceOp) CacheMap(ctx context.Context, g session.Group, index int) (*s 	}  	dgst := digest.FromBytes([]byte(sourceCacheType + ":" + k))- 	if strings.HasPrefix(k, "session:") { 		dgst = digest.Digest("random:" + strings.TrimPrefix(dgst.String(), dgst.Algorithm().String()+":")) 	} +	var resolveResponse map[string]string+	if !strings.HasPrefix(s.op.Source.GetIdentifier(), "local://") {+		resolveResponse = map[string]string{s.op.Source.GetIdentifier(): k}

edit: (left wrong comment before, ignore that if you got notified).

We need to verify that these are the correct hashes for the object. It is not quite the same as the internal cache key in some cases. Eg. https://github.com/moby/buildkit/blob/b2195cec759428a7775a6fe01799b961d14a81bc/source/containerimage/pull.go#L155 adds more info to cache key to make sure platform is correct(+ there is another config based key as well that we should ignore). https://github.com/moby/buildkit/blob/b2195cec759428a7775a6fe01799b961d14a81bc/source/git/gitsource.go#L173-L178 does same for git. In here we would need the raw root image digest, git commit sha, raw sha256 of URL etc.

crazy-max

comment created time in 11 days

Pull request review commentcrazy-max/buildkit

moby/buildkit#2269

 func (s *sourceOp) CacheMap(ctx context.Context, g session.Group, index int) (*s 	}  	dgst := digest.FromBytes([]byte(sourceCacheType + ":" + k))- 	if strings.HasPrefix(k, "session:") { 		dgst = digest.Digest("random:" + strings.TrimPrefix(dgst.String(), dgst.Algorithm().String()+":")) 	} +	var resolveResponse map[string]string+	if !strings.HasPrefix(s.op.Source.GetIdentifier(), "local://") {+		resolveResponse = map[string]string{s.op.Source.GetIdentifier(): k}+	}+ 	return &solver.CacheMap{ 		// TODO: add os/arch-		Digest: dgst,-		Opts:   cacheOpts,+		Digest:          dgst,

I think this digest is not quite what we want. See line 75 where it is already merged with source type. Also I think git adds special things to internal cache key like if the .git is kept in source. In here we want the clean object specific hash. Eg. raw image digest, git commit, direct sha of the url.

crazy-max

comment created time in 11 days

PullRequestReviewEvent

Pull request review commentcrazy-max/buildkit

moby/buildkit#2269

 type Image struct {  	// Variant defines platform variant. To be added to OCI. 	Variant string `json:"variant,omitempty"`++	// BuildInfo defines build dependencies+	BuildInfo source.BuildInfo `json:"moby.buildkit.buildinfo.v0,omitempty"`

This should be base64. I know it makes it more hacky but better not to add big non-spec structures.

crazy-max

comment created time in 11 days

PullRequestReviewEvent

Pull request review commentmoby/buildkit

cache: Refactor out cacheRecord equal* fields

 func (sr *mutableRef) Mount(ctx context.Context, readonly bool, s session.Group) 	return sr.mount(ctx, readonly) } +// must be called holding cacheRecord mu+func (sr *mutableRef) mount(ctx context.Context, readonly bool) (snapshot.Mountable, error) {+	m, err := sr.cm.Snapshotter.Mounts(ctx, getSnapshotID(sr.md))+	if err != nil {+		return nil, errors.Wrapf(err, "failed to mount %s", sr.ID())+	}+	if readonly {+		m = setReadonly(m)+	}+	return m, nil+}+ func (sr *mutableRef) Commit(ctx context.Context) (ImmutableRef, error) { 	sr.cm.mu.Lock() 	defer sr.cm.mu.Unlock()  	sr.mu.Lock() 	defer sr.mu.Unlock() -	return sr.commit(ctx)+	rec := sr.cacheRecord++	ir := &immutableRef{

Snapshot IDs are always internal, they don't matter to me. It is the ref ID that gets stored in the build cache boltdb https://github.com/moby/buildkit/tree/master/solver/bboltcachestorage . And yes there is also the case with the search for mutable that you mentioned where currently I think the metadata was different for these records, but I need to look at it more to understand that case.

sipsma

comment created time in 11 days

PullRequestReviewEvent