profile
viewpoint
Derek McGowan dmcgowan Docker San Francisco

push eventcontainerd/containerd

Lantao Liu

commit sha 8f9eff4fec4da6a8236bd38af5999a383de95f1b

Support disk usage in windows snapshot. Signed-off-by: Lantao Liu <lantaol@google.com>

view details

Derek McGowan

commit sha a176179a0894ca6768a0f993e678f08a7f8553d5

Merge pull request #3785 from Random-Liu/support-disk-usage-windows Support disk usage in windows snapshot.

view details

push time in 7 hours

PR merged containerd/containerd

Support disk usage in windows snapshot.

Add Usage support in windows snapshotter.

For https://github.com/containerd/cri/issues/1299.

/cc @jterry75 Signed-off-by: Lantao Liu lantaol@google.com

+26 -14

14 comments

2 changed files

Random-Liu

pr closed time in 7 hours

issue commentcontainerd/containerd

Deadlock in pulling/unpacking image with containerd 1.3.0

I think seeing the manifest would be helpful. I see a similar case that Lantao mentioned in the code, but with values in the manifest we wouldn't expect. Can you send the anonymized manifest (and at least rootfs section of config), you can send it here, slack, or email.

I think there is a fix for this we should get in to make this case harder to hit, but we need to verify if that is what we are seeing here.

ungureanuvladvictor

comment created time in a day

issue commentcontainerd/containerd

Deadlock in pulling/unpacking image with containerd 1.3.0

I'll take a look, thanks for the report and analysis!

ungureanuvladvictor

comment created time in 2 days

issue commentcontainerd/containerd

containerd can't pull image from Github Docker Package Registry

@ArpithaDR that is a separate issue and a different registry provider. JFrog's registry has a different set of issues that they also need to fix upstream, maybe related to #3556

csantanapr

comment created time in 6 days

push eventdmcgowan/docker

Derek McGowan

commit sha cf4fd443817d0b3d8240231bfa211c9002f4a896

Handle tagless names when matching short identifiers Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 7 days

push eventdmcgowan/containerd

Akihiro Suda

commit sha a2688b972c36bc88407591898fb84f478d3790d2

bring back aufs and zfs Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Phil Estes

commit sha 342ce3e602d13a05b13c7cb642aba5996315e02f

Merge pull request #3789 from AkihiroSuda/vendor-20191030 bring back aufs and zfs

view details

Ameya Gawde

commit sha d21f0f116e29e498eeebf8440e9f43a06cf39cfc

windows process shim installer Signed-off-by: Ameya Gawde <ameya.gawde@docker.com>

view details

Maksym Pavlenko

commit sha 257a7498d00827fbca08078f664cc6b4be27d7aa

Merge pull request #3790 from ameyag/windows-shim-install Windows process shim installer

view details

fahedouch

commit sha 7c016b655b54efd5a4bb97a5a3a0f92c0df9c06c

add tests to server_test Signed-off-by: fahedouch <fahed.dorgaa@gmail.com>

view details

Michael Crosby

commit sha f9d8f0e56ee43594ec529c4d28aec7be7343cd36

Merge pull request #3794 from fahedouch/server_test add tests to server_test

view details

Michael Crosby

commit sha 84879b7c48bfbf33984a55054519ee122d7be3a7

Add CGO makefile flag Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Derek McGowan

commit sha 4d7739527e1d7a328bfc31e905f9825c5b3afefc

Merge pull request #3796 from crosbymichael/cgo Add CGO makefile flag

view details

Phil Estes

commit sha 76aa4e546ef4d3d8f35e525ebe93ec3ee879662a

Use common identifiers package with less restrictive regex Regular expressions in the namespaces package are redundant with the pre-existing validator in the "identifiers" package; replace this custom usage with the validator in "identifiers" Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>

view details

Phil Estes

commit sha fdf7a795bce4abb9488f19a9c4f71f944f0ab00c

Clarify requirements for identifiers validation Fix commentary to be clear that the RFC1035 2.3.1 rules are not explicitly enforced by the identifiers regex. Add a test to show the lessened restrictions for digits to follow hyphen and period. Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>

view details

Michael Crosby

commit sha 356702098d777c5f288262fa5c141bb8f05b0572

Merge pull request #3798 from estesp/common-indentifiers Common identifiers validation

view details

Derek McGowan

commit sha ca73991ecf9eb1f90bf1b0833f8d510a801d248e

Update snapshot parent filter property to always exist The parent property may be empty but always exists. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha df4e1c7a37dba4c694d934959902ba3d2ecbbc73

Support target snapshot references on prepare Allows backend snapshots to bring existing snapshots into a namespace without requiring clients to fully snapshots when the target reference is known. Backend snapshots must explicitly implement this functionality, it is equivalent to sharing across namespaces and is up to the backend to use the label when it is given or ignore it. This enables remote snapshot functionality for a backend to query for a target snapshot before a client has performed any work to create that snapshot. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 7 days

push eventcontainerd/containerd

Michael Crosby

commit sha 84879b7c48bfbf33984a55054519ee122d7be3a7

Add CGO makefile flag Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Derek McGowan

commit sha 4d7739527e1d7a328bfc31e905f9825c5b3afefc

Merge pull request #3796 from crosbymichael/cgo Add CGO makefile flag

view details

push time in 8 days

PR merged containerd/containerd

Add CGO makefile flag

Signed-off-by: Michael Crosby crosbymichael@gmail.com

+4 -3

3 comments

1 changed file

crosbymichael

pr closed time in 8 days

push eventdmcgowan/containerd

Derek McGowan

commit sha d3b384c70d0b37c108baf9d71d84f55969cfbc6a

Support target snapshot references on prepare Allows backend snapshots to bring existing snapshots into a namespace without requiring clients to fully snapshots when the target reference is known. Backend snapshots must explicitly implement this functionality, it is equivalent to sharing across namespaces and is up to the backend to use the label when it is given or ignore it. This enables remote snapshot functionality for a backend to query for a target snapshot before a client has performed any work to create that snapshot. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 8 days

pull request commentcontainerd/containerd

Fixed authorization against Cesenta Docker registry authentication

@skoef what does the response look like today from the POST? If they are returning a 200 with a body that can't be handled properly, we could just fallback in that case, currently our workarounds are just related to status code but we can also consider the body.

skoef

comment created time in 8 days

pull request commentcontainerd/containerd

Fixed authorization against Cesenta Docker registry authentication

@skoef the problem here is that is seems Cesanta is not implementing the POST endpoint at all, however they are also not blocking it and seem to just be handling it like a GET but form data instead of query params. The GET and POST endpoints implement a different protocol (GET was just a custom thing designed by Docker and handles anonymous auth while POST is trying to follow the OAuth specification). I think if there is going to be a workaround related to Cesanta, that workaround should be to fallback to the GET endpoint when the POST fails (this is ideally done through 405 but we have a few other special cases in there). It is important that upstream Cesanta can handle this correctly, that is what that repo is designed to do.

skoef

comment created time in 8 days

Pull request review commentcontainerd/containerd

Support target snapshot references on prepare

 func (s *snapshotter) createSnapshot(ctx context.Context, key, parent string, re 		return nil, err 	} -	var m []mount.Mount+	var (+		target  = base.Labels[labelSnapshotRef]+		bparent string+		bkey    string+		bopts   = []snapshots.Opt{+			snapshots.WithLabels(filterInheritedLabels(base.Labels)),+		}+	)+ 	if err := update(ctx, s.db, func(tx *bolt.Tx) error { 		bkt, err := createSnapshotterBucket(tx, ns, s.name) 		if err != nil { 			return err 		} -		bbkt, err := bkt.CreateBucket([]byte(key))-		if err != nil {-			if err == bolt.ErrBucketExists {-				err = errors.Wrapf(errdefs.ErrAlreadyExists, "snapshot %q", key)+		// Check if target exists, if so, return already exists+		if target != "" {+			if tbkt := bkt.Bucket([]byte(target)); tbkt != nil {+				return errors.Wrapf(errdefs.ErrAlreadyExists, "target snapshot %q", target) 			}+		}++		if bbkt := bkt.Bucket([]byte(key)); bbkt != nil {+			return errors.Wrapf(errdefs.ErrAlreadyExists, "snapshot %q", key)+		}++		if parent != "" {+			pbkt := bkt.Bucket([]byte(parent))+			if pbkt == nil {+				return errors.Wrapf(errdefs.ErrNotFound, "parent snapshot %v does not exist", parent)+			}+			bparent = string(pbkt.Get(bucketKeyName))+		}++		sid, err := bkt.NextSequence()+		if err != nil { 			return err 		}+		bkey = createKey(sid, ns, key)++		return err+	}); err != nil {+		return nil, err+	}++	var (+		m       []mount.Mount+		created string+		rerr    error+	)+	if readonly {+		m, err = s.Snapshotter.View(ctx, bkey, bparent, bopts...)+	} else {+		m, err = s.Snapshotter.Prepare(ctx, bkey, bparent, bopts...)+	}++	// An already exists error should indicate the backend found a snapshot+	// matching a provided target reference.+	if errdefs.IsAlreadyExists(err) {+		if target != "" {+			var tinfo *snapshots.Info+			filter := fmt.Sprintf("labels.\"containerd.io/snapshot.ref\"==%s,parent==%q", target, bparent)+			if err := s.Snapshotter.Walk(ctx, func(ctx context.Context, i snapshots.Info) error {+				if tinfo == nil && i.Kind == snapshots.KindCommitted {+					if i.Labels["containerd.io/snapshot.ref"] != target {+						// Walk did not respect filter+						return nil+					}+					if i.Parent != bparent {+						// Walk did not respect filter+						return nil+					}+					tinfo = &i+				}+				return nil++			}, filter); err != nil {+				return nil, errors.Wrap(err, "failed walking backend snapshots")+			}

We intentionally don't have a naming convention. The metadata will end up adding to the name of the snapshot before sending it to the backend, but there is no limitation on what the backend uses. Previously the backend would have had to parse the name with the pattern used by the metadata store if it wanted the original name, now the original/intended name should be available through the label. However, there is no limitation on what format that is.

I think we can mention more about the naming in the Snapshotter interface definition. Can add that along with the documentation about this feature.

dmcgowan

comment created time in 8 days

Pull request review commentcontainerd/containerd

Support target snapshot references on prepare

 func (s *snapshotter) Update(ctx context.Context, info snapshots.Info, fieldpath 			Labels: filterInheritedLabels(local.Labels), 		} -		if _, err := s.Snapshotter.Update(ctx, inner, fieldpaths...); err != nil {+		// NOTE: Perform this inside the transaction to reduce the+		// chances of out of sync data. The backend snapshotters+		// should perform the Update as fast as possible.+		if info, err = s.Snapshotter.Update(ctx, inner, fieldpaths...); err != nil { 			return err 		}+		updated = true  		return nil 	}); err != nil {-		return snapshots.Info{}, err-	}--	info, err = s.Snapshotter.Stat(ctx, bkey)-	if err != nil {+		if updated {+			log.G(ctx).WithField("snapshotter", s.name).WithField("key", local.Name).WithError(err).Error("transaction failed after updating snapshot backend")

This is to catch the case that the boltdb transaction fails. We have no way of reverting the backend (classic 2 phase commit problem) so we log it. This case is unlikely but if it occurs it may require intervention. In most cases the client should handle this fine, but we need logs to show it occuring. If this starts showing up in client issues then we may need to further mitigate against it.

dmcgowan

comment created time in 8 days

Pull request review commentcontainerd/containerd

Add CGO makefile flag

 DESTDIR ?= /usr/local VERSION=$(shell git describe --match 'v[0-9]*' --dirty='.m' --always) REVISION=$(shell git rev-parse HEAD)$(shell if ! git diff --no-ext-diff --quiet --exit-code; then echo .m; fi) PACKAGE=github.com/containerd/containerd+CGO ?= 0

This is only for the shims, maybe comment it as such? The regular CGO_ENABLED value will be used for the other binaries

crosbymichael

comment created time in 8 days

issue commentcontainerd/containerd

Support remote snapshotter to speed up image pulling

See #3793 for change to core. Client changes to leverage this will be next.

ktock

comment created time in 12 days

PR opened containerd/containerd

Support target snapshot references on prepare

Allows backend snapshots to bring existing snapshots into a namespace without requiring clients to fully snapshots when the target reference is known. Backend snapshots must explicitly implement this functionality, it is equivalent to sharing across namespaces and is up to the backend to use the label when it is given or ignore it.

This enables remote snapshot functionality for a backend to query for a target snapshot before a client has performed any work to create that snapshot.

Further documentation is needed. I will likely create a documented related to remote snapshotters when the remainder of the work is done for those. Additionally, the behavior of snapshotters in regards to errors (and these labels) should be more fully defined in the snapshot interface definition.

There is a change here to change the behavior of the snapshot filter used by walk. Since there is no "not exists" filter for parent, it does not make sense to ever have a parent not exist from the perspective of the filter. This allows snapshots to be filtered for with parent or without parents by parent!="" or parent=="", previously parent=="" would have failed to match.

+554 -50

0 comment

6 changed files

pr created time in 12 days

push eventdmcgowan/containerd

Derek McGowan

commit sha d93bf503eee888676e2f76b398c0df939a8fbc8f

Support target snapshot references on prepare Allows backend snapshots to bring existing snapshots into a namespace without requiring clients to fully snapshots when the target reference is known. Backend snapshots must explicitly implement this functionality, it is equivalent to sharing across namespaces and is up to the backend to use the label when it is given or ignore it. This enables remote snapshot functionality for a backend to query for a target snapshot before a client has performed any work to create that snapshot. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 12 days

push eventdmcgowan/containerd

Akihiro Suda

commit sha f593efdf0c160037c9f831983f62537285739b03

RELEASES.md: 1.1 EOL v1.1 reached EOL on October 23, 2019 Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Akihiro Suda

commit sha 0c01992f9c8cc2794b3d2b4f2ed0b55a4b91ed9e

Merge pull request #3709 from Zyqsempai/3708-added-filters-to-shapsotters-api Added filters to snapshots API

view details

Lantao Liu

commit sha 20e844a227950952e110cf751a8efeb7dfcbf167

Use logrus trace support. Signed-off-by: Lantao Liu <lantaol@google.com>

view details

Phil Estes

commit sha aeec80fca165ab6217cd4c03fa59b1bee9cb0427

Merge pull request #3773 from Random-Liu/use-logrus-trace Use logrus trace support.

view details

Michael Crosby

commit sha edb6f2344a3629ada21b0f267e4db87161e859db

Merge pull request #3780 from AkihiroSuda/1.1-eol RELEASES.md: 1.1 EOL

view details

Manuel Rüger

commit sha 5e5584196b3ca6b9ccfd2fa8603a9b7d25746997

Makefile: Drop vndr whitelist Since autocompletions moved to contrib/ in https://github.com/containerd/containerd/pull/3766 Signed-off-by: Manuel Rüger <manuel@rueg.eu>

view details

Michael Crosby

commit sha cedd3cb16c37d266de18dfda1b8aed71b1d998e0

Merge pull request #3782 from mrueg/vndr-autocompl Makefile: Drop vndr whitelist

view details

Tibor Vass

commit sha d12b53a072dd67aad37046bb4437043fe02d0419

ppc64: -buildmode=pie is not supported I needed this patch to compile containerd for ppc64. Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Lantao Liu

commit sha 78be736622e3eaf5b1f7c9030a930c0b916453bf

Return not found error if no platform matched. Signed-off-by: Lantao Liu <lantaol@google.com>

view details

Michael Crosby

commit sha 62b13256efd11fafd739fc60c5067430574422ec

Merge pull request #3786 from Random-Liu/not-found-error-if-no-platform Return not found error if no platform matched.

view details

Michael Crosby

commit sha 4a1a1bc626830b0a9d0efc8ae154e30d3486d742

Merge pull request #3784 from tiborvass/ppc64 ppc64: -buildmode=pie is not supported

view details

Derek McGowan

commit sha c5f4ae777a4828a5af7f38c76add8c547bf2891c

Update snapshot parent filter property to always exist The parent property may be empty but always exists. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha c03a7da2e6cfa7bc547b2abf6b0045f71845f0ab

Support target snapshot references on prepare Allows backend snapshots to bring existing snapshots into a namespace without requiring clients to fully snapshots when the target reference is known. Backend snapshots must explicitly implement this functionality, it is equivalent to sharing across namespaces and is up to the backend to use the label when it is given or ignore it. This enables remote snapshot functionality for a backend to query for a target snapshot before a client has performed any work to create that snapshot. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 12 days

pull request commentcontainerd/containerd

Support disk usage in windows snapshot.

Is this supposed to the the combined usage or the usage of a single snapshot?

The intent of Usage in the snapshot API is to represent the usage of a single snapshot (not the snapshot + all the parents). Defining "usage" in regards to content which may be shared is tricky, but that is up to the caller to figure out. This should only represent how much disk space would be free if this snapshot is removed.

Random-Liu

comment created time in 12 days

issue commentcontainerd/containerd

Support remote snapshotter to speed up image pulling

@ktock I am working on the unit tests now. I stopped to get the filters branch in so I could use that. Thank you for testing it out!

ktock

comment created time in 13 days

pull request commentcontainerd/containerd

mount: support FUSE helper

I was thinking that mount.proto should have repeated string umountPreCmd; repeated string umountPostCmd;

I don't think that is a road we want to go down. If these mounts require additional behavior we should focus on defining that as a register-able plugin interface rather than adding hooks to existing types. How much of a requirement is that to get this PR useable for what you are doing? I would really like to be able to support this but want to know if I should focus more on getting that plugin stuff working.

AkihiroSuda

comment created time in 14 days

Pull request review commentcontainerd/containerd

Support disk usage in windows snapshot.

 func (s *snapshotter) Commit(ctx context.Context, name, key string, opts ...snap 	} 	defer t.Rollback() -	usage := fs.Usage{-		Size: 0,+	// grab the existing id+	id, _, _, err := storage.GetInfo(ctx, key)+	if err != nil {+		return err 	} -	if _, err = storage.CommitActive(ctx, key, name, snapshots.Usage(usage), opts...); err != nil {-		return errors.Wrap(err, "failed to commit snapshot")+	usage, err := fs.DiskUsage(ctx, s.getSnapshotDir(id))+	if err != nil {+		return err 	} -	if err := t.Commit(); err != nil {-		return err+	if _, err = storage.CommitActive(ctx, key, name, snapshots.Usage(usage), opts...); err != nil {+		return errors.Wrap(err, "failed to commit snapshot") 	}-	return nil+	return t.Commit()

It looks like this is following what the overlay driver is doing, in that case maybe also update the rollback call

	defer func() {
		if err != nil {
			if rerr := t.Rollback(); rerr != nil {
				log.G(ctx).WithError(rerr).Warn("failed to rollback transaction")
			}
		}
	}()
Random-Liu

comment created time in 14 days

pull request commentcontainerd/zfs

Added filters to Walk method signature

I added the commit here since a separate PR would just fail without this PR anyway (since we don't have vendoring here)

Zyqsempai

comment created time in 14 days

push eventZyqsempai/zfs

Derek McGowan

commit sha 456dcfd8803bbaabd31a91d346ce7e37af7a5053

Fix snapshot commit to pass opts to storage Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

bpopovschi

commit sha 73767d7cc85aa4bdb2bca980ac884ef3fd5bfb75

Added filters to Walk method signature Signed-off-by: bpopovschi <zyqsempai@mail.ru>

view details

push time in 14 days

create barnchdmcgowan/zfs

branch : fix-commit-opts

created branch time in 14 days

fork dmcgowan/zfs

ZFS snapshotter plugin for containerd

fork in 14 days

pull request commentcontainerd/zfs

Added filters to Walk method signature

I found the bug, I will open up a separate PR for it. The Commit function wasn't passing along the opts so the labels weren't getting set.

Zyqsempai

comment created time in 14 days

pull request commentcontainerd/zfs

Added filters to Walk method signature

Seems regression?

Walk is a new test, so hard to tell if it is a regression. The issue may have existed previously and been unknown. Still investigating.

Zyqsempai

comment created time in 14 days

issue commentcontainerd/containerd

failed to pull image (centos7 overlayfs) again

@daniel5gh I think in your case ctr is still defaulting to overlayfs even though the daemon isn't. Try setting CONTAINERD_SNAPSHOTTER or with the --snapshotter option to see if that corrects it.

We should likely be disabling the overlay snapshotter on such a filesystem where it is unsupported, although I am not sure what is making that not work in this case. However, in your case, it seems as though you just want to make sure the zfs driver is also being used through ctr (CRI should be using it correctly). You can also try crictl to do your pulls through the same interface that k8s is using.

mg0432062

comment created time in 14 days

pull request commentmoby/moby

daemon: add grpc.WithBlock option

Before containerd serving, dockerd will failed with gRPC error. That bad thing is that restore action will still ignore the any non-NotFound errors and returns running state for already stopped container.

Isn't this just Docker mishandling the errors? My understanding is WithBlock was actually changing the underlying connection to not use a pool and wasn't handling reconnects after that.

fuweid

comment created time in 15 days

issue commentcontainerd/containerd

Add label indexing

The summary above is the only outline we have of this work. To be more detailed, see the current database structure. This work would involve updating the storage to have an indexes bucket under the namespace. Under this bucket would have a per type bucket -> per index bucket -> per value bucket -> list of type primary keys for that value. A new index manager structure would need to be added to handle the creation and removal of these indexes and each individual type would need to handle updating the index values. Additionally the lookup functions would need to be updated to choose the best lookup method based on the provided filters (this is probably the most pure academic work here, the rest is just updating CRUD operations).

dmcgowan

comment created time in 15 days

pull request commentcontainerd/containerd

mount: support FUSE helper

Thanks for the link, if mount.<type> has a defined format, then this make sense. It still doesn't seem necessary to handle fuse in a special way though. Maybe mount.<type>+<subtype> format or something like that, we could then do a binary lookup on mount.<type>.

Mostly yes, but some FUSE might want to register clean up shell command

How would this be registered? The plan for the new resource management proposal is that they would have the equivalent of mount and unmount. In this implementation though, there would have to be something else doing all this.

Anywhere, in the daemon's mount namespace?

I'm trying to think of this from a plugin perspective, if this was functionality we configured in the daemon, would that configuration need to be passed down to the shim.

AkihiroSuda

comment created time in 15 days

pull request commentcontainerd/ttrpc

Revert "Copy codes and status from grpc project"

Maybe having tests in this package? We can create an issue for having grpc compatibility tests, not sure exactly what that would look like as the possible breakages may not be so obvious to test for

crosbymichael

comment created time in 15 days

push eventcontainerd/ttrpc

Michael Crosby

commit sha d4834b09f5931a4909d264d6034b086768c66e72

Revert "Copy codes and status from grpc project" This reverts commit f02233564fb076ad4cb42937d72d194d158e4048. Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Derek McGowan

commit sha 4f1b8fe65a5c6ab7ecd74a227d6cde85b519af35

Merge pull request #57 from crosbymichael/revert-codes-copy Revert "Copy codes and status from grpc project"

view details

push time in 15 days

PR merged containerd/ttrpc

Revert "Copy codes and status from grpc project"

This reverts commit f02233564fb076ad4cb42937d72d194d158e4048.

This isn't going to work as expected because the errdefs package that is used throughout shims and others switches on the grpc codes packages.

Signed-off-by: Michael Crosby crosbymichael@gmail.com

+12 -1151

0 comment

16 changed files

crosbymichael

pr closed time in 15 days

push eventdocker/distribution

Adam Dobrawy

commit sha a994bb839dfded39435fcdd2651d2afdb2a305b9

use latest version of alpine when building the Docker container Signed-off-by: Adam Dobrawy <naczelnik@jawnosc.tk>

view details

Derek McGowan

commit sha dee21c0394b5e1e735412461dbd8d3a769e01799

Merge pull request #2991 from ad-m/patch-1 use latest version of alpine when building the Docker container

view details

push time in 19 days

Pull request review commentcontainerd/imgcrypt

Modified for non-core project

+Stefan Berger <stefanb@linux.ibm.com>

I suggest using a similar format to the project maintainers file and maybe adding something like this to the top

# imgcrypt maintainers
#
# As a containerd sub-project, containerd maintainers are also included from https://github.com/containerd/project/blob/master/MAINTAINERS.
# See https://github.com/containerd/project/blob/master/GOVERNANCE.md for description of maintainer role
#
# MAINTAINERS
# GitHub ID, Name, Email address
lumjjb

comment created time in 19 days

Pull request review commentstefanberger/imgcrypt

Modified for non-core project

+Stefan Berger <stefanb@linux.ibm.com>

I suggest using a similar format to the project maintainers file and maybe adding something like this to the top

# imgcrypt maintainers
#
# As a containerd sub-project, containerd maintainers are also included from https://github.com/containerd/project/blob/master/MAINTAINERS.
# See https://github.com/containerd/project/blob/master/GOVERNANCE.md for description of maintainer role
#
# MAINTAINERS
# GitHub ID, Name, Email address
lumjjb

comment created time in 19 days

push eventdmcgowan/containerd

Michael Crosby

commit sha 25947db049b058fcbce291ef883b8b512e3ea440

Add support for spec generation with host devices Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Eli Uriegas

commit sha 036db34f37617d6a02f07b04fc2d35a91732c6e3

build: Fix manpage generation Seems to be that docs/man/ctr.1.md and docs/man/containerd.1.md were removed in #3637 and were not updated correctly in the Makefile, leading to build failures like: + make man make: *** No rule to make target `man/ctr.1', needed by `man'. Stop. Changes the gen-manpages command to be specific on which manpages are to be generated. Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>

view details

Derek McGowan

commit sha 772aaf127ad8b311c5db7baafb31b5a3bc0e3a9f

Merge pull request #3729 from seemethere/fix_man build: Fix manpage generation

view details

Lantao Liu

commit sha ffcb1cc9be3eda8478d75c46ca02928db43b2693

Fix delete error code on the containerd daemon side. Signed-off-by: Lantao Liu <lantaol@google.com>

view details

Phil Estes

commit sha a6a0c8b6e36415a151d93d096c1c0af9e0bd7977

Merge pull request #3736 from Random-Liu/final-fix-delete-code Fix delete error code on the containerd daemon side.

view details

Michael Crosby

commit sha a6d3f4d30bcdebc97d10bdfefaf34ea00c77165f

Add device opts to ctr --privileged Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Michael Crosby

commit sha 54257bb5e6681971ce6521e88fad1f5da9d7c72c

Add local-fs.target to service file This will ensure that containerd is started after the /etc/fstab entries Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Phil Estes

commit sha 41028772ba66f921c2c4d4972337e936e9a49b0b

Merge pull request #3741 from crosbymichael/localfs Add local-fs.target to service file

view details

Michael Crosby

commit sha 38937f0607267f2effc76bc8fd4855e75ab8eb17

Merge pull request #3718 from crosbymichael/devperms Add spec opts for host device and privileged handling

view details

Michael Crosby

commit sha d5b7bf51aa7dc6217fea04e3e3b6e43289a25746

Move flag.Parse in tests to TestMain This this fixes issues with custom and testing flags in Go 1.3 and should work in previous go versions. Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Phil Estes

commit sha 6af355f3c7a2c1577d5e5e7b769ec26870ad21e9

Merge pull request #3744 from crosbymichael/test-flags Move flag.Parse in tests to TestMain

view details

Michael Crosby

commit sha f8cca26f3c0aea485841e96e3b524acc7ef6f4c1

Handle large output in v2 shim with TTY Reized the I/O buffers to align with the size of the kernel buffers with fifos and move the close aspect of the console to key off of the stdin closing. Fixes #3738 Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Phil Estes

commit sha 57cfc902606e2d601d4fd910291ccbd391234446

Merge pull request #3743 from crosbymichael/v2blocking Handle large output in v2 shim with TTY

view details

Akihiro Suda

commit sha d52cbc19be7a8bb4937e52a57cf32ff2ea3bcb31

snapshots/native: ignore xattr errors during CopyDir `secuity.*` xattrs cannot be copied in most cases For moby/buildkit#1189 Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Wei Fu

commit sha acdcf13d5eaf0dfe0eaeabe7194a82535549bc2b

Merge pull request #3749 from AkihiroSuda/native-copydir-allow-xattr-errors snapshots/native: ignore xattr errors during CopyDir

view details

Evan Cordell

commit sha 7177af84acdddfb3e36b7982cbac982bc7a8d5d5

Allow 202 response code for commit Quay returns this status code when pushing Signed-off-by: Evan Cordell <cordell.evan@gmail.com>

view details

Michael Crosby

commit sha 9c86b8f5ed49c63b887fe94c369c3a797e79c91f

Merge pull request #3750 from ecordell/202-accepted-response Allow 202 response code for commit

view details

Wei Fu

commit sha 074b453ac66797ab93d9570e826ef9c35b079b13

vendor: call vndr to remove useless pkgs and update vendor Signed-off-by: Wei Fu <fuweid89@gmail.com>

view details

Sebastiaan van Stijn

commit sha 6356e55be002df80b98ba59ec98dfd0ece7ec80c

Update Golang 1.12.12 (CVE-2019-17596) Golang 1.12.12 ------------------------------- go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.12 Golang 1.12.11 (CVE-2019-17596) ------------------------------- go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.11 [security] Go 1.13.2 and Go 1.12.11 are released Hi gophers, We have just released Go 1.13.2 and Go 1.12.11 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.2). Invalid DSA public keys can cause a panic in dsa.Verify. In particular, using crypto/x509.Verify on a crafted X.509 certificate chain can lead to a panic, even if the certificates don't chain to a trusted root. The chain can be delivered via a crypto/tls connection to a client, or to a server that accepts and verifies client certificates. net/http clients can be made to crash by an HTTPS server, while net/http servers that accept client certificates will recover the panic and are unaffected. Moreover, an application might crash invoking crypto/x509.(*CertificateRequest).CheckSignature on an X.509 certificate request, parsing a golang.org/x/crypto/openpgp Entity, or during a golang.org/x/crypto/otr conversation. Finally, a golang.org/x/crypto/ssh client can panic due to a malformed host key, while a server could panic if either PublicKeyCallback accepts a malformed public key, or if IsUserAuthority accepts a certificate with a malformed public key. The issue is CVE-2019-17596 and Go issue golang.org/issue/34960. Thanks to Daniel Mandragona for discovering and reporting this issue. We'd also like to thank regilero for a previous disclosure of CVE-2019-16276. The Go 1.13.2 release also includes a fix to the compiler that prevents improper access to negative slice indexes in rare cases. Affected code, in which the compiler can prove that the index is zero or negative, would have resulted in a panic in Go 1.12, but could have led to arbitrary memory read and writes in Go 1.13 and Go 1.13.1. This is Go issue golang.org/issue/34802. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Maksym Pavlenko

commit sha 36c4260e6fa15cb20012cfe95f322ef85cf771ca

Merge pull request #3760 from thaJeztah/bump_golang_1.12.x Update Golang 1.12.12 (CVE-2019-17596)

view details

push time in 19 days

push eventZyqsempai/containerd

Wei Fu

commit sha 074b453ac66797ab93d9570e826ef9c35b079b13

vendor: call vndr to remove useless pkgs and update vendor Signed-off-by: Wei Fu <fuweid89@gmail.com>

view details

Sebastiaan van Stijn

commit sha 6356e55be002df80b98ba59ec98dfd0ece7ec80c

Update Golang 1.12.12 (CVE-2019-17596) Golang 1.12.12 ------------------------------- go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.12 Golang 1.12.11 (CVE-2019-17596) ------------------------------- go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.11 [security] Go 1.13.2 and Go 1.12.11 are released Hi gophers, We have just released Go 1.13.2 and Go 1.12.11 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.2). Invalid DSA public keys can cause a panic in dsa.Verify. In particular, using crypto/x509.Verify on a crafted X.509 certificate chain can lead to a panic, even if the certificates don't chain to a trusted root. The chain can be delivered via a crypto/tls connection to a client, or to a server that accepts and verifies client certificates. net/http clients can be made to crash by an HTTPS server, while net/http servers that accept client certificates will recover the panic and are unaffected. Moreover, an application might crash invoking crypto/x509.(*CertificateRequest).CheckSignature on an X.509 certificate request, parsing a golang.org/x/crypto/openpgp Entity, or during a golang.org/x/crypto/otr conversation. Finally, a golang.org/x/crypto/ssh client can panic due to a malformed host key, while a server could panic if either PublicKeyCallback accepts a malformed public key, or if IsUserAuthority accepts a certificate with a malformed public key. The issue is CVE-2019-17596 and Go issue golang.org/issue/34960. Thanks to Daniel Mandragona for discovering and reporting this issue. We'd also like to thank regilero for a previous disclosure of CVE-2019-16276. The Go 1.13.2 release also includes a fix to the compiler that prevents improper access to negative slice indexes in rare cases. Affected code, in which the compiler can prove that the index is zero or negative, would have resulted in a panic in Go 1.12, but could have led to arbitrary memory read and writes in Go 1.13 and Go 1.13.1. This is Go issue golang.org/issue/34802. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Maksym Pavlenko

commit sha 36c4260e6fa15cb20012cfe95f322ef85cf771ca

Merge pull request #3760 from thaJeztah/bump_golang_1.12.x Update Golang 1.12.12 (CVE-2019-17596)

view details

Phil Estes

commit sha 3e3c5fe129e4d25e5ec215439bc7a0986e49f995

Merge pull request #3759 from fuweid/me-update-vendor vendor: call vndr to remove useless pkgs and update vendor

view details

Sebastiaan van Stijn

commit sha 885232b72f0b7ead6ffec76fe0adfda384572b6f

bump google.golang.org/grpc v1.23.1 full diff: https://github.com/grpc/grpc-go/compare/v1.23.0...v1.23.1 - grpc/grpc-go#3018 server: set and advertise max frame size of 16KB - grpc/grpc-go#3017 grpclb: fix deadlock in grpclb connection cache - Before the fix, if the timer to remove a SubConn fires at the same time NewSubConn cancels the timer, it caused a mutex leak and deadlock. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Phil Estes

commit sha 3bf461ae8ebec194e333a54ec0dfc562b741112c

Move autocomplete files to contrib/ Since recent versions of `vndr` are going to remove the autocomplete scripts from the urfave vendored content, we will just move them into `contrib/` and reference them in the documentation from that location. Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>

view details

Akihiro Suda

commit sha 0d8fc0a487f82b922122ca0280fe1ba3d19b3803

Merge pull request #3767 from thaJeztah/bump_grpc bump google.golang.org/grpc v1.23.1

view details

Phil Estes

commit sha 4523ab734aff7fd4abc0b676b767b1026b826278

Merge pull request #3766 from estesp/move-autocomplete Move autocomplete files to contrib/

view details

Justin Terry (VM)

commit sha 178469e2ae5b7edd9e55601415fe9f494ab22b9a

Update Microsoft/hcsshim vendor Updates Microsoft/hcsshim vendor commit hash to a recent version that now: 1. Supports container stats via the Stats RuntimeV2 gRPC call. 2. Fixes a regression when issuing a resize of the pty after the container has exited which previously in Docker was expected to be a non-error case. 3. Puts in a workaround when using a non-default sandbox size for Windows containers due to a platform bug. This expansion now happens in the go library itself. Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>

view details

Justin Terry (VM)

commit sha 37b56cafc63445721c4475eea4e0994de3ed5118

Add ctr metrics support for Windows/LCOW containers Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>

view details

Akihiro Suda

commit sha 966b1b8e30c9ccf7e0f3127da08d4db30133e3bc

Merge pull request #3775 from jterry75/vendor_hcsshim Update Microsoft/hcsshim vendor

view details

Akihiro Suda

commit sha c224edc5c6350026a7d35a09dce638b0f09e6d44

apply: use naive applier when running in UserNS `OverlayConvertWhiteout` calls `mknod c 0 0` which is not allowed when running in a user namespace, even in Ubuntu kernel. Although there is an alternative hacky way to create whiteouts without calling mknod as Moby `overlay2` actually does(see #3762), let's use naive applier when running in UserNS and call it a day. Close #3762 Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Lantao Liu

commit sha aaccfcbe2b8792e5fa3711811f3025562485e8bb

Fix `containerd config dump`. Signed-off-by: Lantao Liu <lantaol@google.com>

view details

Phil Estes

commit sha f05e19c5c6fa330753b84fe200f887cb3d62df41

Merge pull request #3777 from Random-Liu/fix-containerd-config Fix `containerd config dump`.

view details

Phil Estes

commit sha c59561a08ec1936d4632a6b4868110681216d160

Merge pull request #3763 from AkihiroSuda/disable-mknod00-in-userns apply: use naive applier when running in UserNS

view details

bpopovschi

commit sha e8c14c07c6d9c33df7484bdde4df166627b6b44a

Added filters to snapshots API Signed-off-by: bpopovschi <zyqsempai@mail.ru>

view details

Derek McGowan

commit sha 66aa1d3ef6f69be075f6acf10123f7e8db9112c2

Add snapshot walk implementations Temporarily remove zfs and aufs until interface update Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 19 days

Pull request review commentcontainerd/containerd

Added filters to snapshots API

 k8s.io/kubernetes v1.16.0-rc.2 k8s.io/utils c2654d5206da6b7b6ace12841e8f359bb89b443c sigs.k8s.io/yaml v1.1.0 -# zfs dependencies-github.com/containerd/zfs 2ceb2dbb8154202ed1b8fd32e4ea25b491d7b251-github.com/mistifyio/go-zfs f784269be439d704d3dfa1906f45dd848fed2beb-github.com/google/uuid v1.1.1+# zfs dependencies (temporarily removed)+# github.com/containerd/zfs 2ceb2dbb8154202ed1b8fd32e4ea25b491d7b251+# github.com/mistifyio/go-zfs f784269be439d704d3dfa1906f45dd848fed2beb+# github.com/google/uuid v1.1.1

Moving it outside of the zfs block

Zyqsempai

comment created time in 19 days

Pull request review commentcontainerd/containerd

apply: use naive applier when running in UserNS

 import ( 	"github.com/containerd/containerd/archive" 	"github.com/containerd/containerd/errdefs" 	"github.com/containerd/containerd/mount"+	"github.com/opencontainers/runc/libcontainer/system" 	"github.com/pkg/errors" )  func apply(ctx context.Context, mounts []mount.Mount, r io.Reader) error { 	switch {+	// OverlayConvertWhiteout (mknod c 0 0) doesn't work in userns.+	// https://github.com/containerd/containerd/issues/3762+	case system.RunningInUserNS():+		break

Should we just do the break right at the beginning of the overlay case? AUFS should still work fine right?

AkihiroSuda

comment created time in 20 days

Pull request review commentcontainerd/containerd

apply: use naive applier when running in UserNS

 func apply(ctx context.Context, mounts []mount.Mount, r io.Reader) error { 			} 			return err 		}-		opts := []archive.ApplyOpt{-			archive.WithConvertWhiteout(archive.OverlayConvertWhiteout),+		opts := []archive.ApplyOpt{}+		// OverlayConvertWhiteout (mknod c 0 0) doesn't work in userns.+		// https://github.com/containerd/containerd/issues/3762+		if !system.RunningInUserNS() {

I guess once the process is running that cannot change, right?

AkihiroSuda

comment created time in 20 days

push eventdmcgowan/release-tool

Derek McGowan

commit sha ac2bd48f0d9d23cdde18e0b0fefd8ae6dba9a11c

Add golangci Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 20 days

PR opened containerd/release-tool

Add golangci

Closes #6

+20 -0

0 comment

2 changed files

pr created time in 20 days

create barnchdmcgowan/release-tool

branch : add-lint

created branch time in 20 days

pull request commentcontainerd/containerd

Added filters to snapshots API

@fuweid I agree, I was going to do that and forgot. Commented those out from vendor for now

Zyqsempai

comment created time in 20 days

push eventZyqsempai/containerd

Derek McGowan

commit sha 63879fd57e32f44fe6c3a3096b180358957d8aaf

Add snapshot walk implementations Temporarily remove zfs and aufs until interface update Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 20 days

push eventZyqsempai/containerd

Derek McGowan

commit sha 58608a1c91bbed3dd739ae9159a596ad3bc341a2

Add snapshot walk implementations Temporarily remove zfs and aufs until interface update Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 20 days

pull request commentcontainerd/release-tool

Parse go.mod and compare if present

I could add tests, but I don't think I'll be able to get to it in a timely manner

Not necessary for this PR. We can take care of adding those sort of linting things, saves everyone time rather than manually going through that during review.

haircommander

comment created time in 20 days

pull request commentcontainerd/ttrpc

Copy codes and status pkg from grpc

We can add our copyright above, we just shouldn't remove the old one

crosbymichael

comment created time in 20 days

push eventdmcgowan/containerd

Derek McGowan

commit sha 158844a03bf9f9d711ce1a93e6b0decc4cbc9b45

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha dc29eea2b0bda1fcf149d880b41eb95f01f0c177

Update tar test to show chown error log Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 21 days

Pull request review commentmoby/moby

Update buildkit to containerd leases

 func (p *puller) CacheKey(ctx context.Context, index int) (string, bool, error) 	return k, true, nil } +func (p *puller) getRef(ctx context.Context, diffIDs []layer.DiffID, opts ...cache.RefOption) (cache.ImmutableRef, error) {+	var parent cache.ImmutableRef+	if len(diffIDs) > 1 {+		var err error+		parent, err = p.getRef(ctx, diffIDs[:len(diffIDs)-1], opts...)+		if err != nil {+			return nil, err+		}+		defer parent.Release(context.TODO())+	}+	return p.is.CacheAccessor.GetByBlob(context.TODO(), ocispec.Descriptor{

Why is this also a context.TODO?

tonistiigi

comment created time in 21 days

Pull request review commentmoby/moby

Update buildkit to containerd leases

 func (p *puller) CacheKey(ctx context.Context, index int) (string, bool, error) 	return k, true, nil } +func (p *puller) getRef(ctx context.Context, diffIDs []layer.DiffID, opts ...cache.RefOption) (cache.ImmutableRef, error) {+	var parent cache.ImmutableRef+	if len(diffIDs) > 1 {+		var err error+		parent, err = p.getRef(ctx, diffIDs[:len(diffIDs)-1], opts...)+		if err != nil {+			return nil, err+		}+		defer parent.Release(context.TODO())+	}+	return p.is.CacheAccessor.GetByBlob(context.TODO(), ocispec.Descriptor{+		Annotations: map[string]string{+			"containerd.io/uncompressed": diffIDs[len(diffIDs)-1].String(),

This is currently our recommended way to use this value. Each package can define the constant but we wanted to avoid importing packages just for a constant. This value can have a different form when used as a key or filter.

tonistiigi

comment created time in 21 days

PR merged containerd/containerd

[release/1.2 backport] Update Golang 1.12.12 (CVE-2019-17596)

backport of https://github.com/containerd/containerd/pull/3760

Golang 1.12.12

go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details.

https://github.com/golang/go/issues?q=milestone%3AGo1.12.12

Golang 1.12.11 (CVE-2019-17596)

go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.11

[security] Go 1.13.2 and Go 1.12.11 are released

Hi gophers,

We have just released Go 1.13.2 and Go 1.12.11 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.2).

Invalid DSA public keys can cause a panic in dsa.Verify. In particular, using crypto/x509.Verify on a crafted X.509 certificate chain can lead to a panic, even if the certificates don't chain to a trusted root. The chain can be delivered via a crypto/tls connection to a client, or to a server that accepts and verifies client certificates. net/http clients can be made to crash by an HTTPS server, while net/http servers that accept client certificates will recover the panic and are unaffected.

Moreover, an application might crash invoking crypto/x509.(*CertificateRequest).CheckSignature on an X.509 certificate request, parsing a golang.org/x/crypto/openpgp Entity, or during a golang.org/x/crypto/otr conversation. Finally, a golang.org/x/crypto/ssh client can panic due to a malformed host key, while a server could panic if either PublicKeyCallback accepts a malformed public key, or if IsUserAuthority accepts a certificate with a malformed public key.

The issue is CVE-2019-17596 and Go issue golang.org/issue/34960.

Thanks to Daniel Mandragona for discovering and reporting this issue. We'd also like to thank regilero for a previous disclosure of CVE-2019-16276.

The Go 1.13.2 release also includes a fix to the compiler that prevents improper access to negative slice indexes in rare cases. Affected code, in which the compiler can prove that the index is zero or negative, would have resulted in a panic in Go 1.12, but could have led to arbitrary memory read and writes in Go 1.13 and Go 1.13.1. This is Go issue golang.org/issue/34802.

Signed-off-by: Sebastiaan van Stijn github@gone.nl (cherry picked from commit 6356e55be002df80b98ba59ec98dfd0ece7ec80c) Signed-off-by: Sebastiaan van Stijn github@gone.nl

+3 -3

4 comments

3 changed files

thaJeztah

pr closed time in 21 days

push eventcontainerd/containerd

Sebastiaan van Stijn

commit sha e319caedc4f32d6eb91352c52e008f6c934fb2cc

Update Golang 1.12.12 (CVE-2019-17596) Golang 1.12.12 ------------------------------- go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.12 Golang 1.12.11 (CVE-2019-17596) ------------------------------- go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.11 [security] Go 1.13.2 and Go 1.12.11 are released Hi gophers, We have just released Go 1.13.2 and Go 1.12.11 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.2). Invalid DSA public keys can cause a panic in dsa.Verify. In particular, using crypto/x509.Verify on a crafted X.509 certificate chain can lead to a panic, even if the certificates don't chain to a trusted root. The chain can be delivered via a crypto/tls connection to a client, or to a server that accepts and verifies client certificates. net/http clients can be made to crash by an HTTPS server, while net/http servers that accept client certificates will recover the panic and are unaffected. Moreover, an application might crash invoking crypto/x509.(*CertificateRequest).CheckSignature on an X.509 certificate request, parsing a golang.org/x/crypto/openpgp Entity, or during a golang.org/x/crypto/otr conversation. Finally, a golang.org/x/crypto/ssh client can panic due to a malformed host key, while a server could panic if either PublicKeyCallback accepts a malformed public key, or if IsUserAuthority accepts a certificate with a malformed public key. The issue is CVE-2019-17596 and Go issue golang.org/issue/34960. Thanks to Daniel Mandragona for discovering and reporting this issue. We'd also like to thank regilero for a previous disclosure of CVE-2019-16276. The Go 1.13.2 release also includes a fix to the compiler that prevents improper access to negative slice indexes in rare cases. Affected code, in which the compiler can prove that the index is zero or negative, would have resulted in a panic in Go 1.12, but could have led to arbitrary memory read and writes in Go 1.13 and Go 1.13.1. This is Go issue golang.org/issue/34802. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 6356e55be002df80b98ba59ec98dfd0ece7ec80c) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 435e05fd0d3e367c330868fa70d2b054c6273204

[release/1.2] pin travis to go 1.12.12 Use a fixed version for the release branch. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Derek McGowan

commit sha c2383a5f2c046b1a262209f981420ffd5fa4b354

Merge pull request #3768 from thaJeztah/1.2_backport_bump_golang_1.12.x [release/1.2 backport] Update Golang 1.12.12 (CVE-2019-17596)

view details

push time in 21 days

push eventcontainerd/containerd

Sebastiaan van Stijn

commit sha 23b0ca70feadeda6b9f1f7072e71687c660590ca

Update Golang 1.12.12 (CVE-2019-17596) Golang 1.12.12 ------------------------------- go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.12 Golang 1.12.11 (CVE-2019-17596) ------------------------------- go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.11 [security] Go 1.13.2 and Go 1.12.11 are released Hi gophers, We have just released Go 1.13.2 and Go 1.12.11 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.2). Invalid DSA public keys can cause a panic in dsa.Verify. In particular, using crypto/x509.Verify on a crafted X.509 certificate chain can lead to a panic, even if the certificates don't chain to a trusted root. The chain can be delivered via a crypto/tls connection to a client, or to a server that accepts and verifies client certificates. net/http clients can be made to crash by an HTTPS server, while net/http servers that accept client certificates will recover the panic and are unaffected. Moreover, an application might crash invoking crypto/x509.(*CertificateRequest).CheckSignature on an X.509 certificate request, parsing a golang.org/x/crypto/openpgp Entity, or during a golang.org/x/crypto/otr conversation. Finally, a golang.org/x/crypto/ssh client can panic due to a malformed host key, while a server could panic if either PublicKeyCallback accepts a malformed public key, or if IsUserAuthority accepts a certificate with a malformed public key. The issue is CVE-2019-17596 and Go issue golang.org/issue/34960. Thanks to Daniel Mandragona for discovering and reporting this issue. We'd also like to thank regilero for a previous disclosure of CVE-2019-16276. The Go 1.13.2 release also includes a fix to the compiler that prevents improper access to negative slice indexes in rare cases. Affected code, in which the compiler can prove that the index is zero or negative, would have resulted in a panic in Go 1.12, but could have led to arbitrary memory read and writes in Go 1.13 and Go 1.13.1. This is Go issue golang.org/issue/34802. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 6356e55be002df80b98ba59ec98dfd0ece7ec80c) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 7f6f2c7d4fbcae8d95b4561107400e0d77701a29

[release/1.3] pin travis to go 1.12.12 Use a fixed version for the release branch. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Derek McGowan

commit sha 7af311b4200b464a79c340b4e3a2799f8906ee8d

Merge pull request #3769 from thaJeztah/1.3_backport_bump_golang_1.12.x [release/1.3 backport] Update Golang 1.12.12 (CVE-2019-17596)

view details

push time in 21 days

PR merged containerd/containerd

[release/1.3 backport] Update Golang 1.12.12 (CVE-2019-17596)

backport of https://github.com/containerd/containerd/pull/3760

Golang 1.12.12

go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details.

https://github.com/golang/go/issues?q=milestone%3AGo1.12.12

Golang 1.12.11 (CVE-2019-17596)

go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.12.11

[security] Go 1.13.2 and Go 1.12.11 are released

Hi gophers,

We have just released Go 1.13.2 and Go 1.12.11 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.2).

Invalid DSA public keys can cause a panic in dsa.Verify. In particular, using crypto/x509.Verify on a crafted X.509 certificate chain can lead to a panic, even if the certificates don't chain to a trusted root. The chain can be delivered via a crypto/tls connection to a client, or to a server that accepts and verifies client certificates. net/http clients can be made to crash by an HTTPS server, while net/http servers that accept client certificates will recover the panic and are unaffected.

Moreover, an application might crash invoking crypto/x509.(*CertificateRequest).CheckSignature on an X.509 certificate request, parsing a golang.org/x/crypto/openpgp Entity, or during a golang.org/x/crypto/otr conversation. Finally, a golang.org/x/crypto/ssh client can panic due to a malformed host key, while a server could panic if either PublicKeyCallback accepts a malformed public key, or if IsUserAuthority accepts a certificate with a malformed public key.

The issue is CVE-2019-17596 and Go issue golang.org/issue/34960.

Thanks to Daniel Mandragona for discovering and reporting this issue. We'd also like to thank regilero for a previous disclosure of CVE-2019-16276.

The Go 1.13.2 release also includes a fix to the compiler that prevents improper access to negative slice indexes in rare cases. Affected code, in which the compiler can prove that the index is zero or negative, would have resulted in a panic in Go 1.12, but could have led to arbitrary memory read and writes in Go 1.13 and Go 1.13.1. This is Go issue golang.org/issue/34802.

Signed-off-by: Sebastiaan van Stijn github@gone.nl (cherry picked from commit 6356e55be002df80b98ba59ec98dfd0ece7ec80c) Signed-off-by: Sebastiaan van Stijn github@gone.nl

+3 -3

6 comments

3 changed files

thaJeztah

pr closed time in 21 days

push eventdmcgowan/containerd

Derek McGowan

commit sha 35ee42b2b36810866c617dc3c90ca6305ee035a3

Add arm64 build to Travis Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 7f72a9c8550452a26ed0999035a3870874a911d7

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 05a4849d7f6e7d8e796f3ac581cb45d5ba5c2705

Update tar test to show chown error log Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 21 days

pull request commentcontainerd/containerd

mount: support FUSE helper

We are trying to come up with a generic way to handle resources (specifically mounts and fuse processes) for 1.4. However, for these binaries it seems we could potentially make these plugable on their own. A few questions I have since I am not as familiar with FUSE.

  1. Will unmounting the target be sufficient for cleanup
  2. Could this be done with a sequence of mount -t commands, what are these binaries additionally doing?
  3. Does the mount need to be performed here inside of the containerd daemon or the shim?
AkihiroSuda

comment created time in 21 days

Pull request review commentcontainerd/containerd

mount: support FUSE helper

 func mountAt(chdir string, source, target, fstype string, flags uintptr, data st 	} 	return errors.Wrap(sys.FMountat(f.Fd(), source, target, fstype, flags, data), "failed to mountat") }++func (m *Mount) mountWithHelper(helperBinary, typePrefix, target string) error {+	// helperBinary: "mount.fuse3"+	// target: "/foo/merged"+	// m.Type: "fuse3.fuse-overlayfs"+	// command: "mount.fuse3 overlay /foo/merged -o lowerdir=/foo/lower2:/foo/lower1,upperdir=/foo/upper,workdir=/foo/work -t fuse-overlayfs"+	args := []string{m.Source, target}+	for _, o := range m.Options {+		args = append(args, "-o", o)+	}+	args = append(args, "-t", strings.TrimPrefix(m.Type, typePrefix))+	cmd := exec.Command(helperBinary, args...)

I don't think this is generic enough. The options should probably just be the arguments.

AkihiroSuda

comment created time in 21 days

push eventdmcgowan/containerd

Derek McGowan

commit sha af2621499a040677a7ffea58051e48a83c952d91

fixup: remove dmesg

view details

Derek McGowan

commit sha 0b4da0a0e9fb81191efced2687c0ecf43ee12b73

Update tar test to show chown error log Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 21 days

push eventdmcgowan/containerd

Derek McGowan

commit sha 7b62f1848dc159fc6f508b48dda91fdb39ad5138

Add arm64 build to Travis Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha d78eae13393238c9d48282ffdc82bb78ece9df25

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 21 days

push eventdmcgowan/containerd

Derek McGowan

commit sha 26994b5bf42ffcaa1e1db70711c753449747cbe4

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 21 days

push eventdmcgowan/containerd

Derek McGowan

commit sha a5d47874102ec10213b7a949650c0e4edf73aff1

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in 21 days

Pull request review commentcontainerd/containerd

apply: use naive applier when running in UserNS

 func apply(ctx context.Context, mounts []mount.Mount, r io.Reader) error { 			} 			return err 		}-		opts := []archive.ApplyOpt{-			archive.WithConvertWhiteout(archive.OverlayConvertWhiteout),+		opts := []archive.ApplyOpt{}+		// OverlayConvertWhiteout (mknod c 0 0) doesn't work in userns.+		// https://github.com/containerd/containerd/issues/3762+		if !system.RunningInUserNS() {

This will need to fall through to do the temp mount at the end of the function. Maybe just add to the case statement. Maybe do a sync.Once, not sure if that tradeoff is worth it.

AkihiroSuda

comment created time in 22 days

push eventdmcgowan/containerd

Derek McGowan

commit sha 35141005772920e19e2b1ad80a6aa4cc6cd8dcbc

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

push eventdmcgowan/containerd

Derek McGowan

commit sha 0f9d3a322afeb242d2c8fe31c9851d691a714cdd

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

push eventdmcgowan/containerd

Derek McGowan

commit sha 3bab9b1a5663f14ee5333e38dd95a00ba36e2ee3

Add sysinfo script Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

push eventdmcgowan/containerd

Derek McGowan

commit sha 95f8083352b599fab23a860c8534309ac2f2dee6

Add arm64 build to Travis Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

push eventdmcgowan/containerd

Derek McGowan

commit sha c491fa77b7e6dcb3ef1b7e5128c4bdf69b45dacd

Add arm64 build to Travis Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

PR opened containerd/containerd

Add arm64 build to Travis

Opening as draft until Travis builds verified

+3 -0

0 comment

1 changed file

pr created time in a month

create barnchdmcgowan/containerd

branch : travis-arm64

created branch time in a month

issue openedcontainerd/containerd

Add binary upgrade tests

Add a new test suite, possibly in contrib, which focuses on integration testing with the built binaries and capable of doing upgrade testing from previous releases. These tests may use ctr to do the integration testing, however only the latest version of ctr should be used even when upgrade testing (API is stable but ctr is not).

created time in a month

push eventZyqsempai/containerd

Derek McGowan

commit sha ee085d7ad300aa30f002517f4cd495c500f19097

Add snapshot walk implementations Temporarily remove zfs and aufs until interface update Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

push eventZyqsempai/containerd

Derek McGowan

commit sha 4ee715ec1eccd72ba3bcbe277a6b310981ef1860

Add snapshot walk implementations Temporarily remove zfs and aufs until interface update Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

push eventZyqsempai/containerd

Manuel Rüger

commit sha b5fa55b0c2837d033614c2c2ab1488d097aec2a0

ctr: Enable shell autocompletion Signed-off-by: Manuel Rüger <manuel@rueg.eu>

view details

Phil Estes

commit sha 309c9c25baa81db884645a9a1539f5a537b515d6

Merge pull request #3706 from mrueg/enable-autocmpl ctr: Enable shell autocompletion

view details

Phil Estes

commit sha e33f70385b6664e7fed5312b0972d6c38952162d

Add more detail on releases Add more detail to the README.md where releases are found and also add the documentation for how to use the new auto-completion capabilities provided by urfave and enabled in a recent PR. Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>

view details

Michael Crosby

commit sha f277665edba8c08ffc8ce9a9348d73a2765841a0

Bump runc to 1b8a1eeec3f337ab5d94f28980 Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Phil Estes

commit sha 4c2d740a2ae6a741be5f62fe81e5a4fd42927d65

Merge pull request #3713 from crosbymichael/runc-bump2 Bump runc to 1b8a1eeec3f337ab5d94f28980

view details

Michael Crosby

commit sha be6bead423dccf804add1cbe4b053c260fba6579

Merge pull request #3712 from estesp/add-completion-docs Add more detail on releases to README

view details

Lantao Liu

commit sha 18be6e37140e778dffd91804dab2bc66ba54493f

Use cached state instead of `runc state`. Signed-off-by: Lantao Liu <lantaol@google.com>

view details

Michael Crosby

commit sha 38a0013c70944d09876f96854c3bbcb9984e39d3

Merge pull request #3711 from Random-Liu/avoid-unnecessary-runc-state Use cached state instead of `runc state`.

view details

Maksym Pavlenko

commit sha e4e7e554d5ddc8291f64be897b595ec0f9aa0d15

Make client's WithLease more flexible. It's sometimes useful to specify a custom lease configuration. In this case clients have to deal with LeaseManager manually to create/defer delete lease (essentially dup WithLease logic). This PR adds lease opts to WithLease, so options can be specified inline. Signed-off-by: Maksym Pavlenko <makpav@amazon.com>

view details

Michael Crosby

commit sha 2a6d13b2ca2d529103885cfc02e6cc57963eb439

Merge pull request #3719 from mxpv/lease_opts Make client's WithLease more flexible

view details

Derek McGowan

commit sha 772032598a4282a50ccee29e9cc23be88bd74b2f

Fix flaky btrfs test Add logging and move the creation of the snapshotter inside the attempt loop to catch cases where the mountinfo may not be updated yet. When all attempts are reached there is no reason to create the snapshotter as the unmount has already occurred. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Sebastiaan van Stijn

commit sha fe947e83dce45d108bb364262a1ce2c56a3828c3

bump runc v1.0.0-rc9 No code changes since the last vendor bump Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

fahed dorgaa

commit sha c0984941e38447933a5599a7c8233ed8fc218622

adding go version to client description Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com> gofmt version.go Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com> add a Godoc Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com> go fmt version.go Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com> add sapces to comment Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

view details

Akihiro Suda

commit sha ea8cb3ac462695d471b14e12e94ad4d302514387

Merge pull request #3721 from thaJeztah/bump_runc_1.0.0-rc9 bump runc v1.0.0-rc9

view details

Maksym Pavlenko

commit sha 635dbf251a8e325064bdc3695a373628b035ec38

Merge pull request #3720 from dmcgowan/fix-flaky-btrfs Fix flaky btrfs test

view details

Michael Crosby

commit sha 8d0d1ebb6375f953178bfe394b3380fbc2c10a42

Merge pull request #3716 from fahedouch/add_go_version_to_cli adding go version to client description

view details

Michael Crosby

commit sha 25947db049b058fcbce291ef883b8b512e3ea440

Add support for spec generation with host devices Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Wei Fu

commit sha eaef634ef79d474d78713d637a593379618c5680

version: use runtime.Version() directly Signed-off-by: Wei Fu <fuweid89@gmail.com>

view details

Michael Crosby

commit sha 901bcb2231466229d27aee8d38a6e2fcdc95015e

Add distribution subpkgs to core Ref: #3554 Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

view details

Phil Estes

commit sha c58e1e20d6810beee6f4e6a375a63ba6d266db95

Merge pull request #3725 from fuweid/me-use-runtime-version version: use runtime.Version() directly

view details

push time in a month

create barnchdmcgowan/containerd

branch : filter-snapshot-walk

created branch time in a month

pull request commentmoby/buildkit

Update storage management from root labels to leases

Needs rebase

tonistiigi

comment created time in a month

Pull request review commentmoby/buildkit

Update storage management from root labels to leases

 func NewWorkerOpt(root string, snFactory SnapshotterFactory, rootless bool, proc 	for k, v := range labels { 		xlabels[k] = v 	}+	snap := containerdsnapshot.NewSnapshotter(snFactory.Name, mdb.Snapshotter(snFactory.Name), "buildkit", idmap)+	lm := leaseutil.WithNamespace(ctdmetadata.NewLeaseManager(mdb), "buildkit")+	if err := cache.MigrateV2(context.TODO(), filepath.Join(root, "metadata.db"), filepath.Join(root, "metadata_v2.db"), c, snap, lm); err != nil {

Is this context going to be passed in? The calling function signature looks like it could use some attention

tonistiigi

comment created time in a month

pull request commentmoby/moby

Add flag to configure containerd healthcheck delay.

Thoughts?

Waiting for an exit seems to make more sense for me too. I am not quite sure what conditions the healthcheck is really protecting against now, OOM kills and crashes?

cpuguy83

comment created time in a month

Pull request review commentcontainerd/containerd

Added filters to snapshots API

 type Snapshotter interface { 	Remove(ctx context.Context, key string) error  	// Walk all snapshots in the snapshotter. For each snapshot in the-	// snapshotter, the function will be called.-	Walk(ctx context.Context, fn func(context.Context, Info) error) error+	// snapshotter, the function will be called, unless filters were used.+	// Zero or more filters may be provided as strings. Only events that match+	// *any* of the provided filters will be sent on the channel. The filters use+	// the standard containerd filters package syntax.+	Walk(ctx context.Context, fn func(context.Context, Info) error, filters ...string) error

The problem with a custom func is that it gives no performance advantage. The filters allow the backend to limit what gets sent back as well as the possibility of using indexing. If we don't fix it now, it will just get harder to fix in the future. Adding any new function would be the same issue.

Zyqsempai

comment created time in a month

pull request commentmoby/moby

Add flag to configure containerd healthcheck delay.

What do you find to be a reasonable value here? Wondering why not just change this to a more reasonable value since this option does not apply when containerd is not a child process.

cpuguy83

comment created time in a month

Pull request review commentmoby/moby

Add flag to configure containerd healthcheck delay.

 func WithPlugin(name string, conf interface{}) DaemonOpt { 		return nil 	} }++// WithHealthcheckDelay configures the freqeucny that healthchecks

Typo in the comment

cpuguy83

comment created time in a month

Pull request review commentcontainerd/containerd

[release/1.3] build: Fix manpage generation

 func run() error { 		"containerd": command.App(), 		"ctr":        app.New(), 	}-	dir := flag.Arg(0)-	for name, app := range apps {-		// clear out the usage as we use banners that do not display in man pages-		app.Usage = ""-		data, err := app.ToMan()-		if err != nil {-			return err-		}-		if _, err := os.Stat(dir); os.IsNotExist(err) {-			os.Mkdir(dir, os.ModePerm)-		}-		if err := ioutil.WriteFile(filepath.Join(dir, fmt.Sprintf("%s.1", name)), []byte(data), 0644); err != nil {-			return err-		}+	name := flag.Arg(0)+	dir := flag.Arg(1)+	app, ok := apps[name]+	if !ok {+		return fmt.Errorf("Invalid application '%s'", name)

I though the linters were complaining about this, maybe just a warning. Should fix upstream though if we have to, fine for now

seemethere

comment created time in a month

Pull request review commentcontainerd/containerd

[release/1.3] build: Fix manpage generation

 script:   - go build -i .   - make check   - if [ "$GOOS" = "linux" ]; then make check-protos check-api-descriptors; fi+  - if [ "$TRAVIS_GOOS" = "linux" ]; then make man ; fi

TRAVIS_GOOS is just defined in the env section and used in this yml, it has no other meaning. Not sure why GOOS is used above, but TRAVIS_GOOS should be used here since we set it and know what it is set to.

seemethere

comment created time in a month

push eventdmcgowan/containerd

Derek McGowan

commit sha f6c2401168384e88084791a8edf3f58ec608bdd9

WIP: support target snapshots on prepare Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

push eventdmcgowan/containerd

Derek McGowan

commit sha d9e4c5e5ed5be4bf28cbd36753307ab84be302b6

WIP: support target snapshots on prepare Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

issue commentcontainerd/containerd

Support remote snapshotter to speed up image pulling

My WIP of this is here https://github.com/dmcgowan/containerd/tree/prepare-snapshot-target Mostly just figuring out the metadata store side right now and making sure it is backwards compatible.

Basically it allows an ErrAlreadyExists to be returned when passing in containerd.io/snapshot.ref. The backend snapshotter may also return that and the metadata store will handle it by calling a stat to get the snapshot info then adding it to the metadata store. The ErrAlreadyExists will be returned to the client at that point to re-check whether the target snapshot is already there.

ktock

comment created time in a month

push eventdmcgowan/containerd

Derek McGowan

commit sha c8981ccf8199fedbf5197996d854e6035eabeb30

WIP: support target snapshots on prepare Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

create barnchdmcgowan/containerd

branch : prepare-snapshot-target

created branch time in a month

Pull request review commentopencontainers/artifacts

Artifacts Spec

+# Open Container Initiative++## Artifacts Specification++This specification defines using the [index][image-index] and [manifest][image-manifest] to define new artifact types, persisted within the [OCI Distribution Spec][distribution-spec]++The goal of this specification is to standardize artifact distribution, leveraging the infrastructure used to run [OCI Distribution Spec][distribution-spec] compliant registries.++## Table of Contents++- [Scope](#scope)+  - [Future](#future-scope)+- [Defining OCI Artifact Types](#defining-oci-artifact-types)+- [Defining a Unique Artifact Type](#defining-a-unique-artifact-type)+- [Defining Supported Layer Types](#defining-supported-layer-types)+- [Optional: Defining Config Schema](#optional-defining-config-schema)+- [Optional: Artifact Publisher Manifest](#optional-defining-artifact-publisher-manifests)+  - [Artifact Publisher Manifest Example](#artifact-publisher-manifest-type-example) as a [Well Known Type][def-well-known-types]++## Scope++The scope of Artifacts v1.0 is based on what exists in 1.0 of [manifest][image-manifest].++This includes individual artifacts that use [OCI manifest][image-manifest] including [OCI Image][image-spec], [Helm][helm] and [Singularity][singularity].++### Future Scope++Future versions of artifacts will support collections of artifacts using [OCI Index][image-index].++To support artifacts that represent a collection of other artifacts, a means to identify an index is a type of artifact will be required.++## Defining OCI Artifact Types++As registries become content addressable distribution points, tools that pull artifacts must know if they can operate on the artifact. Artifact types are equivalent to file extensions. When users open files, the host operating system typically launches the appropriate program. When users open a file, from within a program, the open dialog filters to the supported types. When search or security software scan the contents of a storage solution, the software must to know how to process the different types of content. When users view the contents of a storage solution, they see the textual and visual indications of the type. OCI Artifacts provides these core capabilities to [OCI distribution spec][distribution-spec] based registries.++Authoring an OCI Artifacts involves the following steps:++- Define a unique type+- Define the format for other tools to operate upon the type+- Define human elements, such as an icon and localized string to be displayed to users+- Optionally, publish as a [well-known type][def-well-known-types] for registries to consume++Defining a unique type, string and logo; artifacts can be represented as the following:++|Icon|Artifact|`config.mediaType`|+|-|-|-|+|<img src="https://github.com/opencontainers/artwork/blob/master/oci/icon/color/oci-icon-color.png?raw=true" width=30x>|[OCI Image][image-spec]|`application/vnd.oci.image.config.v1+json`|+|<img src="https://github.com/helm/helm-www/blob/master/themes/helm/static/img/apple-touch-icon.png?raw=true" width=30x>|[Helm Chart](https://helm.sh)|`application/vnd.cncf.helm.chart.config.v1+json`|+|<img src="https://github.com/sylabs/singularity/blob/master/docs/logos/singularity_v3.png?raw=true" width=30x>|[Singularity][singularity], by [Sylabs][sylabs]|`application/vnd.sylabs.sif.config.v1+json`|++## Defining a Unique Artifact Type++Defining the unique type involves uniqueness for computer processing, and uniqueness for humans.++For computer processing, artifacts are defined by setting `manifest.config.mediaType` to a globally unique value.++> **Note:** The `config.mediaType` of `application/vnd.oci.image.config.v1+json` is reserved for artifacts intended to be instanced by [docker][docker] and [containerd][containerd].  +*Each artifact MUST have its own unique type.*++ The following `mediaType` format is used to differentiate the type of artifact:++`application/vnd.`[org|company]`.`[objectType]`.`[optionalSubType]`.config.`[version]`+json`++- **`org|company`** - represents an open source foundation (`oci`, `cncf`) or a company (`microsoft`, `ibm`).+- **`objectType`** - a value representing the short name of the type. The combination of the org|company and object type should provide human identification and uniqueness for computers indexing.+- **`optionalSubType`** - provides additional extensibility of an `objectType`+- **`version`** - provides artifact authors the ability to revision their schemas and formatting, enabling tools to identify which format they will process.++## Defining Supported Layer Types++Artifacts are intended to have content. The content of an artifact is represented through a collection of blobs as provided by the [distribution spec][distribution-spec]. How the blobs are re-constituted, and whether the blobs are ordinal layers is a decision of the artifact author.++As an example, [OCI Images][image-layer] are represented through an ordinal collection of compressed files. Each blob represents a layer. Each layer overlays the previous layer.++Other artifacts may be represented as a single file, such as config artifact representing a deployment specification. Other artifacts may include a config blob, and a collection of binaries compressed as another blob. By separating the blob, the artifact author can benefit from layer de-duplication and concurrent downloading of each blob.++### Layer File Format++Layers are persisted as blobs within registries. The blobs can be a single file or a collection of files. The persistance format is up to the artifact author. They may choose to persist individual files with their native or custom format, such as `.config`, `.json`, `.bin`, `.sif`, and compress them with various formats such as `.gzip`.++Large files can benefit from compression when being transferred across the network. However, decompression takes time and compute cycles. For smaller, individual files, the decompression may take longer than downloading the file in its original format.++### Layer Versioning++Layers MUST be versioned to future proof new enhancements that may evolve. How the artifact tooling processes forward and backward compatibility is up to the artifact and tooling authors.++### Defining Layer Types++Artifact layers are expected to be unique, per artifact author. A layer that represents an OCI Image is not expected to be shared with a Helm Chart. To differentiate layers, artifact authors SHOULD create unique artifact layer types.

What does it mean "expected to be unique"? New types should only be defined if it is actually defining something new. If the layer is just a tar without any special processing, then just a standard tar type should be used. If the layer uses the OCI layer definitions, it should just use that. In reality the bottommost layer in an OCI image could be defined as "application/tar" (this wouldn't be supported by most clients), but there is no need to define something new unless it is actually unique.

SteveLasker

comment created time in a month

Pull request review commentcontainerd/project

Add definition of core and non-core subprojects

 maintainer has already discussed the candidacy with the candidate and a maintainer is willing to be a sponsor by opening the pull request. The candidate becomes a maintainer once the pull request is merged. -## Adding sub-projects--Similar to adding maintainers, new sub projects can be added to containerd-GitHub organization as long as they adhere to the CNCF-[charter](https://www.cncf.io/about/charter/) and mission. After a project-proposal has been announced on a public forum (GitHub issue or mailing list),-the existing maintainers are  given five business days to discuss the new+## Adding subprojects++containerd subprojects are divided into two flavors currently: **core** and+**non-core**. Most of the repositories within the containerd GitHub organization are+"core" to the delivery of the containerd runtime project's releases. For example,+the Golang implementations of the cgroups, runc, and console APIs, the ttrpc+implementation, and various plugins like the CRI implementation. These, among a+handful of other repositories, are all **core** and required pieces which culminate+in officially supported and maintained releases of containerd.++Non-core projects have strong affiliation with the containerd runtime, but+operate in some ways similar to the traditional `contrib/` directory in many+open source projects. They are most likely separately maintained, and potentially+have a different release cycle and support guarantee than the main containerd project+and its core subprojects. They may be written in other languages and therefore+require different skills, developer tools, and CI systems than the core projects.+For these reasons, non-core subprojects will be treated differently than core+projects within the governance of the containerd organization.++Both core and non-core subprojects must adhere to the CNCF+[charter](https://www.cncf.io/about/charter/) and mission.++### Adding core subprojects++New core subprojects can request to be added to the containerd GitHub+organization by submitting a project proposal via public forum (a+`containerd/project` GitHub issue is the easiest way to provide this proposal).+The existing maintainers are given five business days to discuss the new project, raise objections and cast their vote. Projects must be approved by at least 66% of the current maintainers by adding their vote.  If a project is approved, a maintainer will add the project to the containerd GitHub organization, and make an announcement on a public forum. +To make this transition to being a core subproject easier, please prepare your+repository with the DCO signoff checks, the fileheader check, and by adding the+appropriate Apache 2 license file to your project root as well as the boilerplate+header to all applicable source files.

I don't think we should have integration requirements. I think being part of core means it is supported as part of a release though, being in the main release binary or tarballs will always make that true but we could foresee other cases. I think from the release logic is makes sense though since anyone releasing code that is part of a containerd release must be a "core" maintainer. So promoting a project to core would just be for that purpose and bringing in those maintainers as project level maintainers.

estesp

comment created time in a month

push eventdmcgowan/docker

Sebastiaan van Stijn

commit sha 1be2cc2568ddd52a3b27d607a9c9f7c9347d3c50

Makefile: force using buildkit if USE_BUILDX is not set Before this change: ``` unset DOCKER_BUILDKIT make build docker build --build-arg=CROSS=false -t "docker-dev:require-buildkit" -f "Dockerfile" . Sending build context to Docker daemon 50.01MB Error response from daemon: Dockerfile parse error line 17: Unknown flag: mount make: *** [build] Error 1 ``` After this change: ``` unset DOCKER_BUILDKIT make build docker build --build-arg=CROSS=false -t "docker-dev:require-buildkit" -f "Dockerfile" . [+] Building 5.2s (71/71) FINISHED => [internal] load .dockerignore 0.1s ... ... => => exporting layers 0.9s => => writing image sha256:1ea4128a0e7f3bdee47de1675252609d9d6071e32da24a2aafee9fba96b2404b 0.0s => => naming to docker.io/library/docker-dev:require-buildkit 0.0s ... Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Tibor Vass

commit sha 448db5a783a0fa989796589877ab7bd7f1869b56

Merge pull request #40060 from thaJeztah/require_buildkit Makefile: force using buildkit if USE_BUILDX is not set

view details

Derek McGowan

commit sha 50a2dc26476da6b1f27cd57111d736141d398c20

Use containerd client to pull images Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 8cf54745fa034090a05f2178ba7386ab54ecb418

Use containerd to list images Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 821bd9c9addd8c4cbb74d8e45a893e126bf68866

Use containerd to create tags Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha f865456bfbf9da5b9c9c3b50478a19570c87e2d3

Add image cache Use cache for delete and listing images Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 3407e28734ca96e48f2598175fb053b36f758eea

Add image tagging by id and reference Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 749b8d92885fab28337618ffb3e9aae1951870bf

Add image deletion Fix conflict and update removal to use cache and not stores. Layer removal is still missing. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 62b218908f161aa504401539806440fc399bcf7f

Add support for layers Unpack and delete layers Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 03bce362e78cd6dd079d78c2a847b8ff1b8faff7

Add support for inspect Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha bc80d663375347b8f62a946930860d35ef30e333

Update image history to use containerd Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 6ea3769abb4c328ecacf72b22fb59601d5812cf9

Add support for docker run Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 3e17c8ccd753d7f9ecbe9b723a3553035dfc6a22

Update image service functions to use containerd Updates info functions to query the containerd image cache. Updates related call stack to pass down context. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Anda Xu

commit sha 8a130fe423d5c1366a82872d4b2f803cbada0f92

display docker pull progress for containerd integration Signed-off-by: Anda Xu <axu@knox.edu>

view details

Derek McGowan

commit sha 19edf6f14d6cd31992125fdd382ccf1eebbdf705

Add support for commit Fix dangling image management to not rely on reference parsing Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha bfffc6e0f51f3430ff4cf57b3c0f21d64b5f5f2d

Add parent label to committed images Fix layer references being taken on startup Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha e3d46c582cf337f38f5c4fca52eb7d95fdeb44d1

Update events handler to use containerd Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 4ad44160147a3affd3596918f1612c667a0107e9

Add push support through containerd Uses containerd client to push. Temporarily resolves current platform on push to match current behavior. Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 6657a3c264e1e417aa82541f7e3132a3e29f973f

Update images to use manifest as image ID Lookup images without using caching Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

Derek McGowan

commit sha 7b9bbc9efa3f3a005114262e83b87af39ee627e9

Add layer label to content Adds equivalent label as the snapshotter gc label Signed-off-by: Derek McGowan <derek@mcgstyle.net>

view details

push time in a month

Pull request review commentcontainerd/project

Add definition of core and non-core subprojects

 maintainer has already discussed the candidacy with the candidate and a maintainer is willing to be a sponsor by opening the pull request. The candidate becomes a maintainer once the pull request is merged. -## Adding sub-projects--Similar to adding maintainers, new sub projects can be added to containerd-GitHub organization as long as they adhere to the CNCF-[charter](https://www.cncf.io/about/charter/) and mission. After a project-proposal has been announced on a public forum (GitHub issue or mailing list),-the existing maintainers are  given five business days to discuss the new+## Adding subprojects++containerd subprojects are divided into two flavors currently: **core** and+**non-core**. Most of the repositories within the containerd GitHub organization are+"core" to the delivery of the containerd runtime project's releases. For example,+the Golang implementations of the cgroups, runc, and console APIs, the ttrpc+implementation, and various plugins like the CRI implementation. These, among a+handful of other repositories, are all **core** and required pieces which culminate+in officially supported and maintained releases of containerd.++Non-core projects have strong affiliation with the containerd runtime, but+operate in some ways similar to the traditional `contrib/` directory in many+open source projects. They are most likely separately maintained, and potentially+have a different release cycle and support guarantee than the main containerd project+and its core subprojects. They may be written in other languages and therefore+require different skills, developer tools, and CI systems than the core projects.+For these reasons, non-core subprojects will be treated differently than core+projects within the governance of the containerd organization.++Both core and non-core subprojects must adhere to the CNCF+[charter](https://www.cncf.io/about/charter/) and mission.++### Adding core subprojects++New core subprojects can request to be added to the containerd GitHub+organization by submitting a project proposal via public forum (a+`containerd/project` GitHub issue is the easiest way to provide this proposal).+The existing maintainers are given five business days to discuss the new project, raise objections and cast their vote. Projects must be approved by at least 66% of the current maintainers by adding their vote.  If a project is approved, a maintainer will add the project to the containerd GitHub organization, and make an announcement on a public forum. +To make this transition to being a core subproject easier, please prepare your+repository with the DCO signoff checks, the fileheader check, and by adding the+appropriate Apache 2 license file to your project root as well as the boilerplate+header to all applicable source files.++### Adding non-core subprojects++Non-core subprojects will also submit a project proposal via public forum, and+should state that the project is expected to be **non-core**.++The proposal should include a proposed list of maintainers who will manage+the non-core project and provide general information on support, releases,+stability, and any additional detail useful for the containerd maintainers to+understand the scope and nature of the project.++The existing maintainers are given five business days to discuss the new+project, raise objections and cast their vote. Projects must be approved by at+least 50% of the current maintainers by adding their vote.

I prefer the higher threshold until it becomes a problem. If we get maintainers filibustering then we can discuss that, but generally these will be rare (just a few quorum votes per month) and asking for a simple yes/no vote is not too much to ask of a majority of maintainers.

estesp

comment created time in a month

Pull request review commentcontainerd/project

Add definition of core and non-core subprojects

 maintainer has already discussed the candidacy with the candidate and a maintainer is willing to be a sponsor by opening the pull request. The candidate becomes a maintainer once the pull request is merged. -## Adding sub-projects--Similar to adding maintainers, new sub projects can be added to containerd-GitHub organization as long as they adhere to the CNCF-[charter](https://www.cncf.io/about/charter/) and mission. After a project-proposal has been announced on a public forum (GitHub issue or mailing list),-the existing maintainers are  given five business days to discuss the new+## Adding subprojects++containerd subprojects are divided into two flavors currently: **core** and+**non-core**. Most of the repositories within the containerd GitHub organization are+"core" to the delivery of the containerd runtime project's releases. For example,+the Golang implementations of the cgroups, runc, and console APIs, the ttrpc+implementation, and various plugins like the CRI implementation. These, among a+handful of other repositories, are all **core** and required pieces which culminate+in officially supported and maintained releases of containerd.++Non-core projects have strong affiliation with the containerd runtime, but+operate in some ways similar to the traditional `contrib/` directory in many+open source projects. They are most likely separately maintained, and potentially+have a different release cycle and support guarantee than the main containerd project+and its core subprojects. They may be written in other languages and therefore+require different skills, developer tools, and CI systems than the core projects.+For these reasons, non-core subprojects will be treated differently than core+projects within the governance of the containerd organization.++Both core and non-core subprojects must adhere to the CNCF+[charter](https://www.cncf.io/about/charter/) and mission.++### Adding core subprojects++New core subprojects can request to be added to the containerd GitHub+organization by submitting a project proposal via public forum (a+`containerd/project` GitHub issue is the easiest way to provide this proposal).+The existing maintainers are given five business days to discuss the new project, raise objections and cast their vote. Projects must be approved by at least 66% of the current maintainers by adding their vote.  If a project is approved, a maintainer will add the project to the containerd GitHub organization, and make an announcement on a public forum. +To make this transition to being a core subproject easier, please prepare your+repository with the DCO signoff checks, the fileheader check, and by adding the+appropriate Apache 2 license file to your project root as well as the boilerplate+header to all applicable source files.++### Adding non-core subprojects++Non-core subprojects will also submit a project proposal via public forum, and+should state that the project is expected to be **non-core**.++The proposal should include a proposed list of maintainers who will manage+the non-core project and provide general information on support, releases,+stability, and any additional detail useful for the containerd maintainers to+understand the scope and nature of the project.++The existing maintainers are given five business days to discuss the new+project, raise objections and cast their vote. Projects must be approved by at+least 50% of the current maintainers by adding their vote.++If a project is approved, a maintainer will add the project to the containerd+GitHub organization and provide write access for that repository to the proposed+maintainer list, as well as make an announcement on a public forum.++Unlike core projects, non-core project maintainers can become reviewers and

The wording in this paragraph is somewhat unclear. I think we should clarify here that maintainers of non-core projects are only maintainers of that subproject and not maintainers of any other containerd projects, as such, they are not required to participate in project level voting. On the other hand, containerd maintainers listed in this repository are responsible for maintaining all containerd projects, core and non-core. As such, they will have write access to all sub-projects and there is no need for containerd maintainers to be listed as maintainers of non-core projects.

estesp

comment created time in a month

Pull request review commentcontainerd/project

Add definition of core and non-core subprojects

 maintainer has already discussed the candidacy with the candidate and a maintainer is willing to be a sponsor by opening the pull request. The candidate becomes a maintainer once the pull request is merged. -## Adding sub-projects--Similar to adding maintainers, new sub projects can be added to containerd-GitHub organization as long as they adhere to the CNCF-[charter](https://www.cncf.io/about/charter/) and mission. After a project-proposal has been announced on a public forum (GitHub issue or mailing list),-the existing maintainers are  given five business days to discuss the new+## Adding subprojects++containerd subprojects are divided into two flavors currently: **core** and+**non-core**. Most of the repositories within the containerd GitHub organization are+"core" to the delivery of the containerd runtime project's releases. For example,+the Golang implementations of the cgroups, runc, and console APIs, the ttrpc+implementation, and various plugins like the CRI implementation. These, among a+handful of other repositories, are all **core** and required pieces which culminate+in officially supported and maintained releases of containerd.++Non-core projects have strong affiliation with the containerd runtime, but+operate in some ways similar to the traditional `contrib/` directory in many+open source projects. They are most likely separately maintained, and potentially+have a different release cycle and support guarantee than the main containerd project+and its core subprojects. They may be written in other languages and therefore+require different skills, developer tools, and CI systems than the core projects.+For these reasons, non-core subprojects will be treated differently than core+projects within the governance of the containerd organization.++Both core and non-core subprojects must adhere to the CNCF+[charter](https://www.cncf.io/about/charter/) and mission.++### Adding core subprojects++New core subprojects can request to be added to the containerd GitHub+organization by submitting a project proposal via public forum (a+`containerd/project` GitHub issue is the easiest way to provide this proposal).+The existing maintainers are given five business days to discuss the new project, raise objections and cast their vote. Projects must be approved by at least 66% of the current maintainers by adding their vote.  If a project is approved, a maintainer will add the project to the containerd GitHub organization, and make an announcement on a public forum. +To make this transition to being a core subproject easier, please prepare your+repository with the DCO signoff checks, the fileheader check, and by adding the+appropriate Apache 2 license file to your project root as well as the boilerplate+header to all applicable source files.++### Adding non-core subprojects++Non-core subprojects will also submit a project proposal via public forum, and+should state that the project is expected to be **non-core**.++The proposal should include a proposed list of maintainers who will manage+the non-core project and provide general information on support, releases,+stability, and any additional detail useful for the containerd maintainers to+understand the scope and nature of the project.++The existing maintainers are given five business days to discuss the new+project, raise objections and cast their vote. Projects must be approved by at+least 50% of the current maintainers by adding their vote.

I think we should keep the 66% here as well. The project requirements are different but should still get full consensus.

estesp

comment created time in a month

issue commentcontainerd/ttrpc

TTRPC Rust implementation

Sounds good.

The process is mentioned here https://github.com/containerd/project/blob/master/GOVERNANCE.md#adding-sub-projects

Open an issue in https://github.com/containerd/project proposing the move of the repository (original repo and new repo name). We will make sure it has everything set up to move (such as maintainer list, license, code of conduct) then vote on it. We also might also discuss some related governance changes to handle maintenance of this sub-project so it can have its own set of maintainers.

sameo

comment created time in a month

more