profile
viewpoint
Brian Goff cpuguy83 @Microsoft Seattle, WA

containerd/ttrpc 153

GRPC for low-memory environments

cpuguy83/chef-openldap-server 1

OpenLDAP Server Cookbook for Chef

cpuguy83/aad-pod-identity 0

Assign Azure Active Directory Identities to kubernetes applications.

cpuguy83/acs-engine 0

Azure Container Service Engine - provision and deploy container orchestrators on Azure: Kubernetes, DC/OS, and Docker Swarm.

cpuguy83/activeldap 0

ActiveLdap provides an object oriented interface to LDAP.

cpuguy83/aks-engine 0

AKS Engine: Units of Kubernetes on Azure!

cpuguy83/aws-cli 0

Universal Command Line Interface for Amazon Web Services

cpuguy83/azure-aci 0

Things releated Azure Container Instance

issue commentmoby/moby

Feature request: option to re-run container with updated image

Again, the feature already exists.

"No" does not only have a negative value. "No" is much simpler than "yes" because "yes" often leads to mistakes that we have to support long term. "No" also doesn't mean always no, it can mean "not right now". Yes, however, is forever.

jbemmel

comment created time in 7 hours

issue commentmoby/moby

Need to read ImagePush result for it to work

This seems to be working as expected. Since the API is not asynchronous, the early return (and as such I'd expect closing the response) is considered a cancellation.

I will say I am on my phone and haven't looked deeply at this, but based on the description I'm not sure this should be unexpected.

BrunoMCBraga

comment created time in 9 hours

issue commentmoby/moby

Move logdrivers to separate binaries

Also thinking we should extract the entire log handling infra out so someone could swap the default (equivalent of what we have today) for something custom since log handling is often contentious... it could also be much more efficiently.

cpuguy83

comment created time in 12 hours

push eventmoby/moby

Wei Fu

commit sha 9ed0504592d338890a37e18999f98d69d7103f2d

daemon: add grpc.WithBlock option WithBlock makes sure that the following containerd request is reliable. In one edge case with high load pressure, kernel kills dockerd, containerd and containerd-shims caused by OOM. When both dockerd and containerd restart, but containerd will take time to recover all the existing containers. Before containerd serving, dockerd will failed with gRPC error. That bad thing is that restore action will still ignore the any non-NotFound errors and returns running state for already stopped container. It is unexpected behavior. And we need to restart dockerd to make sure that anything is OK. It is painful. Add WithBlock can prevent the edge case. And n common case, the containerd will be serving in shortly. It is not harm to add WithBlock for containerd connection. Signed-off-by: Wei Fu <fuweid89@gmail.com> (cherry picked from commit 9f73396dabf087a8dd5fa74296c2cd4c188ff889) Signed-off-by: Wei Fu <fuweid89@gmail.com>

view details

Brian Goff

commit sha 679115602f8c33b4bcdee776393a4cc388c287a7

Merge pull request #40555 from fuweid/cp1903-40137 [19.03 backport] daemon: add grpc.WithBlock option

view details

push time in 14 hours

issue closedmoby/moby

[19.03] backport #40137 grpc.WithBlock

Tracking issue to backport #40137 to 19.03

closed time in 14 hours

cpuguy83

PR merged moby/moby

[19.03 backport] daemon: add grpc.WithBlock option area/runtime status/2-code-review

WithBlock makes sure that the following containerd request is reliable.

In one edge case with high load pressure, kernel kills dockerd, containerd and containerd-shims caused by OOM. When both dockerd and containerd restart, but containerd will take time to recover all the existing containers. Before containerd serving, dockerd will failed with gRPC error. That bad thing is that restore action will still ignore the any non-NotFound errors and returns running state for already stopped container. It is unexpected behavior. And we need to restart dockerd to make sure that anything is OK.

It is painful. Add WithBlock can prevent the edge case. And n common case, the containerd will be serving in shortly. It is not harm to add WithBlock for containerd connection.

Signed-off-by: Wei Fu fuweid89@gmail.com (cherry picked from commit 9f73396dabf087a8dd5fa74296c2cd4c188ff889) Signed-off-by: Wei Fu fuweid89@gmail.com

From #40137 Fix: https://github.com/moby/moby/issues/40554

+18 -0

1 comment

1 changed file

fuweid

pr closed time in 14 hours

issue commentmoby/moby

Feature request: option to re-run container with updated image

What you are looking for is the "service" object, and it is part of swarm.

"docker service create --name fooService ... foo:v1" "docker service update --image=foo:v2 fooService"

jbemmel

comment created time in a day

delete branch cpuguy83/docker

delete branch : 19.03_stats_use_cond_var

delete time in a day

PR merged moby/moby

Update Golang 1.13.8 impact/changelog status/2-code-review

full diff: https://github.com/golang/go/compare/go1.13.7...go1.13.8

go1.13.8 (released 2020/02/12) includes fixes to the runtime, the crypto/x509, and net/http packages. See the Go 1.13.8 milestone on the issue tracker for details.

https://github.com/golang/go/issues?q=milestone%3AGo1.13.8+label%3ACherryPickApproved

- Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: -->

- A picture of a cute animal (not mandatory but encouraged)

+4 -4

1 comment

4 changed files

thaJeztah

pr closed time in a day

push eventmoby/moby

Sebastiaan van Stijn

commit sha 3f7503f98a9e0752d3df7fc53fc920e539b4fffe

Update Golang 1.13.8 full diff: https://github.com/golang/go/compare/go1.13.7...go1.13.8 go1.13.8 (released 2020/02/12) includes fixes to the runtime, the crypto/x509, and net/http packages. See the Go 1.13.8 milestone on the issue tracker for details. https://github.com/golang/go/issues?q=milestone%3AGo1.13.8+label%3ACherryPickApproved Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha d706420b5d0f16822f4b2e9d717270728393c60c

Merge pull request #40532 from thaJeztah/bump_golang_1.13.8 Update Golang 1.13.8

view details

push time in a day

PR merged moby/moby

Update windows container utility to fix compilation on gcc-mingw-w64 8.3 area/project process/cherry-pick status/2-code-review
  • bump windows-container-utility aa1ba87e99b68e0113bd27ec26c60b88f9d4ccd9
    • full diff: https://github.com/docker/windows-container-utility/compare/e004a1415a433447369e315b9d7df357102be0d2...aa1ba87e99b68e0113bd27ec26c60b88f9d4ccd9
    • Use standard include paths instead of hard-coding
  • revert 25a1bf53d29e6424d4e9688952129a03c62fdef6 "Fix containerutility compilation on gcc-mingw-w64 8.3" (part of https://github.com/moby/moby/pull/39880)
+1 -8

2 comments

1 changed file

thaJeztah

pr closed time in a day

push eventmoby/moby

Sebastiaan van Stijn

commit sha 5125f8b304b860383f277e6d52bb72abfe820a30

bump windows-container-utility aa1ba87e99b68e0113bd27ec26c60b88f9d4ccd9 full diff: https://github.com/docker/windows-container-utility/compare/e004a1415a433447369e315b9d7df357102be0d2...aa1ba87e99b68e0113bd27ec26c60b88f9d4ccd9 changes: - Use standard include paths instead of hard-coding Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 79b130ac31127fbc1a3bfb35b4743a7b9d3d6283

Revert "Fix containerutility compilation on gcc-mingw-w64 8.3" This reverts commit 25a1bf53d29e6424d4e9688952129a03c62fdef6. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha 68ba9881249a24293572add595a0a9cf2bc3c1b8

Merge pull request #40526 from thaJeztah/bump_container_utility Update windows container utility to fix compilation on gcc-mingw-w64 8.3

view details

push time in a day

issue openedmoby/moby

[19.03] backport #40137 grpc.WitBlock

Tracking issue to backport #40137 to 19.03

created time in a day

pull request commentmoby/moby

daemon: add grpc.WithBlock option

Added cherry-pick since this seems like something we'd want on 19.03

@fuweid Did you want to open a backport?

fuweid

comment created time in a day

push eventmoby/moby

Wei Fu

commit sha 9f73396dabf087a8dd5fa74296c2cd4c188ff889

daemon: add grpc.WithBlock option WithBlock makes sure that the following containerd request is reliable. In one edge case with high load pressure, kernel kills dockerd, containerd and containerd-shims caused by OOM. When both dockerd and containerd restart, but containerd will take time to recover all the existing containers. Before containerd serving, dockerd will failed with gRPC error. That bad thing is that restore action will still ignore the any non-NotFound errors and returns running state for already stopped container. It is unexpected behavior. And we need to restart dockerd to make sure that anything is OK. It is painful. Add WithBlock can prevent the edge case. And n common case, the containerd will be serving in shortly. It is not harm to add WithBlock for containerd connection. Signed-off-by: Wei Fu <fuweid89@gmail.com>

view details

Brian Goff

commit sha 62bd5a33f7074f64f28217b07b9efcd4c714a71d

Merge pull request #40137 from fuweid/me-wait-for-remote-containerd-before-reload daemon: add grpc.WithBlock option

view details

push time in a day

PR merged moby/moby

daemon: add grpc.WithBlock option area/runtime status/2-code-review

WithBlock makes sure that the following containerd request is reliable.

In one edge case with high load pressure, kernel kills dockerd, containerd and containerd-shims caused by OOM. When both dockerd and containerd restart, but containerd will take time to recover all the existing containers. Before containerd serving, dockerd will failed with gRPC error. That bad thing is that restore action will still ignore the any non-NotFound errors and returns running state for already stopped container. It is unexpected behavior. And we need to restart dockerd to make sure that anything is OK.

It is painful. Add WithBlock can prevent the edge case. And n common case, the containerd will be serving in shortly. It is not harm to add WithBlock for containerd connection.

Signed-off-by: Wei Fu fuweid89@gmail.com

+18 -0

8 comments

1 changed file

fuweid

pr closed time in a day

push eventcpuguy83/docker-jruby

David Rodríguez

commit sha 052e8a683993e9752525864845abc5545d299971

Bump version to 9.2.10.0

view details

Brian Goff

commit sha f4246779f59c1a8d69baa0926187b39ba439f718

Merge pull request #59 from deivid-rodriguez/9_2_10_0 Bump version to 9.2.10.0

view details

push time in a day

PR merged cpuguy83/docker-jruby

Bump version to 9.2.10.0

Jruby 9.2.10.0 is out!

https://www.jruby.org/2020/02/18/jruby-9-2-10-0.html

+9 -9

0 comment

5 changed files

deivid-rodriguez

pr closed time in a day

pull request commentmoby/moby

Upstream logging changes from Enterprise Edition

Interesting failure on ppc64

[2020-02-20T05:10:10.105Z] --- FAIL: TestTemplatedConfig (2.24s)

[2020-02-20T05:10:10.105Z] panic: reflect: call of reflect.Value.IsNil on struct Value [recovered]

[2020-02-20T05:10:10.105Z] 	panic: reflect: call of reflect.Value.IsNil on struct Value

[2020-02-20T05:10:10.105Z] 

[2020-02-20T05:10:10.105Z] goroutine 99 [running]:

[2020-02-20T05:10:10.105Z] testing.tRunner.func1(0xc00081a000)

[2020-02-20T05:10:10.105Z] 	/usr/local/go/src/testing/testing.go:874 +0x380

[2020-02-20T05:10:10.105Z] panic(0x534840, 0xc0007df2c0)

[2020-02-20T05:10:10.105Z] 	/usr/local/go/src/runtime/panic.go:679 +0x1bc

[2020-02-20T05:10:10.105Z] reflect.Value.IsNil(...)

[2020-02-20T05:10:10.105Z] 	/usr/local/go/src/reflect/value.go:1073

[2020-02-20T05:10:10.105Z] github.com/docker/docker/vendor/gotest.tools/v3/assert.assert(0x6b7980, 0xc00081a000, 0xc0008874d8, 0x604268, 0x57b180, 0xc0007f48e0, 0xc000887638, 0x1, 0x1, 0x0)

[2020-02-20T05:10:10.105Z] 	/go/src/github.com/docker/docker/vendor/gotest.tools/v3/assert/assert.go:123 +0x6d0

[2020-02-20T05:10:10.105Z] github.com/docker/docker/vendor/gotest.tools/v3/assert.NilError(0x6b7980, 0xc00081a000, 0x6abf20, 0xc0007f48e0, 
SamWhited

comment created time in a day

Pull request review commentmoby/moby

Remove 'deny mount' in the apparmor template

 profile {{.Name}} flags=(attach_disconnected,mediate_deleted) {   deny @{PROC}/sysrq-trigger rwklx,   deny @{PROC}/kcore rwklx, -  deny mount,

I'm not particularly familiar with the code in this package. I do seem to recall we used to generate these profiles on the fly at least to support privileged, I don't know what we are doing now.

danifv

comment created time in a day

issue closedmoby/moby

docker build: "error creating overlay mount to /var/lib/docker/overlay2/xxx/merged: device or resource busy"

Description

When running docker build (parallel - around 3-6 other builds) on our Gitlab CI instance I encounter a lot of failed builds because of this error:

---> a4d65b2fc73e
Step 9/19 : RUN ["crystal", "spec", "--error-on-warnings", "--no-debug"]
error creating overlay mount to /var/lib/docker/overlay2/209a7dfb23bbf26404576987d2b3e1e847f3c6afcc328b201afe75f6d8de2639/merged: device or resource busy

This happens randomly. Sometimes there are 12 concurrent build jobs running and nothing happens, but then there is single build job (=multiple docker builds running at once) and this happens. Sadly I don't know when this started occuring exactly, so I don't know since what docker or linux version is the cause.

NOTE: there is no push going on at the same time, just other docker builds. Verified multiple times that this is caused by docker build alone, so concurrent push isn't the cause.

Doing docker system prune -af was suggested in docker/for-linux#711, but that did not help.

I found only one similar issue to this one (moby/moby#39751), but that seems to be about docker push. Don't know if this is related, regression or duplicity. Either way, that issue is closed. Other issues are for incompatibility with newer versions of linux kernel. Someone mentioned running with overlay driver instead of overlay2, or switching off concurrent builds in Gitlab, but I don't consider any of that to be a fix.

Steps to reproduce the issue:

  1. run docker build multiple times at the same time
  2. watch the world burn, receive failed build messages, watch your happiness go down
  3. ... don't profit.

Describe the results you received: Unexpected random error messages regarding overlay2.

Describe the results you expected: No failed docker builds :-)

Additional information you deem important (e.g. issue happens only occasionally): Happens randomly, multiple times a day. Gitlab runner is configured to use docker via mounted /var/run/docker.sock.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:25:58 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:24:29 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 25
  Running: 3
  Paused: 0
  Stopped: 22
 Images: 748
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-6-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 23.54GiB
 Name: dev
 ID: ESO2:YKCY:PW7K:XKWK:VNTK:S6UB:Y6TO:CLAN:JT4K:PR22:IVCU:UWQ5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

VMware VM running on iSCSI SSD array. VM has 8 cores and 24GB of memory. Running Debian buster. Docker CE via official docker repository.

closed time in a day

NoICE

issue commentmoby/moby

docker build: "error creating overlay mount to /var/lib/docker/overlay2/xxx/merged: device or resource busy"

Seems like since this is a kernel issue and is fixed in various newer kernel patches, I'm going to go ahead and close this, but feel free to discuss.

Thank you! 😇

NoICE

comment created time in a day

push eventmoby/moby

Sebastiaan van Stijn

commit sha 12c7541f1f2d616967f9eecce182789de7e2a238

vendor: update opencontainers/selinux v1.3.1 full diff: https://github.com/opencontainers/selinux/compare/5215b1806f52b1fcc2070a8826c542c9d33cd3cf...v1.3.1 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha 498e7185c69d295c9ebf65e199326361f2717769

Merge pull request #40546 from thaJeztah/update_selinux_v1.3.1 vendor: update opencontainers/selinux v1.3.1

view details

push time in a day

PR merged moby/moby

vendor: update opencontainers/selinux v1.3.1 area/security/selinux process/cherry-picked status/2-code-review

full diff: https://github.com/opencontainers/selinux/compare/5215b1806f52b1fcc2070a8826c542c9d33cd3cf...v1.3.1

+18 -4

0 comment

3 changed files

thaJeztah

pr closed time in a day

issue commentdocker/for-linux

Error "copying between containers is not supported" when trying to copy to local path containing ":"

Seems like the CLI would need to support escaping the :. Only thing I could say for now is "don't do that".

This seems purely a problem with the CLI's parsing, the API should not have such an issue.

julien-lecomte

comment created time in a day

issue commentmoby/moby

failed to get event and rpc error "connect: connection refused"

Sounds likely to be a socket or something left behind preventing containerd from starting.

Can you post logs from containerd?

journalctl -fu containerd
xieyanker

comment created time in 2 days

issue closedmoby/moby

Docker not removing old containers

Hi,

i am facing strange problem whenever i am deploying new code i am seeing multiple problems due to multiple container because its not shutting down old containers.

utcsnwvdixyc srv-captain--upsteam-popup replicated 3/1 img-captain-upsteam-popup:16

closed time in 2 days

hamza-younas94

issue commentmoby/moby

Docker not removing old containers

In order to troubleshoot we'll need more information.

Can you describe what you are doing in detail?

Additionally this repo includes an issue template, could you fill that info in? I'm going to close since none of this information is available, but if you can post that just ping and I will re-open.

Thank you! 😇

hamza-younas94

comment created time in 2 days

issue commentmoby/moby

Move logdrivers to separate binaries

I haven't decided personally. If it's a separate binary it will still come packaged with docker anyway. So they will always be available.

cpuguy83

comment created time in 2 days

delete branch cpuguy83/docker

delete branch : stats_use_cond_var

delete time in 2 days

delete branch cpuguy83/aks-engine

delete branch : azure_containerd

delete time in 2 days

issue commentmoby/moby

Allow log line max size to be configurable

Possibly related to that: https://github.com/moby/moby/issues/40517

I would like to move logdrivers out of the daemon. Spending some time thinking about it and speaking with others, it will likely involve more than just moving the drivers themselves and likely copy the all the log handling into a separate bin (not just the driver/drivers). If we make that bin configurable someone should be able to inject their own behavior.

Ideally the communication there is as efficient as possible.

someword

comment created time in 2 days

issue closedmoby/moby

userns, buildkit: docker build creates directories with wrong permissions

Description

When using --userns-remap=default and DOCKER_BUILDKIT=1 docker build COPY creates non existing directory hierarchy with wrong owner.

This doesn't happen without userns remapping or without BuildKit.

Steps to reproduce the issue:

Dockerfile with COPY targeting non-existent directory. Ex:

FROM alpine
COPY Dockerfile /root/foo/
RUN  touch /root/bar
RUN  stat -c '%n %u/%g' /root/foo /root/foo/Dockerfile /root/bar
RUN  touch /root/foo/bar

Describe the results you received:

The directory created by COPY, /root/foo, belongs to nobody as confirmed by stat. Attempt to create a file in /root/foo fails with Permission denied error.

$ DOCKER_BUILDKIT=1 docker build . --progress=plain --no-cache
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 195B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.0s

#4 [1/5] FROM docker.io/library/alpine
#4 CACHED

#5 [internal] load build context
#5 transferring context: 189B done
#5 DONE 0.0s

#6 [2/5] COPY Dockerfile /root/foo/
#6 DONE 0.1s

#7 [3/5] RUN  touch /root/bar
#7 DONE 0.5s

#8 [4/5] RUN  stat -c '%n %u/%g' /root/foo /root/foo/Dockerfile /root/bar
#8 0.633 /root/foo 65534/65534
#8 0.633 /root/foo/Dockerfile 0/0
#8 0.633 /root/bar 0/0
#8 DONE 0.7s

#9 [5/5] RUN  touch /root/foo/bar
#9 0.638 touch: /root/foo/bar: Permission denied
#9 ERROR: executor failed running [/bin/sh -c touch /root/foo/bar]: runc did not terminate sucessfully
------
 > [5/5] RUN  touch /root/foo/bar:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c touch /root/foo/bar]: runc did not terminate sucessfully

Describe the results you expected:

/root/foo should be owned by the mapped root user and it should be possible to create files in /root/foo.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:28:22 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 26
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
  userns
 Kernel Version: 4.18.0-18-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.852GiB
 Name: xu-1
 ID: 7LCF:ROWQ:RALP:RSBF:2DCG:36K2:VBOI:OQVU:5ADJ:D7XO:KSLL:SYVH
 Docker Root Dir: /var/lib/docker/231072.231072
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

userns_remap, BuildKit

closed time in 2 days

mejedi

issue commentmoby/moby

userns, buildkit: docker build creates directories with wrong permissions

It seems like this should have closed after merging #40440

mejedi

comment created time in 2 days

pull request commentmoby/moby

Use condition variable to wake stats collector.

Issue this is improving is that the kubelet hits the stats endpoint to collect stats and it takes almost 2s per container. This changes it to just over 1s per container.

Plus it's just not great to wake the daemon every second for no reason.

cpuguy83

comment created time in 2 days

PR opened moby/moby

Use condition variable to wake stats collector.

Backport of #40481


Before the collection goroutine wakes up every 1 second (as configured). This sleep interval is in case there are no stats to collect we don't end up in a tight loop.

Instead use a condition variable to signal that a collection is needed. This prevents us from waking the goroutine needlessly when there is no one looking for stats.

For now I've kept the sleep just moved it to the end of the loop, which gives some space between collections.

Signed-off-by: Brian Goff cpuguy83@gmail.com (cherry picked from commit e75e6b0e31428c00047bc814746aff4b4c7c90ad) Signed-off-by: Brian Goff cpuguy83@gmail.com

<!-- Please make sure you've read and understood our contributing guidelines; https://github.com/moby/moby/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with git commit -s **

For additional information on our contributing process, read our contributing guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx"

Please provide the following information: -->

- What I did

- How I did it

- How to verify it

- Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: -->

- A picture of a cute animal (not mandatory but encouraged)

+15 -10

0 comment

1 changed file

pr created time in 2 days

create barnchcpuguy83/docker

branch : 19.03_stats_use_cond_var

created branch time in 2 days

push eventmoby/moby

Sebastiaan van Stijn

commit sha c6afabf3b335163091ec978ca878072f7bb94f4b

update containerd runtime v1.2.13 The thirteenth patch release for `containerd` 1.2 fixes a regression introduced in v1.2.12 that caused container/shim to hang on single core machines, fixes an issue with blkio, and updates the Golang runtime to 1.12.17. * Fix container pid race condition * Update containerd/cgroups dependency to address blkio issue * Set octet-stream content-type on PUT request * Pin to libseccomp 2.3.3 to preserve compatibility with hosts that do not have libseccomp 2.4 or higher installed * Update Golang runtime to 1.12.17, which includes a fix to the runtime full diff: https://github.com/containerd/containerd/compare/v1.2.12...v1.2.13 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha d1cf6d13038ea64c86f94342d4d8cf73654eab2c

Merge pull request #40540 from thaJeztah/19.03_update_containerd_1.2.13 [19.03] update containerd runtime v1.2.13

view details

push time in 2 days

PR merged moby/moby

[19.03] update containerd runtime v1.2.13 area/runtime impact/changelog status/2-code-review
  • addresses https://github.com/moby/moby/issues/40514 docker run/build using echo commands hangs forever, centos 7
  • addresses https://github.com/docker-library/docker/issues/216 Docker-in-Docker 19.03.6 hangs during execution
  • addresses https://gitlab.com/gitlab-org/gitlab-runner/issues/6697 Job execution hangs when latest docker:dind (version 19.03.6 and "floating" ones) is used
  • addresses https://gitlab.com/gitlab-com/support-forum/issues/5194 CI runs that involve the building of a Docker container are hanging causing a CI timeout

The thirteenth patch release for containerd 1.2 fixes a regression introduced in v1.2.12 that caused container/shim to hang on single core machines, fixes an issue with blkio, and updates the Golang runtime to 1.12.17.

  • Fix container pid race condition
  • Update containerd/cgroups dependency to address blkio issue
  • Set octet-stream content-type on PUT request
  • Pin to libseccomp 2.3.3 to preserve compatibility with hosts that do not have libseccomp 2.4 or higher installed
  • Update Golang runtime to 1.12.17, which includes a fix to the runtime

full diff: https://github.com/containerd/containerd/compare/v1.2.12...v1.2.13

+1 -1

1 comment

1 changed file

thaJeztah

pr closed time in 2 days

push eventmoby/moby

Sebastiaan van Stijn

commit sha 55af2904620f38db560009e52f441a4af3c76e91

Update Golang 1.12.17 full diff: https://github.com/golang/go/compare/go1.12.16...go1.12.17 go1.12.17 (released 2020/02/12) includes a fix to the runtime. See the Go 1.12.17 milestone on the issue tracker for details: https://github.com/golang/go/issues?q=milestone%3AGo1.12.17+label%3ACherryPickApproved Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha e145add0ef9b7fd990f41fc24d891956f1747628

Merge pull request #40533 from thaJeztah/19.03_update_golang_1.12.17 [19.03] Update Golang 1.12.17

view details

push time in 2 days

PR merged moby/moby

[19.03] Update Golang 1.12.17 impact/changelog status/2-code-review

full diff: https://github.com/golang/go/compare/go1.12.16...go1.12.17

go1.12.17 (released 2020/02/12) includes a fix to the runtime. See the Go 1.12.17 milestone on the issue tracker for details:

https://github.com/golang/go/issues?q=milestone%3AGo1.12.17+label%3ACherryPickApproved

- Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: -->

+4 -4

2 comments

4 changed files

thaJeztah

pr closed time in 2 days

pull request commentmoby/moby

test: amend some misuse of channel in test functions

It's a flakey test that seems to have gotten more flakey recently.

lzhfromustc

comment created time in 3 days

issue commentvirtual-kubelet/virtual-kubelet

Virtual-Kubelet Metrics

I don't think anyone is looking into this yet.

Smithx10

comment created time in 3 days

issue commentvirtual-kubelet/virtual-kubelet

Virtual-Kubelet Metrics

It is still relevant. Ideally it would find a nice home like we have for the tracing package.

Smithx10

comment created time in 3 days

issue commentmoby/moby

Allow log line max size to be configurable

Yes a custom driver is susceptible to those limits.

Yes I think we should consider a model that allows for pluging into the stream itself rather than breaking into discrete messages in the daemon. We'd still need to support the existing model, but a new model where we just don't care about the stream except to wire it up to some collection service would be great.

someword

comment created time in 4 days

issue commentmoby/moby

Allow log line max size to be configurable

SizedLogger is an interface specifically to optimize how cloudwatch does buffering.

Partial log messages now include extra metadata that allow you to stitch the message back together more easily.

The issue here is that we cannot have an infinite buffer to store logs lines so it has to be split at some point. Enabling this to be configurable is effectively the same as an infinite buffer.

someword

comment created time in 4 days

issue commentdocker/for-linux

docker pull behind proxy does unnecessary dns lookup

Why are you using a proxy instead of a mirror configuration?

Also note that your strace on "docker pull" is tracing the client, but the actual pull (including dns resolution) would happen on the daemon process.

tesharp

comment created time in 6 days

delete branch cpuguy83/buildx

delete branch : k8s_priority

delete time in 6 days

Pull request review commentmoby/moby

test: amend some misuse of channel in test functions

 func TestMaxDownloadAttempts(t *testing.T) { 				})  			progressChan := make(chan progress.Progress)-			progressDone := make(chan struct{})

I think the intention of this was to wait for the goroutine to finish in the main test, so the wait is just missed. Really I guess we could just pull from progressChan on a timeout to ensure it's closed (at the end of the test).

lzhfromustc

comment created time in 7 days

Pull request review commentmoby/moby

test: amend some misuse of channel in test functions

 func (s *DockerDaemonSuite) TestDaemonStartWithoutColors(c *testing.T) {  	go func() { 		io.Copy(b, p)-		done <- true

These changes just seem to be reversing what's happening here. Is there something else happening that I'm missing?

The intent of this is to signal to the main test that the copy is done hence the send in the goroutine. The change seems to be flipping that to have the main test effectively ask "are you done" which I find to be a little more confusing.

lzhfromustc

comment created time in 7 days

pull request commentmoby/moby

Fix go-swagger to work with Go 1.14

Master tests do not like this.

[2020-02-14T19:58:01.544Z] + docker run --rm -t --privileged -v /home/ubuntu/workspace/moby_master/bundles:/go/src/github.com/docker/docker/bundles -v /home/ubuntu/workspace/moby_master/.git:/go/src/github.com/docker/docker/.git --name docker-pr367 -e DOCKER_EXPERIMENTAL -e DOCKER_GITCOMMIT=b2b80564dbcf6b0643a4c29a405c17bb91840dae -e DOCKER_GRAPHDRIVER -e TEST_FORCE_VALIDATE -e VALIDATE_REPO=https://github.com/moby/moby.git -e VALIDATE_BRANCH= docker:b2b80564dbcf6b0643a4c29a405c17bb91840dae hack/validate/default

[2020-02-14T19:58:02.108Z] 0 adds, 0 deletions; nothing to validate! :)

[2020-02-14T19:58:07.376Z] Congratulations!  Seccomp profile generation is done correctly.

[2020-02-14T19:58:07.376Z] Congratulations!  Packages in "./pkg/..." are safely isolated from internal code.

[2020-02-14T19:58:15.485Z] Congratulations!  The swagger spec at "api/swagger.yaml" is valid against swagger specification 2.0

[2020-02-14T19:58:17.381Z] The result of hack/generate-swagger-api.sh differs

[2020-02-14T19:58:17.381Z] 

[2020-02-14T19:58:17.381Z] diff --git a/api/types/container/container_wait.go b/api/types/container/container_wait.go

[2020-02-14T19:58:17.381Z] index 49e05ae669..c2e69b5b25 100644

[2020-02-14T19:58:17.381Z] --- a/api/types/container/container_wait.go

[2020-02-14T19:58:17.381Z] +++ b/api/types/container/container_wait.go

[2020-02-14T19:58:17.381Z] @@ -6,14 +6,6 @@ package container // import "github.com/docker/docker/api/types/container"

[2020-02-14T19:58:17.381Z]  // See hack/generate-swagger-api.sh

[2020-02-14T19:58:17.381Z]  // ----------------------------------------------------------------------------

[2020-02-14T19:58:17.381Z]  

[2020-02-14T19:58:17.381Z] -// ContainerWaitOKBodyError container waiting error, if any

[2020-02-14T19:58:17.381Z] -// swagger:model ContainerWaitOKBodyError

[2020-02-14T19:58:17.381Z] -type ContainerWaitOKBodyError struct {

[2020-02-14T19:58:17.381Z] -

[2020-02-14T19:58:17.381Z] -	// Details of an error

[2020-02-14T19:58:17.381Z] -	Message string `json:"Message,omitempty"`

[2020-02-14T19:58:17.381Z] -}

[2020-02-14T19:58:17.381Z] -

[2020-02-14T19:58:17.381Z]  // ContainerWaitOKBody OK response to ContainerWait operation

[2020-02-14T19:58:17.381Z]  // swagger:model ContainerWaitOKBody

[2020-02-14T19:58:17.381Z]  type ContainerWaitOKBody struct {

[2020-02-14T19:58:17.381Z] @@ -26,3 +18,11 @@ type ContainerWaitOKBody struct {

[2020-02-14T19:58:17.381Z]  	// Required: true

[2020-02-14T19:58:17.381Z]  	StatusCode int64 `json:"StatusCode"`

[2020-02-14T19:58:17.381Z]  }

[2020-02-14T19:58:17.381Z] +

[2020-02-14T19:58:17.381Z] +// ContainerWaitOKBodyError container waiting error, if any

[2020-02-14T19:58:17.381Z] +// swagger:model ContainerWaitOKBodyError

[2020-02-14T19:58:17.381Z] +type ContainerWaitOKBodyError struct {

[2020-02-14T19:58:17.381Z] +

[2020-02-14T19:58:17.381Z] +	// Details of an error

[2020-02-14T19:58:17.381Z] +	Message string `json:"Message,omitempty"`

[2020-02-14T19:58:17.381Z] +}

[2020-02-14T19:58:17.381Z] 

[2020-02-14T19:58:17.381Z] Please update api/swagger.yaml with any API changes, then 

[2020-02-14T19:58:17.381Z] run hack/generate-swagger-api.sh.

script returned exit code 1
thaJeztah

comment created time in 8 days

issue commentcontainerd/containerd

Pod is stuck in terminating due to containerd-shim unmount error.

We fixed this in docker with a few changes, but all these were specific to how docker was configuring itself... nevertheless there's a bunch of detail into the reasoning behind the changes in those PR's.

https://github.com/moby/moby/pull/36055 https://github.com/moby/moby/pull/36096 https://github.com/moby/moby/pull/36047

payall4u

comment created time in 8 days

issue commentcontainerd/containerd

Pod is stuck in terminating due to containerd-shim unmount error.

Cross posting from https://github.com/containerd/containerd/pull/4021#issuecomment-585876327

What typically happens in cases like this is you there is a mount marked as private that gets copied into a new mount namespace. A new mount namespace is created for every container, for systemd services that have MountPropagation or PrivateTmp defined, and these types of things. When those namespaces are created they get a copy of the root namespace, anything that has a private mount cannot be unmounted until all the namespaces are shut down. Mounts get marked private depending on the propagation defined on their root mount or if explicitly set.... so for example if you have /var/foo mounted and /var is mounted with mount private propagation, /var/foo will inherit the private propagation.

In this case MNT_DETACH only detaches the mount and hides very real problems. Even if you remove the mountpoint the data will not be freed until (possibly?) a reboot or all other namespaces with copies of that mount in them are shut down.

payall4u

comment created time in 8 days

issue commentdocker/for-linux

Docker sees a previous image as latest after successfully tagging a new one

@tonistiigi Any idea what's happening here?

SHA when doing docker images --no-trunc is different than what is reported by build, and likewise reported after pushing and then pulling.

bra-fsn

comment created time in 8 days

issue openedcontainerd/cri

Option to pass http headers to registry

We have a request to be able to pass extra http headers for requests to a registry. The docker CLI currently has this option in ~/.docker/config.json called httpHeaders, the client passes those to the dockerd, and dockerd passes those to the registries.

I suspect this could be something that can be put into the containerd config.

An example of what would be passed through: https://github.com/Azure/acr/blob/master/docs/http-headers.md#header-values

created time in 8 days

pull request commentmoby/moby

Dockerfile: make static binary target simpler to consume

Hmm, possibly it is expected since dockerd is the only thing that's dynamic. Probably the dynbinary target is not very useful except for testing that hack/make.sh dynbinary works.

tiborvass

comment created time in 8 days

pull request commentmoby/moby

Dockerfile: make static binary target simpler to consume

Same for dynbinary.

tiborvass

comment created time in 9 days

pull request commentmoby/moby

Dockerfile: make static binary target simpler to consume

SGTM

tiborvass

comment created time in 9 days

push eventcpuguy83/docker

Sargun Dhillon

commit sha 569bcd316cec23c0d45a387f950dea2783cddd02

copy: Add benchmark, and size checking to tests This adds a basic benchmark function, and writes some data during the tests for copy. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Refactor, and separate out metadata copy This is a step to begin copying the file data concurrently with the metadata. It separates out the metadata functions which are used for standard file copying, symlink copies, and directory copies. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Copy file data in the background, while copying metadata This is based on the following benchmarks on EXT4 filesystem with the built-in benchmark test: No background copy: 1081927574 ns/op 1082520482 ns/op Background copy, chan size 1: 1569585224 ns/op Background copy, chan size 10: 924814623 ns/op Background copy, chan size 100: 643039679 ns/op Background copy, chan size 500: 698984799 ns/op It plateaued at ~100, and then started getting slower as I went further. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Do dumb multiplexing of copiers, so we have parallel file copy This starts 8 workers (tuned) based on EXT4, and XFS results, which inodes are randomly sorted across and data is copied. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Process directory metadata concurrently This also moves copying of directory metadata into the background copying workers. This shows about another 15% in terms of speed on EXT4. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Add small optimization to remove unncessary chmod call All syscalls we make to make files set the permissions bits. If are setting any permissions bits that conflict with the umask, then we need to call chmod, otherwise our initial file creation call will work fine. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Copy directory mtimes asyncronously This has quite a few changes in it as it brings in the filepath walkdir code into copy, so we can make further speed enhancements here. It now copies the mtime of the directory using the same mechanism as the other file data / metadata copies, and it serializes writes to the diven directory in order to make sure that's the last "update" that arrives. Lastly, it fixes problems with failures, and deadlocks around blocked channels if the workers exit, but our walk continues. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Remove the usage of sync.WaitGroup on every inode We know based on stat from the source inode whether or not it is hardlinked based on the nlinks field. Iff this is greater than 1 then we send over a shared inode with a mutex. This also reduces the total amount of memory / book keeping that needs to be done by the worker threads, and the logic to handle these hard links is in the main walkr. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Allow API consumers to specify concurrency This extends the API and adds the function DirCopyWithConcurrency, which allows the user to specify the copy concurrency for odd shaped hardware. Signed-off-by: Sargun Dhillon <sargun@sargun.me> Signed-off-by: Brian Goff <cpuguy83@gmail.com>

view details

Sargun Dhillon

commit sha bf0bd1d74c50186c6108e49d856ee24a0b425dda

layer, vfs: Remove switching copy method during tests This removes a legacy change where the layer tests temporarily changed the dirCopy mechanism for the VFS driver. Signed-off-by: Sargun Dhillon <sargun@sargun.me> vfs: Allow users to specify copy concurrency This allows users to specify the copy concurrency in the VFS graphdriver using copy.concurrency. Signed-off-by: Sargun Dhillon <sargun@sargun.me> overlay: Allow specifying copy concurrency This adds the driver opt to the overlay(1) driver to allow specifying copy.concurrency. Signed-off-by: Sargun Dhillon <sargun@sargun.me> Signed-off-by: Brian Goff <cpuguy83@gmail.com>

view details

push time in 9 days

push eventcpuguy83/docker

Sargun Dhillon

commit sha 3d6d7bcff92635ad108e975fa8cfe0bf2a671ffe

copy: Add benchmark, and size checking to tests This adds a basic benchmark function, and writes some data during the tests for copy. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Refactor, and separate out metadata copy This is a step to begin copying the file data concurrently with the metadata. It separates out the metadata functions which are used for standard file copying, symlink copies, and directory copies. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Copy file data in the background, while copying metadata This is based on the following benchmarks on EXT4 filesystem with the built-in benchmark test: No background copy: 1081927574 ns/op 1082520482 ns/op Background copy, chan size 1: 1569585224 ns/op Background copy, chan size 10: 924814623 ns/op Background copy, chan size 100: 643039679 ns/op Background copy, chan size 500: 698984799 ns/op It plateaued at ~100, and then started getting slower as I went further. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Do dumb multiplexing of copiers, so we have parallel file copy This starts 8 workers (tuned) based on EXT4, and XFS results, which inodes are randomly sorted across and data is copied. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Process directory metadata concurrently This also moves copying of directory metadata into the background copying workers. This shows about another 15% in terms of speed on EXT4. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Add small optimization to remove unncessary chmod call All syscalls we make to make files set the permissions bits. If are setting any permissions bits that conflict with the umask, then we need to call chmod, otherwise our initial file creation call will work fine. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Copy directory mtimes asyncronously This has quite a few changes in it as it brings in the filepath walkdir code into copy, so we can make further speed enhancements here. It now copies the mtime of the directory using the same mechanism as the other file data / metadata copies, and it serializes writes to the diven directory in order to make sure that's the last "update" that arrives. Lastly, it fixes problems with failures, and deadlocks around blocked channels if the workers exit, but our walk continues. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Remove the usage of sync.WaitGroup on every inode We know based on stat from the source inode whether or not it is hardlinked based on the nlinks field. Iff this is greater than 1 then we send over a shared inode with a mutex. This also reduces the total amount of memory / book keeping that needs to be done by the worker threads, and the logic to handle these hard links is in the main walkr. Signed-off-by: Sargun Dhillon <sargun@sargun.me> copy: Allow API consumers to specify concurrency This extends the API and adds the function DirCopyWithConcurrency, which allows the user to specify the copy concurrency for odd shaped hardware. Signed-off-by: Sargun Dhillon <sargun@sargun.me> Signed-off-by: Brian Goff <cpuguy83@gmail.com>

view details

Sargun Dhillon

commit sha 3759682d9ef1c1d29c2bafb93318ad155715b637

layer, vfs: Remove switching copy method during tests This removes a legacy change where the layer tests temporarily changed the dirCopy mechanism for the VFS driver. Signed-off-by: Sargun Dhillon <sargun@sargun.me> vfs: Allow users to specify copy concurrency This allows users to specify the copy concurrency in the VFS graphdriver using copy.concurrency. Signed-off-by: Sargun Dhillon <sargun@sargun.me> overlay: Allow specifying copy concurrency This adds the driver opt to the overlay(1) driver to allow specifying copy.concurrency. Signed-off-by: Sargun Dhillon <sargun@sargun.me> Signed-off-by: Brian Goff <cpuguy83@gmail.com>

view details

push time in 9 days

PR opened moby/moby

Reviewers
Concurrent graph copy

This is a carry of #38034 Closes #38034

I squashed commits for an easier rebase.


  • What I did I made copies concurrent for VFS, and Overlay1.

  • How I did it I started with adding a benchmark to the tests to give me a baseline. On my test system that gave me about ~1081927574 ns/op. I slowly improved the performance throughout the commits, and increased performance to about 552743671 ns/op in the final tests. I parallizes

  • How to verify it There are attached benchmarks, and tests

  • Description for the changelog Make VFS Faster.

+516 -145

0 comment

7 changed files

pr created time in 9 days

delete tag moby/moby

delete tag : v19.03.6

delete time in 9 days

create barnchcpuguy83/docker

branch : concurrent_graph_copy

created branch time in 9 days

issue commentAzure/AKS

Support of OCI container images (built with buildah and podman)

MS Moby 3.0.10 has this fix.

mathieu-benoit

comment created time in 9 days

issue commentmoby/moby

--platform doesn't work if image with tag is already pulled

@thaJeztah This should be fixed before moving buildkit to default.

cpuguy83

comment created time in 9 days

issue openedmoby/moby

--platform doesn't work if image with tag is already pulled

When passing --platform for run or even build (with buildkit), if the image with the given tag is already in the daemon, platform is ignored.

This ends up in some really surprising results when trying to build multiple architectures on a daemon.

created time in 9 days

issue commentmoby/moby

Move logdrivers to separate binaries

I think so, or at least a recommended package. That could be the deprecation process, assuming there is a desire to deprecate... basically phase out from the main package.

cpuguy83

comment created time in 9 days

issue commentmoby/moby

Move logdrivers to separate binaries

I don't think so because it's only a restructure, not a removal.

cpuguy83

comment created time in 9 days

issue commentmoby/moby

Move logdrivers to separate binaries

We already do have external ones via plugins which incur extra overhead in encode/decode/transport. Depending on how we build the exec plugins, it could even pass the raw fd to consume.

cpuguy83

comment created time in 9 days

push eventmoby/moby

Mike Bush

commit sha f282dde8773399e5127a2fcdfe29b3661d0222c1

Fixes #33434 - API docs to specify using base64url Specify base64url rather than base64. Also correct other links to the base64url section of RFC4648 Signed-off-by: Mike Bush <mpbush@gmail.com>

view details

Brian Goff

commit sha 31a86c4ab209528b77d6048f8c99c363ffd4c884

Merge pull request #34958 from platy/#33434-api-doc-base64url Fixes #33434 - API docs to specify using base64url

view details

push time in 9 days

PR merged moby/moby

Fixes #33434 - API docs to specify using base64url area/api area/docs process/cherry-pick status/3-docs-review

Specify base64url rather than base64. Also correct other links to the base64url section of RFC4648

Fixes #33434

+9 -9

1 comment

1 changed file

platy

pr closed time in 9 days

issue closedmoby/moby

Documentation does not indicate that X-Registry-Auth base64 encoding must be URL-safe

Description

The documentation does not indicate that the encoding for X-Registry-Auth should be URL-safe, but the underlying code expects such encoding.

This is particular problematic on image pulls, where Docker will silently ignore the header if it's improperly encoded. This also makes the issue hard to troubleshoot: many valid auth headers will encode identically with URL-safe and unsafe encodings.

Steps to reproduce the issue:

  1. Use a library that uses regular base64 encoding for the X-Registry-Auth header, e.g. https://github.com/swipely/docker-api
  2. Use a username and password that cause the X-Registry-Auth to contain URL-unsafe characters, such as /.
  3. Try to pull and image that requires authentication.

Describe the results you received:

Pull fails with "image not found".

Describe the results you expected:

Docker throws an error (that'd be great), or more realistically the documentation indicates which encoding should have been used so I have an authoritative reference when I make a PR to fix the client library 😄

Thanks!

closed time in 9 days

krallin

issue commentmoby/moby

Move logdrivers to separate binaries

Thoughts?

@samuelkarp (AWS) @tianon (packaing guru)

cpuguy83

comment created time in 9 days

issue openedmoby/moby

Move logdrivers to separate binaries

Currently dockerd imports 10's (100's?) of thousands of lines of code for cloud providers and other logging providers.

It would be nice to move these into separate binaries to slim down dockerd. Possibly even nicer if someone can just stick a binary into the right folder and take advantage of it as a logging driver.

created time in 9 days

issue commentmoby/moby

libnetwork re-exec hads lots of overhead

@arkodg @fcrisciani maybe would like to chime in here.

cpuguy83

comment created time in 9 days

issue openedmoby/moby

libnetwork re-exec hads lots of overhead

The re-exec for setting up container networking has a lot of overhead since it has to re-initialize the daemon multiple times for a single container... even with --net=none and --net=host

In the short-term I propose that the re-exec calls get moved to a separate binary. There is some discussion on this here: https://github.com/docker/libnetwork/pull/1987

In the longer term I wonder if we can utilize the create/start split for configuring the container networking. Currently it's using a runc-hook (as I recall).

created time in 9 days

pull request commentmoby/moby

new storage driver: fuse-overlayfs

SGTM

Even though we are moving to containerd, I think this is good to bring in. As @tonistiigi pointed out on the maintainers call, this doesn't pose any new problems for that migration.

AkihiroSuda

comment created time in 9 days

Pull request review commentmoby/moby

Add stats options to not prime the stats

 func (s *containerRouter) getContainersStats(ctx context.Context, w http.Respons 	if !stream { 		w.Header().Set("Content-Type", "application/json") 	}+	var oneShot bool+	if versions.GreaterThanOrEqualTo(httputils.VersionFromContext(ctx), "1.v1") {

lol

cpuguy83

comment created time in 9 days

pull request commentmoby/moby

os.Environ() on "docker-applyLayer" like on "docker-untar"

Is this still needed? I honestly get great reproducibility with buildkit.

feeloo007

comment created time in 9 days

PR closed moby/moby

Reviewers
Support file mounts on Windows 1803 and above impact/changelog platform/windows status/2-code-review

Windows 1803 introduces the bind filter driver, bindflt.sys, which introduces several mounting improvements over the previous directory symbolic link approach. These improvements include:

  • No longer appear as symbolic links, improving compatibilty.
  • Can be mounted on top of existing directories and files.
  • Can be files, instead of only directories.

This change addresses the last point by allowing non-directory file mounts on Windows 1803 (build 17134) and greater. The first two improvements already work without this, or any other change.

This change also rearranges the code such that the Windows and LCOW mount parsers are only built for, and tested on Windows, and the Linux parser is only built for, and tested on Linux. This allows the Windows parser to call system.GetOSVersion, which only exists for Windows.

Signed-off-by: John Stephens johnstep@docker.com

+63 -56

5 comments

8 changed files

johnstep

pr closed time in 9 days

pull request commentmoby/moby

Support file mounts on Windows 1803 and above

Going to close this since it seems to be abandoned. We could still take a change to support file mounts if someone is interested in picking up this work.

johnstep

comment created time in 9 days

pull request commentmoby/moby

Fixes #33434 - API docs to specify using base64url

I went ahead and rebased this.

platy

comment created time in 9 days

push eventplaty/moby

Sebastiaan van Stijn

commit sha e4611b3e074c48e90ea2ea2fc138ede2ce87fb36

Merge pull request #39683 from tonistiigi/builder-metadata-timestamps builder-next: ensure timestamps set for metadata commands

view details

Sebastiaan van Stijn

commit sha 6ae46aeabf056180067dd6af8d5d8588d6075c31

make.ps1: Run-IntegrationTests(): set working directory for test suite This function changed to the correct working directory before starting the tests (which is the same as on Linux), however the `ProcessStartInfo` process does not inherit this working directory, which caused Windows tests to be running with a different working directory as Linux (causing files used in tests to not be found). From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8 > When `UseShellExecute` is `true`, the fully qualified name of the directory that contains > the process to be started. When the `UseShellExecute` property is `false`, the working > directory for the process to be started. The default is an empty string (`""`). This patch sets the `ProcessStartInfo.WorkingDirectory` to the correct working directory before starting the process. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e554ab558985b686c9c3427275a5e016aa1cdb76

Allow system.MkDirAll() to be used as drop-in for os.MkDirAll() also renamed the non-windows variant of this file to be consistent with other files in this package Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 097c09eb60a9f9c2d2fc646b569fc89761b9dfff

Merge pull request #39679 from jterry75/revendor_go-winio Update Microsoft/go-winio v0.4.14

view details

Sebastiaan van Stijn

commit sha 5858a99267822b93e2d304d876bab84d05b227c6

Builder: fix "COPY --from" to non-existing directory on Windows This fixes a regression introduced in 6d87f19142f86b8fee75af721b583a306202f228, causing `COPY --from` to fail if the target directory does not exist: ``` FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1 RUN echo "Hello World" > /hello FROM mcr.microsoft.com/windows/servercore:ltsc2019 COPY --from=s1 /hello /hello/another/world ``` Would produce an error: ``` Step 4/4 : COPY --from=s1 /hello /hello/another/world failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect. ``` The cause for this was that Go's `os.MkdirAll()` does not support/detect volume GUID paths (`\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}`), and as a result attempted to create the volume as a directory (`\\?`), causing it to fail. This patch replaces `os.MkdirAll()` with our own `system.MkdirAll()` function, which is capable of detecting GUID volumes. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kirill Kolyshkin

commit sha 838843bbad0a056baea502b4cf65215e0b041224

Merge pull request #39698 from thaJeztah/fix_windows_integration_pwd make.ps1: Run-IntegrationTests(): set working directory for test suite

view details

Kirill Kolyshkin

commit sha 150530564a142f264c9820a488585b093ae11d87

Merge pull request #39695 from thaJeztah/fix_copy_on_windows Builder: fix "COPY --from" to non-existing directory on Windows

view details

Deep Debroy

commit sha 4d5b6260bc595d5d9787c67ae887e83432911380

Fix regression in handling of NotFound err during startup Signed-off-by: Deep Debroy <ddebroy@docker.com>

view details

Sebastiaan van Stijn

commit sha 4cc9dc73ba533177a342394649d730b91e84e6f5

Merge pull request #39703 from ddebroy/fix-39623 Fix regression in handling of NotFound err during startup ENGCORE-929

view details

Sebastiaan van Stijn

commit sha 1ea8b413d126c4b4dea51f7b012b7c636e3bb177

initBridgeDriver: minor cleanup and linting fixes Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f8cde0b32d86fa2df71ec65adc3d45f862b3ea33

docker-py: deselect broken experimental tests These tests are fixed upstream, but those fixes are not yet in a released version. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha c2f70da793d8d4ab7ece860dd56e6d0c2970ff11

Merge pull request #38859 from kolyshkin/journald Fixes for reading journald logs

view details

Sebastiaan van Stijn

commit sha a43123cab1368fa1dab972481a2951ddbe1e2c66

Consistently use DOCKER_EXPERIMENTAL=1 instead or =y Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e856b46cfb29a9b637013369df0f57e6a06e1add

Jenkinsfile: remove "experimental" stage All tests that require experimental either spin up a separate daemon, or use the main daemon if experimental is enabled. This patch - allows enabling "experimental" for stages through an environment variable - enables experimental by default on all stages, so that some of these tests don't have to start a new daemon. - removes the seaprate "experimental" stage, because it was running exactly the same tests as the "janky" stage. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Tibor Vass

commit sha c5c11f9cefd4dde73dbb70f9ae245db7c1826b66

Merge pull request #39709 from thaJeztah/remove_experimental Jenkinsfile: remove "experimental" stage

view details

Kir Kolyshkin

commit sha b283dff3ff872b89dc4f2e70efe3af398bbf9423

Jenkinsfile: avoid errors from find There are many errors like this one: > 01:39:28.750 find: ‘bundles/test-integration/dbc77018d39a5/root/overlay2/f49953a883daceee60a481dd8e1e37b0f806d309258197d6ba0f6871236d3d47/work/work’: Permission denied (probably caused by bad permissions) These directories are not to be looked at when we search for logs, so let's exclude them. It's not super easy to do in find, here is some kind of an explanation for find arguments ``` PATTERN ACTION OR PATTERN ACTION -path X -prune -o -type f [AND] (-name A -o name B) -print ``` (here -o means OR, while AND is implicit) While at it, - let the find know we're only looking for files, not directories - remove a subshell and || true - remove `-name integration.test` (there are no such files) Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

view details

Sebastiaan van Stijn

commit sha 901d30b6b4b7f2ceaf289e4fc88d01d8d281520b

Merge pull request #39685 from kolyshkin/jenkins-find Jenkinsfile: avoid errors from find

view details

Sebastiaan van Stijn

commit sha 6f5c377ddce449bdf297fd4cab1f71e57788f883

docker-py: skip PullImageTest::test_pull_invalid_platform and remove `PullImageTest::test_build_invalid_platform` from the list, which was a copy/paste error in f8cde0b32d86fa2df71ec65adc3d45f862b3ea33 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Olli Janatuinen

commit sha 8660330173e5053e274cf12860079f132cbaa9fa

Unit test for getOrphan Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>

view details

Sebastiaan van Stijn

commit sha 90af4ba5e7fa5a75387a06f8987c30217f05d618

Merge pull request #39714 from thaJeztah/fix_docker_py_deselects docker-py: skip PullImageTest::test_pull_invalid_platform

view details

push time in 9 days

issue commentkubernetes/kubernetes

Add HTTP headers to ACR

HTTP headers are configured when creating the client. You can set them afterwards but this is deprecated.

Here's the deprecated method: https://github.com/moby/moby/blob/3af8d484b132a492bd6f159e454560b7d7691fa9/client/client.go#L293-L297

The updated one is an function option passed when creating the client: https://github.com/moby/moby/blob/28d7dba41d0c0d9c7f0dafcc79d3c59f2b3f5dc3/client/options.go#L114-L120

feiskyer

comment created time in 9 days

pull request commentcontainerd/containerd

Add flag on umount

Now that I'm at my laptop...

What typically happens in cases like this is you there is a mount marked as private that gets copied into a new mount namespace. A new mount namespace is created for every container, for systemd services that have MountPropagation or PrivateTmp defined, and these types of things. When those namespaces are created they get a copy of the root namespace, anything that has a private mount cannot be unmounted until all the namespaces are shut down. Mounts get marked private depending on the propagation defined on their root mount or if explicitly set.... so for example if you have /var/foo mounted and /var is mounted with mount private propagation, /var/foo will inherit the private propagation.

payall4u

comment created time in 9 days

Pull request review commentcontainerd/containerd

[release/1.2 backport] Pin to libseccomp 2.3.3

+#!/usr/bin/env bash++#   Copyright The containerd Authors.++#   Licensed under the Apache License, Version 2.0 (the "License");+#   you may not use this file except in compliance with the License.+#   You may obtain a copy of the License at++#       http://www.apache.org/licenses/LICENSE-2.0++#   Unless required by applicable law or agreed to in writing, software+#   distributed under the License is distributed on an "AS IS" BASIS,+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+#   See the License for the specific language governing permissions and+#   limitations under the License.+++#+# Builds and installs runc to /usr/local/go/bin based off+# the commit defined in vendor.conf

Ah on my phone didn't notice this was a backport.

hakman

comment created time in 10 days

Pull request review commentcontainerd/containerd

[release/1.2 backport] Pin to libseccomp 2.3.3

+#!/usr/bin/env bash++#   Copyright The containerd Authors.++#   Licensed under the Apache License, Version 2.0 (the "License");+#   you may not use this file except in compliance with the License.+#   You may obtain a copy of the License at++#       http://www.apache.org/licenses/LICENSE-2.0++#   Unless required by applicable law or agreed to in writing, software+#   distributed under the License is distributed on an "AS IS" BASIS,+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+#   See the License for the specific language governing permissions and+#   limitations under the License.+++#+# Builds and installs runc to /usr/local/go/bin based off+# the commit defined in vendor.conf

Ah on my phone didn't notice this was a backport.

hakman

comment created time in 10 days

pull request commentcontainerd/containerd

Add flag on umount

Why are you getting a busy error? This sounds like a configuration problem.

The problem with using MNT_DETACH is the underlying space will not be freed until the mount is no longer in use. If a pod is stuck trying to unmount it sounds like the space will never be freed.

payall4u

comment created time in 10 days

Pull request review commentcontainerd/containerd

[release/1.2 backport] Pin to libseccomp 2.3.3

+#!/usr/bin/env bash++#   Copyright The containerd Authors.++#   Licensed under the Apache License, Version 2.0 (the "License");+#   you may not use this file except in compliance with the License.+#   You may obtain a copy of the License at++#       http://www.apache.org/licenses/LICENSE-2.0++#   Unless required by applicable law or agreed to in writing, software+#   distributed under the License is distributed on an "AS IS" BASIS,+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+#   See the License for the specific language governing permissions and+#   limitations under the License.+++#+# Builds and installs runc to /usr/local/go/bin based off+# the commit defined in vendor.conf

This doesn't seem like the right comment.

hakman

comment created time in 10 days

PR opened docker/buildx

Make k8s driver priority lower

Otherwise it ends up being default and it's probably not the normal case.

+4 -2

0 comment

2 changed files

pr created time in 10 days

create barnchcpuguy83/buildx

branch : k8s_priority

created branch time in 10 days

pull request commentcontainerd/containerd

Makefile: remove FORCE dependency

p.s. ran into this trying to optimize build in qemu by pre-building manpages outside qemu, but saw it was rebuilding even though the files were already generated.

AkihiroSuda

comment created time in 10 days

pull request commentcontainerd/containerd

Makefile: remove FORCE dependency

I'm not sure I understand this argument to revert the change. Wouldn't you want something like make clean in the case when you want new binaries?

AkihiroSuda

comment created time in 10 days

push eventmoby/moby

Sebastiaan van Stijn

commit sha da6c1429d0b468fe8cb7c0cf41b59d5d3647391b

docker-py: skip flaky AttachContainerTest::test_attach_no_stream (again) This test was disabled in the past, but re-enabled when we upgraded docker-py to 4.2.0. The test looks to be still flaky though, so skipping it again: ``` [2020-02-10T23:40:44.429Z] =================================== FAILURES =================================== [2020-02-10T23:40:44.429Z] __________________ AttachContainerTest.test_attach_no_stream ___________________ [2020-02-10T23:40:44.429Z] tests/integration/api_container_test.py:1250: in test_attach_no_stream [2020-02-10T23:40:44.429Z] assert output == 'hello\n'.encode(encoding='ascii') [2020-02-10T23:40:44.429Z] E AssertionError: assert b'' == b'hello\n' [2020-02-10T23:40:44.429Z] E Right contains more items, first extra item: 104 [2020-02-10T23:40:44.429Z] E Use -v to get the full diff [2020-02-10T23:40:44.429Z] ------- generated xml file: /src/bundles/test-docker-py/junit-report.xml ------- ```` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha 38ce975b7691c367ac3366a84c1b813387382aa9

Merge pull request #40500 from thaJeztah/docker_py_skip_flaky docker-py: skip flaky AttachContainerTest::test_attach_no_stream (again)

view details

push time in 10 days

PR merged moby/moby

Reviewers
docker-py: skip flaky AttachContainerTest::test_attach_no_stream (again) area/testing process/cherry-pick status/2-code-review

relates to https://github.com/docker/docker-py/issues/2513

This test was disabled in the past (https://github.com/moby/moby/pull/39848), but re-enabled when we upgraded docker-py to 4.2.0 (https://github.com/moby/moby/pull/40467).

The test looks to be still flaky though, so skipping it again:

[2020-02-10T23:40:44.429Z] =================================== FAILURES ===================================
[2020-02-10T23:40:44.429Z] __________________ AttachContainerTest.test_attach_no_stream ___________________
[2020-02-10T23:40:44.429Z] tests/integration/api_container_test.py:1250: in test_attach_no_stream
[2020-02-10T23:40:44.429Z]     assert output == 'hello\n'.encode(encoding='ascii')
[2020-02-10T23:40:44.429Z] E   AssertionError: assert b'' == b'hello\n'
[2020-02-10T23:40:44.429Z] E     Right contains more items, first extra item: 104
[2020-02-10T23:40:44.429Z] E     Use -v to get the full diff
[2020-02-10T23:40:44.429Z] ------- generated xml file: /src/bundles/test-docker-py/junit-report.xml -------
+2 -1

6 comments

1 changed file

thaJeztah

pr closed time in 10 days

PR opened opencontainers/runc

Use "command -v" shell builtin instead of "which"

Took me awhile to figure out why it wasn't finding my go-md2man... centos image doesn't have which in it. So this is just a little more convenient and one less required build dep.

+1 -1

0 comment

1 changed file

pr created time in 10 days

create barnchcpuguy83/runc

branch : no_whiches

created branch time in 10 days

pull request commentmoby/moby

Jenkinsfile: temporarily pin windows image to 10.0.17763.973

Let's go with the previous tag, CI to rebuild the VM images is in progress due to some GitHub org refactorings.

@StefanScherer Do you mean we shouldn't merge this?

thaJeztah

comment created time in 11 days

pull request commentdocker/docker-ce

Bump Version to 19.03.6

Should be the same commit as rc2?

arkodg

comment created time in 11 days

push eventmoby/moby

Sebastiaan van Stijn

commit sha 562880b276edb9130eab6adaab27d8aeb41ec388

Fix more goimports ``` daemon/logger/splunk/splunk_test.go:33: File is not `goimports`-ed (goimports) envKey: "a", envRegexKey: "^foo", labelsKey: "b", tagKey: "c", integration/build/build_test.go:41: File is not `goimports`-ed (goimports) rm: false, forceRm: false, integration/image/remove_unix_test.go:49: File is not `goimports`-ed (goimports) Root: d.Root, ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha c51c65a217236da4cef0413637660b79f319741f

Merge pull request #40502 from thaJeztah/fix_goimports Fix more goimports

view details

push time in 11 days

more