profile
viewpoint
Drew Erny dperny @docker San Francisco, CA http://www.dperny.net FLOSS Machine. Outside of my official work obligations, I am happy to help anyone contribute to the various projects I work on in any way I can! Get in touch!

issue commentmoby/moby

Dockerd eats too much RAM

PR is opened on Swarmkit.

goetas

comment created time in 6 hours

PR opened docker/swarmkit

Fix leaking log subscription contexts

- What I did

Fixes leaking subscription contexts, and slightly alters the close procedure for PublishLogs streams. This in turn prevents leaking whole connections, and (hopefully) fixes moby/moby#39916

- How I did it

Very carefully, over the course of six or so hours

- How to test it

Merge it and see if the complaints dry up.

- Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: -->

+26 -4

0 comment

1 changed file

pr created time in 6 hours

push eventdperny/swarmkit-1

Drew Erny

commit sha 5c241fa6fd7fbbbcadefd4c0408bc63925372a70

Fix leaking subscription contexts Fixes leaking subscription contexts, and slightly alters the close procedure for PublishLogs streams. This in turn prevents leaking whole connections, and (hopefully) fixes moby/moby#39916 Signed-off-by: Drew Erny <derny@mirantis.com>

view details

push time in 6 hours

create barnchdperny/swarmkit-1

branch : fix-logs-leak

created branch time in 6 hours

issue commentmoby/moby

Dockerd eats too much RAM

I've been looking at this for a couple of hours. Without the ability to reproduce this, I have to be very thorough in my analysis so that any fix I try is very likely to solve the problem.

It seems that, first off, about 8 gigs of memory in use is a consequence of service logs. Service logs handling is shockingly complex, which makes this a bit difficult to tease out. Additionally, I don't have a super great understanding of gRPC streams anymore. But, from what I can gather, there exists some possibility for a leak if we try to close a gRPC stream without trying to receive any messages, the stream doesn't get properly closed.

The documentation for (*grpc.ClientConn).NewStream says:

To ensure resources are not leaked due to the stream returned, one of the following actions must be performed:

  1. Call Close on the ClientConn.
  2. Cancel the context provided.
  3. Call RecvMsg until a non-nil error is returned. A protobuf-generated client-streaming RPC, for instance, might use the helper function CloseAndRecv (note that CloseSend does not Recv, therefore is not guaranteed to release all resources).
  4. Receive a non-nil, non-io.EOF error from Header or SendMsg.

If none of the above happen, a goroutine and a context will be leaked, and grpc will not call the optionally-configured stats handler with a stats.End message.

In the swarmkit agent, we have a function, which is supposed to close the log stream. It does not call Recv, so perhaps this is the root cause of the leak. This function is called when either the agent attempts to send logs when the context has been canceled, or when the agent explicitly cancels the log stream.

https://github.com/moby/moby/blob/063bd4c96ae633a1aa830b7f8a646b689c14e5c8/vendor/github.com/docker/swarmkit/agent/agent.go#L544-L553

However, it should be the case that the context provided is canceled, so I took a deeper look at that.

Publish function on the Agent is called from one place: the (*agent.worker).Subscribe method, which passes the context provided to it. This context, in turn is passed from the Agent's main run loop, in this block of code:

https://github.com/moby/moby/blob/063bd4c96ae633a1aa830b7f8a646b689c14e5c8/vendor/github.com/docker/swarmkit/agent/agent.go#L297-L314

We can see here that a new sub-context is created for each subscription, and that it should be canceled when the subscription is closed. However, taking a look at the data from the pprof profile, we can see this:

    297            .          .           		case sub := <-session.subscriptions: 
    298            .          .           			if sub.Close { 
    299            .          .           				if cancel, ok := subscriptions[sub.ID]; ok { 
    300            .          .           					cancel() 
    301            .          .           				} 
    302            .          .           				delete(subscriptions, sub.ID) 
    303            .          .           				continue 
    304            .          .           			} 
    305            .          .            
    306            .          .           			if _, ok := subscriptions[sub.ID]; ok { 
    307            .          .           				// Duplicate subscription 
    308            .          .           				continue 
    309            .          .           			} 
    310            .          .            
    311            .   500.52MB           			subCtx, subCancel := context.WithCancel(ctx) 
    312        221MB      221MB           			subscriptions[sub.ID] = subCancel 
    313            .          .           			// TODO(dperny) we're tossing the error here, that seems wrong 
    314            .          .           			go a.worker.Subscribe(subCtx, sub) 

It seems that the contexts are never getting canceled, which means that subscriptions are never get Close messages. Now the question is why?

The Logs code is complex, and its server-side respresentation is in the package github.com/docker/swarmkit/manager/logbroker. The agent connects to the manager, creating a stream through which the manager sends new requests for logs to the agent (the ListenSubscriptions stream). When a user requests logs, the manager creates a new stream to send logs down to the user (the SubscribeLogs stream), and send a message down the ListenSubscriptions stream asking for those logs. The agent then creates a third stream (the PublishLogs stream) through which it sends logs to the manager. The stream, in particular, that we know is not getting closed is the PublishLogs stream on the agent's side. These three streams all exist in separate concurrent routines, and communicate over a complex series of channels.

The problem is, basically, this:

  • The ListenSubscriptions stream starts a watch on a queue. It knows a subscription is relevant to it if the subscription contains the NodeID of the agent.
  • When a subscription is finished, it is updated to say that it is closed and then published to this queue.
  • When the server side of the PublishLogs call receives a close message, it removes the node ID from the subscription.
  • Therefore, if the PublishLogs call finishes before the Subscription is closed, the Close message is never sent to the agent.
  • The close messages is never sent, because the Close message is only intended to stop a logs call in progress, not to clean up dangling streams.

In order to guard against leaks occurring when the manager never gets the opportunity to send the Close message (like if the manager crashes), we need to be able to cancel the context and remove it from the subscriptions map that it's being held on from. Additionally, in the interest of being Correct, we need to call Recv after we close the PublishLogs stream on the agent side. I'll make those changes and open a PR on swarmkit.

Whoever's idea it was to take logs out of experimental, they're an idiot.

goetas

comment created time in 6 hours

issue commentmoby/moby

Dockerd eats too much RAM

@mhemrg were you using docker service logs at the time of taking that profile? If so, how was it being used?

goetas

comment created time in 9 hours

pull request commentdocker/cli

Add jobs support to CLI

Removed WIP. The support for jobs upstream was merged.

dperny

comment created time in a day

push eventdperny/cli

Drew Erny

commit sha 73f2e459ecbd5e1c7966d131e3a6e6776fb6ce86

WIP: Add jobs support to CLI * Added two new modes accepted by the `--mode` flag * `replicated-job` creates a replicated job * `global-job` creates a global job. * When using `replicated-job` mode, the `replicas` flag sets the `TotalCompletions` parameter of the job. This is the total number of tasks that will run * Added a new flag, `max-concurrent`, for use with `replicated-job` mode. This flag sets the `MaxConcurrent` parameter of the job, which is the maximum number of replicas the job will run simultaneously. * When using `replicated-job` or `global-job` mode, using any of the update parameter flags will result in an error, as jobs cannot be updated in the traditional sense. * Updated the `docker service ls` UI to include the completion status (completed vs total tasks) if the service is a job. * Updated the progress bars UI for service creation and update to support jobs. For jobs, there is displayed a bar covering the overall progress of the job (the number of tasks completed over the total number of tasks to complete). * Added documentation explaining the use of the new flags, and of jobs in general. Signed-off-by: Drew Erny <derny@mirantis.com>

view details

push time in a day

issue commentmoby/moby

Dockerd eats too much RAM

Good lord this code doesn't need debugging it needs an exorcism.

goetas

comment created time in 4 days

issue commentmoby/moby

Dockerd eats too much RAM

I am not 100% sure how to read a profile graph like this, to be completely honest, but am I seeing over half a gig of contexts????

goetas

comment created time in 4 days

pull request commentdocker/cli

WIP: Add jobs support to CLI

@mathroc for whatever reason, I had not considered the possibility of an --rm flag, actually. I'm unsure how to implement it correctly, and it certainly won't make it into this release. That said, if my distant memories of being a mediocre ruby-on-rails developer are somewhat in tact, database migrations should be idempotent, mitigating the possibility of screwing things up by accidentally re-running a job.

Second, to create a job without starting it, you can just set --replicas to 0, which should be valid. Or, for a global job, set a placement constraint that cannot be met.

dperny

comment created time in 4 days

pull request commentdocker/cli

WIP: Add jobs support to CLI

Added nolint: gocyclo to the (*serviceOptions).ToService method. In exchange for this concession, I have added a documentation comment to that method.

dperny

comment created time in 5 days

push eventdperny/cli

Drew Erny

commit sha d34f273ac003923e4735ac48af23977b91826ede

WIP: Add jobs support to CLI * Added two new modes accepted by the `--mode` flag * `replicated-job` creates a replicated job * `global-job` creates a global job. * When using `replicated-job` mode, the `replicas` flag sets the `TotalCompletions` parameter of the job. This is the total number of tasks that will run * Added a new flag, `max-concurrent`, for use with `replicated-job` mode. This flag sets the `MaxConcurrent` parameter of the job, which is the maximum number of replicas the job will run simultaneously. * When using `replicated-job` or `global-job` mode, using any of the update parameter flags will result in an error, as jobs cannot be updated in the traditional sense. * Updated the `docker service ls` UI to include the completion status (completed vs total tasks) if the service is a job. * Updated the progress bars UI for service creation and update to support jobs. For jobs, there is displayed a bar covering the overall progress of the job (the number of tasks completed over the total number of tasks to complete). * Added documentation explaining the use of the new flags, and of jobs in general. Signed-off-by: Drew Erny <derny@mirantis.com>

view details

push time in 5 days

push eventdperny/cli

Drew Erny

commit sha fc6a2cd9f9ac5019d77a336ddb6217bb320b2aca

WIP: Add jobs support to CLI * Added two new modes accepted by the `--mode` flag * `replicated-job` creates a replicated job * `global-job` creates a global job. * When using `replicated-job` mode, the `replicas` flag sets the `TotalCompletions` parameter of the job. This is the total number of tasks that will run * Added a new flag, `max-concurrent`, for use with `replicated-job` mode. This flag sets the `MaxConcurrent` parameter of the job, which is the maximum number of replicas the job will run simultaneously. * When using `replicated-job` or `global-job` mode, using any of the update parameter flags will result in an error, as jobs cannot be updated in the traditional sense. * Updated the `docker service ls` UI to include the completion status (completed vs total tasks) if the service is a job. * Updated the progress bars UI for service creation and update to support jobs. For jobs, there is displayed a bar covering the overall progress of the job (the number of tasks completed over the total number of tasks to complete). * Added documentation explaining the use of the new flags, and of jobs in general. Signed-off-by: Drew Erny <derny@mirantis.com>

view details

push time in 5 days

pull request commentmoby/moby

Add support for swarm jobs

CLI PR is open, pending the merge of this PR.

dperny

comment created time in 5 days

PR opened docker/cli

Reviewers
WIP: Add jobs support to CLI

<!-- Please make sure you've read and understood our contributing guidelines; https://github.com/docker/cli/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with git commit -s **

For additional information on our contributing process, read our contributing guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx"

Please provide the following information: -->

This commit is currently WIP pending the merge of the engine PR for jobs

- What I did

Add support to the CLI for swarm jobs (moby/moby#40307).

- How I did it

  • Added two new modes accepted by the --mode flag
    • replicated-job creates a replicated job
    • global-job creates a global job.
  • When using replicated-job mode, the replicas flag sets the TotalCompletions parameter of the job. This is the total number of tasks that will run
  • Added a new flag, max-concurrent, for use with replicated-job mode. This flag sets the MaxConcurrent parameter of the job, which is the maximum number of replicas the job will run simultaneously.
  • When using replicated-job or global-job mode, using any of the update parameter flags will result in an error, as jobs cannot be updated in the traditional sense.
  • Updated the docker service ls UI to include the completion status (completed vs total tasks) if the service is a job.
  • Updated the progress bars UI for service creation and update to support jobs. For jobs, there is displayed a bar covering the overall progress of the job (the number of tasks completed over the total number of tasks to complete).
  • Added documentation explaining the use of the new flags, and of jobs in general.

- How to verify it

Includes automated tests for all major changes.

- Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> Added CLI support for swarm jobs.

+1152 -20

0 comment

16 changed files

pr created time in 5 days

push eventdperny/cli

Michael Irwin

commit sha b3a1681fd81cae1abe67b1a2c0ad6f137593c6ea

Add clarification when using USER UID:GI Signed-off-by: Michael Irwin <mikesir87@gmail.com>

view details

Nick Adcock

commit sha 154a1f6df8f3ce65f980b47bb4a4ff8c41c4c5c0

Reverse order of long-form ports Reverses the order long-form port options when converted to short-form to correctly match the documentation and `docker service create`. Post change `-p published=8111,target=8112` is the equivalent of `8111:8112` Signed-off-by: Nick Adcock <nick.adcock@docker.com>

view details

Sebastiaan van Stijn

commit sha c8e27296b74e7e283c7f215f4ab8930e28929dcc

Merge pull request #2252 from zappy-shu/2250-reverse-port-long-format-order Reverse order of long-form ports

view details

Aleksander Piotrowski

commit sha c2c7503d498db3fba5054430fe4589a0bc0b7739

Convert ports before parsing. Refactor code to allow mixed notation with -p flag. Signed-off-by: Aleksander Piotrowski <apiotrowski312@gmail.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha c804b1083ea7e5252ef5e53364f2d8e17af00f95

Dockerfile.e2e: update compose v1.25.1, enable buildkit Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Silvin Lubecki

commit sha 0ab564b1c868647382ea072663e64d5bdd6fdfbc

Merge pull request #2251 from thaJeztah/1962-fix-port-bug Convert ports before parsing.

view details

Silvin Lubecki

commit sha ddde460613e0a647f1d8a1079772ee9036c8b336

Merge pull request #2258 from thaJeztah/e2e_update_compose Dockerfile.e2e: update compose v1.25.1, enable buildkit

view details

Sebastiaan van Stijn

commit sha 6e5528b6501a6635691b5495d2a41dfc233668a1

e2e: fix formatting of comments Comments should have a leading space unless the comment is for special purposes (go:generate, nolint:) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 73dcf50d5a53ba97312fdcf9df8638da6bee9bc8

cli/command: fix formatting of comments, and minor linting issues Comments should have a leading space unless the comment is for special purposes (go:generate, nolint:) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e7f720b0a6cbc4611194426f91e2392962faacc3

cli/config: fix formatting of comments Comments should have a leading space unless the comment is for special purposes (go:generate, nolint:) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 1d6445dc99a5f617fa7b1b6c54f499838941b8de

opts: fix formatting of comments Comments should have a leading space unless the comment is for special purposes (go:generate, nolint:) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Silvin Lubecki

commit sha d05846d2cb1917d06bd72b551b5a37552cee53e9

Merge pull request #2260 from thaJeztah/fix_comment_formatting Fix comment formatting and minor linting issue

view details

Sebastiaan van Stijn

commit sha c6d10b6da090b24d73de453986470fc054b0187b

Merge pull request #2091 from mikesir87/user-group-info Add clarification when using USER UID:GID

view details

Drew Erny

commit sha c2e992aee10965126d56c0fe531bb790ef6ebf57

WIP: Add jobs support to CLI Adds support to the CLI for managing swarm jobs. Signed-off-by: Drew Erny <derny@mirantis.com>

view details

push time in 5 days

push eventdperny/cli

Drew Erny

commit sha 229d433c33316ab06e9c6c42a650c01f64793205

WIP: Add jobs support to CLI Adds support to the CLI for managing swarm jobs. Signed-off-by: Drew Erny <derny@mirantis.com>

view details

push time in 6 days

push eventdperny/cli

Akhil Mohan

commit sha e1daa22ba15894febfe84a21bf4705391c8e50c3

Modify docs to specify working of device flag Modify the docker run docs, to specify working of --device flag in containers that are started in privileged mode. The custom device permissions that are given to a device in privileged mode are ignored and goes with `rwm` by default Signed-off-by: Akhil Mohan <akhil.mohan@mayadata.io>

view details

Sebastiaan van Stijn

commit sha b104da4a094faf888409c98f0979908576be230d

bump containerd/continuity f2a389ac0a02ce21c09edd7344677a601970f41c full diff: https://github.com/containerd/continuity/compare/aaeac12a7ffcd198ae25440a9dff125c2e2703a7...f2a389ac0a02ce21c09edd7344677a601970f41c Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 4b267469b9b4769ac5f8a2cf3fa5e640efdc201e

bump hashicorp/golang-lru v0.5.3 full diff: https://github.com/hashicorp/golang-lru/compare/0fb14efe8c47ae851c0034ed7a448854d3d34cf3...v0.5.3 - hashicorp/golang-lru#53 remove defer keyword to avoid overhead - hashicorp/golang-lru#56 lru.Get(): avoid nil pointer dereference - hashicorp/golang-lru#57 Adds LRU cache resize - hashicorp/golang-lru#58 lru: don't kill the return values of Remove and RemoveOldest Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha cf3f902df4de63cdd88f993c4b279da3c7ea4842

update runc to v1.0.0-rc8-92-g84373aaa (CVE-2019-16884) full diff: https://github.com/opencontainers/runc/compare/v1.0.0-rc8...3e425f80a8c931f88e6d94a8c831b9d5aa481657 - opencontainers/runc#2010 criu image path permission error when checkpoint rootless container - opencontainers/runc#2028 Update to Go 1.12 and drop obsolete versions - opencontainers/runc#2029 Update dependencies - opencontainers/runc#2034 Support for logging from children processes - opencontainers/runc#2035 specconv: always set "type: bind" in case of MS_BIND - opencontainers/runc#2038 `r.destroy` can defer exec in `runner.run` method - opencontainers/runc#2041 Change the permissions of the notify listener socket to rwx for everyone - opencontainers/runc#2042 libcontainer: intelrdt: add missing destroy handler in defer func - opencontainers/runc#2047 Move systemd.Manager initialization into a function in that module - opencontainers/runc#2057 main: not reopen /dev/stderr - closes opencontainers/runc#2056 Runc + podman|cri-o + systemd issue with stderr - closes kubernetes/kubernetes#77615 kubelet fails starting CRI-O containers (Ubuntu 18.04 + systemd cgroups driver) - closes cri-o/cri-o#2368 Joining worker node not starting flannel or kube-proxy / CRI-O error "open /dev/stderr: no such device or address" - opencontainers/runc#2061 libcontainer: fix TestGetContainerState to check configs.NEWCGROUP - opencontainers/runc#2065 Fix cgroup hugetlb size prefix for kB - opencontainers/runc#2067 libcontainer: change seccomp test for clone syscall - opencontainers/runc#2074 Update dependency libseccomp-golang - opencontainers/runc#2081 Bump CRIU to 3.12 - opencontainers/runc#2089 doc: First process in container needs `Init: true` - opencontainers/runc#2094 Skip searching /dev/.udev for device nodes - closes opencontainers/runc#2093 HostDevices() race with older udevd versions - opencontainers/runc#2098 man: fix man-pages - opencontainers/runc#2103 cgroups/fs: check nil pointers in cgroup manager - opencontainers/runc#2107 Make get devices function public - opencontainers/runc#2113 libcontainer: initial support for cgroups v2 - opencontainers/runc#2116 Avoid the dependency on cgo through go-systemd/util package - removes github.com/coreos/pkg as dependency - opencontainers/runc#2117 Remove libcontainer detection for systemd features - fixes opencontainers/runc#2117 Cache the systemd detection results - opencontainers/runc#2119 libcontainer: update masked paths of /proc - relates to #36368 Add /proc/keys to masked paths - relates to #38299 Masked /proc/asound - relates to #37404 Add /proc/acpi to masked paths (CVE-2018-10892) - opencontainers/runc#2122 nsenter: minor fixes - opencontainers/runc#2123 Bump x/sys and update syscall for initial Risc-V support - opencontainers/runc#2125 cgroup: support mount of cgroup2 - opencontainers/runc#2126 libcontainer/nsenter: Don't import C in non-cgo file - opencontainers/runc#2129 Only allow proc mount if it is procfs - addresses opencontainers/runc#2129 AppArmor can be bypassed by a malicious image that specifies a volume at /proc (CVE-2019-16884) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha c07f50afab43ea67bd5720b392d5fc33f3a16ab6

bump buildkit f7042823e340d38d1746aa675b83d1aca431cee3 full diff: https://github.com/moby/buildkit/compare/10cef0c6e178bcaca1ad02b041a96b1091f52071...f7042823e340d38d1746aa675b83d1aca431cee3 no local changes, other than updated go.mod Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 22a291f703b2511daaa8e0d325ab48523e817558

config: preserve ownership and permissions on configfile When running `docker login` or `docker logout`, the CLI updates the configuration file by creating a temporary file, to replace the old one (if exists). When using `sudo`, this caused the file to be created as `root`, making it inaccessible to the current user. This patch updates the CLI to fetch permissions and ownership of the existing configuration file, and applies those permissions to the new file, so that it has the same permissions as the existing file (if any). Currently, only done for "Unix-y" systems (Mac, Linux), but can be implemented for Windows in future if there's a need. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 6732347e55bcdf1d763aec806ed84e301c710bf9

bump containerd v1.3.0 full diff: https://github.com/containerd/containerd/compare/7c1e88399ec0b0b077121d9d5ad97e647b11c870...v1.3.0 This also adds back containerd/ttrpc as a dependency, which is referenced by the BuildKit client (indirectly) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 69f216f6e4f9c5b7663564deca474ed508ddf306

Fix docker ps --format with templating functions Before this patch, using a template that used templating functions (such as `lower` or `json`) caused the command to fail in the pre-processor step (in `buildContainerListOptions`): docker ps --format='{{upper .Names}}' template: :1:8: executing "" at <.Names>: invalid value; expected string This problem was due to the pre-processing using a different "context" type than was used in the actual template, and custom functions to not be defined when instantiating the Go template. With this patch, using functions in templates works correctly: docker ps --format='{{upper .Names}}' MUSING_NEUMANN ELOQUENT_MEITNER Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha aef6b04a7ce7dd56f0fa4facafaca976aff46c16

Fix docker ps table headers with custom format and "split" or "join" Update the list of overrides for table headers so that columns using split or join will produce the correct table header. Before this patch: docker ps --format='table {{split .Names "/"}}' [NAMES] [unruffled_mclean] [eloquent_meitner] [sleepy_grothendieck] With this patch applied: docker ps --format='table {{split .Names "/"}}' NAMES [unruffled_mclean] [eloquent_meitner] [sleepy_grothendieck] Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 54ba28f4020b51978ef99b0132dc838408c34b48

docker info: list CLI plugins alphabetically Before this change, plugins were listed in a random order: Client: Debug Mode: false Plugins: doodle: Docker Doodles all around! 🐳 🎃 (thaJeztah, v0.0.1) shell: Open a browser shell on the Docker Host. (thaJeztah, v0.0.1) app: Docker Application (Docker Inc., v0.8.0) buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker) With this change, plugins are listed alphabetically: Client: Debug Mode: false Plugins: app: Docker Application (Docker Inc., v0.8.0) buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker) doodle: Docker Doodles all around! 🐳 🎃 (thaJeztah, v0.0.1) shell: Open a browser shell on the Docker Host. (thaJeztah, v0.0.1) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ef37a8a57c6821cf5e4c2ee4192ac80b263fba4e

Fix "make build" (non-containerized) on macOS macOS doesn't ship with the GNU version of `date`, which causes the command to fail if the `--rfc-3339 ns` format option is used. Given that we don't need the build-time with nanosecond precision, this patch changes the format used, so that the CLI binary can be built on the host (outside of a container); Before this change, `make binary` would fail: DISABLE_WARN_OUTSIDE_CONTAINER=1 make binary WARNING: binary creates a Linux executable. Use cross for macOS or Windows. ./scripts/build/binary make: *** [binary] Error 1 With this change, the binary can be built on the host: DISABLE_WARN_OUTSIDE_CONTAINER=1 make binary WARNING: binary creates a Linux executable. Use cross for macOS or Windows. ./scripts/build/binary Building statically linked build/docker-darwin-amd64 While the previous version formatted (and parsed) the date with nanoseconds precision, that level of precision is not actually used; ```go func reformatDate(buildTime string) string { t, errTime := time.Parse(time.RFC3339Nano, buildTime) if errTime == nil { return t.Format(time.ANSIC) } return buildTime } ``` Both the old, and new input will yield the same output: ```go fmt.Println(reformatDate("2019-12-31T13:41:44.846741804+00:00")) // Tue Dec 31 13:41:44 2019 fmt.Println(reformatDate("2019-12-31T13:41:44Z")) // Tue Dec 31 13:41:44 2019 ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 4beea585315e47b0e52509cff043cf9425ee3eec

Bump Golang 1.12.14 go1.12.14 (released 2019/12/04) includes a fix to the runtime. See the Go 1.12.14 milestone on our issue tracker for details: https://github.com/golang/go/issues?q=milestone%3AGo1.12.14+label%3ACherryPickApproved Update Golang 1.12.13 ------------------------ go1.12.13 (released 2019/10/31) fixes an issue on macOS 10.15 Catalina where the non-notarized installer and binaries were being rejected by Gatekeeper. Only macOS users who hit this issue need to update. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Tonis Tiigi

commit sha ce42168940e69f94ca7431de5187c3bf87c41d13

docs: document from platform Co-Authored-By: Tibor Vass <tiborvass@users.noreply.github.com> Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Vincent Demeester

commit sha 389fa742ff0ebfeda5d94735ac0ca09eac8ecc4f

Merge pull request #2232 from thaJeztah/fixup_docker_ps_formatting Fix some issues with docker ps --format

view details

Vincent Demeester

commit sha a2e252c92a55938e8ab5336059a0d4e5a4d561ec

Merge pull request #2234 from thaJeztah/bump_golang_1.12.14 Bump Golang 1.12.14

view details

Oscar Wieman

commit sha 2d6aec78a10ab70c673375de97aa2da569e80dd7

Change example networks Signed-off-by: Oscar Wieman <oscar@oscarr.nl>

view details

Sebastiaan van Stijn

commit sha 43c131f0ae1380af7e33173cd45df8ec0d2df82f

Merge pull request #2238 from thaJeztah/carry_platform_docs docs: document from platform (carry 2003)

view details

Sebastiaan van Stijn

commit sha 9efca8e7b44fd8d83fee2019264e822fbb17ad08

Merge pull request #2231 from oscrx/patch-1 Change example networks

view details

Silvin Lubecki

commit sha f870440c359c7f225032705c33c7f4ef2c09c600

Merge pull request #2153 from thaJeztah/bump_containerd_1.3.0 bump containerd and dependencies to v1.3.0

view details

Silvin Lubecki

commit sha f7f4a19328f139347a33aa5e728a06927a0036b7

Merge pull request #2236 from thaJeztah/plugin_sorting docker info: list CLI plugins alphabetically

view details

push time in 7 days

push eventdperny/docker

Drew Erny

commit sha 30d9fe30b1c1bf52f15a41e0b106a1542a167e04

Add swarm jobs Adds support for ReplicatedJob and GlobalJob service modes. These modes allow running service which execute tasks that exit upon success, instead of daemon-type tasks. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 8 days

Pull request review commentmoby/moby

Add support for swarm jobs

 func ServiceFromGRPC(s swarmapi.Service) (types.Service, error) { 	service.CreatedAt, _ = gogotypes.TimestampFromProto(s.Meta.CreatedAt) 	service.UpdatedAt, _ = gogotypes.TimestampFromProto(s.Meta.UpdatedAt) +	// If the service is a job, include JobStatus.+	if curSpec.Mode.ReplicatedJob != nil || curSpec.Mode.GlobalJob != nil {

Sure. I can make that change. It shouldn't make a difference one way or another.

dperny

comment created time in 8 days

pull request commentmoby/moby

Add support for swarm jobs

Swarmkit will only schedule the number of Tasks needed to reach TotalCompletions, so there won't be any Tasks still running when TotalCompletions is reached. MaxConcurrent establishes the upper bound on how many are scheduled.

dperny

comment created time in 8 days

Pull request review commentmoby/moby

Add support for swarm jobs

 type Service struct { 	// listing all tasks for a service, an operation that could be 	// computation and network expensive. 	ServiceStatus *ServiceStatus `json:",omitempty"`++	// JobStatus is the status of a Service which is in one of ReplicatedJob or+	// GlobalJob modes. It is absent on Replicated and Global services.+	JobStatus *JobStatus `json:",omitempty"`

ServiceStatus is only available when doing a List operation of Services, where it is used to store the result of server-side computation of task states for the service. That is, ServiceStatus is an optimization of a common operation (counting desired/running task counts). JobStatus is an intrinsic part of the Service object, not derived or computed.

dperny

comment created time in 8 days

push eventdperny/docker

Drew Erny

commit sha 0839fff5d1f72f06e63c7c9cf8c869402d018e0b

Add swarm jobs Adds support for ReplicatedJob and GlobalJob service modes. These modes allow running service which execute tasks that exit upon success, instead of daemon-type tasks. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 8 days

pull request commentmoby/moby

Add support for swarm jobs

No longer WIP.

dperny

comment created time in 8 days

push eventdperny/docker

Drew Erny

commit sha f3d6eff41da920e53702aee54a5d8ffb1ed46930

Add swarm jobs Adds support for ReplicatedJob and GlobalJob service modes. These modes allow running service which execute tasks that exit upon success, instead of daemon-type tasks. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 8 days

push eventdocker/swarmkit

Drew Erny

commit sha 788e8a61cf87a96f4c05fa79d0b38abe7cde8068

Add Job protos Adds protocol buffers for implementing Jobs in swarmkit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 67ceea9542ec0a487610cbdd4c24aaa33476ce43

Add minimal Replicated Job Orchestrator and Tests Adds orchestrators for replicated and global jobs, and the basic tests. This commit exists mostly to keep the Ginkgo in mostly its own commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha a4da9ec1fdeedc835d31583fc8ce1d48dcd548fb

Add service reconcilation for replicated jobs Adds service reconciliation logic for the replicated jobs orchestrator. This code does not function in production, and is not actually called except from the tests. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 3b160c83991643effb0fb54bfc0c35c9fc2051aa

Add global job orchestrator skeleton Expands the skeleton structure of the global jobs orchestrator. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 06fabe5432f8f9de52ac61e2447b3755e5116b17

Refactor replicated job orchestrator and add initialization Refactors the replicated job orchestrator to make testing simpler, and then adds initialization logic to it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 8140d068d946f75ff5d4391d41191c96325a544a

Add store event logic to replicated jobs orchestrator Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha e5962e7d08d29e14b67282387aded7d51d4cd297

Refactor global job orchestrator Refactors the global job orchestrator along the same lines as the replicated job orchestrator, in order to better decouple the event-driven orchestrator logic from the reconciliation logic. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 0d91115a74cfe30ff63454afc30a3f5141da9230

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha fb8a7199e509164c6517f3e6427f5353e8d76aa1

Add controlapi support for job services Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha b9321f79657867749301d4b4c4052e16fd8a6f89

Wire up jobs orchestrator to manager Adds the jobs orchestrator to the swarmkit manager. Jobs orchestrator will now start and run with the manager. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 5fb683f0a9fab24194b87889004f9b4b9d826f48

Add jobs orchestrator to controlapi integration tests Adds the beginnings of the integration tests between the controlapi and the jobs orchestrator. These tests don't actually check much more than creation right now, as update handling for the jobs orchestrator is pending, but further tests will be able to leverage the groundwork here to a high degree. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 9fec8973672102d56784dc7663d907bf2d0891a2

Support restarting failing jobs tasks In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed. Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1. Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed. Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 42e77978035e617c33f4ef4cfe6699f535ae5023

Update components to support jobs Updates manager components to support jobs, which have a desired state of Completed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 6ad221378a3b3b7e07e650cb344ea3d42f34cd64

Update ListServiceStatuses for Jobs Updates the ListServiceStatuses RPC to work with jobs. Includes adding a new field to the responses showing the number of completed Tasks in a job. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 39596e5df663a2dbf9f8ad8d859c19106748ab3d

Add support for updating global jobs Adds support for updating global jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 9959bf70121295add98ff140e7232c47a90e25d1

Add support for updating replicated jobs Adds support for updating replicated jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 17cd5b172ae6fb2a0a9be7b70f8278f2e84cf397

Add controlapi integration test for updating replicated jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha f61b9f747f767c0b8005908f55c06d2c04455e0a

Revert #2899 This reverts the commits from #2899, which shut down tasks of old iterations. This is the wrong approach and the right approach will be fixed in a later commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 53677047eb94e93220096fb92b96387d7b7ff1d1

Fix some jobs TODOs Addresses a couple of TODOs in the jobs orchestrator. Most notably, updates the reconcilers to set the DesiredState of Tasks belonging to older job iterations to Remove, which will cause them to be cleaned up and deleted. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2345d9220b3c807b7d5651a4f7a6021f812f6294

Add tests for jobs integration with RestartSupervisor Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 8 days

PR merged docker/swarmkit

[Jobs] Merge Jobs Support into Master

Over time, development of the Swarm Jobs feature has taken place on a separate feature-jobs branch, to avoid disrupting the downstream engine until the feature was complete. I'm comfortable, now, saying that Jobs is complete enough to merge into the swarmkit master branch.

Jobs integration with the engine is tracked in moby/moby#40307.

+201534 -910

3 comments

253 changed files

dperny

pr closed time in 8 days

pull request commentdocker/swarmkit

[Jobs] Merge Jobs Support into Master

Alright, I'm merging it. Here we go.

dperny

comment created time in 8 days

MemberEvent

push eventdocker/swarmkit

Drew Erny

commit sha eae24f8ddc916c1c4c746c90e27ceace4187d3d6

Remove global job node creation time check Removes code from the global job reconciler that prevented global jobs from executing on newly created nodes. This behavior probably would not have worked well in real-world use, and closes off a few use cases (like using a global job to perform some initialization on newly-created nodes). Signed-off-by: Drew Erny <derny@mirantis.com>

view details

push time in 11 days

PR opened docker/swarmkit

[Jobs] Merge Jobs Support into Master

Over time, development of the Swarm Jobs feature has taken place on a separate feature-jobs branch, to avoid disrupting the downstream engine until the feature was complete. I'm comfortable, now, saying that Jobs is complete enough to merge into the swarmkit master branch.

Jobs integration with the engine is tracked in moby/moby#40307.

+201534 -910

0 comment

253 changed files

pr created time in 11 days

push eventdperny/docker

Sebastiaan van Stijn

commit sha 20e3b5ba2cc17a5cba47b99e4f74fb9b6b6ab5c7

api/types: minor BuilderVersion refactor Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 9709f6b95e4604956125eaddbb8a4a46b4eb9005

api/server: build: use locally scoped variables Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 04203d13fb52ae41a156506831ce44abcc726dd6

api/server: build: refactor for better readability - construct the initial options as a literal - move validation for windows up, and fail early - move all API-version handling together Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Akihiro Suda

commit sha 612343618dd7dad7cf023e6263d693ab37507a92

cgroup2: use shim V2 * Requires containerd binaries from containerd/containerd#3799 . Metrics are unimplemented yet. * Works with crun v0.10.4, but `--security-opt seccomp=unconfined` is needed unless using master version of libseccomp ( containers/crun#156, seccomp/libseccomp#177 ) * Doesn't work with master runc yet * Resource limitations are unimplemented Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Akihiro Suda

commit sha 409bbdc3212a37e7a21a70eeae0b44e96509f54d

cgroup2: enable resource limitation enable resource limitation by disabling cgroup v1 warnings resource limitation still doesn't work with rootless mode (even with systemd mode) Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Akihiro Suda

commit sha 19baeaca267d5710907ac1b3c3972d44725fe8ad

cgroup2: enable cgroup namespace by default For cgroup v1, we were unable to change the default because of compatibility issue. For cgroup v2, we should change the default right now because switching to cgroup v2 is already breaking change. See also containers/libpod#4363 containers/libpod#4374 Privileged containers also use cgroupns=private by default. https://github.com/containers/libpod/pull/4374#issuecomment-549776387 Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Akihiro Suda

commit sha 491531c12bf60a538eeeb37e22b19729a1a65bb8

cgroup2: mark cpu-rt-{period,runtime} unimplemented Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Sebastiaan van Stijn

commit sha 7e0afd4934528d89e09bd850490db6477222df07

swagger: move ContainerState to definitions Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 79c877cfa71f7929facda0160bfec12ebed4e0ef

swagger: restore bind options information This information was added to an older version of the API documentation (through 164ab2cfc9900a5e9a8037d41faea2bfdf3d64f1 and 5213a0a67ec635a45e640364e8aa9bf5f431625e), but only added in the "docs" branch. This patch copies the information to the swagger file. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 9ae7196775dddeb49be5195e00572a55b3c4658a

swagger: add missing container Health docs Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 07efe6a0a7c8de362fe3fcf5201ae50578b34309

Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2 Bumps the vendoring of github.com/docker/swarmkit to the above commit, which is the current master at commit time. Most notably, this includes a change making the ingress network respect the default address pool. Because of this change, a change to network integration tests was needed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Sebastiaan van Stijn

commit sha 5dbfae694919b1aff76f360061b4065349bf1900

hack/make: ignore failure to stop apparmor ``` ---> Making bundle: .integration-daemon-stop (in bundles/test-integration) ++++ cat bundles/test-integration/docker.pid +++ kill 13137 +++ /etc/init.d/apparmor stop Leaving: AppArmorNo profiles have been unloaded. Unloading profiles will leave already running processes permanently unconfined, which can lead to unexpected situations. To set a process to complain mode, use the command line tool 'aa-complain'. To really tear down all profiles, run 'aa-teardown'." script returned exit code 255 ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 4e3ab9e9fbca682f75eb350c8ad4312282869a03

Dockerfile: switch golang image to "buster" variant, and update btrfs packages The btrfs-tools was a transitional package, and no longer exists: > Package btrfs-tools > stretch (oldstable) (admin): transitional dummy package > 4.7.3-1: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x It must be replaced either by `btrfs-progs` or `libbtrfs-dev` (which has just the development headers) > Package: libbtrfs-dev (4.20.1-2) > Checksumming Copy on Write Filesystem utilities (development headers) Note that the `libbtrfs-dev` package is not available on Debian stretch (only in stretch-backports) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha bb0472bd23806327a1ff1ea87343a6a0cadc21e9

Dockerfile: switch to iptables-legacy to match the host CI runs on Ubuntu 16.04 machines, which use iptables (legacy), but Debian buster uses nftables. Because of this, DNS resolution does not work if the daemon configures iptables. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 25a1bf53d29e6424d4e9688952129a03c62fdef6

Fix containerutility compilation on gcc-mingw-w64 8.3 The makefile for this binary has version 6.3 hardcoded, which causes compilation on 8.3 to fail: ``` Building: bundles/cross/windows/amd64/containerutility.exe In file included from /usr/x86_64-w64-mingw32/include/minwindef.h:163, from /usr/x86_64-w64-mingw32/include/windef.h:8, from /usr/x86_64-w64-mingw32/include/windows.h:69, from containerutility.h:3, from argumentstream.cpp:1: /usr/x86_64-w64-mingw32/include/winnt.h:1554:11: fatal error: x86intrin.h: No such file or directory # include <x86intrin.h> ^~~~~~~~~~~~~ compilation terminated. ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e6c1820ef5de8c198b4ddec74440a7ea7b331194

Merge pull request #40174 from AkihiroSuda/cgroup2 support cgroup2

view details

Sebastiaan van Stijn

commit sha d64156970f4d79038e16ddc93d82bf95b004e93f

Merge pull request #40309 from dperny/bump-swarmkit Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2

view details

Brian Goff

commit sha 73d407fa57b7da57f3f8ea03e4a76d333f4fdeed

Merge pull request #40348 from thaJeztah/swagger_add_healthcheck Swagger: add missing container Health docs, and add ContainerState as definition

view details

Sam Whited

commit sha 02bc2769fb12b617e572b166b4851fa3f908580c

Add myself as a curator Signed-off-by: Sam Whited <sam@samwhited.com>

view details

Sebastiaan van Stijn

commit sha 954d729ab85cf254f96a461482771646830da774

Merge pull request #40079 from thaJeztah/cleanup_build API: cleanup build endpoint code

view details

push time in 11 days

pull request commentmoby/moby

WIP: Add support for swarm jobs

@thaJeztah Yes, it does, but in any case, the development on Jobs has happened all in a feature branch in Swarmkit, and that feature branch has not quite yet been merged into swarmkit master.

dperny

comment created time in 12 days

create barnchdperny/cli

branch : swarm-jobs

created branch time in 12 days

push eventdperny/docker

Drew Erny

commit sha 7e19e9e619072e418d2201c9e695d615fa704306

WIP: add update test and service status Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 12 days

pull request commentmoby/moby

WIP: Add support for swarm jobs

It seems that in CI, the Jobs test takes too long to complete 20 iterations. I've cut down the total to 7, which is less than the minimum completion I saw, which was 8.

dperny

comment created time in 12 days

push eventdocker/swarmkit

Drew Erny

commit sha 374122fd29db51abbeac555700e3528119c41ff8

Remove global job node creation time check Removes code from the global job reconciler that prevented global jobs from executing on newly created nodes. This behavior probably would not have worked well in real-world use, and closes off a few use cases (like using a global job to perform some initialization on newly-created nodes). Signed-off-by: Drew Erny <derny@mirantis.com>

view details

Drew Erny

commit sha ae4359a17fcc338aa7c8d56c916c8b323fae63b8

Merge pull request #2924 from dperny/global-job-timing [Jobs] Remove global job node creation time check

view details

push time in 13 days

PR merged docker/swarmkit

[Jobs] Remove global job node creation time check

Removes code from the global job reconciler that prevented global jobs from executing on newly created nodes. This behavior probably would not have worked well in real-world use, and closes off a few use cases (like using a global job to perform some initialization on newly-created nodes).

+1 -76

1 comment

2 changed files

dperny

pr closed time in 13 days

PR opened docker/swarmkit

[Jobs] Remove global job node creation time check

Removes code from the global job reconciler that prevented global jobs from executing on newly created nodes. This behavior probably would not have worked well in real-world use, and closes off a few use cases (like using a global job to perform some initialization on newly-created nodes).

+1 -76

0 comment

2 changed files

pr created time in 13 days

create barnchdperny/swarmkit-1

branch : global-job-timing

created branch time in 13 days

push eventdocker/swarmkit

Drew Erny

commit sha 2345d9220b3c807b7d5651a4f7a6021f812f6294

Add tests for jobs integration with RestartSupervisor Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha e981d2eaa913d2c54bd52d35d0ca39690cbd4f6c

Merge pull request #2923 from dperny/jobs-restart-tests Add tests for jobs integration with RestartSupervisor

view details

push time in 13 days

PR merged docker/swarmkit

Add tests for jobs integration with RestartSupervisor

Adds integration tests which verify that the RestartSupervisor works correctly with jobs.

+326 -51

1 comment

2 changed files

dperny

pr closed time in 13 days

push eventdperny/docker

Drew Erny

commit sha dbd0ef627c214b827a09a23e309d7fdd0c6f9006

WIP: add update test and service status Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 13 days

push eventdperny/docker

Drew Erny

commit sha c930baa64ef3687ed68e49beb009d4d812dbc9ba

WIP: add update test and service status Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 14 days

PR opened docker/swarmkit

Add tests for jobs integration with RestartSupervisor

Adds integration tests which verify that the RestartSupervisor works correctly with jobs.

+326 -51

0 comment

2 changed files

pr created time in 14 days

push eventdperny/swarmkit-1

Drew Erny

commit sha 004962fa3d76cbaf5c65289e24c7337ba035179e

Remove t.Log from mock task controller Previous, worker_test.go included a call to t.Log in the mock task controller. This presented a problem: because goroutines spawned by a test are not exited when the test completes, any holdover goroutines, like the one spawned to close the task controller, live on. This creates a problem when that goroutine calls t.Log on a test that has already completed. This t.Log call doesn't add enough information to bother worrying about. Further, these goroutines, though they may outlive the test, don't actually affect anything or matter, so there's no rush to fix the test to ensure all goroutines exit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 123bb7489afa2eef84a4e52d966d44ffa2792055

Merge pull request #2919 from dperny/remove-test-log Remove t.Log from mock task controller

view details

Drew Erny

commit sha 5453adb444d20eeca64212ef5ece6018f7e3dcce

Revert "YOLO see if this works" This reverts commit 42085d2f8e43a3ed90ed289d3f3ed3de57837100. Turns out it's a bad idea to merge a commit with the word "YOLO" in it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 24fb4cfe8af56803640180c5592bf32da732ced2

Merge pull request #2920 from dperny/revert-yolo Revert "YOLO see if this works"

view details

Drew Erny

commit sha 788e8a61cf87a96f4c05fa79d0b38abe7cde8068

Add Job protos Adds protocol buffers for implementing Jobs in swarmkit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 67ceea9542ec0a487610cbdd4c24aaa33476ce43

Add minimal Replicated Job Orchestrator and Tests Adds orchestrators for replicated and global jobs, and the basic tests. This commit exists mostly to keep the Ginkgo in mostly its own commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha a4da9ec1fdeedc835d31583fc8ce1d48dcd548fb

Add service reconcilation for replicated jobs Adds service reconciliation logic for the replicated jobs orchestrator. This code does not function in production, and is not actually called except from the tests. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 3b160c83991643effb0fb54bfc0c35c9fc2051aa

Add global job orchestrator skeleton Expands the skeleton structure of the global jobs orchestrator. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 06fabe5432f8f9de52ac61e2447b3755e5116b17

Refactor replicated job orchestrator and add initialization Refactors the replicated job orchestrator to make testing simpler, and then adds initialization logic to it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 8140d068d946f75ff5d4391d41191c96325a544a

Add store event logic to replicated jobs orchestrator Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha e5962e7d08d29e14b67282387aded7d51d4cd297

Refactor global job orchestrator Refactors the global job orchestrator along the same lines as the replicated job orchestrator, in order to better decouple the event-driven orchestrator logic from the reconciliation logic. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 0d91115a74cfe30ff63454afc30a3f5141da9230

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha fb8a7199e509164c6517f3e6427f5353e8d76aa1

Add controlapi support for job services Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha b9321f79657867749301d4b4c4052e16fd8a6f89

Wire up jobs orchestrator to manager Adds the jobs orchestrator to the swarmkit manager. Jobs orchestrator will now start and run with the manager. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 5fb683f0a9fab24194b87889004f9b4b9d826f48

Add jobs orchestrator to controlapi integration tests Adds the beginnings of the integration tests between the controlapi and the jobs orchestrator. These tests don't actually check much more than creation right now, as update handling for the jobs orchestrator is pending, but further tests will be able to leverage the groundwork here to a high degree. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 9fec8973672102d56784dc7663d907bf2d0891a2

Support restarting failing jobs tasks In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed. Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1. Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed. Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 42e77978035e617c33f4ef4cfe6699f535ae5023

Update components to support jobs Updates manager components to support jobs, which have a desired state of Completed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 6ad221378a3b3b7e07e650cb344ea3d42f34cd64

Update ListServiceStatuses for Jobs Updates the ListServiceStatuses RPC to work with jobs. Includes adding a new field to the responses showing the number of completed Tasks in a job. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 39596e5df663a2dbf9f8ad8d859c19106748ab3d

Add support for updating global jobs Adds support for updating global jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 9959bf70121295add98ff140e7232c47a90e25d1

Add support for updating replicated jobs Adds support for updating replicated jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 14 days

push eventdocker/swarmkit

Drew Erny

commit sha 004962fa3d76cbaf5c65289e24c7337ba035179e

Remove t.Log from mock task controller Previous, worker_test.go included a call to t.Log in the mock task controller. This presented a problem: because goroutines spawned by a test are not exited when the test completes, any holdover goroutines, like the one spawned to close the task controller, live on. This creates a problem when that goroutine calls t.Log on a test that has already completed. This t.Log call doesn't add enough information to bother worrying about. Further, these goroutines, though they may outlive the test, don't actually affect anything or matter, so there's no rush to fix the test to ensure all goroutines exit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 123bb7489afa2eef84a4e52d966d44ffa2792055

Merge pull request #2919 from dperny/remove-test-log Remove t.Log from mock task controller

view details

Drew Erny

commit sha 5453adb444d20eeca64212ef5ece6018f7e3dcce

Revert "YOLO see if this works" This reverts commit 42085d2f8e43a3ed90ed289d3f3ed3de57837100. Turns out it's a bad idea to merge a commit with the word "YOLO" in it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 24fb4cfe8af56803640180c5592bf32da732ced2

Merge pull request #2920 from dperny/revert-yolo Revert "YOLO see if this works"

view details

Drew Erny

commit sha 788e8a61cf87a96f4c05fa79d0b38abe7cde8068

Add Job protos Adds protocol buffers for implementing Jobs in swarmkit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 67ceea9542ec0a487610cbdd4c24aaa33476ce43

Add minimal Replicated Job Orchestrator and Tests Adds orchestrators for replicated and global jobs, and the basic tests. This commit exists mostly to keep the Ginkgo in mostly its own commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha a4da9ec1fdeedc835d31583fc8ce1d48dcd548fb

Add service reconcilation for replicated jobs Adds service reconciliation logic for the replicated jobs orchestrator. This code does not function in production, and is not actually called except from the tests. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 3b160c83991643effb0fb54bfc0c35c9fc2051aa

Add global job orchestrator skeleton Expands the skeleton structure of the global jobs orchestrator. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 06fabe5432f8f9de52ac61e2447b3755e5116b17

Refactor replicated job orchestrator and add initialization Refactors the replicated job orchestrator to make testing simpler, and then adds initialization logic to it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 8140d068d946f75ff5d4391d41191c96325a544a

Add store event logic to replicated jobs orchestrator Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha e5962e7d08d29e14b67282387aded7d51d4cd297

Refactor global job orchestrator Refactors the global job orchestrator along the same lines as the replicated job orchestrator, in order to better decouple the event-driven orchestrator logic from the reconciliation logic. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 0d91115a74cfe30ff63454afc30a3f5141da9230

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha fb8a7199e509164c6517f3e6427f5353e8d76aa1

Add controlapi support for job services Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha b9321f79657867749301d4b4c4052e16fd8a6f89

Wire up jobs orchestrator to manager Adds the jobs orchestrator to the swarmkit manager. Jobs orchestrator will now start and run with the manager. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 5fb683f0a9fab24194b87889004f9b4b9d826f48

Add jobs orchestrator to controlapi integration tests Adds the beginnings of the integration tests between the controlapi and the jobs orchestrator. These tests don't actually check much more than creation right now, as update handling for the jobs orchestrator is pending, but further tests will be able to leverage the groundwork here to a high degree. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 9fec8973672102d56784dc7663d907bf2d0891a2

Support restarting failing jobs tasks In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed. Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1. Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed. Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 42e77978035e617c33f4ef4cfe6699f535ae5023

Update components to support jobs Updates manager components to support jobs, which have a desired state of Completed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 6ad221378a3b3b7e07e650cb344ea3d42f34cd64

Update ListServiceStatuses for Jobs Updates the ListServiceStatuses RPC to work with jobs. Includes adding a new field to the responses showing the number of completed Tasks in a job. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 39596e5df663a2dbf9f8ad8d859c19106748ab3d

Add support for updating global jobs Adds support for updating global jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 9959bf70121295add98ff140e7232c47a90e25d1

Add support for updating replicated jobs Adds support for updating replicated jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 14 days

pull request commentmoby/moby

Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2

I think there may be some flakiness, as the last force push succeeded, but the previous on did not. I'm going to force a few more times to verify that it was a transient issue and not a flaky test.

dperny

comment created time in 14 days

push eventdperny/swarmkit-1

Drew Erny

commit sha 5c524cebe894701d6af59dbd1367a372c99305c2

Add tests for jobs integration with RestartSupervisor Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 14 days

push eventdperny/docker

Drew Erny

commit sha 07efe6a0a7c8de362fe3fcf5201ae50578b34309

Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2 Bumps the vendoring of github.com/docker/swarmkit to the above commit, which is the current master at commit time. Most notably, this includes a change making the ingress network respect the default address pool. Because of this change, a change to network integration tests was needed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 14 days

push eventdperny/docker

Drew Erny

commit sha 7ccf10d3d119a49ca074ce35e1af0ed1cd3fdf2b

WIP: add update test and service status Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 15 days

create barnchdperny/swarmkit-1

branch : jobs-restart-tests

created branch time in 15 days

push eventdperny/docker

Sebastiaan van Stijn

commit sha 6bc45b09e7b06cd68e61c26f0630d09279136d75

docker-py: re-enable tests that were fixed in v4.1.0 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f2b25e498f8ea8bfd30f166717146e11e17e948d

docker-py: skip broken ImageCollectionTest::test_pull_multiple The ImageCollectionTest.test_pull_multiple test performs a `docker pull` without a `:tag` specified) to pull all tags of the given repository (image). After pulling the image, the image(s) pulled are checked to verify if the list of images contains the `:latest` tag. However, the test assumes that all tags of the image are tags for the same version of the image (same digest), and thus a *single* image is returned, which is not always the case. Currently, the `hello-world:latest` and `hello-world:linux` tags point to a different digest, therefore the `client.images.pull()` returns multiple images: one image for digest, making the test fail: =================================== FAILURES =================================== ____________________ ImageCollectionTest.test_pull_multiple ____________________ tests/integration/models_images_test.py:90: in test_pull_multiple assert len(images) == 1 E AssertionError: assert 2 == 1 E + where 2 = len([<Image: 'hello-world:linux'>, <Image: 'hello-world:latest'>]) This patch temporarily skips the broken test until it is fixed upstream. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Tianon Gravi

commit sha a9507c6f76627fdc092edc542d5a7ef4a6df5eec

Merge pull request #40343 from thaJeztah/skip_broken_docker_py_test docker-py: skip broken ImageCollectionTest::test_pull_multiple, and re-enable fixed tests

view details

Drew Erny

commit sha 8356566cf9eaec2977e7fa23e3b3a18fe149e0f2

Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2 Bumps the vendoring of github.com/docker/swarmkit to the above commit, which is the current master at commit time. Most notably, this includes a change making the ingress network respect the default address pool. Because of this change, a change to network integration tests was needed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 15 days

push eventdperny/docker

wenlxie

commit sha 03b3ec1dd52bb45eaa73864b62177c13c348c639

make --device works at privileged mode Signed-off-by: wenlxie <wenlxie@ebay.com>

view details

Akhil Mohan

commit sha 35b9e6989f0421ba2fb71a7404058859c586b438

Make `--device` flag work in privileged mode When a container is started in privileged mode, the device mappings provided by `--device` flag was ignored. Now the device mappings will be considered even in privileged mode. Signed-off-by: Akhil Mohan <akhil.mohan@mayadata.io>

view details

Jan Chren

commit sha 9608dc5470e74f1ee34fd6477547303b31868576

Document `message` parameter to `/images/create` This parameter was introduced 4 years ago in b857dadb334fe0660abce4ae73a8b5b5c7f92c33 as part of https://github.com/moby/moby/pull/15711, but has never made it to the API docs. Signed-off-by: Jan Chren (rindeal) <dev.rindeal@gmail.com>

view details

Akhil Mohan

commit sha 86ebbe16de82933a916f67887aedd86d45a51882

remove host directory check Signed-off-by: Akhil Mohan <akhil.mohan@mayadata.io>

view details

Brian Goff

commit sha 03163f6825e64dbb127cb7cf1a4deb642b823c39

Merge pull request #40291 from akhilerm/privileged-device 35991- make `--device` works at privileged mode

view details

Brian Goff

commit sha a874c42edac24ab5c22d56e49e9262eec6fd8e63

Merge pull request #39700 from rindeal/patch-1 Document `message` parameter to `/images/create`

view details

Sebastiaan van Stijn

commit sha 6bc45b09e7b06cd68e61c26f0630d09279136d75

docker-py: re-enable tests that were fixed in v4.1.0 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f2b25e498f8ea8bfd30f166717146e11e17e948d

docker-py: skip broken ImageCollectionTest::test_pull_multiple The ImageCollectionTest.test_pull_multiple test performs a `docker pull` without a `:tag` specified) to pull all tags of the given repository (image). After pulling the image, the image(s) pulled are checked to verify if the list of images contains the `:latest` tag. However, the test assumes that all tags of the image are tags for the same version of the image (same digest), and thus a *single* image is returned, which is not always the case. Currently, the `hello-world:latest` and `hello-world:linux` tags point to a different digest, therefore the `client.images.pull()` returns multiple images: one image for digest, making the test fail: =================================== FAILURES =================================== ____________________ ImageCollectionTest.test_pull_multiple ____________________ tests/integration/models_images_test.py:90: in test_pull_multiple assert len(images) == 1 E AssertionError: assert 2 == 1 E + where 2 = len([<Image: 'hello-world:linux'>, <Image: 'hello-world:latest'>]) This patch temporarily skips the broken test until it is fixed upstream. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Tianon Gravi

commit sha a9507c6f76627fdc092edc542d5a7ef4a6df5eec

Merge pull request #40343 from thaJeztah/skip_broken_docker_py_test docker-py: skip broken ImageCollectionTest::test_pull_multiple, and re-enable fixed tests

view details

Drew Erny

commit sha d8b9a9ba34842c6921c499580095523dafa908ad

WIP: add swarm jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 15 days

push eventdocker/swarmkit

Drew Erny

commit sha 55a87cfd91f1e2ed6e224fe0cc14637ce127a7d7

Revert #2899 This reverts the commits from #2899, which shut down tasks of old iterations. This is the wrong approach and the right approach will be fixed in a later commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2be4f6c2308016fbc57fa5d22a832ee03dff7b97

Fix some jobs TODOs Addresses a couple of TODOs in the jobs orchestrator. Most notably, updates the reconcilers to set the DesiredState of Tasks belonging to older job iterations to Remove, which will cause them to be cleaned up and deleted. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha d1f40647e01ea436159075a8886a0acd33d51254

Merge pull request #2921 from dperny/jobs-fix-todos [Jobs] Support updates, add logging, etc.

view details

push time in 18 days

PR merged docker/swarmkit

[Jobs] Support updates, add logging, etc.

Fixes some TODOs in the code, adds some logging.

Also, more importantly, fixes updating jobs, so that it is done the right way. Tasks, instead of being shut down, are moved to the desired state of Remove, which allows the Task Reaper to do its work.

+284 -242

1 comment

8 changed files

dperny

pr closed time in 18 days

push eventdperny/docker

wenlxie

commit sha 03b3ec1dd52bb45eaa73864b62177c13c348c639

make --device works at privileged mode Signed-off-by: wenlxie <wenlxie@ebay.com>

view details

Akhil Mohan

commit sha 35b9e6989f0421ba2fb71a7404058859c586b438

Make `--device` flag work in privileged mode When a container is started in privileged mode, the device mappings provided by `--device` flag was ignored. Now the device mappings will be considered even in privileged mode. Signed-off-by: Akhil Mohan <akhil.mohan@mayadata.io>

view details

Sebastiaan van Stijn

commit sha a218e9b7b0ba5422288e79c7703e71f7ea41f85b

Update Golang 1.13.5 go1.13.5 (released 2019/12/04) includes fixes to the go command, the runtime, the linker, and the net/http package. See the Go 1.13.5 milestone on our issue tracker for details: https://github.com/golang/go/issues?q=milestone%3AGo1.13.5+label%3ACherryPickApproved Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Jan Chren

commit sha 9608dc5470e74f1ee34fd6477547303b31868576

Document `message` parameter to `/images/create` This parameter was introduced 4 years ago in b857dadb334fe0660abce4ae73a8b5b5c7f92c33 as part of https://github.com/moby/moby/pull/15711, but has never made it to the API docs. Signed-off-by: Jan Chren (rindeal) <dev.rindeal@gmail.com>

view details

Brian Goff

commit sha 9177ac322afe2371376591094116be59f3fef054

Fix output targets getting double nested. Targets are going to bundles/bundles instead of just bundles/. This is because there is `bundles` in the actual built binaries as well as the output dir being set to bundles. Signed-off-by: Brian Goff <cpuguy83@gmail.com>

view details

Akihiro Suda

commit sha 8ca8f8bd6598d44d344da407663a57584a0dc6a2

Merge pull request #40326 from thaJeztah/bump_golang_1.13.5 Update Golang 1.13.5

view details

Akihiro Suda

commit sha e6f6c35b79028e3f59ee99a5482641ac6d2e7f23

Merge pull request #40331 from cpuguy83/fix_bundle_output_dir Fix output targets getting double nested.

view details

Akhil Mohan

commit sha 86ebbe16de82933a916f67887aedd86d45a51882

remove host directory check Signed-off-by: Akhil Mohan <akhil.mohan@mayadata.io>

view details

Brian Goff

commit sha 03163f6825e64dbb127cb7cf1a4deb642b823c39

Merge pull request #40291 from akhilerm/privileged-device 35991- make `--device` works at privileged mode

view details

Brian Goff

commit sha a874c42edac24ab5c22d56e49e9262eec6fd8e63

Merge pull request #39700 from rindeal/patch-1 Document `message` parameter to `/images/create`

view details

Drew Erny

commit sha 05e7f2e1d1ae63ed81bb61c8f22bdb0d419f8006

Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2 Bumps the vendoring of github.com/docker/swarmkit to the above commit, which is the current master at commit time. Most notably, this includes a change making the ingress network respect the default address pool. Because of this change, a change to network integration tests was needed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 18 days

PR opened docker/swarmkit

[Jobs] Support updates, add logging, etc.

Fixes some TODOs in the code, adds some logging.

Also, more importantly, fixes updating jobs, so that it is done the right way. Tasks, instead of being shut down, are moved to the desired state of Remove, which allows the Task Reaper to do its work.

+284 -242

0 comment

8 changed files

pr created time in 19 days

push eventdperny/swarmkit-1

Drew Erny

commit sha dbddc47738b0a76d4680e1ce02f49b6714e49909

Add support for updating global jobs Adds support for updating global jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 1ba945ed0525588258898d150491754e93a7b93d

Add support for updating replicated jobs Adds support for updating replicated jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2bbdc74c4948c2885dcf1dd3a6eac81abecbbbb3

Add controlapi integration test for updating replicated jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha feab4999c1fccb2bd75af40ea81f631c46c237e1

Update ListServiceStatuses for Jobs Updates the ListServiceStatuses RPC to work with jobs. Includes adding a new field to the responses showing the number of completed Tasks in a job. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 5b97b0b0ed077cb27b71a6d771bbe6cbc3a7bddb

Merge pull request #2916 from dperny/jobs-service-status [Jobs] Update ListServiceStatuses for Jobs

view details

Drew Erny

commit sha 0c71f744ba2eaf79e780d1cf4cb9a2205d673e8a

Merge pull request #2899 from dperny/jobs-updates [Jobs] Support job updates

view details

Drew Erny

commit sha 55a87cfd91f1e2ed6e224fe0cc14637ce127a7d7

Revert #2899 This reverts the commits from #2899, which shut down tasks of old iterations. This is the wrong approach and the right approach will be fixed in a later commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2be4f6c2308016fbc57fa5d22a832ee03dff7b97

Fix some jobs TODOs Addresses a couple of TODOs in the jobs orchestrator. Most notably, updates the reconcilers to set the DesiredState of Tasks belonging to older job iterations to Remove, which will cause them to be cleaned up and deleted. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 19 days

push eventdperny/docker

Drew Erny

commit sha ddb986cfd7efd316192404b845d5b0beed97c155

WIP: add swarm jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 19 days

push eventdperny/docker

Sebastiaan van Stijn

commit sha 39ad39d22020a06f25b06ca45d252f7d822485e8

bump google.golang.org/grpc v1.23.1 full diff: https://github.com/grpc/grpc-go/compare/v1.23.0...v1.23.1 - grpc/grpc-go#3018 server: set and advertise max frame size of 16KB - grpc/grpc-go#3017 grpclb: fix deadlock in grpclb connection cache - Before the fix, if the timer to remove a SubConn fires at the same time NewSubConn cancels the timer, it caused a mutex leak and deadlock. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

HuanHuan Ye

commit sha 203ba72fc549b5ce306e78030579300f6e5ffe10

integration-cli: Fix `SA1019: httputil.ClientConn is deprecated` Rewrite sockRequestHijack to requestHijack which use writable Transport's Response.Body to replace deprecated hijacked httputil.ClientConn. ``` // As of Go 1.12, the Body will also implement io.Writer // on a successful "101 Switching Protocols" response, // as used by WebSockets and HTTP/2's "h2c" mode. Body io.ReadCloser ```. TestPostContainersAttach and TestExecResizeImmediatelyAfterExecStart replace all sockRequestHijack to requestHijack. Signed-off-by: HuanHuan Ye <logindaveye@gmail.com>

view details

Sebastiaan van Stijn

commit sha 5a74a736a89944b856681fe5926f09c35e48c092

TestParseRemoteURL: use subtests Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 86594739b08a0fe9a25db38934d51f2fc23d88c3

builder/remotecontext: use net/url instead of urlutil urlutil.IsUrl() was merely checking if the url had a http(s):// prefix, which is just as well handled through using url.Parse() Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

vikrambirsingh

commit sha 25c5a55261c3a856264c19d3b252351566a210a6

Check for OS Type and skip within the test Premature check for OS type means that the test will never even get to run on other OS types. This will cause it to be always flagged as a failure on such OS types. Signed-off-by: vikrambirsingh <vikrambir.singh@docker.com>

view details

Justen Martin

commit sha 36f6cc11a458d51aeed2c181257cbbb6a6640739

refactored integration/service/instead_test.go to ues unique resource names Signed-off-by: Justen Martin <jmart@the-coder.com>

view details

Till Wegmüller

commit sha bbf1edae4f5df9219a008fd67bb8c4f1cabee68d

Allow client consumers like traefik to compile on illumos Signed-off-by: Till Wegmüller <toasterson@gmail.com>

view details

Sebastiaan van Stijn

commit sha 4cdb796b5453d4c2fe7dde85cadd2a56725e3452

pkg/system/windows: remove obsolete comment Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 33d8492ce413780e4e339631386541410677ecf3

pkg/system/windows: remove unnecessary conversions Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 81f9edc7b0efce763ad89a7b62f0546e5a12aff6

pkg/parsers: remove use of deprecated system.GetOSVersion() Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 044b74e33bc499e360d1793f4c0d5975032c2d6d

daemon: remove use of deprecated system.GetOSVersion() Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f572df7c22c1f48b9a2ddd51519797a5b6a2eda4

pkg/system: deprecate constants in favor of golang.org/x/sys/windows Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f22ff19668e220fcc06182ca3dae14fd086e0a4a

builder: remove use of deprecated pkg/system constants Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 9aa55d97e564329f9b1c5b0b2c09c5f2bc1cc578

bump containerd/go-runc a2952bc25f5116103a8b78f3817f6df759aa7def full diff: https://github.com/containerd/go-runc/compare/e029b79d8cda8374981c64eba71f28ec38e5526f...a2952bc25f5116103a8b78f3817f6df759aa7def - containerd/go-runc#56 fix typo in successfully Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 90aa0901da11582da7639cc1ab202260183799e0

api/server/router: use consts for HTTP methods Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 580d3677c80398fb0fffb12a6a7db784ec1a135d

integration-cli: normalize comment formatting Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha c9b2a3cff51a6403f4eac319af5a794b0dbdeba4

api: normalize comment formatting Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 92ad849327d8ed5640bc7d0fbcba3fcf56f8a962

integration: normalize comment formatting Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 0fb563078469c76fbd16535d7a2bf10772cbec3a

pkg/system: normalize comment formatting Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ecb898dcb9065c8e9bcf7bb79fd160dea1c859b8

pkg/archive: normalize comment formatting Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

push time in 19 days

pull request commentmoby/moby

Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2

@thaJeztah I fixed it. Additionally, I hand-copied the test changes from #39966.

dperny

comment created time in 19 days

push eventdperny/docker

Drew Erny

commit sha e5797a099827cdd7a117007e0aa5216222e3fb69

Bump swarmkit to 24fb4cfe8af56803640180c5592bf32da732ced2 Bumps the vendoring of github.com/docker/swarmkit to the above commit, which is the current master at commit time. Most notably, this includes a change making the ingress network respect the default address pool. Because of this change, a change to network integration tests was needed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 19 days

push eventdocker/swarmkit

Drew Erny

commit sha 5453adb444d20eeca64212ef5ece6018f7e3dcce

Revert "YOLO see if this works" This reverts commit 42085d2f8e43a3ed90ed289d3f3ed3de57837100. Turns out it's a bad idea to merge a commit with the word "YOLO" in it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 24fb4cfe8af56803640180c5592bf32da732ced2

Merge pull request #2920 from dperny/revert-yolo Revert "YOLO see if this works"

view details

push time in 22 days

PR merged docker/swarmkit

Revert "YOLO see if this works"

This reverts commit 42085d2f8e43a3ed90ed289d3f3ed3de57837100.

Turns out it's a bad idea to merge a commit with the word "YOLO" in it.

This commit, when vendored into the docker engine, causes a test to hang. See moby/moby#40309.

The thing this was attempting to fix was worked around in #2919.

/cc @thaJeztah

+7 -5

2 comments

1 changed file

dperny

pr closed time in 22 days

pull request commentmoby/moby

Bump swarmkit to d509e31c1fda18ef8224ffd19a34b4aaaff7c4da

https://github.com/docker/swarmkit/pull/2920

dperny

comment created time in 22 days

push eventdperny/swarmkit-1

Drew Erny

commit sha 5453adb444d20eeca64212ef5ece6018f7e3dcce

Revert "YOLO see if this works" This reverts commit 42085d2f8e43a3ed90ed289d3f3ed3de57837100. Turns out it's a bad idea to merge a commit with the word "YOLO" in it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 22 days

PR opened docker/swarmkit

Revert "YOLO see if this works"

This reverts commit 42085d2f8e43a3ed90ed289d3f3ed3de57837100.

Turns out it's a bad idea to merge a commit with the word "YOLO" in it.

This commit, when vendored into the docker engine, causes a test to hang. See moby/moby#40309.

The thing this was attempting to fix was worked around in #2919.

/cc @thaJeztah

+7 -5

0 comment

1 changed file

pr created time in 22 days

create barnchdperny/swarmkit-1

branch : revert-yolo

created branch time in 22 days

pull request commentmoby/moby

Bump swarmkit to d509e31c1fda18ef8224ffd19a34b4aaaff7c4da

I'm 100% sure the problem is TestDockerSwarmSuite/TestSwarmVolumePlugin

dperny

comment created time in 22 days

pull request commentmoby/moby

Bump swarmkit to d509e31c1fda18ef8224ffd19a34b4aaaff7c4da

I'm 80% sure the test failure is in TestSwarmVolumePlugin.

dperny

comment created time in 22 days

push eventdocker/swarmkit

Drew Erny

commit sha dbddc47738b0a76d4680e1ce02f49b6714e49909

Add support for updating global jobs Adds support for updating global jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 1ba945ed0525588258898d150491754e93a7b93d

Add support for updating replicated jobs Adds support for updating replicated jobs by adding code to shut down tasks belonging to previous job iterations. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2bbdc74c4948c2885dcf1dd3a6eac81abecbbbb3

Add controlapi integration test for updating replicated jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 0c71f744ba2eaf79e780d1cf4cb9a2205d673e8a

Merge pull request #2899 from dperny/jobs-updates [Jobs] Support job updates

view details

push time in 22 days

PR merged docker/swarmkit

[Jobs] Support job updates

Adds code to put old tasks in the Shutdown state when jobs are updated, allowing updates to proceed correctly.

+160 -6

1 comment

5 changed files

dperny

pr closed time in 22 days

push eventdocker/swarmkit

Drew Erny

commit sha feab4999c1fccb2bd75af40ea81f631c46c237e1

Update ListServiceStatuses for Jobs Updates the ListServiceStatuses RPC to work with jobs. Includes adding a new field to the responses showing the number of completed Tasks in a job. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 5b97b0b0ed077cb27b71a6d771bbe6cbc3a7bddb

Merge pull request #2916 from dperny/jobs-service-status [Jobs] Update ListServiceStatuses for Jobs

view details

push time in 22 days

PR merged docker/swarmkit

[Jobs] Update ListServiceStatuses for Jobs

<!-- Please make sure you've read and understood our contributing guidelines; https://github.com/docker/swarmkit/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with git commit -s **

For additional information on our contributing process, read our contributing guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx"

Please provide the following information: -->

- What I did

Updates the ListServiceStatuses RPC to work with Jobs

- How I did it

Added a new field to the response, CompletedTasks, which is the number of tasks of the current job iteration completed.

- How to test it

New code is unit tested.

+374 -178

1 comment

5 changed files

dperny

pr closed time in 22 days

push eventdocker/swarmkit

Drew Erny

commit sha 004962fa3d76cbaf5c65289e24c7337ba035179e

Remove t.Log from mock task controller Previous, worker_test.go included a call to t.Log in the mock task controller. This presented a problem: because goroutines spawned by a test are not exited when the test completes, any holdover goroutines, like the one spawned to close the task controller, live on. This creates a problem when that goroutine calls t.Log on a test that has already completed. This t.Log call doesn't add enough information to bother worrying about. Further, these goroutines, though they may outlive the test, don't actually affect anything or matter, so there's no rush to fix the test to ensure all goroutines exit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 123bb7489afa2eef84a4e52d966d44ffa2792055

Merge pull request #2919 from dperny/remove-test-log Remove t.Log from mock task controller

view details

push time in 22 days

PR merged docker/swarmkit

Remove t.Log from mock task controller

Previous, worker_test.go included a call to t.Log in the mock task controller. This presented a problem: because goroutines spawned by a test are not exited when the test completes, any holdover goroutines, like the one spawned to close the task controller, live on. This creates a problem when that goroutine calls t.Log on a test that has already completed.

This t.Log call doesn't add enough information to bother worrying about. Further, these goroutines, though they may outlive the test, don't actually affect anything or matter, so there's no rush to fix the test to ensure all goroutines exit.

This ought to fix a frequently observed test flake.

+4 -8

1 comment

1 changed file

dperny

pr closed time in 22 days

push eventdperny/docker

HuanHuan Ye

commit sha 203ba72fc549b5ce306e78030579300f6e5ffe10

integration-cli: Fix `SA1019: httputil.ClientConn is deprecated` Rewrite sockRequestHijack to requestHijack which use writable Transport's Response.Body to replace deprecated hijacked httputil.ClientConn. ``` // As of Go 1.12, the Body will also implement io.Writer // on a successful "101 Switching Protocols" response, // as used by WebSockets and HTTP/2's "h2c" mode. Body io.ReadCloser ```. TestPostContainersAttach and TestExecResizeImmediatelyAfterExecStart replace all sockRequestHijack to requestHijack. Signed-off-by: HuanHuan Ye <logindaveye@gmail.com>

view details

vikrambirsingh

commit sha 25c5a55261c3a856264c19d3b252351566a210a6

Check for OS Type and skip within the test Premature check for OS type means that the test will never even get to run on other OS types. This will cause it to be always flagged as a failure on such OS types. Signed-off-by: vikrambirsingh <vikrambir.singh@docker.com>

view details

Sebastiaan van Stijn

commit sha 4cdb796b5453d4c2fe7dde85cadd2a56725e3452

pkg/system/windows: remove obsolete comment Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 33d8492ce413780e4e339631386541410677ecf3

pkg/system/windows: remove unnecessary conversions Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 81f9edc7b0efce763ad89a7b62f0546e5a12aff6

pkg/parsers: remove use of deprecated system.GetOSVersion() Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 044b74e33bc499e360d1793f4c0d5975032c2d6d

daemon: remove use of deprecated system.GetOSVersion() Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f572df7c22c1f48b9a2ddd51519797a5b6a2eda4

pkg/system: deprecate constants in favor of golang.org/x/sys/windows Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f22ff19668e220fcc06182ca3dae14fd086e0a4a

builder: remove use of deprecated pkg/system constants Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 9aa55d97e564329f9b1c5b0b2c09c5f2bc1cc578

bump containerd/go-runc a2952bc25f5116103a8b78f3817f6df759aa7def full diff: https://github.com/containerd/go-runc/compare/e029b79d8cda8374981c64eba71f28ec38e5526f...a2952bc25f5116103a8b78f3817f6df759aa7def - containerd/go-runc#56 fix typo in successfully Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 90aa0901da11582da7639cc1ab202260183799e0

api/server/router: use consts for HTTP methods Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Olli Janatuinen

commit sha 446fe7f263e26a5d8817770c0c4250175d4f528c

Make sure that layers are removed from layerdb after succefull layer removal Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>

view details

Sebastiaan van Stijn

commit sha 5bb4f4818bebd7fdc243c35f7f0db86dc11eb0d6

libcontainerd: move hcsshim import to windows-only file This reduces the dependency-graph when building packages for Linux only. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

liuxiaodong

commit sha ee5aa536d16269a48a57aebc8c1d4a2732031e68

update vendor golang.org/x/sys to 6d18c012aee9febd81bbf9806760c8c4480e870d full diff: https://github.com/golang/sys/compare/c990c680b611ac1aeb7d8f2af94a825f98d69720...6d18c012aee9febd81bbf9806760c8c4480e870d execute "docker run -it {image name} /bin/sh" no response on mip64el platform; EpollEvent struct define lack a field named PadFd; add the field "PadFd"; execute docker run -it {image} /bin/sh for test after rebuilded. Signed-off-by: liuxiaodong <liuxiaodong@loongson.cn> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Akihiro Suda

commit sha 658723badd2fe2db6aacfddec17d78df2997a905

rootless: fix proxying UDP packets UDP reply packets were not proxied: https://github.com/rootless-containers/rootlesskit/issues/86 The issue was fixed in RootlessKit v0.7.1: https://github.com/rootless-containers/rootlesskit/pull/87 Full changes since v0.7.0: https://github.com/rootless-containers/rootlesskit/compare/v0.7.0...v0.7.1 Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Brian Goff

commit sha f9c8717d878ff2277c71d0ebc4abf213f96a9dc2

Merge pull request #40268 from olljanat/garbage-collector-remove-layerdb-values Make sure that layers are removed from layerdb after succefull layer …

view details

Akihiro Suda

commit sha 8b8c01dd87e85666aabe4c50e4a0e860106b7ee3

Merge pull request #40074 from yedamao/fix-integration-cli integration-cli: Fix `SA1019: httputil.ClientConn is deprecated`

view details

Tianon Gravi

commit sha 3f2d1eb95de61a1a8cfcd1ffdea837cc868c7953

Merge pull request #40317 from AkihiroSuda/bump-up-rootlesskit rootless: fix proxying UDP packets

view details

Tibor Vass

commit sha a9416c67da9fdacefa909d86e9271aeb6a9a9f05

Merge pull request #40259 from thaJeztah/more_constants api/server/router: use consts for HTTP methods

view details

Sebastiaan van Stijn

commit sha c8ecedbde98240f6d1cb330ca81299d306e6ee1c

Revert "hack: temporarily disable deprecate-integration-cli" This reverts commit 5f67568bfd08a10f51a819cae8696f591881f26b. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e21019fa51f192022c89b97b8dd51c1046b622ac

hack/validate/default: fix minor linting issue Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

push time in 22 days

PR opened docker/swarmkit

Remove t.Log from mock task controller

Previous, worker_test.go included a call to t.Log in the mock task controller. This presented a problem: because goroutines spawned by a test are not exited when the test completes, any holdover goroutines, like the one spawned to close the task controller, live on. This creates a problem when that goroutine calls t.Log on a test that has already completed.

This t.Log call doesn't add enough information to bother worrying about. Further, these goroutines, though they may outlive the test, don't actually affect anything or matter, so there's no rush to fix the test to ensure all goroutines exit.

This ought to fix a frequently observed test flake.

+4 -8

0 comment

1 changed file

pr created time in a month

create barnchdperny/swarmkit-1

branch : remove-test-log

created branch time in a month

pull request commentdocker/swarmkit

[WIP] Cleanup routine for old tasks.db tasks

I have honestly no idea what the purpose of the task database is, so I'm a bit nervous to merge code related to it. I'll spend some time looking into it and give you an answer, though. I know that this is a rather disruptive problem for most users.

olljanat

comment created time in a month

create barnchdperny/swarmkit-1

branch : jobs-fix-todos

created branch time in a month

push eventdperny/docker

Drew Erny

commit sha ced23b392e913fb8de68823d80e345e8b2c66de1

Bump swarmkit to d509e31c1fda18ef8224ffd19a34b4aaaff7c4da Bumps the vendoring of github.com/docker/swarmkit to the above commit, which is the current master at commit time. Most notably, this includes a change making the ingress network respect the default address pool. Because of this change, a change to network integration tests was needed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

issue openeddocker/cli

Swarm Jobs CLI

Swarm jobs is coming, and it needs the CLI designed.

Behavior

Currently, internally and from the API, Jobs are presented as a different Service mode. This is because they follow mostly the same rules as Services, with the key distinction that the Tasks spawned by a Job are designed to enter a terminal state of Complete and stop executing, instead of being restarted.

There are two modes for job, ReplicatedJob, and GlobalJob, which are analogous to Replicated and Global services.

A ReplicatedJob has two parameters: the maximum number of concurrent Tasks (MaxConcurrent) to execute (analogous to the Replicas parameter of a replicated Service) and the total number of desired completed Tasks (TotalCompletions). A running ReplicatedJob executes up to MaxConcurrent Tasks at the same time, and starts a new Task every time one is completed, until the sum of the Running and Completed tasks is equal to TotalCompletions.

A GlobalJob has no parameters, like a Global service, being primarily configured through its placement constraints. It executes on Task on each Node (that matches constraints), starting new Tasks only if a running Task fails (enters a terminal state other than Completed).

Jobs cannot be updated while in progress, only interrupted and restarted. Any chance to the ServiceSpec while a job is in progress will result in stopping all of that job's Tasks, and starting a whole new set of Tasks with the new spec. This behavior can be changed in the future, but is present for now. Because jobs cannot be updated, flags specifying update parameters have no effect.

To re-execute a job that has already been executed, the user can pass --force, which will cause the job Spec to change and result in a re-execution of all tasks. Each execution of a Job, whether fresh or interrupting, increments a JobIteration version number. This number is present on Tasks belonging to a Job, and is used to differentiate which Tasks belong to the current run of the job, and which Tasks belong to previous runs.

For ReplicatedJobs, the --replicas flag can be repurposed to mean MaxConcurrent. Additionally, if no other options are specified, the value of TotalCompletions would be set equal to the value of --replicas, meaning all Tasks desired for the job would execute simultaneously. ReplicatedJobs would require an additional flag, perhaps --total or --total-completions, which could specify the desired TotalCompletions independent of the value of --replicas.

When doing an ls of jobs, instead of displaying two numbers, Running over Desired Tasks, four numbers are needed to express the state of the job: the number of Running Tasks, the number of tasks desired to be Running, the number of Tasks Completed, and the number of Tasks desired to be completed. The resulting list table would look roughly like this:

ID            NAME     MODE           REPLICAS            IMAGE
c8wgl7q4ndfd  mybatch  replicated-job  5/5 (5/10 Complete) somejobimage
dmu1ept4cxcf  redis    replicated      3/3                 redis:3.0.6
iwe3278osahj  mongo    global          7/7                 mongo:3.3

From the CLI side, there is a smallish UI decision to make with how to represent jobs. There are, as far as I can feel, three approaches

First Option: a --job flag

A flag, --job, could be added, which specifies that the service being executed is a Job-type service. This flag could be used in conjunction with the existing --mode flag to specify any of the now 4 possible modes. So, to create a job, the full command would be something like docker service create --job --mode replicated myimage

Second Option: Separate Modes

Instead of adding a --job flag, we express jobs as they are: different Service modes. This would simply add replicated-job and global-job as possible values to --mode. The command here would be docker service create --mode replicated-job myimage

Third Option: A docker job Sub-command

This is perhaps the most radical option, but has the opportunity to express the best UI. We would create a separate sub-command, importing most of the important bits from the existing docker service sub-command, and present jobs as a separate user interface object. docker service ls would only list Replicated and Global service, docker job ls would only list ReplicatedJobs and GlobalJobs. Any of the flags that are not applicable to jobs would not be present on the docker job sub-command, and the total-completions flag would not be present on the docker service sub-command. The command here would be docker job create --mode replicated myimage. In this case, the ls table would look slightly different:

ID            NAME       MODE        EXECUTING  PROGRESS  IMAGE
c8wgl7q4ndfd  mybatch    replicated  5/5        5/10      somejobimage
dmu1ept4cxcf  systask    global      0/0        6/6       someuser/updater:3.0.6
iwe3278osahj  backup     global      3/3        3/6       someuser/backup:3.3

Absent any dissenting opinion, I will go with Option 1 and add a --job flag

created time in a month

push eventdperny/docker

Drew Erny

commit sha bb94bda448d4e768e551632b7f8852da9f27aca5

WIP: add swarm jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

PR opened moby/moby

Bump swarmkit to d509e31c1fda18ef8224ffd19a34b4aaaff7c4da

Full diff: docker/swarmkit@7dded76...d509e31

changes included:

  • docker/swarmkit#2867 Only update non-terminal tasks on node removal
    • related to docker/swarmkit#2806 Fix leaking task resources when nodes are deleted
  • docker/swarmkit#2880 Bump to golang 1.12.9
  • docker/swarmkit#2886 Bump vendoring to match current docker/docker master
    • regenerates protobufs
  • docker/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network
    • fixes issue where specifying --default-addr-pool for docker swarm init is not picked up by ingress network
  • A few vendoring bumps in swarmkit.

Notably, because of docker/swarmkit#2890, a change to integration tests is needed. This change has been documented in a comment.

Supercedes #39953

/cc @thaJeztah

+14621 -4524

0 comment

19 changed files

pr created time in a month

create barnchdperny/docker

branch : bump-swarmkit

created branch time in a month

push eventdperny/docker

Drew Erny

commit sha fff938e3141fb9be45df7739b0f836a8b1620226

WIP: add swarm jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

push eventdperny/docker

Drew Erny

commit sha c01b3fe416e421fbd5025e519a99ef2850561c73

WIP: add swarm jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

PR opened docker/swarmkit

[Jobs] Update ListServiceStatuses for Jobs

<!-- Please make sure you've read and understood our contributing guidelines; https://github.com/docker/swarmkit/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with git commit -s **

For additional information on our contributing process, read our contributing guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx"

Please provide the following information: -->

- What I did

Updates the ListServiceStatuses RPC to work with Jobs

- How I did it

Added a new field to the response, CompletedTasks, which is the number of tasks of the current job iteration completed.

- How to test it

New code is unit tested.lusion in the changelog: -->

+374 -178

0 comment

5 changed files

pr created time in a month

push eventdperny/swarmkit-1

Drew Erny

commit sha feab4999c1fccb2bd75af40ea81f631c46c237e1

Update ListServiceStatuses for Jobs Updates the ListServiceStatuses RPC to work with jobs. Includes adding a new field to the responses showing the number of completed Tasks in a job. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

create barnchdperny/swarmkit-1

branch : jobs-service-status

created branch time in a month

pull request commentmoby/moby

WIP: Add support for swarm jobs

Mostly running this PR as WIP because my local dev box is giving me trouble, and I want to run against CI a bit to see if the problem is more general.

dperny

comment created time in a month

push eventdperny/docker

Drew Erny

commit sha 5290757bad2577457d3e7fdeea871566b04ead44

WIP: add swarm jobs Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

PR opened moby/moby

WIP: Add support for swarm jobs

<!-- Please make sure you've read and understood our contributing guidelines; https://github.com/moby/moby/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with git commit -s **

For additional information on our contributing process, read our contributing guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx"

Please provide the following information: -->

- What I did

WIP: Adds support for Jobs in Swarm Mode

- How I did it

Uses the feature-jobs branch of the github.com/docker/swarmkit repo as the vendoring target. This branch is not merged yet into Swarm master.

+16979 -4694

0 comment

40 changed files

pr created time in a month

push eventdperny/docker

Drew Erny

commit sha bb42843a25621eade43745699cd925c781ab65d6

WIP: add swarm jobs

view details

push time in a month

push eventdocker/swarmkit

Drew Erny

commit sha 1cd986f773fa8bef633122b091bfb3844157cacb

Update components to support jobs Updates manager components to support jobs, which have a desired state of Completed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2d735994dfc5ab64e00434bbab9ebcb64b315257

Merge pull request #2915 from dperny/jobs-update-components [Jobs] Update components to support jobs

view details

push time in a month

PR merged docker/swarmkit

[Jobs] Update components to support jobs

Updates various components in swarmkit to work with Jobs, which have Complete as the desired terminal state.

+27 -9

1 comment

4 changed files

dperny

pr closed time in a month

push eventdperny/swarmkit-1

Drew Erny

commit sha 1cd986f773fa8bef633122b091bfb3844157cacb

Update components to support jobs Updates manager components to support jobs, which have a desired state of Completed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

more