profile
viewpoint
Drew Erny dperny @docker San Francisco, CA http://www.dperny.net FLOSS Machine. Outside of my official work obligations, I am happy to help anyone contribute to the various projects I work on in any way I can! Get in touch!

create barnchdperny/stacks-1

branch : fix-manager

created branch time in 12 days

pull request commentdocker/cli

Services: use ServiceStatus on API v1.41 and up

LGTM, thanks! Adding this functionality was actually on my todo list for the sprint, so I'm sorry I didn't communicate that.

thaJeztah

comment created time in 16 days

pull request commentmoby/moby

Add support for sending down service Running and Desired task counts

bless you sebastiaan you magnificent man

dperny

comment created time in a month

push eventdocker/swarmkit

Trapier Marshall

commit sha a4e520aff16b607c0cf5558fbcfc0dc2cf41d9d8

Fix nil pointer deref in dump-snapshot --redacted container.PullOptions is nil if the service is deployed without `--with-registry-auth`. Observed on client+server 19.03.2. Signed-off-by: Trapier Marshall <trapier.marshall@docker.com>

view details

Drew Erny

commit sha 8a69c0da0d0f002c4d6b4611aa4f1e40f5ccd659

Merge pull request #2897 from trapier/dump-snapshot-redacted-nil-deref Fix nil pointer deref in dump-snapshot --redacted [FIELD-2081]

view details

Sebastiaan van Stijn

commit sha dfe3c44f35c0d845ed3934c23e63aa7006eaac64

bump dustin/go-humanize v1.0.0 full diff: https://github.com/dustin/go-humanize/compare/8929fe90cee4b2cb9deb468b51fb34eba64d1bf0...v1.0.0 probably most relevant change: - https://github.com/dustin/go-humanize/commit/64dbdae0d393b7d71480a6dace78456396b55286 Add space between the numbers and units - closes dustin/go-humanize#21 Space between numerical value and unit symbol for SI output - related: dustin/go-humanize#32 no space between bytes and units Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 12eaba61051a1bac6803735a5c473e7c0ef5c23a

bump beorn7/perks v1.0.1 full diff: https://github.com/beorn7/perks/compare/e7f67b54abbeac9c40a31de0f81159e4cafebd6a...v1.0.1 - Add go module support - Create v1.0.0 (to play better with go modules) - Lower Go requirement in go.mod Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 8eb38c8c345ac7be26dfaf2bf7de16b1cb291f17

Merge pull request #2904 from thaJeztah/bump_beorn7_perks_1.0.1 bump beorn7/perks v1.0.1

view details

Drew Erny

commit sha d509e31c1fda18ef8224ffd19a34b4aaaff7c4da

Merge pull request #2903 from thaJeztah/bump_go_humanize_1.0.0 bump dustin/go-humanize v1.0.0

view details

Drew Erny

commit sha d2ffe20e1255a510a5d4a32bb3431b0422c2e676

Add Job protos Adds protocol buffers for implementing Jobs in swarmkit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 49a205d2868a2c3cf160546bb634f7f98927426c

Add minimal Replicated Job Orchestrator and Tests Adds orchestrators for replicated and global jobs, and the basic tests. This commit exists mostly to keep the Ginkgo in mostly its own commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 110f01cb1547e6840b1d0d8ba4e1bf53c1d9edac

Add service reconcilation for replicated jobs Adds service reconciliation logic for the replicated jobs orchestrator. This code does not function in production, and is not actually called except from the tests. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 457dc357cdd8aef6a945042d5547734be635e702

Add global job orchestrator skeleton Expands the skeleton structure of the global jobs orchestrator. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha f0a0381314551d021fe933994ce1e1dc0df2ddec

Refactor replicated job orchestrator and add initialization Refactors the replicated job orchestrator to make testing simpler, and then adds initialization logic to it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 7fde6319271d97d47c0670fd8365722eebb1c214

Add store event logic to replicated jobs orchestrator Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha fdee6fb82dd53c3214980d50a6fde04d29e55742

Refactor global job orchestrator Refactors the global job orchestrator along the same lines as the replicated job orchestrator, in order to better decouple the event-driven orchestrator logic from the reconciliation logic. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 9b8cc5af14cbbbfe2a18978198dc0ba4d3b11e17

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha bb23c6f1d80130b5d7bccc2d79f1a21a2ec23af0

Add controlapi support for job services Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 61da429ff350406b65a8383bc90b75a8e993cf9d

Wire up jobs orchestrator to manager Adds the jobs orchestrator to the swarmkit manager. Jobs orchestrator will now start and run with the manager. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha e65b4a3e37a0b7cea433676e31798240bc839798

Add jobs orchestrator to controlapi integration tests Adds the beginnings of the integration tests between the controlapi and the jobs orchestrator. These tests don't actually check much more than creation right now, as update handling for the jobs orchestrator is pending, but further tests will be able to leverage the groundwork here to a high degree. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 8927a173bb51e586f64f3547c9626c57ba757189

Support restarting failing jobs tasks In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed. Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1. Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed. Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

PR merged docker/swarmkit

[Jobs] Support restarting failing jobs tasks

In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed.

Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1.

Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed.

Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed.

This does not yet fully include code necessary to handle node updates.

+792 -117

1 comment

9 changed files

dperny

pr closed time in a month

push eventdocker/swarmkit

Drew Erny

commit sha bfd10e1d35b84c5792b7ccc52dd7699ea762df09

Support restarting failing jobs tasks In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed. Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1. Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed. Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha d9a581f21ec88534dced0feea70ad6046ec9892d

Merge pull request #2907 from dperny/jobs-restart-supervisor [Jobs] Support restarting failing jobs tasks

view details

push time in a month

push eventdperny/docker

Kamil Domański

commit sha 186e22d26e7cf6e4d6f718257c653e496850914a

include IPv6 address of linked containers in /etc/hosts Signed-off-by: Kamil Domański <kamil@domanski.co>

view details

Brian Goff

commit sha a5f237c2b54177ffe45cf371461db1892e092452

Use FILE_SHARE_DELETE for log files on Windows. This fixes issues where one goroutine tries to delete or rename a file while another goroutine has the file open (e.g. a log reader). Signed-off-by: Brian Goff <cpuguy83@gmail.com>

view details

Devon Estes

commit sha cb2a36a89c1fb73b5b9ea3e9df8977f2b3139ad1

Add ability to handle index acknowledgment with splunk log driver Previously there was no way for the splunk log driver to work if index acknowledgment was set on the HEC, and it would in fact fail silently. This will now allow users to specify if index acknowledgment is set and will work with that setting. Signed-off-by: Devon Estes <devon.c.estes@gmail.com>

view details

Kunal Kushwaha

commit sha 8b7bbf180fc65013bc9ec0269b4a475d3eb038ee

builder entitlements configutation added. buildkit supports entitlements like network-host and security-insecure. this patch aims to make it configurable through daemon.json file. by default network-host is enabled & secuirty-insecure is disabled. Signed-off-by: Kunal Kushwaha <kunal.kushwaha@gmail.com>

view details

Tibor Vass

commit sha 023b072288eab3c9e768d4aeeb917f27f06034c7

homedir: add cgo or osusergo buildtag constraints for unix This is to ensure that users of the homedir package cannot compile statically (CGO_ENABLED=0) without also setting the osusergo build tag. Signed-off-by: Tibor Vass <tibor@docker.com>

view details

HuanHuan Ye

commit sha 8498ee7514cfb166197f637cc3dd69194de5f16b

Fix pkg/pools staticcheck SA6002 change bufferPool use pointer instead byte slice Signed-off-by: HuanHuan Ye <logindaveye@gmail.com>

view details

Sebastiaan van Stijn

commit sha f4c172e6b99fbce874a2a69f84e4560c8e310d33

integration-cli: fix golint (copy/paste whoops) These were accidentally wrong due to a sloppy copy/paste issue. Interestingly, CI passed on the PR that added it (6397dd4d3123e0a1b89298d0a2cfe5388410a74f), possibly because of this issue, it stopped linting? WARN [runner/golint] Golint: can't lint 4 files: no file name for file &{Doc:<nil> Package:23044677 Name:quota Decls:[0xc02cc3fa40 0xc02cc3fac0 0xc02cc3fb40 0xc02cc3fbc0 0xc02cc62ab0 0xc02cc62c00] Scope:scope 0xc02cc5c340 { var ErrQuotaNotSupported type errQuotaNotSupported } Imports:[0xc02cc62930] Unresolved:[errdefs nil string] Comments:[0xc02cbc9ae0 0xc02cbc9c60]} Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 5c891ea9ca9084e81429395d99fdad451d2cffaf

integration-cli: fix DockerNetworkSuite not being run Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 404d87ec6946aaa9c130b64c0c75514a2fcd50c0

AppArmor: add missing rules for running in userns Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 279ddb31ef03c48436c17e1a3e96df84fd64e559

testutil: update WithStorageDriver to use daemon.Option as return type Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ce2e8e37d05a2d80241caadbc8a7489461371f43

testutil: update WithTestLogger to use daemon.Option as return type Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 554d9cec257335cdd6af60fa747c8bd6095922b6

testutil: update WithExperimental signature to be a daemon.Option Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f60d6ee4bc742ff12ff295507e73d24407b64fba

testutil: update WithInitsignature to be a daemon.Option Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 23457f05a9e15e2c9d78c0df407783ed1b1fc33b

Update mailmap and authors Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kirill Kolyshkin

commit sha 09ee529cfe1ffb3b6625a6797ecd5705d134f324

Merge pull request #40016 from thaJeztah/fix_dockernetworksuite integration-cli: fix DockerNetworkSuite not being run

view details

Kirill Kolyshkin

commit sha 58653d097ceaaa18a34336bbeae750b37e0d3ceb

Merge pull request #40014 from thaJeztah/fix_golint_copy_pasta integration-cli: fix golint (copy/paste whoops)

view details

Tibor Vass

commit sha 0f9c4fa00b7d0843d7a8aa51e5b96c6c2e1cae72

Merge pull request #40009 from yedamao/fix-pkg-pool Fix pkg/pools staticcheck SA6002

view details

Sebastiaan van Stijn

commit sha 0b5140dd66d6f1662c1d65675b49cd198301ac35

Merge pull request #39994 from tiborvass/homedir-buildtags homedir: add cgo or osusergo buildtag constraints for unix

view details

Kirill Kolyshkin

commit sha b93f68ab4c5b82549691d1050e94d9b138eb9342

Merge pull request #40013 from thaJeztah/fix_daemon_ops_type testutil: change some remaining options to be a daemon.Option

view details

Tibor Vass

commit sha 9732185e073789529ad285e7b97646b6ab840f8c

Merge pull request #39144 from kunalkushwaha/builder-entitilement-confg builder entitlements configuration added.

view details

push time in a month

push eventdocker/swarmkit

Olli Janatuinen

commit sha f970ef971e3bd409315bf8bb7d964035007697ec

Increased wait time on test utils WaitForCluster and WatchTaskCreate Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com> (cherry picked from commit 5f167cab731ee75bc6cf3888fd2a4a5b5194f924) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 2d28e8fda58d489bd17b1b92c60c8a71ebae762a

attempt to fix weirdly broken tests Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit be26111c4a48c44fac04c17c69fd2504aea6db91) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha f5adf368c385661a37ac17304f821aefa68b8f3c

Fix flaky tests It is likely that a large portion of test flakiness, especially in CI, comes from the fact that swarmkit components under test are started in goroutines, but those goroutines never have an opportunity to run. This adds code ensuring those goroutines are scheduled and run, which should hopefully solve many inexplicably flaky tests. Additionally, increased test timeouts, to hopefully cover a few more flaky cases. Finally, removed direct use of the atomic package, in favor of less efficient but higher-level mutexes. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 06a356671bc11e4fd5d754f257f9c5f93ec5c563) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 5fca4d731cd6c28628fbeade010efee518309996

Fix update out of sequence A simple but old error has recently become evident. Due to the fact that we read an object and then write it back across the boundaries of a transaction, it is possible for the task object to have changed in between transactions. This would cause the attempt to write out the old task to suffer an "Update out of sequence" error. This fix simply reads the latest version of the task back out within the boundary of a transaction to avoid the race. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit d68ac46e3b11d7384472677d210bb0ce941284dc) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 91bc626d5ed0fd23820b30aa37f4adb87ed4b9da

Merge pull request #2902 from thaJeztah/18.03_backport_fix_update_out_of_sequence [18.03 backport] Fix update out of sequence

view details

push time in a month

PR merged docker/swarmkit

[18.03 backport] Fix update out of sequence

relates to

  • https://github.com/moby/moby/pull/39531 integration-cli: fix swarm tests flakiness

Backports of

  • https://github.com/docker/swarmkit/pull/2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
  • https://github.com/docker/swarmkit/pull/2771 Allow using Configs as CredentialSpecs
    • second commit only (attempt to fix weirdly broken tests)
  • https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
  • https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence
# https://github.com/docker/swarmkit/pull/2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
git cherry-pick -s -S -x 5f167cab731ee75bc6cf3888fd2a4a5b5194f924

# second commit from https://github.com/docker/swarmkit/pull/2771 Allow using Configs as CredentialSpecs
git cherry-pick -s -S -x be26111c4a48c44fac04c17c69fd2504aea6db91

# https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
git cherry-pick -s -S -x 06a356671bc11e4fd5d754f257f9c5f93ec5c563

# https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence
git cherry-pick -s -S -x d68ac46e3b11d7384472677d210bb0ce941284dc

Minor conflict in imports when cherry-picking https://github.com/docker/swarmkit/pull/2808 (06a356671bc11e4fd5d754f257f9c5f93ec5c563) because is not in this branch (which would require updating the gRPC version that's used; see https://github.com/docker/swarmkit/pull/2827#issuecomment-462920863)

diff --cc manager/orchestrator/replicated/update_test.go
index ccc084b8,53d6da72..00000000
--- a/manager/orchestrator/replicated/update_test.go
+++ b/manager/orchestrator/replicated/update_test.go
@@@ -1,7 -1,8 +1,12 @@@
  package replicated
  
  import (
++<<<<<<< HEAD
 +      "sync/atomic"
++=======
+       "context"
+       "sync"
++>>>>>>> 06a35667... Fix flaky tests
        "testing"
        "time"
+171 -70

5 comments

8 changed files

thaJeztah

pr closed time in a month

push eventdocker/swarmkit

Olli Janatuinen

commit sha 6a454da90ce5065a76f57734f15f62ea59e95604

Increased wait time on test utils WaitForCluster and WatchTaskCreate Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com> (cherry picked from commit 5f167cab731ee75bc6cf3888fd2a4a5b5194f924) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 6a2573e3e576827453ca3a8d37c8edfbb47c9721

attempt to fix weirdly broken tests Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit be26111c4a48c44fac04c17c69fd2504aea6db91) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 6db8d2da5d34e8fdc68fc2172f093269c3a9c046

Fix flaky tests It is likely that a large portion of test flakiness, especially in CI, comes from the fact that swarmkit components under test are started in goroutines, but those goroutines never have an opportunity to run. This adds code ensuring those goroutines are scheduled and run, which should hopefully solve many inexplicably flaky tests. Additionally, increased test timeouts, to hopefully cover a few more flaky cases. Finally, removed direct use of the atomic package, in favor of less efficient but higher-level mutexes. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 06a356671bc11e4fd5d754f257f9c5f93ec5c563) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha ba217b76c4c856b8d727ae46a88859efbb170c7f

add golangci-lint gometalinter is deprecated, and golangci-lint is its recommended successor. This commit adds golangci-lint as the linter for swarmkit. In addition, golangci-lint found a few issues in the code that were not yet identified, and so those issues have been fixed. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 27c2d27e23e76243bb49ec5d803a6e40b3f96f7a) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

nmengin

commit sha 177caa5af91a2cacbccf42426145fdb464a48c8b

Set bigger grpc value to initialize connection broker Signed-off-by: nmengin <nicolas@containo.us> (cherry picked from commit 127e816ed8c8de4c981e02c9d0b45c70d3baa824) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha bfce89e3eeb7ff6cd3e2dc9b887e4a291f13a2b0

Fix update out of sequence A simple but old error has recently become evident. Due to the fact that we read an object and then write it back across the boundaries of a transaction, it is possible for the task object to have changed in between transactions. This would cause the attempt to write out the old task to suffer an "Update out of sequence" error. This fix simply reads the latest version of the task back out within the boundary of a transaction to avoid the race. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit d68ac46e3b11d7384472677d210bb0ce941284dc) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 5c86095cef3ff480e69486da50f18fd1b3a0de78

Merge pull request #2900 from thaJeztah/18.09_backport_fix_update_out_of_sequence [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets

view details

push time in a month

PR merged docker/swarmkit

[18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets

~built on top of https://github.com/docker/swarmkit/pull/2836 ([18.09 backport] Switch to go 1.11)~ rebased

relates to

  • https://github.com/moby/moby/pull/39531 integration-cli: fix swarm tests flakiness
  • https://github.com/docker/engine/pull/346 [18.09 backport] integration-cli: fix swarm tests flakiness

Backports of

  • https://github.com/docker/swarmkit/pull/2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
  • https://github.com/docker/swarmkit/pull/2771 Allow using Configs as CredentialSpecs
    • second commit only (attempt to fix weirdly broken tests)
  • https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
  • https://github.com/docker/swarmkit/pull/2866 Swap gometalinter for golangci-lint
  • https://github.com/docker/swarmkit/pull/2869 Increase max recv gRPC message size to initialize connection broker
  • related / similar to https://github.com/moby/moby/pull/38103 / https://github.com/docker/engine/pull/102 cluster: set bigger grpc limit for array requests
  • related / similar to https://github.com/moby/moby/pull/39306 Increase max recv gRPC message size for nodes and secrets
  • fixes https://github.com/docker/swarmkit/issues/2733 Error generated when messages size is too big
  • https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence

Cherry-pick were clean, no conflicts

# https://github.com/docker/swarmkit/pull/2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
git cherry-pick -s -S -x 5f167cab731ee75bc6cf3888fd2a4a5b5194f924

# second commit from https://github.com/docker/swarmkit/pull/2771 Allow using Configs as CredentialSpecs
git cherry-pick -s -S -x be26111c4a48c44fac04c17c69fd2504aea6db91

# https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
git cherry-pick -s -S -x 06a356671bc11e4fd5d754f257f9c5f93ec5c563

# https://github.com/docker/swarmkit/pull/2866 Swap gometalinter for golangci-lint
git cherry-pick -s -S -x 27c2d27e23e76243bb49ec5d803a6e40b3f96f7a

# https://github.com/docker/swarmkit/pull/2869 Increase max recv gRPC message size to initialize connection broker
git cherry-pick -s -S -x 127e816ed8c8de4c981e02c9d0b45c70d3baa824

# https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence
git cherry-pick -s -S -x d68ac46e3b11d7384472677d210bb0ce941284dc
+202 -105

7 comments

20 changed files

thaJeztah

pr closed time in a month

push eventdocker/stacks

Jason Schroeder

commit sha 9d297033c750b3976264b285190772be7ac86af8

The new algorithm with multiple and layered 100% code coverage tests. The tests represent high-level API scenarios in order to get the coverage and roughly document the what-if scenarios. What remains: 1. The dispatcher and eventing relationship needs some discussion. 2. Names for things could be changed 3. There was some discussion about special cases for changing networks for a service. 4. The tests need some work to fix the descriptions, round out checks and fix legibility

view details

Jason Schroeder

commit sha 9af83fb4991c94c67ece7e3b5933bace81f9a692

Missing linter fixes

view details

Jason Schroeder

commit sha b6898ef5ee22d9f522c46f340e4ea5fba2579068

Updated apline version to shake off linter issues

view details

Drew Erny

commit sha 43d1ab5f3fe21dc65021cfbbae4e330b74d240c5

Merge pull request #61 from undertheflowerpot/FirstTry The new algorithm with multiple and layered 100% code coverage tests.

view details

push time in a month

PR merged docker/stacks

The new algorithm with multiple and layered 100% code coverage tests.

The tests represent high-level API scenarios in order to get the coverage and roughly document the what-if scenarios.

What remains:

  1. The dispatcher and eventing relationship needs some discussion.
  2. Names for things could be changed
  3. There was some discussion about special cases for changing networks for a service.
  4. The tests need some work to fix the descriptions, round out checks and fix legibility
+3427 -878

1 comment

27 changed files

undertheflowerpot

pr closed time in a month

pull request commentdocker/stacks

The new algorithm with multiple and layered 100% code coverage tests.

Sorry I haven't reviewed this yet. It's very thick. I've looked at it now, though, and it looks good to me.

undertheflowerpot

comment created time in a month

push eventdperny/docker

Drew Erny

commit sha 87ba762c5f8a35c9970f37b448b4486a86b9e4f6

Add support for sending down service Running and Desired task counts Adds a new ServiceStatus field to the Service object, which includes the running and desired task counts. This new field is gated behind a "status" query parameter. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

pull request commentmoby/moby

Add support for sending down service Running and Desired task counts

I've rebased this PR but I'm not clear on what's wrong. Something with swagger is failing, but it's panicking, not outputting useful information about the failure.

dperny

comment created time in a month

PR merged docker/swarmkit

[18.09 backport] Switch to go 1.11

backport of;

  • https://github.com/docker/swarmkit/pull/2752 Switch to go 1.11
  • daf87201f686710974b282d4c0b0c8422e55dac9 Bump Golang 1.11.13
    • taken from #2880

Whitespace changes are caused by the fact that gofmt from go-1.11 uses a different heuristic as to how to format the file, making the source code that was OK for go-1.10 causing a warning with go-1.11.

NOTE this whitespace change makes the gofmt from go-1.10 complain, so please upgrade your golang.

[v2: regen pb files]

Signed-off-by: Kir Kolyshkin kolyshkin@gmail.com (cherry picked from commit fd2d7f2ef925282e8cd2920efa253151749102ad) Signed-off-by: Sebastiaan van Stijn github@gone.nl

<!-- Please make sure you've read and understood our contributing guidelines; https://github.com/docker/swarmkit/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with git commit -s **

For additional information on our contributing process, read our contributing guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx"

Please provide the following information: -->

- What I did

- How I did it

- How to test it

- Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: -->

+25 -25

5 comments

10 changed files

thaJeztah

pr closed time in a month

push eventdocker/swarmkit

Kir Kolyshkin

commit sha 8028c5a78796373be3d3f664055b8d31847a44c0

Switch to go 1.11 Whitespace changes are caused by the fact that gofmt from go-1.11 uses a different heuristic as to how to format the file, making the source code that was OK for go-1.10 causing a warning with go-1.11. NOTE this whitespace change makes the gofmt from go-1.10 complain, so please upgrade your golang. [v2: regen pb files] Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit fd2d7f2ef925282e8cd2920efa253151749102ad) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 26b47ab6f34db170dd5225b384672a17765dc47b

Bump Golang 1.11.13 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit daf87201f686710974b282d4c0b0c8422e55dac9) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ace6803427caadadc5fa130a09f2c968d8dd23c7

Bump Golang 1.12.9 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit be528e80a841ae89e1bd7539201f12c4de496ab3) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 2da1ccb80da73e8eb744ed03019b09ef82575a3a

Update tests for new output with Go 1.12 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 43fac9c8049f7dd2ae42d87f60d2a88f86f63853) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 5cfdd802269822c795774ec8342f0e95dd0fa129

YOLO see if this works ``` panic: Log in goroutine after TestWorkerUpdate has completed goroutine 1175 [running]: testing.(*common).logDepth(0xc00019a100, 0xc0000396a0, 0x1d, 0x3) /usr/local/go/src/testing/testing.go:634 +0x51a testing.(*common).log(...) /usr/local/go/src/testing/testing.go:614 testing.(*common).Log(0xc00019a100, 0xc00006fb00, 0x1, 0x1) /usr/local/go/src/testing/testing.go:642 +0x79 github.com/docker/swarmkit/agent.(*mockTaskController).Remove(0xc00020a780, 0x13b6f40, 0xc0000b4010, 0x13b6f40, 0xc0000b4010) /home/circleci/.go_workspace/src/github.com/docker/swarmkit/agent/worker_test.go:604 +0x8e github.com/docker/swarmkit/agent.reconcileTaskState.func1(0xc0004dac60) /home/circleci/.go_workspace/src/github.com/docker/swarmkit/agent/worker.go:270 +0x137 created by github.com/docker/swarmkit/agent.reconcileTaskState /home/circleci/.go_workspace/src/github.com/docker/swarmkit/agent/worker.go:313 +0x1a6f FAIL github.com/docker/swarmkit/agent 4.244s make: *** [coverage] Error 1 Exited with code 2 ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 42085d2f8e43a3ed90ed289d3f3ed3de57837100) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha deb207126471705c379e8a126bbd61c2c2876056

Merge pull request #2901 from thaJeztah/18.09_backport_bump_golang_1.12.9 [18.09 backport] Bump to golang 1.12.9

view details

push time in a month

PR merged docker/swarmkit

[18.09 backport] Bump to golang 1.12.9

built on top of / replaces https://github.com/docker/swarmkit/pull/2836 [18.09 backport] Switch to go 1.11 closes https://github.com/docker/swarmkit/pull/2836 [18.09 backport] Switch to go 1.11

backports of:

  • https://github.com/docker/swarmkit/pull/2880 Bump to golang 1.12.9

cherry-picks were clean; no conflicts

# https://github.com/docker/swarmkit/pull/2880 Bump to golang 1.12.9
git cherry-pick -s -S -x \
	be528e80a841ae89e1bd7539201f12c4de496ab3 \
	43fac9c8049f7dd2ae42d87f60d2a88f86f63853 \
	42085d2f8e43a3ed90ed289d3f3ed3de57837100
+34 -36

5 comments

12 changed files

thaJeztah

pr closed time in a month

push eventdperny/docker

Tibor Vass

commit sha a8608b5b67c77169276863cf31c6bc89a9ab3d8c

homedir: remove idtools and libcontainer's user package dependencies About github.com/opencontainers/runc/libcontainer/user: According to https://github.com/opencontainers/runc/commit/195d8d544aca731301d8116821c75e1244dd5a0c this package has two functions: - Have a static implementation of user lookup, which is now supported in the os/user stdlib package with the osusergo build tag, but wasn't at the time. - Have extra functions that os/user doesn't have, but none of those are used in homedir. Since https://github.com/moby/moby/pull/11287, homedir depended directly on libcontainer's user package for CurrentUser(). This is being replaced with os/user.Current(), because all of our static binaries are compiled with the osusergo tag, and for dynamic libraries it is more correct to use libc's implementation than parsing /etc/passwd. About github.com/docker/docker/pkg/idtools: Only dependency was from GetStatic() which uses idtools.LookupUID(uid). The implementation of idtools.LookupUID just calls to github.com/opencontainers/runc/libcontainer/user.LookupUid or fallbacks to exec-ing to getent (since https://github.com/moby/moby/pull/27599). This patch replaces calls to homedir.GetStatic by homedir.Get(), opting out of supporting nss lookups in static binaries via exec-ing to getent for the homedir package. If homedir package users need to support nss lookups, they are advised to compile dynamically instead. Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Sam Whited

commit sha ae0a878b863511d0455413b137d0f79b56e5d2f4

testutil, integration: untangle image dependency Signed-off-by: Sam Whited <sam@samwhited.com>

view details

Chris Price

commit sha c21a3cf432bd89f9ceb5724b8a90f30f789433a5

Add variant to image.Image and legacy builder This commit adds the image variant to the image.(Image) type and updates related functionality. Images built from another will inherit the OS, architecture and variant. Note that if a base image does not specify an architecture, the local machine's architecture is used for inherited images. On the other hand, the variant is set equal to the parent image's variant, even when the parent image's variant is unset. The legacy builder is also updated to allow the user to specify a '--platform' argument on the command line when creating an image FROM scratch. A complete platform specification, including variant, is supported. The built image will include the variant, as will any derived images. Signed-off-by: Chris Price <chris.price@docker.com>

view details

Tibor Vass

commit sha f3d8b8ae74804ba6a0386186c1c65bd1d556075a

Jenkinsfile: move integration step cleanup to amd64 where it was intended to be Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Sebastiaan van Stijn

commit sha 0f0e3163b5278e7eb047313d22c645fbaee26884

daemon/info: remove use of docker/go-connections The `docker/go-connections` package was only used for a quite generic utility. This patch removes the use of the package by replacing the `GetProxyEnv` utility with a local function that's based on the one in golang.org/x/net/http/httpproxy: https://github.com/golang/net/blob/c21de06aaf072cea07f3a65d6970e5c7d8b6cd6d/http/httpproxy/proxy.go#L100-L107 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha d31642c6b5fcd5d1bfddf423189f91f70327840c

Merge pull request #39984 from tiborvass/fix-jenkins-cleanup-amd64-integration Jenkinsfile: move integration step cleanup to amd64 where it was intended to be

view details

John Howard

commit sha 8988448729dbff6a5bd308f93c40ae702216468d

Remove refs to jhowardmsft from .go code Signed-off-by: John Howard <jhoward@microsoft.com>

view details

Sebastiaan van Stijn

commit sha 4e9cffae051a85e045d72f1fc048846edfb31db4

Merge pull request #39988 from microsoft/jjh/jjh-gocode Remove refs to jhowardmsft from .go code

view details

Sebastiaan van Stijn

commit sha b323c6e9aeb6c18b25cd77b6ecafc761785b8c65

Jenkinsfile: update references to repositories that were moved Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 83fd212f2cb71aae2f4a5a60c893c2bd01e59b72

Dockerfile.windows: update references to repositories that were moved Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 5175ed54e58d2027e54571ef384eed54626f6c89

hack/ci/windows.ps1 update references to repositories that were moved Also updated the related docs. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e553a03627c20cbedb97f5f8cb4b02a0ff47c54e

AppArmor: remove rules for linkgraph.db SQLite database Commit 0f9f99500c40f2a46682967ca358cd2346fd5e13 removed the use of SQLite for managing container links, and commit f8119bb7a76b5c42defb6e0a2dc67bd77ad29a5e removed the migration tool, and SQLite dependency. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Felipe Ruhland

commit sha 8107d4485216835a1347839d42cd1f657a9b37d8

Fix Engine API version history typo Signed-off-by: Felipe Ruhland <felipe.ruhland@gmail.com>

view details

John Howard

commit sha fd820c4d6526da4df07ae7456c8d46f20a61f26d

Merge pull request #39990 from thaJeztah/update_moved_repositories Update links/references to transferred repositories

view details

Tibor Vass

commit sha 39e6def2194045cb206160b66bf309f486bd7e64

Merge pull request #39177 from pricec/image-variant Add variant to image.Image and legacy builder

view details

Tibor Vass

commit sha b4c9b4abb1213ab17565f309cb72470b720ef33c

Merge pull request #39980 from SamWhited/image_dependency testutil, integration: untangle image dependency

view details

Sebastiaan van Stijn

commit sha 2c6d3689924a12efb213419f80d4d3898206cddd

Merge pull request #39975 from tiborvass/homedir-less-deps homedir: remove idtools and libcontainer's user package dependencies

view details

Tibor Vass

commit sha b6684a403c99aaf6be5b8ce0bef3c6650fcdcd12

Merge pull request #39985 from thaJeztah/inline_proxyenv daemon/info: remove use of docker/go-connections

view details

Brian Goff

commit sha 40b9333523d9c128bb6746f80b8e698b21feb6e0

Merge pull request #39993 from feliperuhland/fix-doc-inline-code Fix Engine API version history typo

view details

Jintao Zhang

commit sha c4ec02b0af1753fc958553aed383e66552e7ffcc

Update containerd to v1.2.10 Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>

view details

push time in a month

push eventdperny/swarmkit-1

Drew Erny

commit sha bfd10e1d35b84c5792b7ccc52dd7699ea762df09

Support restarting failing jobs tasks In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed. Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1. Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed. Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in a month

PR opened docker/swarmkit

[Jobs] Support restarting failing jobs tasks

In order to make the jobs reconcilers work correctly with the restart supervisor, they have been altered to never replace failed tasks directly Replacing failed tasks is the purview of the restart supervisor. The jobs reconcilers will only create new tasks when needed.

Additionally, this alters the behavior of the replicated job reconciler with regards to slots -- each new task will get a new slot, and when the job is completed, there will be a Completed task in each slot from 0 to TotalCompletions-1.

Then, makes the tweaks necessary for the Restart Supervisor to support Jobs, which are different from other services in that they deliberately have a desired state of Completed.

Finally, wires up the replicated and global orchestrators to call the restart supervisor to restart tasks that have failed.

This does not yet fully include code necessary to handle node updates.

+792 -117

0 comment

9 changed files

pr created time in 2 months

create barnchdperny/swarmkit-1

branch : jobs-restart-supervisor

created branch time in 2 months

push eventdocker/swarmkit

Sebastiaan van Stijn

commit sha dfe3c44f35c0d845ed3934c23e63aa7006eaac64

bump dustin/go-humanize v1.0.0 full diff: https://github.com/dustin/go-humanize/compare/8929fe90cee4b2cb9deb468b51fb34eba64d1bf0...v1.0.0 probably most relevant change: - https://github.com/dustin/go-humanize/commit/64dbdae0d393b7d71480a6dace78456396b55286 Add space between the numbers and units - closes dustin/go-humanize#21 Space between numerical value and unit symbol for SI output - related: dustin/go-humanize#32 no space between bytes and units Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha d509e31c1fda18ef8224ffd19a34b4aaaff7c4da

Merge pull request #2903 from thaJeztah/bump_go_humanize_1.0.0 bump dustin/go-humanize v1.0.0

view details

push time in 2 months

PR merged docker/swarmkit

bump dustin/go-humanize v1.0.0

full diff: https://github.com/dustin/go-humanize/compare/8929fe90cee4b2cb9deb468b51fb34eba64d1bf0...v1.0.0

probably most relevant change:

  • https://github.com/dustin/go-humanize/commit/64dbdae0d393b7d71480a6dace78456396b55286 Add space between the numbers and units
    • closes dustin/go-humanize#21 Space between numerical value and unit symbol for SI output
    • related: dustin/go-humanize#32 no space between bytes and units
+231 -64

2 comments

11 changed files

thaJeztah

pr closed time in 2 months

pull request commentdocker/swarmkit

bump beorn7/perks v1.0.1

LT (Look Taken)

thaJeztah

comment created time in 2 months

push eventdocker/swarmkit

Sebastiaan van Stijn

commit sha 12eaba61051a1bac6803735a5c473e7c0ef5c23a

bump beorn7/perks v1.0.1 full diff: https://github.com/beorn7/perks/compare/e7f67b54abbeac9c40a31de0f81159e4cafebd6a...v1.0.1 - Add go module support - Create v1.0.0 (to play better with go modules) - Lower Go requirement in go.mod Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 8eb38c8c345ac7be26dfaf2bf7de16b1cb291f17

Merge pull request #2904 from thaJeztah/bump_beorn7_perks_1.0.1 bump beorn7/perks v1.0.1

view details

push time in 2 months

PR merged docker/swarmkit

bump beorn7/perks v1.0.1

full diff: https://github.com/beorn7/perks/compare/e7f67b54abbeac9c40a31de0f81159e4cafebd6a...v1.0.1

  • Add go module support
  • Create v1.0.0 (to play better with go modules)
  • Lower Go requirement in go.mod
+4 -1

2 comments

2 changed files

thaJeztah

pr closed time in 2 months

push eventdperny/docker

Sebastiaan van Stijn

commit sha 961119db21b95504f819a31dfadd7115757fffb3

Dockerfile: set GO111MODULE=off Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 38e4ae3bca76b9558eb44993c4208b41114c4597

Bump Golang version 1.13.0 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 0620990307aaf8ada706ffb6c5dc0628c92d84af

hack/test/unit: fix custom TESTFLAGS not working The `-test.timeout=5m` was glued directly after the current `TESTFLAGS`, causing them to be non-functional; Before: make TESTDEBUG=1 TESTDIRS='github.com/docker/docker/pkg/filenotify' TESTFLAGS='-test.run TestPollerEvent' test-unit + mkdir -p bundles + gotestsum --format=standard-quiet --jsonfile=bundles/go-test-report.json --junitfile=bundles/junit-report.xml -- -tags 'netgo seccomp libdm_no_deferred_remove' -cover -coverprofile=bundles/profile.out -covermode=atomic -test.run TestPollerEvent-test.timeout=5m github.com/docker/docker/pkg/filenotify testing: warning: no tests to run ok github.com/docker/docker/pkg/filenotify 0.003s coverage: 0.0% of statements [no tests to run] DONE 0 tests in 0.298s After: make TESTDEBUG=1 TESTDIRS='github.com/docker/docker/pkg/filenotify' TESTFLAGS='-test.run TestPollerEvent' test-unit + mkdir -p bundles + gotestsum --format=standard-quiet --jsonfile=bundles/go-test-report.json --junitfile=bundles/junit-report.xml -- -tags 'netgo seccomp libdm_no_deferred_remove' -cover -coverprofile=bundles/profile.out -covermode=atomic -test.run TestPollerEvent -test.timeout=5m github.com/docker/docker/pkg/filenotify ok github.com/docker/docker/pkg/filenotify 0.608s coverage: 44.7% of statements DONE 1 tests in 0.922s This was introduced in 42f0a0db75a921145c7f519f7b550e1392890da2 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ac9ef840ef94ff66266d3d8b9d32caf570d3b93f

integration-cli: update TestCreateWithWorkdir for Hyper-V isolation Hyper-V isolated containers do not allow file-operations on a running container. This test currently uses `docker cp` to verify that the WORKDIR was automatically created, which cannot be done while the container is running. ``` FAIL: docker_cli_create_test.go:302: DockerSuite.TestCreateWithWorkdir assertion failed: Command: d:\CI-7\CI-f3768a669\binary\docker.exe cp foo:c:\home\foo\bar c:\tmp ExitCode: 1 Error: exit status 1 Stdout: Stderr: Error response from daemon: filesystem operations against a running Hyper-V container are not supported Failures: ExitCode was 1 expected 0 Expected no error ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sam Whited

commit sha 41adef29f57b0050c4e9c6c9606fae2b7ddf129b

testutil/daemon: group options under type Signed-off-by: Sam Whited <sam@samwhited.com>

view details

Tibor Vass

commit sha 84928be6059084c608f6d071f7121ab42bee721c

hack: have integration-cli use gotestsum codepath Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Tibor Vass

commit sha f1c1cd436ae1252778417121e77d084172f628a1

integration-cli: move each test suite to its own TestX testing function Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Justen Martin

commit sha 23ab331979fd3eb6d190c19606ef4b93faabbb0a

Removed deprecated CloseNotifier logic Signed-off-by: Justen Martin <jmart@the-coder.com>

view details

Tibor Vass

commit sha f470698c2c10c2382080fab34f83088450fabdd6

Jenkinsfile: ensure all containers are cleaned up By convention, containers spawned by jenkins jobs have the name: docker-pr${BUILD_NUMBER} That works fine for jobs with a single container. This commit cleans up when multiple containers are spawned with the convention that their names share the same "docker-pr${BUILD_NUMBER}-" prefix. Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Tibor Vass

commit sha 5feaae02fbd321d6c2dcbdd5006253861d35b1f7

Merge pull request #39957 from tiborvass/jenkins-cleanup-all-containers Jenkinsfile: ensure all containers are cleaned up

view details

Tibor Vass

commit sha 3acf0cc795afae6c598ea2271895f203edc2f279

Merge pull request #39930 from thaJeztah/fix_TESTFLAGS hack/test/unit: fix custom TESTFLAGS not working

view details

Sebastiaan van Stijn

commit sha 6397dd4d3123e0a1b89298d0a2cfe5388410a74f

integration-cli: fix golint issues ``` docker/integration-cli/checker/checker.go Line 12: warning: exported type Compare should have comment or be unexported (golint) Line 14: warning: exported function False should have comment or be unexported (golint) Line 20: warning: exported function True should have comment or be unexported (golint) Line 26: warning: exported function Equals should have comment or be unexported (golint) Line 32: warning: exported function Contains should have comment or be unexported (golint) Line 38: warning: exported function Not should have comment or be unexported (golint) Line 52: warning: exported function DeepEquals should have comment or be unexported (golint) Line 58: warning: exported function HasLen should have comment or be unexported (golint) Line 64: warning: exported function IsNil should have comment or be unexported (golint) Line 70: warning: exported function GreaterThan should have comment or be unexported (golint) Line 76: warning: exported function NotNil should have comment or be unexported (golint) ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 851b000641c195630ed05313a7ad27895e4d8780

integration-cli: enable golangci-lint Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Lukas Heeren

commit sha ce61a1ed98119fa723b0f49fb5d1dc654fd793c6

Adding ability to change max download attempts Moby works perfectly when you are in a situation when one has a good and stable internet connection. Operating in area's where internet connectivity is likely to be lost in undetermined intervals, like a satellite connection or 4G/LTE in rural area's, can become a problem when pulling a new image. When connection is lost while image layers are being pulled, Moby will try to reconnect up to 5 times. If this fails, the incompletely downloaded layers are lost will need to be completely downloaded again during the next pull request. This means that we are using more data than we might have to. Pulling a layer multiple times from the start can become costly over a satellite or 4G/LTE connection. As these techniques (especially 4G) quite common in IoT and Moby is used to run Azure IoT Edge devices, I would like to add a settable maximum download attempts. The maximum download attempts is currently set at 5 (distribution/xfer/download.go). I would like to change this constant to a variable that the user can set. The default will still be 5, so nothing will change from the current version unless specified when starting the daemon with the added flag or in the config file. I added a default value of 5 for DefaultMaxDownloadAttempts and a settable max-download-attempts in the daemon config file. It is also added to the config of dockerd so it can be set with a flag when starting the daemon. This value gets stored in the imageService of the daemon when it is initiated and can be passed to the NewLayerDownloadManager as a parameter. It will be stored in the LayerDownloadManager when initiated. This enables us to set the max amount of retries in makeDownoadFunc equal to the max download attempts. I also added some tests that are based on maxConcurrentDownloads/maxConcurrentUploads. You can pull this version and test in a development container. Either create a config `file /etc/docker/daemon.json` with `{"max-download-attempts"=3}``, or use `dockerd --max-download-attempts=3 -D &` to start up the dockerd. Start downloading a container and disconnect from the internet whilst downloading. The result would be that it stops pulling after three attempts. Signed-off-by: Lukas Heeren <lukas-heeren@hotmail.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 78d137dd23bc72c89dc7980ef22a78859c1779fd

integration-cli: add more debugging for TestSwarmClusterRotateUnlockKey This test was updated in b79adac339173bf8bb6de6d0a061a97973c4b62b, but is still flaky; ``` 20:24:13 FAIL: docker_cli_swarm_test.go:1333: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey 20:24:13 20:24:13 Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/3/DockerSwarmSuite.TestSwarmClusterRotateUnlockKey 20:24:13 [d6f95e679cb65] waiting for daemon to start 20:24:13 [d6f95e679cb65] waiting for daemon to start 20:24:13 [d6f95e679cb65] daemon started 20:24:13 20:24:13 Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/3/DockerSwarmSuite.TestSwarmClusterRotateUnlockKey 20:24:13 [d204a02ba4780] waiting for daemon to start 20:24:13 [d204a02ba4780] waiting for daemon to start 20:24:13 [d204a02ba4780] daemon started 20:24:13 20:24:13 [d204a02ba4780] joining swarm manager [d6f95e679cb65]@0.0.0.0:2477, swarm listen addr 0.0.0.0:2478 20:24:13 Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/3/DockerSwarmSuite.TestSwarmClusterRotateUnlockKey 20:24:13 [d873d6a842829] waiting for daemon to start 20:24:13 [d873d6a842829] waiting for daemon to start 20:24:13 [d873d6a842829] daemon started 20:24:13 20:24:13 [d873d6a842829] joining swarm manager [d6f95e679cb65]@0.0.0.0:2477, swarm listen addr 0.0.0.0:2479 20:24:13 [d204a02ba4780] Stopping daemon 20:24:13 [d204a02ba4780] exiting daemon 20:24:13 [d204a02ba4780] Daemon stopped 20:24:13 [d204a02ba4780] waiting for daemon to start 20:24:13 [d204a02ba4780] waiting for daemon to start 20:24:13 [d204a02ba4780] daemon started 20:24:13 20:24:13 [d873d6a842829] Stopping daemon 20:24:13 [d873d6a842829] exiting daemon 20:24:13 [d873d6a842829] Daemon stopped 20:24:13 [d873d6a842829] waiting for daemon to start 20:24:13 [d873d6a842829] waiting for daemon to start 20:24:13 [d873d6a842829] daemon started 20:24:13 20:24:13 docker_cli_swarm_test.go:1413: 20:24:13 c.Assert(err, checker.IsNil, check.Commentf("%s", outs)) 20:24:13 ... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc000934240), Stderr:[]uint8(nil)} ("exit status 1") 20:24:13 ... Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online. 20:24:13 20:24:13 20:24:13 [d6f95e679cb65] Stopping daemon 20:24:13 [d6f95e679cb65] exiting daemon 20:24:13 [d6f95e679cb65] Daemon stopped 20:24:13 [d204a02ba4780] Stopping daemon 20:24:13 [d204a02ba4780] exiting daemon 20:24:13 [d204a02ba4780] Daemon stopped 20:24:13 [d873d6a842829] Stopping daemon 20:24:13 [d873d6a842829] exiting daemon 20:24:13 [d873d6a842829] Daemon stopped ``` The interesting bit there is that the retry loop should have a 3 second sleep before retrying, but looking at the failure above, the test started (and failed) within a second, which means that a different error / output was returned. This patch adds some additional debugging to that test to see if we can catch the reason this test is still flaky. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Stefan Scherer

commit sha 48662075434da5a13b1316df1ac7488a81e4cf0a

Zap a fixed folder, add build number to folder inside Signed-off-by: Stefan Scherer <stefan.scherer@docker.com>

view details

Brian Goff

commit sha 0f18e434b5366ae48a6eddd378c22dfcc583d6ce

Merge pull request #39911 from tiborvass/gotestsum-integration-cli hack: have integration-cli use gotestsum codepath

view details

Sebastiaan van Stijn

commit sha 41ee87c6816aea0d7e06d8484e09d1b9cfd595d5

Merge pull request #39942 from SamWhited/daemon_ops_type testutil/daemon: group options under type

view details

Vikram bir Singh

commit sha 7de4e130898fc1cf74f8c4fec24416c3f7d3589b

Disable TestPsListContainersFilterExited (Windows) On account of being flaky on both RS1 and RS5. Co-Authored-By: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com> Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>

view details

Tibor Vass

commit sha 8f2ae8f73923bda14b01a949d5a7e23e63c5eb41

Merge pull request #39945 from vikramhh/disable_TestPsListContainersFilterExited_on_windows Disable TestPsListContainersFilterExited (Windows)

view details

push time in 2 months

PR opened docker/swarmkit

[Jobs] Support job updates

Adds code to put old tasks in the Shutdown state when jobs are updated, allowing updates to proceed correctly.

+254 -2

0 comment

5 changed files

pr created time in 2 months

create barnchdperny/swarmkit-1

branch : jobs-updates

created branch time in 2 months

push eventdocker/swarmkit

Drew Erny

commit sha e5dcefdf66e223d742e0272f5580b606bd9bbf65

Add jobs orchestrator to controlapi integration tests Adds the beginnings of the integration tests between the controlapi and the jobs orchestrator. These tests don't actually check much more than creation right now, as update handling for the jobs orchestrator is pending, but further tests will be able to leverage the groundwork here to a high degree. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 438cdc04067489a401c80b64cbd516545c7d41a8

Merge pull request #2896 from dperny/jobs-controlapi-integration [Jobs] controlapi integration tests

view details

push time in 2 months

PR merged docker/swarmkit

[Jobs] controlapi integration tests

This is the beginning of a set of tests that verifies that things created through the controlapi are correctly operated on by the jobs orchestrator. These tests are intended to ferret out any mismatches between what the orchestrator expects the controlapi to do in terms of setting up a service object and what it actually does.

+163 -0

3 comments

1 changed file

dperny

pr closed time in 2 months

pull request commentdocker/swarmkit

[Jobs] controlapi integration tests

kicking off a rebuild to figure out if the spontaneous disaster in CI has equally spontaneously resolved itself. if so, i will merge.

dperny

comment created time in 2 months

push eventdocker/swarmkit

Trapier Marshall

commit sha a4e520aff16b607c0cf5558fbcfc0dc2cf41d9d8

Fix nil pointer deref in dump-snapshot --redacted container.PullOptions is nil if the service is deployed without `--with-registry-auth`. Observed on client+server 19.03.2. Signed-off-by: Trapier Marshall <trapier.marshall@docker.com>

view details

Drew Erny

commit sha 8a69c0da0d0f002c4d6b4611aa4f1e40f5ccd659

Merge pull request #2897 from trapier/dump-snapshot-redacted-nil-deref Fix nil pointer deref in dump-snapshot --redacted [FIELD-2081]

view details

push time in 2 months

PR merged docker/swarmkit

Fix nil pointer deref in dump-snapshot --redacted [FIELD-2081]

container.PullOptions is nil if the service is deployed without --with-registry-auth. Observed on client+server 19.03.2.

- What I did

Fixed a panic in swarm-rafttool dump-snapshot --redact.

stack and affected frame:

$ echo -e 'c\nbt\nframe 4' | dlv exec bin/swarm-rafttool -- -d /tmp/swarmdump  dump-snapshot --redact
Type 'help' for list of commands.
2019-09-20 11:36:08.938888 W | wal: ignored file 0000000000000000-0000000000000000.wal.broken in wal
Active members:
 NodeID=p1rqbkyjtikf53p1rrov4wk81, RaftID=39c366065096262f, Addr=192.168.30.101:2377

Removed members:
 RaftID=2ae2e29a7ad988cc
 RaftID=31a2a4a4c22e495c
 RaftID=2c73df5477fe943e
 RaftID=791d06f8748db411
 RaftID=11143b659a8b722d

> [unrecovered-panic] runtime.fatalpanic() /usr/lib/go/src/runtime/panic.go:847 (hits goroutine(1):1 total:1) (PC: 0x42fef0)
Warning: debugging optimized function
        runtime.curg._panic.arg: interface {}(string) "runtime error: invalid memory address or nil pointer dereference"
   842: // fatalpanic implements an unrecoverable panic. It is like fatalthrow, except
   843: // that if msgs != nil, fatalpanic also prints panic messages and decrements
   844: // runningPanicDefers once main is blocked from exiting.
   845: //
   846: //go:nosplit
=> 847: func fatalpanic(msgs *_panic) {
   848:         pc := getcallerpc()
   849:         sp := getcallersp()
   850:         gp := getg()
   851:         var docrash bool
   852:         // Switch to the system stack to avoid any stack growth, which
 0  0x000000000042fef0 in runtime.fatalpanic
    at /usr/lib/go/src/runtime/panic.go:847
 1  0x000000000042f952 in runtime.gopanic
    at /usr/lib/go/src/runtime/panic.go:722
 2  0x000000000044453c in runtime.panicmem
    at /usr/lib/go/src/runtime/panic.go:199
 3  0x000000000044453c in runtime.sigpanic
    at /usr/lib/go/src/runtime/signal_unix.go:394
 4  0x0000000000dc17b3 in main.dumpSnapshot
    at ./cmd/swarm-rafttool/dump.go:203
 5  0x0000000000dc2f50 in main.glob..func3
    at ./cmd/swarm-rafttool/main.go:97
 6  0x0000000000db8b24 in github.com/docker/swarmkit/vendor/github.com/spf13/cobra.(*Command).execute
    at ./vendor/github.com/spf13/cobra/command.go:565
 7  0x0000000000db9235 in github.com/docker/swarmkit/vendor/github.com/spf13/cobra.(*Command).ExecuteC
    at ./vendor/github.com/spf13/cobra/command.go:656
 8  0x0000000000dc285d in main.main
    at ./cmd/swarm-rafttool/main.go:185
 9  0x000000000043169e in runtime.main
    at /usr/lib/go/src/runtime/proc.go:203
10  0x000000000045cc61 in runtime.goexit
    at /usr/lib/go/src/runtime/asm_amd64.s:1357
> [unrecovered-panic] runtime.fatalpanic() /usr/lib/go/src/runtime/panic.go:847 (hits goroutine(1):1 total:1) (PC: 0x42fef0)
Warning: debugging optimized function
Frame 4: ./cmd/swarm-rafttool/dump.go:203 (PC: dc17b3)
   198:                 }
   199:                 for _, task := range s.Store.Tasks {
   200:                         if task != nil {
   201:                                 if container := task.Spec.GetContainer(); container != nil {
   202:                                         container.Env = []string{"ENVVARS REDACTED"}
=> 203:                                         container.PullOptions.RegistryAuth = "REDACTED"
   204:                                 }
   205:                         }
   206:                 }
   207:                 for _, service := range s.Store.Services {
   208:                         if service != nil {
exit

nil here:

$ echo -e 'c\nbt\nframe 4\nlocals -v container' | dlv exec bin/swarm-rafttool -- -d /home/trapier/go/src/github.com/docker/certified-infrastructure/vagrant/swarm  dump-snapshot --redact  |& grep PullOptions |tail -n1
        PullOptions: *github.com/docker/swarmkit/api.ContainerSpec_PullOptions nil,

- How I did it

Test for nil on container.PullOptions.

- How to test it

Doubt it matters, but docker client+server 19.03.2.

  1. Set the snapshot interval low enough to trigger rotation within a reasonable amount of time

    docker swarm update --snapshot-interval 200
    
  2. Deploy a service without --with-registry-auth that will churn into the snapshot:

    docker service create --replicas 200 --update-parallelism 0 alpine sh -c 'exit 0'
    
  3. Wait for mtime on /var/lib/docker/swarm/raft/*snap*/* to change.

  4. Stop engine.

  5. Run rafttool:

    swarm-rafttool -d /var/lib/docker/swarm dump-snapshot --redact
    

FAIL if panic. PASS if no panic.

- Description for the changelog N/A

+6 -2

1 comment

1 changed file

trapier

pr closed time in 2 months

pull request commentdocker/swarmkit

[WIP][18.03 backport] dependency changes for newer Go versions

@thaJeztah I reran vndr and make generate and pushed up the results. Hopefully this will fix the broken test and we can merge.

thaJeztah

comment created time in 2 months

push eventthaJeztah/swarmkit

Drew Erny

commit sha 648725d9ec10fb737adf259ea43e93290c43ebe1

Revendor, because something was broken Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 2 months

pull request commentdocker/swarmkit

[Jobs] controlapi integration tests

Updated the comments, and switched to using the tempUnixSocket variable when appropriate.

dperny

comment created time in 2 months

push eventdperny/swarmkit-1

Drew Erny

commit sha e5dcefdf66e223d742e0272f5580b606bd9bbf65

Add jobs orchestrator to controlapi integration tests Adds the beginnings of the integration tests between the controlapi and the jobs orchestrator. These tests don't actually check much more than creation right now, as update handling for the jobs orchestrator is pending, but further tests will be able to leverage the groundwork here to a high degree. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 2 months

Pull request review commentdocker/swarmkit

[Jobs] controlapi integration tests

+package jobs++import (+	. "github.com/onsi/ginkgo"+	. "github.com/onsi/gomega"++	"context"+	"io/ioutil"+	"net"+	"os"+	"time"++	"google.golang.org/grpc"++	"github.com/docker/swarmkit/api"+	"github.com/docker/swarmkit/manager/controlapi"+	"github.com/docker/swarmkit/manager/orchestrator/testutils"+	"github.com/docker/swarmkit/manager/state/store"+	stateutils "github.com/docker/swarmkit/manager/state/testutils"+)++var _ = Describe("Integration between the controlapi and jobs orchestrator", func() {+	// These tests verify that the control api and the jobs orchestrator+	// cooperate and work correctly.++	var (+		// the object store is shared between the controlapi server and the+		// orchestrator. makes an end-run around the manager, allowing us to+		// avoid having to start the whole manager apparatus to test this one+		// integration point+		s *store.MemoryStore+		o *Orchestrator++		client api.ControlClient++		// we need all of these variables captured so that we can close them+		// out in AfterEach+		server         *controlapi.Server+		grpcServer     *grpc.Server+		conn           *grpc.ClientConn+		tempUnixSocket string++		// serverDone is a channel that is closed when the goroutine running+		// the server exits.+		serverDone <-chan struct{}+		// orchestratorDone is likewise closed when the orchestrator exits+		orchestratorDone <-chan struct{}+	)++	BeforeEach(func() {+		// By statements are just extra documentation of what's going on, no+		// functional difference besides some test log output+		By("setting up the controlapi client and server")+		// a lot of this setup is taken from manager/controlapi/server_test.go,+		// but it's unexported. in any case, re-writing it in ginkgo isn't+		// unwarranted.+		s = store.NewMemoryStore(&stateutils.MockProposer{})+		server = controlapi.NewServer(s, nil, nil, nil, nil)++		// we need a temporary unix socket to server on+		temp, err := ioutil.TempFile("", "test-socket")+		// i'm not sure why we immediately close this out, i'm not well versed+		// in file system shenanigans, but this is what the controlapi unit+		// tests do

that's likely the case, but I was unsure.

dperny

comment created time in 2 months

Pull request review commentdocker/swarmkit

[Jobs] controlapi integration tests

+package jobs++import (+	. "github.com/onsi/ginkgo"+	. "github.com/onsi/gomega"++	"context"+	"io/ioutil"+	"net"+	"os"+	"time"++	"google.golang.org/grpc"++	"github.com/docker/swarmkit/api"+	"github.com/docker/swarmkit/manager/controlapi"+	"github.com/docker/swarmkit/manager/orchestrator/testutils"+	"github.com/docker/swarmkit/manager/state/store"+	stateutils "github.com/docker/swarmkit/manager/state/testutils"+)++var _ = Describe("Integration between the controlapi and jobs orchestrator", func() {+	// These tests verify that the control api and the jobs orchestrator+	// cooperate and work correctly.++	var (+		// the object store is shared between the controlapi server and the+		// orchestrator. makes an end-run around the manager, allowing us to+		// avoid having to start the whole manager apparatus to test this one+		// integration point+		s *store.MemoryStore+		o *Orchestrator++		client api.ControlClient++		// we need all of these variables captured so that we can close them+		// out in AfterEach+		server         *controlapi.Server+		grpcServer     *grpc.Server+		conn           *grpc.ClientConn+		tempUnixSocket string++		// serverDone is a channel that is closed when the goroutine running+		// the server exits.+		serverDone <-chan struct{}+		// orchestratorDone is likewise closed when the orchestrator exits+		orchestratorDone <-chan struct{}+	)++	BeforeEach(func() {+		// By statements are just extra documentation of what's going on, no+		// functional difference besides some test log output+		By("setting up the controlapi client and server")+		// a lot of this setup is taken from manager/controlapi/server_test.go,+		// but it's unexported. in any case, re-writing it in ginkgo isn't+		// unwarranted.+		s = store.NewMemoryStore(&stateutils.MockProposer{})+		server = controlapi.NewServer(s, nil, nil, nil, nil)++		// we need a temporary unix socket to server on+		temp, err := ioutil.TempFile("", "test-socket")+		// i'm not sure why we immediately close this out, i'm not well versed+		// in file system shenanigans, but this is what the controlapi unit+		// tests do+		Expect(err).ToNot(HaveOccurred())+		Expect(temp.Close()).ToNot(HaveOccurred())+		Expect(os.Remove(temp.Name())).ToNot(HaveOccurred())++		tempUnixSocket = temp.Name()+		lis, err := net.Listen("unix", temp.Name())

🤔 probably.

dperny

comment created time in 2 months

Pull request review commentdocker/swarmkit

[Jobs] controlapi integration tests

+package jobs++import (+	. "github.com/onsi/ginkgo"+	. "github.com/onsi/gomega"++	"context"+	"io/ioutil"+	"net"+	"os"+	"time"++	"google.golang.org/grpc"++	"github.com/docker/swarmkit/api"+	"github.com/docker/swarmkit/manager/controlapi"+	"github.com/docker/swarmkit/manager/orchestrator/testutils"+	"github.com/docker/swarmkit/manager/state/store"+	stateutils "github.com/docker/swarmkit/manager/state/testutils"+)++var _ = Describe("Integration between the controlapi and jobs orchestrator", func() {+	// These tests verify that the control api and the jobs orchestrator+	// cooperate and work correctly.++	var (+		// the object store is shared between the controlapi server and the+		// orchestrator. makes an end-run around the manager, allowing us to+		// avoid having to start the whole manager apparatus to test this one+		// integration point+		s *store.MemoryStore+		o *Orchestrator++		client api.ControlClient++		// we need all of these variables captured so that we can close them+		// out in AfterEach+		server         *controlapi.Server+		grpcServer     *grpc.Server+		conn           *grpc.ClientConn+		tempUnixSocket string++		// serverDone is a channel that is closed when the goroutine running+		// the server exits.+		serverDone <-chan struct{}+		// orchestratorDone is likewise closed when the orchestrator exits+		orchestratorDone <-chan struct{}+	)++	BeforeEach(func() {+		// By statements are just extra documentation of what's going on, no+		// functional difference besides some test log output+		By("setting up the controlapi client and server")+		// a lot of this setup is taken from manager/controlapi/server_test.go,+		// but it's unexported. in any case, re-writing it in ginkgo isn't+		// unwarranted.+		s = store.NewMemoryStore(&stateutils.MockProposer{})+		server = controlapi.NewServer(s, nil, nil, nil, nil)++		// we need a temporary unix socket to server on+		temp, err := ioutil.TempFile("", "test-socket")+		// i'm not sure why we immediately close this out, i'm not well versed+		// in file system shenanigans, but this is what the controlapi unit+		// tests do+		Expect(err).ToNot(HaveOccurred())+		Expect(temp.Close()).ToNot(HaveOccurred())+		Expect(os.Remove(temp.Name())).ToNot(HaveOccurred())++		tempUnixSocket = temp.Name()+		lis, err := net.Listen("unix", temp.Name())+		Expect(err).ToNot(HaveOccurred())++		grpcServer = grpc.NewServer()+		api.RegisterControlServer(grpcServer, server)++		serverDone = testutils.EnsureRuns(func() {+			_ = grpcServer.Serve(lis)+		})++		// the controlapi tests use the older grpc.Dial with a manually entered+		// timeout. avoid that mess by just using the DialTimeout function+		// instead.+		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)+		// cancel after dial has completed is a no-op, but if we don't cancel,+		// linters will (probably) complain about a leaked context.+		defer cancel()+		conn, err = grpc.DialContext(+			ctx, "unix:"+temp.Name(),+			// block on making this connection, to avoid the tests failing for+			// funny reasons related to this connection being established async+			grpc.WithBlock(),+			grpc.WithInsecure(),+		)+		Expect(err).ToNot(HaveOccurred())++		client = api.NewControlClient(conn)++		By("setting up the orchestrator")+		o = NewOrchestrator(s)+		orchestratorDone = testutils.EnsureRuns(func() {+			o.Run(context.Background())+		})+	})++	AfterEach(func() {+		conn.Close()+		grpcServer.Stop()+		// wait for the server to stop+		<-serverDone+		s.Close()+		os.RemoveAll(tempUnixSocket)++		o.Stop()+		<-orchestratorDone+	})++	It("should create the requisite tasks for a new replicated job", func() {+		spec := &api.ServiceSpec{+			Annotations: api.Annotations{+				Name: "testService",+			},+			Mode: &api.ServiceSpec_ReplicatedJob{+				ReplicatedJob: &api.ReplicatedJob{+					MaxConcurrent:    3,+					TotalCompletions: 5,+				},+			},+			Task: api.TaskSpec{+				Runtime: &api.TaskSpec_Container{+					Container: &api.ContainerSpec{+						Image: "image",+					},+				},+			},+		}++		resp, err := client.CreateService(context.Background(), &api.CreateServiceRequest{+			Spec: spec,+		})+		Expect(err).ToNot(HaveOccurred())+		Expect(resp).ToNot(BeNil())+		Expect(resp.Service).ToNot(BeNil())++		// Eventually is like Expect but it does polling, which is what we want+		// If the function has two return values, this will additionally assert+		// that the second value is nil

Yeah, I just happened to read this in the Ginkgo docs. It actually works for any number of arguments, not just 2, and will assert that any past the first are all nil.

dperny

comment created time in 2 months

Pull request review commentdocker/swarmkit

[Jobs] controlapi integration tests

+package jobs++import (+	. "github.com/onsi/ginkgo"+	. "github.com/onsi/gomega"++	"context"+	"io/ioutil"+	"net"+	"os"+	"time"++	"google.golang.org/grpc"++	"github.com/docker/swarmkit/api"+	"github.com/docker/swarmkit/manager/controlapi"+	"github.com/docker/swarmkit/manager/orchestrator/testutils"+	"github.com/docker/swarmkit/manager/state/store"+	stateutils "github.com/docker/swarmkit/manager/state/testutils"+)++var _ = Describe("Integration between the controlapi and jobs orchestrator", func() {+	// These tests verify that the control api and the jobs orchestrator+	// cooperate and work correctly.++	var (+		// the object store is shared between the controlapi server and the+		// orchestrator. makes an end-run around the manager, allowing us to+		// avoid having to start the whole manager apparatus to test this one+		// integration point+		s *store.MemoryStore+		o *Orchestrator++		client api.ControlClient++		// we need all of these variables captured so that we can close them+		// out in AfterEach+		server         *controlapi.Server+		grpcServer     *grpc.Server+		conn           *grpc.ClientConn+		tempUnixSocket string++		// serverDone is a channel that is closed when the goroutine running+		// the server exits.+		serverDone <-chan struct{}+		// orchestratorDone is likewise closed when the orchestrator exits+		orchestratorDone <-chan struct{}+	)++	BeforeEach(func() {+		// By statements are just extra documentation of what's going on, no+		// functional difference besides some test log output+		By("setting up the controlapi client and server")+		// a lot of this setup is taken from manager/controlapi/server_test.go,+		// but it's unexported. in any case, re-writing it in ginkgo isn't+		// unwarranted.+		s = store.NewMemoryStore(&stateutils.MockProposer{})+		server = controlapi.NewServer(s, nil, nil, nil, nil)++		// we need a temporary unix socket to server on+		temp, err := ioutil.TempFile("", "test-socket")+		// i'm not sure why we immediately close this out, i'm not well versed+		// in file system shenanigans, but this is what the controlapi unit+		// tests do+		Expect(err).ToNot(HaveOccurred())+		Expect(temp.Close()).ToNot(HaveOccurred())+		Expect(os.Remove(temp.Name())).ToNot(HaveOccurred())++		tempUnixSocket = temp.Name()+		lis, err := net.Listen("unix", temp.Name())+		Expect(err).ToNot(HaveOccurred())++		grpcServer = grpc.NewServer()+		api.RegisterControlServer(grpcServer, server)++		serverDone = testutils.EnsureRuns(func() {+			_ = grpcServer.Serve(lis)+		})++		// the controlapi tests use the older grpc.Dial with a manually entered+		// timeout. avoid that mess by just using the DialTimeout function+		// instead.+		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)+		// cancel after dial has completed is a no-op, but if we don't cancel,+		// linters will (probably) complain about a leaked context.+		defer cancel()+		conn, err = grpc.DialContext(+			ctx, "unix:"+temp.Name(),+			// block on making this connection, to avoid the tests failing for+			// funny reasons related to this connection being established async+			grpc.WithBlock(),+			grpc.WithInsecure(),+		)+		Expect(err).ToNot(HaveOccurred())++		client = api.NewControlClient(conn)++		By("setting up the orchestrator")+		o = NewOrchestrator(s)+		orchestratorDone = testutils.EnsureRuns(func() {+			o.Run(context.Background())+		})+	})++	AfterEach(func() {+		conn.Close()+		grpcServer.Stop()+		// wait for the server to stop+		<-serverDone+		s.Close()+		os.RemoveAll(tempUnixSocket)++		o.Stop()+		<-orchestratorDone+	})++	It("should create the requisite tasks for a new replicated job", func() {+		spec := &api.ServiceSpec{+			Annotations: api.Annotations{+				Name: "testService",+			},+			Mode: &api.ServiceSpec_ReplicatedJob{+				ReplicatedJob: &api.ReplicatedJob{+					MaxConcurrent:    3,+					TotalCompletions: 5,+				},+			},+			Task: api.TaskSpec{+				Runtime: &api.TaskSpec_Container{+					Container: &api.ContainerSpec{+						Image: "image",+					},+				},+			},+		}++		resp, err := client.CreateService(context.Background(), &api.CreateServiceRequest{+			Spec: spec,+		})+		Expect(err).ToNot(HaveOccurred())+		Expect(resp).ToNot(BeNil())+		Expect(resp.Service).ToNot(BeNil())++		// Eventually is like Expect but it does polling, which is what we want+		// If the function has two return values, this will additionally assert+		// that the second value is nil+		Eventually(func() ([]*api.Task, error) {+			// make sure that we can read the tasks back out through the+			// controlapi, not just through the store+			taskListResp, err := client.ListTasks(+				context.Background(),+				&api.ListTasksRequest{+					Filters: &api.ListTasksRequest_Filters{+						ServiceIDs: []string{resp.Service.ID},+						DesiredStates: []api.TaskState{+							api.TaskStateCompleted,+						},+					},+				},+			)+			return taskListResp.Tasks, err+		}).Should(HaveLen(3))

I considered it, but I think that would be out of the scope of the integration tests? The touch points between these two components are actually fairly small, and I don't want to spend too much time testing things that are already tested.

There will be more meat to the tests when we cover updates.

dperny

comment created time in 2 months

PR opened docker/swarmkit

[Jobs] controlapi integration tests

This is the beginning of a set of tests that verifies that things created through the controlapi are correctly operated on by the jobs orchestrator. These tests are intended to ferret out any mismatches between what the orchestrator expects the controlapi to do in terms of setting up a service object and what it actually does.

+164 -0

0 comment

1 changed file

pr created time in 2 months

create barnchdperny/swarmkit-1

branch : jobs-controlapi-integration

created branch time in 2 months

push eventdperny/docker

Olli Janatuinen

commit sha 8660330173e5053e274cf12860079f132cbaa9fa

Unit test for getOrphan Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>

view details

Pavel Tikhomirov

commit sha b469933b063169718987865b8b1215cb7befd1a6

integration-cli/requirements: Skip windows specific isolation requirements on non-windows After the commit faaffd5d6d7f ("Windows:Disable 2 restart test when Hyper-V") some tests became skipped on linux: SKIP: docker_cli_restart_test.go:167: DockerSuite.TestRestartContainerSuccess (unmatched requirement IsolationIsProcess) SKIP: docker_cli_restart_test.go:240: DockerSuite.TestRestartPolicyAfterRestart (unmatched requirement IsolationIsProcess) But AFAIU it is highly unlikely that we actually meant to skip them on linux. https://github.com/moby/moby/issues/39625 Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

view details

Andrew Hsu

commit sha 039eb05ac87d52fe0b268c090f137eaa14754eed

skip win-RS1 on PRs unless the checkbox is checked Signed-off-by: Andrew Hsu <andrewhsu@docker.com> Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Andrew Hsu

commit sha e653943e8befaff83d6ace9b6f943f0cc8027ae4

run integration-cli stages on s390x and ppc64le if not a PR check Essentially, run on merge to target branch which may or may not be master branch. Could be 19.03 branch, for example. See: https://jenkins.io/doc/book/pipeline/syntax/ Signed-off-by: Andrew Hsu <andrewhsu@docker.com>

view details

Sebastiaan van Stijn

commit sha 64b3d12686e1a4ad4669e8c195cacafd76ea265c

Jenkinsfile: remove redundant -f Dockerfile Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Tibor Vass

commit sha 984ed95ed7e3903841f8df8f5278db4a12ec3733

Merge pull request #39789 from tiborvass/andrewhsu-jenkinsfile Jenkinsfile: reduce CI time by disabling windows RS1

view details

Arnaud Rebillout

commit sha 667c87ef4f277009803602ebd0813b65fff84090

profiles: Fix file permissions on json files json files should not be executable I think. Signed-off-by: Arnaud Rebillout <arnaud.rebillout@collabora.com>

view details

Akihiro Suda

commit sha 4b0371fb36a958589319ab7c501ff4bc22645cfa

Merge pull request #39936 from elboulangero/fix-profiles-json-file-permissions profiles: Fix file permissions on json files

view details

Sebastiaan van Stijn

commit sha 2e7d48f2db138d5421c424172fa2babbb53cec4b

Merge pull request #39688 from Snorch/skip-windows-specific-isolation-requirements-on-non-windows integration-cli/requirements: Skip windows specific isolation require…

view details

Sebastiaan van Stijn

commit sha e894aae2a60b9a1ff84ae9ba3c24d2b5df9c4140

Merge pull request #39715 from olljanat/getorplan-unit-test Unit test for getOrphan

view details

Michael Crosby

commit sha 92cc603036ddadd608762a6f69e72b9b2bef651e

Merge pull request #39932 from thaJeztah/remove_redundant_dockerfile Jenkinsfile: remove redundant -f Dockerfile

view details

Sebastiaan van Stijn

commit sha 8695176d11d7d45c3ac3b75bd790a2a7ac7d41de

daemon/seccomp_linux.go: fix error-capitalization (golint) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 5ded7886c3b313b0396eef0761c378075ad85d81

daemon/cluster: fix unused context (staticcheck) ``` daemon/cluster/nodes.go:69:36: SA4009: argument ctx is overwritten before first use (staticcheck) 13:06:14 return c.lockedManagerAction(func(ctx context.Context, state nodeState) error { ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 41cfcac7fc93a932b10f0e97f4b1d53f43dcb7dd

DriverBenchDiffApplyN: ignore empty branch (staticcheck) suppressing the "SA9003: empty branch (staticcheck)" instead of commenting-out or removing these lines because removing/commenting these lines causes a ripple effect of changes, and there's still a to-do below. ``` 13:06:14 daemon/graphdriver/graphtest/graphbench_unix.go:175:3: SA9003: empty branch (staticcheck) 13:06:14 if applyDiffSize != diffSize { ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 178af761b757782b45a89eb8d44d01495de635cb

TestParseRelease: fix missing assert Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ab599b5cdc11686528966ead215c72868b6b973b

pkg/pools/pools.go: suppress SA6002: argument should be pointer-like for now ``` 13:06:14 pkg/pools/pools.go:75:13: SA6002: argument should be pointer-like to avoid allocations (staticcheck) 13:06:14 bp.pool.Put(b) ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e334eeeed89983e0c734843cf8e52f63674df08e

TestMoveToSubdir: use sort.Strings() (gosimple) ``` 13:06:14 pkg/directory/directory_test.go:182:2: S1032: should use sort.Strings(...) instead of sort.Sort(sort.StringSlice(...)) (gosimple) 13:06:14 sort.Sort(sort.StringSlice(results)) ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha f6f58f38586b5e166e3ee99b63d6856441bfe422

daemon.getEndpointInNetwork() is only used on Windows ``` 13:06:14 daemon/network.go:964:6: U1000: func `getEndpointInNetwork` is unused (unused) 13:06:14 func getEndpointInNetwork(name string, n libnetwork.Network) (libnetwork.Endpoint, error) { ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha c2532d56b0c80a0300d9aae9c24c3ba077bab2a9

volume Create: fix incorrect file permissions (staticcheck) ``` 14:01:54 volume/local/local.go:175:80: SA9002: file mode '600' evaluates to 01130; did you mean '0600'? (staticcheck) 14:01:54 if err = ioutil.WriteFile(filepath.Join(filepath.Dir(path), "opts.json"), b, 600); err != nil { ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 56e690f340e030027ed1b5503bbde06e5a879518

cluster/executor: remove unused containerConfig.endpoint() Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

push time in 2 months

pull request commentmoby/moby

Add support for sending down service Running and Desired task counts

a test seems to have flaked. i'm going to rebase and push and see if it fails again

dperny

comment created time in 2 months

push eventdocker/swarmkit

Drew Erny

commit sha 99a65a992799f1df218818e30d00a0049950e265

Fix leaking task resources when nodes are deleted When a node is deleted, its tasks are asked to restart, which involves putting them into a desired state of Shutdown. However, the Allocator will not deallocate a task which is not in an actual state of a terminal state. Once a node is deleted, the only opportunity for its tasks to recieve updates and be moved to a terminal state is when the function moving those tasks to TaskStateOrphaned is called, 24 hours after the node enters the Down state. However, if a leadership change occurs, then that function will never be called, and the tasks will never be moved to a terminal state, leaking resources. With this change, upon node deletion, all of its tasks will be moved to TaskStateOrphaned, allowing those tasks' resources to be cleaned up. Additionally, as part of this backport, avoid using the gogo types.TimestampNow function, which does not exist in the vendored version. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 8467e6a43f9cee66f46efb457b2c22f46ae5da0b) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 847a883dfd5a23bc81a7422bb73d292706660bf5

Only update non-terminal tasks on node removal. When a node is removed, its tasks are set in state ORPHANED. This does not need to be done for tasks that are already in a terminal state, and if all tasks in all states are updated, the size of the transaction may grow too large to process, and node removal becomes impossible. This changes to only set non-terminal tasks to state ORPHANED, and terminal tasks are left alone. Cherry pick does not apply cleanly, but the fix is rather simple. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit d5df26594f5b74a9de41e376206c36abffc961fe) Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 7ef676983601d839dfa65d33d1e2df37446bdf2c

Merge pull request #2841 from thaJeztah/18.03_backport_fix_leaking_task_resources [WIP][18.03 backport] Fix leaking task resources when nodes are deleted

view details

push time in 2 months

PR merged docker/swarmkit

[WIP][18.03 backport] Fix leaking task resources when nodes are deleted

this is for ENGCORE-711 and ENGCORE-937

backport of https://github.com/docker/swarmkit/pull/2806 and https://github.com/docker/swarmkit/pull/2867 for the bump_v18.03 branch

When a node is deleted, its tasks are asked to restart, which involves putting them into a desired state of Shutdown. However, the Allocator will not deallocate a task which is not in an actual state of a terminal state. Once a node is deleted, the only opportunity for its tasks to recieve updates and be moved to a terminal state is when the function moving those tasks to TaskStateOrphaned is called, 24 hours after the node enters the Down state. However, if a leadership change occurs, then that function will never be called, and the tasks will never be moved to a terminal state, leaking resources.

With this change, upon node deletion, all of its tasks will be moved to TaskStateOrphaned, allowing those tasks' resources to be cleaned up.

+67 -25

15 comments

2 changed files

thaJeztah

pr closed time in 2 months

push eventdocker/stacks

Jason Schroeder

commit sha 7c329ddda0c3398f74cc21622ddfaf3442cdb05b

Bringing SnapshotStack related changes to the API, these changes, or ones compatible, are needed to make the reconcilation alterations possible.

view details

Jason Schroeder

commit sha 6ccb1d454885e8b21e80f6a185e751bfe88ffa04

types.StackLabel for resources should not be stored in the types.Stack. And collecting the fake text fixtures utilities into one file.

view details

Jason Schroeder

commit sha c8fc9cb82cad5aa65da43d6b5984bd23690b126d

Adding some testing improvements from the pending reconciliation improvements

view details

Drew Erny

commit sha 899389a60437a35f2a07030bb723db019b0e376e

Merge pull request #60 from undertheflowerpot/FirstTry Bringing SnapshotStack related changes to the API,

view details

push time in 2 months

PR merged docker/stacks

Bringing SnapshotStack related changes to the API,

these changes, including the Snapshot concept, or something compatible, are needed to make the reconcilation alterations possible.

+815 -550

0 comment

24 changed files

undertheflowerpot

pr closed time in 2 months

push eventdocker/swarmkit

Drew Erny

commit sha e2ceb582b28180797bcc4688c10009186978fa9a

Wire up jobs orchestrator to manager Adds the jobs orchestrator to the swarmkit manager. Jobs orchestrator will now start and run with the manager. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 27284dbb2e9d9aa760c55121f0d4e5f043ff6243

Merge pull request #2894 from dperny/wire-up-jobs-orchestrator [Jobs] Wire up jobs orchestrator to manager

view details

push time in 2 months

PR merged docker/swarmkit

[Jobs] Wire up jobs orchestrator to manager

Adds the jobs orchestrator to the swarmkit manager. Jobs orchestrator will now start and run with the manager.

+11 -0

2 comments

1 changed file

dperny

pr closed time in 2 months

pull request commentdocker/swarmkit

[Jobs] Wire up jobs orchestrator to manager

Testing this would be a lot of test rigging for not a lot of payoff.

dperny

comment created time in 2 months

push eventdocker/swarmkit

Arko Dasgupta

commit sha 54cd59ff976756c9a675e03399ce4a02b5653168

Remove hardcoded IPAM config subnet value for ingress network Fixes: https://docker.atlassian.net/browse/ENGORC-2651 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com> (cherry picked from commit 9ccb20b27fa9a0e7b88800d7e196fa030da2a6a7) Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

view details

Drew Erny

commit sha b0bc4017ad110cd20898d8c44be03c1e78e4e979

Merge pull request #2892 from arkodg/bump_v18.09 [18.09 backport] Remove hardcoded IPAM config subnet value for ingress network

view details

push time in 2 months

PR merged docker/swarmkit

[18.09 backport] Remove hardcoded IPAM config subnet value for ingress network

backport of https://github.com/docker/swarmkit/pull/2890 Fixes: https://docker.atlassian.net/browse/ENGORC-2651

Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com (cherry picked from commit 9ccb20b27fa9a0e7b88800d7e196fa030da2a6a7) Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com

+2 -6

2 comments

1 changed file

arkodg

pr closed time in 2 months

push eventdocker/swarmkit

Arko Dasgupta

commit sha 9b09fa72c5aaccd34264aa2b6bed07833668e51a

Remove hardcoded IPAM config subnet value for ingress network Fixes: https://docker.atlassian.net/browse/ENGORC-2651 (cherry picked from commit 9ccb20b27fa9a0e7b88800d7e196fa030da2a6a7) Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

view details

Drew Erny

commit sha f35d9100f2c6ac810cc8d7de6e8f93dcc7a42d29

Merge pull request #2891 from arkodg/bump_v19.03 [19.03 backport] Remove hardcoded IPAM config subnet value for ingress network

view details

push time in 2 months

PR merged docker/swarmkit

[19.03 backport] Remove hardcoded IPAM config subnet value for ingress network

backport of https://github.com/docker/swarmkit/pull/2890 Fixes: https://docker.atlassian.net/browse/ENGORC-2651

Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com (cherry picked from commit 9ccb20b27fa9a0e7b88800d7e196fa030da2a6a7) Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com

+2 -6

2 comments

1 changed file

arkodg

pr closed time in 2 months

PR opened docker/swarmkit

[Jobs] Wire up jobs orchestrator to manager

Adds the jobs orchestrator to the swarmkit manager. Jobs orchestrator will now start and run with the manager.

+11 -0

0 comment

1 changed file

pr created time in 2 months

create barnchdperny/swarmkit-1

branch : wire-up-jobs-orchestrator

created branch time in 2 months

push eventdocker/swarmkit

Drew Erny

commit sha 23e108d462ce7d2f57c9d4501031201cf1a8b5b0

Add controlapi support for job services Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha c3605e0176e2954310a84641680a7757764853f4

Merge pull request #2893 from dperny/add-jobs-controlapi Add controlapi support for job services

view details

push time in 2 months

PR merged docker/swarmkit

Add controlapi support for job services

Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services.

+317 -7

2 comments

2 changed files

dperny

pr closed time in 2 months

push eventdperny/swarmkit-1

Arko Dasgupta

commit sha 9ccb20b27fa9a0e7b88800d7e196fa030da2a6a7

Remove hardcoded IPAM config subnet value for ingress network Fixes: https://docker.atlassian.net/browse/ENGORC-2651 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

view details

Drew Erny

commit sha a8bbe7de43b42fc3a8430819ed577e1218dc64aa

Merge pull request #2890 from arkodg/remove-ingress-hardcoded-subnet Remove hardcoded IPAM config subnet value for ingress network

view details

Drew Erny

commit sha e00554d91de7d0eb73f6ba6a8cf0dfff6e624eb6

Add Job protos Adds protocol buffers for implementing Jobs in swarmkit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 83e85e765d39b8e576dacecd37a8c8f21571ba0e

Add minimal Replicated Job Orchestrator and Tests Adds orchestrators for replicated and global jobs, and the basic tests. This commit exists mostly to keep the Ginkgo in mostly its own commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2b50c51c1794530271ff386e17181825a1f6f05f

Add service reconcilation for replicated jobs Adds service reconciliation logic for the replicated jobs orchestrator. This code does not function in production, and is not actually called except from the tests. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 0ff3113198c6a6112ec1119ee6b6e0525bc7d455

Add global job orchestrator skeleton Expands the skeleton structure of the global jobs orchestrator. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha f8f44b036797278c1c4f10d73681c6d90511e33f

Refactor replicated job orchestrator and add initialization Refactors the replicated job orchestrator to make testing simpler, and then adds initialization logic to it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 4598e8d37763aff92481f2dc7f7d7bce5250359b

Add store event logic to replicated jobs orchestrator Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha acdada5cee5acee253690effa017e026ba85a847

Refactor global job orchestrator Refactors the global job orchestrator along the same lines as the replicated job orchestrator, in order to better decouple the event-driven orchestrator logic from the reconciliation logic. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 38317eeeb63abe015b5eb9dfa25e7b7c5dcf6272

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 23e108d462ce7d2f57c9d4501031201cf1a8b5b0

Add controlapi support for job services Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 2 months

push eventdocker/swarmkit

Arko Dasgupta

commit sha 9ccb20b27fa9a0e7b88800d7e196fa030da2a6a7

Remove hardcoded IPAM config subnet value for ingress network Fixes: https://docker.atlassian.net/browse/ENGORC-2651 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

view details

Drew Erny

commit sha a8bbe7de43b42fc3a8430819ed577e1218dc64aa

Merge pull request #2890 from arkodg/remove-ingress-hardcoded-subnet Remove hardcoded IPAM config subnet value for ingress network

view details

Drew Erny

commit sha e00554d91de7d0eb73f6ba6a8cf0dfff6e624eb6

Add Job protos Adds protocol buffers for implementing Jobs in swarmkit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 83e85e765d39b8e576dacecd37a8c8f21571ba0e

Add minimal Replicated Job Orchestrator and Tests Adds orchestrators for replicated and global jobs, and the basic tests. This commit exists mostly to keep the Ginkgo in mostly its own commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2b50c51c1794530271ff386e17181825a1f6f05f

Add service reconcilation for replicated jobs Adds service reconciliation logic for the replicated jobs orchestrator. This code does not function in production, and is not actually called except from the tests. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 0ff3113198c6a6112ec1119ee6b6e0525bc7d455

Add global job orchestrator skeleton Expands the skeleton structure of the global jobs orchestrator. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha f8f44b036797278c1c4f10d73681c6d90511e33f

Refactor replicated job orchestrator and add initialization Refactors the replicated job orchestrator to make testing simpler, and then adds initialization logic to it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 4598e8d37763aff92481f2dc7f7d7bce5250359b

Add store event logic to replicated jobs orchestrator Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha acdada5cee5acee253690effa017e026ba85a847

Refactor global job orchestrator Refactors the global job orchestrator along the same lines as the replicated job orchestrator, in order to better decouple the event-driven orchestrator logic from the reconciliation logic. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 38317eeeb63abe015b5eb9dfa25e7b7c5dcf6272

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 2 months

pull request commentdocker/swarmkit

Add controlapi support for job services

i'm rebasing the feature-jobs branch because i think there was some kind of vendoring issue on it which is what's causing this PR to fail right now.

dperny

comment created time in 2 months

push eventdperny/docker

Arko Dasgupta

commit sha f3a3ea0d3c7f4d4da035db871d3c8a8bbb51371f

Fix flaky TestServiceWithDefaultAddressPoolInit 1.This commit replaces serviceRunningCount with swarm.RunningTasksCount to accurately check if the service is running with the accurate number of instances or not. serviceRunningCount was only checking the ServiceList and was not checking if the tasks were running or not This adds a safe barrier to execute docker network inspect commands for overlay networks which get created asynchronously via Swarm 2. Make sure client connections are closed 3. Make sure every service and network name is unique 4. Make sure services and networks are cleaned up Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

view details

Arko Dasgupta

commit sha a65dee30fc36ed974940b699ed22ab9242eac6cf

Move defer method to the top right after New is called Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

view details

Jintao Zhang

commit sha f8f6f7c2a0e1e1e8b541b29b0f1bdae44964e714

cleanup: remove SetDead function Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>

view details

Sebastiaan van Stijn

commit sha 556d26c07d068d92fd896428ab4ac890554239d7

pkg/term: refactor TestEscapeProxyRead - use subtests to make it clearer what the individual test-cases are, and to prevent tests from depending on values set by the previous test(s). - remove redundant messages in assert (gotest.tools already prints a useful message if assertions fail). Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ce77a804b86886489fae47aa15a3ee5a0a8815e0

docker-py: skip flaky AttachContainerTest::test_attach_no_stream Seen failing a couple of times: ``` [2019-09-02T08:40:15.796Z] =================================== FAILURES =================================== [2019-09-02T08:40:15.796Z] __________________ AttachContainerTest.test_attach_no_stream ___________________ [2019-09-02T08:40:15.796Z] tests/integration/api_container_test.py:1250: in test_attach_no_stream [2019-09-02T08:40:15.796Z] assert output == 'hello\n'.encode(encoding='ascii') [2019-09-02T08:40:15.796Z] E AssertionError: assert b'' == b'hello\n' [2019-09-02T08:40:15.796Z] E Right contains more items, first extra item: 104 [2019-09-02T08:40:15.796Z] E Use -v to get the full diff ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Jintao Zhang

commit sha e6fce00ec83df2f23523b836f647b8f3df97953f

TestCase: use `icmd.RunCmd` instead `icmd.StartCmd` Use `cli.Docker` instead `dockerCmdWithResult`. Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>

view details

Sebastiaan van Stijn

commit sha 54d021ef8f56f90cc4f7fc604190560361dd6b0c

awslogs: remove unused eventBuffer update We return immediately after this, so no need to update eventBuffer: ``` 16:04:35 daemon/logger/awslogs/cloudwatchlogs.go:554:5: SA4006: this value of `eventBuffer` is never used (staticcheck) 16:04:35 eventBuffer = eventBuffer[:0] ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ef2872132d40ba56b29f7c8a08492ebe3abb1f7c

awslogs: replace deprecated session.New() with session.NewSession() ``` 16:04:35 daemon/logger/awslogs/cloudwatchlogs.go:312:25: SA1019: session.New is deprecated: Use NewSession functions to create sessions instead. NewSession has the same functionality as New except an error can be returned when the func is called instead of waiting to receive an error until a request is made. (staticcheck) 16:04:35 return ec2metadata.New(session.New()) ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kir Kolyshkin

commit sha 2e0cafb01bf7871cb66f98da6123d36ffe043823

awslogs: refactor create() Get rid of too many nested if statements. Remove the redundand check for err != nil, fixing the following lint issue: > daemon/logger/awslogs/cloudwatchlogs.go:452:10: nilness: tautological condition: non-nil != nil (govet) > if err != nil { > ^ Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

view details

Sebastiaan van Stijn

commit sha 6e5a304675648a9f3d9b0ae95e7610337c8bd6c3

container.ConfigFilePath: use same signature on Windows This made my IDE unhappy; `ConfigFilePath` is an exported function, so it makes sense to use the same signature for both Linux and Windows. This patch also adds error handling (same as on Linux), even though the current implementation will never return an error (it's good practice to handle errors, so I assumed this would be the right approach) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha e128f175089d2508dae0a39f4ec86d4733d1ee44

Rename variable for consistency Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 6ee61f54935d77d3d5667b4a3ba8f0be6c3de23d

Jenkinsfile: create bundles for Windows stages CI already stores the logs of the test daemon, so we might as well store them as artifacts ``` [2019-09-03T12:49:39.835Z] INFO: Tidying up at end of run [2019-09-03T12:49:39.835Z] INFO: Saving daemon under test log (d:\CI-2\CI-3593e7622\dut.out) to C:\windows\TEMP\CIDUT.out [2019-09-03T12:49:39.835Z] INFO: Saving daemon under test log (d:\CI-2\CI-3593e7622\dut.err) to C:\windows\TEMP\CIDUT.err ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 1fbadd76b7ceab6e5bb683a7471f5cae8e715495

Jenkinsfile: Windows: enabled debug-mode for daemon under test Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Tobias Klauser

commit sha 7aeb3efcb40c907e9d19cd75bac2ad88aaf7fa19

Use unix.Uname instead of shelling out to uname on darwin/freebsd Reuse the linux implementation based on Uname from golang.org/x/sys/unix for darwin and freebsd. Signed-off-by: Tobias Klauser <tklauser@distanz.ch>

view details

Tibor Vass

commit sha 30166b9a4bc93a263ff084e1b01ea797bb9f3393

Merge pull request #39858 from thaJeztah/Jenkinsfile_windows_bundles Jenkinsfile: create bundles for Windows stages

view details

Sebastiaan van Stijn

commit sha 651d146fa4d199ec928a1cd073a37d5a86dcb27f

Poule: remove random assign Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Brian Goff

commit sha 3b23f90339678b9e59ee95cb24b95f99bd458526

Merge pull request #39872 from thaJeztah/remove_auto_assign Poule: remove random assign

view details

Vikram bir Singh

commit sha ebf12dbda08633375ab12387255d3f617ee9be38

Reimplement iteration over fileInfos in getOrphan. 1. Reduce complexity due to nested if blocks by using early return/continue 2. Improve logging Changes suggested as a part of code review comments in 39748 Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com>

view details

Jintao Zhang

commit sha 9ef9a337f6751c5d2d67e70dccb4bc0489915dbf

Update containerd to v1.2.9 Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>

view details

Tibor Vass

commit sha ef2890c295bda0a353286b3de86ac9a7bfb4cfe7

Merge pull request #39848 from thaJeztah/docker_py_disable_flaky docker-py: skip flaky AttachContainerTest::test_attach_no_stream

view details

push time in 2 months

issue commentmoby/moby

Dockerd eats too much RAM

I know it's a bit of a pain, so I really appreciate you getting that heap dump. If it is a memory leak, the heap dump will show it pretty obviously, and it'll be like a 5 minute fix more than likely.

goetas

comment created time in 2 months

issue commentmoby/moby

Dockerd eats too much RAM

gonna need to figure out how to profile swarm over a long period of time, as i suspect the problem may be a memory leak.

goetas

comment created time in 2 months

PR opened docker/swarmkit

Add controlapi support for job services

Adds support to the controlapi for creating and updating job-mode services. This still does not include correct plumbing to execute job-type services.

+308 -9

0 comment

2 changed files

pr created time in 2 months

push eventdocker/swarmkit

Drew Erny

commit sha 3d8c0ca4adbc13fc392a4deec4f718d4062e2b5f

Fix flaky tests It is likely that a large portion of test flakiness, especially in CI, comes from the fact that swarmkit components under test are started in goroutines, but those goroutines never have an opportunity to run. This adds code ensuring those goroutines are scheduled and run, which should hopefully solve many inexplicably flaky tests. Additionally, increased test timeouts, to hopefully cover a few more flaky cases. Finally, removed direct use of the atomic package, in favor of less efficient but higher-level mutexes. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 06a356671bc11e4fd5d754f257f9c5f93ec5c563) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha caa9a343fa571bf3648a86cbf6306442c863baac

add golangci-lint gometalinter is deprecated, and golangci-lint is its recommended successor. This commit adds golangci-lint as the linter for swarmkit. In addition, golangci-lint found a few issues in the code that were not yet identified, and so those issues have been fixed. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 27c2d27e23e76243bb49ec5d803a6e40b3f96f7a) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

nmengin

commit sha ea1a2bd3d6b9045e9405dee7b825fd73135efbac

Set bigger grpc value to initialize connection broker Signed-off-by: nmengin <nicolas@containo.us> (cherry picked from commit 127e816ed8c8de4c981e02c9d0b45c70d3baa824) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha bcd123c34b994e8604e80ba044ce43e2f58e55b6

Fix update out of sequence A simple but old error has recently become evident. Due to the fact that we read an object and then write it back across the boundaries of a transaction, it is possible for the task object to have changed in between transactions. This would cause the attempt to write out the old task to suffer an "Update out of sequence" error. This fix simply reads the latest version of the task back out within the boundary of a transaction to avoid the race. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit d68ac46e3b11d7384472677d210bb0ce941284dc) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha bbe341867eae1615faf8a702ec05bfe986e73e06

Merge pull request #2889 from thaJeztah/19.03_backport_fix_update_out_of_sequence [19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets

view details

push time in 2 months

PR merged docker/swarmkit

[19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets

relates to

  • https://github.com/moby/moby/pull/39531 integration-cli: fix swarm tests flakiness
  • https://github.com/docker/engine/pull/345 [19.03 backport] integration-cli: fix swarm tests flakiness

Backports of

  • https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
  • https://github.com/docker/swarmkit/pull/2866 Swap gometalinter for golangci-lint
  • https://github.com/docker/swarmkit/pull/2869 Increase max recv gRPC message size to initialize connection broker
  • related / similar to https://github.com/moby/moby/pull/38103 / https://github.com/docker/engine/pull/102 cluster: set bigger grpc limit for array requests
  • related / similar to https://github.com/moby/moby/pull/39306 Increase max recv gRPC message size for nodes and secrets
  • fixes https://github.com/docker/swarmkit/issues/2733 Error generated when messages size is too big
  • https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence

Cherry-pick were clean, no conflicts

# https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
git cherry-pick -s -S -x 06a356671bc11e4fd5d754f257f9c5f93ec5c563

# https://github.com/docker/swarmkit/pull/2866 Swap gometalinter for golangci-lint
git cherry-pick -s -S -x 27c2d27e23e76243bb49ec5d803a6e40b3f96f7a

# https://github.com/docker/swarmkit/pull/2869 Increase max recv gRPC message size to initialize connection broker
git cherry-pick -s -S -x 127e816ed8c8de4c981e02c9d0b45c70d3baa824

# https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence
git cherry-pick -s -S -x d68ac46e3b11d7384472677d210bb0ce941284dc
+135 -113

6 comments

18 changed files

thaJeztah

pr closed time in 2 months

create barnchdperny/swarmkit-1

branch : add-jobs-controlapi

created branch time in 2 months

push eventdocker/swarmkit

Sebastiaan van Stijn

commit sha daf87201f686710974b282d4c0b0c8422e55dac9

Bump Golang 1.11.13 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha be528e80a841ae89e1bd7539201f12c4de496ab3

Bump Golang 1.12.9 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 43fac9c8049f7dd2ae42d87f60d2a88f86f63853

Update tests for new output with Go 1.12 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 42085d2f8e43a3ed90ed289d3f3ed3de57837100

YOLO see if this works ``` panic: Log in goroutine after TestWorkerUpdate has completed goroutine 1175 [running]: testing.(*common).logDepth(0xc00019a100, 0xc0000396a0, 0x1d, 0x3) /usr/local/go/src/testing/testing.go:634 +0x51a testing.(*common).log(...) /usr/local/go/src/testing/testing.go:614 testing.(*common).Log(0xc00019a100, 0xc00006fb00, 0x1, 0x1) /usr/local/go/src/testing/testing.go:642 +0x79 github.com/docker/swarmkit/agent.(*mockTaskController).Remove(0xc00020a780, 0x13b6f40, 0xc0000b4010, 0x13b6f40, 0xc0000b4010) /home/circleci/.go_workspace/src/github.com/docker/swarmkit/agent/worker_test.go:604 +0x8e github.com/docker/swarmkit/agent.reconcileTaskState.func1(0xc0004dac60) /home/circleci/.go_workspace/src/github.com/docker/swarmkit/agent/worker.go:270 +0x137 created by github.com/docker/swarmkit/agent.reconcileTaskState /home/circleci/.go_workspace/src/github.com/docker/swarmkit/agent/worker.go:313 +0x1a6f FAIL github.com/docker/swarmkit/agent 4.244s make: *** [coverage] Error 1 Exited with code 2 ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 958d149179db019aef3a065f23b35455b2dd54ca

Merge pull request #2880 from thaJeztah/bump_golang Bump to golang 1.12.9

view details

Drew Erny

commit sha 9562ffc58d55bc9aad8445a84d4953d661cd91db

Bump vendoring to match current docker/docker master Updates vendor.conf so that the versions used all match the corresponding versions in github.com/docker/docker. Bumping the vendored version of gogo/protobuf required a few changes to the swarmkit code for generating protos. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha e0f62d18ab5fcb3afc8b0ecdc6d33a020b8128c7

Merge pull request #2886 from dperny/bump-vendoring Bump vendoring to match current docker/docker master

view details

Drew Erny

commit sha e65e043e8dfdc68c6db01c43455594f8c8172611

Add Job protos Adds protocol buffers for implementing Jobs in swarmkit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 8e31407036f444bcfef2dd4460ce19caaaec1a3f

Add minimal Replicated Job Orchestrator and Tests Adds orchestrators for replicated and global jobs, and the basic tests. This commit exists mostly to keep the Ginkgo in mostly its own commit. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 17ddee33c274c1efe2b7aa82c291287c5c2a11a8

Add service reconcilation for replicated jobs Adds service reconciliation logic for the replicated jobs orchestrator. This code does not function in production, and is not actually called except from the tests. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 7ce22f57e0e0fb72851d2722314d4d0769d9b923

Add global job orchestrator skeleton Expands the skeleton structure of the global jobs orchestrator. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 31672b78a70e292445fe29c3219a28e185f04482

Refactor replicated job orchestrator and add initialization Refactors the replicated job orchestrator to make testing simpler, and then adds initialization logic to it. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 170ebc5fc4a2d86a954bef255a9e4a9222fe8ed6

Add store event logic to replicated jobs orchestrator Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha a4864a6b7e7be4355252e692744ac860e90f8419

Refactor global job orchestrator Refactors the global job orchestrator along the same lines as the replicated job orchestrator, in order to better decouple the event-driven orchestrator logic from the reconciliation logic. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 900ae8c87910f5b2e1a9ead6ed99d7c7ab75ec80

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 2 months

pull request commentdocker/swarmkit

[19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets

test is known to be flaky despite many attempts to improve it.

thaJeztah

comment created time in 2 months

push eventdocker/swarmkit

Arko Dasgupta

commit sha 9ccb20b27fa9a0e7b88800d7e196fa030da2a6a7

Remove hardcoded IPAM config subnet value for ingress network Fixes: https://docker.atlassian.net/browse/ENGORC-2651 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

view details

Drew Erny

commit sha a8bbe7de43b42fc3a8430819ed577e1218dc64aa

Merge pull request #2890 from arkodg/remove-ingress-hardcoded-subnet Remove hardcoded IPAM config subnet value for ingress network

view details

push time in 2 months

PR merged docker/swarmkit

Remove hardcoded IPAM config subnet value for ingress network

Fixes: https://docker.atlassian.net/browse/ENGORC-2651

+2 -6

5 comments

1 changed file

arkodg

pr closed time in 2 months

pull request commentdocker/swarmkit

[WIP][18.03 backport] Fix leaking task resources when nodes are deleted

known flaky test in the codebase that i have spent many hours on and still not fixed.

thaJeztah

comment created time in 2 months

pull request commentdocker/swarmkit

[WIP][18.03 backport] Fix leaking task resources when nodes are deleted

@kolyshkin just rebased and force-pushed.

thaJeztah

comment created time in 2 months

push eventthaJeztah/swarmkit

Jean Rouge

commit sha 748751b900c8cd16b32c36debce19022cd942dd2

Adding a Dockerfile and making it easy to use it for dev Credit goes to dperny (https://github.com/docker/swarmkit/pull/2687) **- What I did** Adds a Dockerfile for the swarmkit project, to easily get off the ground. Modifies the Makefile to make intelligent use of Docker. Also made small clean up changes to the Makefile. **- How I did it** Modifies the Makefile to have two paths: containerized.mk, which builds the docker image and forwards any make targets to a container, and direct.mk, which encompasses the old Makefile's workflow. By default, nothing will run inside a container. Set the environment variable `DOCKER_SWARMKIT_USE_CONTAINER` to use dockerized making. Also leverages docker-sync for synchronizing code to the container if the `DOCKER_SWARMKIT_USE_DOCKER_SYNC` env variable is set; comes in handy on Macs, for example. **- How to test it** Set `DOCKER_SWARMKIT_USE_CONTAINER` and verify that your favorite make targets all work! Signed-off-by: Jean Rouge <jer329@cornell.edu> (cherry picked from commit f8c048cd8c00d3530bbdd87e160c6d0a6e30ac42) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kir Kolyshkin

commit sha 344718a037d6724015c1b2adfbb88f4893eed5bb

go {build,test}: rm -i option Looks like it was used to get faster incremental builds. Nowdays (since Go 1.10) there is no need to use it, because go build cache is used [1]. This fixes `make binaries` on my system where golang is installed as read-only snap: > $ make binaries > 🐳 bin/swarmd > go build runtime/cgo: open /snap/go/2635/pkg/linux_amd64/runtime/cgo.a: read-only file system > direct.mk:100: recipe for target 'bin/swarmd' failed [1] https://groups.google.com/forum/#!msg/golang-dev/qfa3mHN4ZPA/X2UzjNV1BAAJ Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit cb50952a3a15a58bbbb76003994a694d7a658872) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kir Kolyshkin

commit sha 24212502490316923e5a01d45eb2466c95703c93

Makefile: use gometalinter Instead of running many small source code checkers and linters one by one, let's use gometalinter that runs them all in parallel. While at it, remove the individual make targets (fmt, vet, lint, ineffassign, and misspell) and replace with a single check target. One thing it provides is faster source validation. BEFORE: real 0m24.025s user 1m2.646s sys 0m3.860s (note these timings are without building binaries, which for some reason was a dependency of the vet target) AFTER: real 0m6.330s user 0m20.255s sys 0m1.019s In addition to this, it is now way easier to add/remove the checks, as well as to filter out some errors from linters that we consider false positives. [v2: add 2m deadline] [v3: use gometalinter --install; move configuration to .json] Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit 318574db9d8b5953f966b7f7e0e8803fb4de03b4) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kir Kolyshkin

commit sha 262160c49afc4218bd1a1aea5c2cd9bbf0ba7eea

gometalinter: add deadcode linter ...and fix the following initial bunch of warnings: agent/session.go:18:1:warning: errSessionDisconnect is unused (deadcode) agent/errors.go:7:1:warning: errTaskNoController is unused (deadcode) agent/errors.go:7:1:warning: errTaskStatusUpdateNoChange is unused (deadcode) agent/errors.go:7:1:warning: errTaskNotAssigned is unused (deadcode) agent/errors.go:7:1:warning: errTaskInvalid is unused (deadcode) ca/transport.go:21:1:warning: timeoutError is unused (deadcode) ca/config.go:29:1:warning: nodeCSRFilename is unused (deadcode) cmd/swarmctl/node/common.go:58:1:warning: changeNodeMembership is unused (deadcode) integration/cluster.go:44:1:warning: newTestCluster is unused (deadcode) manager/orchestrator/global/global.go:590:1:warning: isTaskCompleted is unused (deadcode) Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit 04ae7e3c35c50f0d28c27b79ccbb45db2c28f780) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kir Kolyshkin

commit sha 17d7a875a7741088fd609e0ca61992e3b1c0cc34

gometalinter: add deadcode, goimports, unconvert ... and fix some warnings from unconvert: ca/config.go:627:58:warning: unnecessary conversion (unconvert) manager/allocator/cnmallocator/portallocator.go:410:38:warning: unnecessary conversion (unconvert) manager/allocator/cnmallocator/portallocator.go:415:50:warning: unnecessary conversion (unconvert) manager/controlapi/service.go:200:47:warning: unnecessary conversion (unconvert) manager/controlapi/service.go:210:45:warning: unnecessary conversion (unconvert) manager/controlapi/service.go:220:35:warning: unnecessary conversion (unconvert) manager/dispatcher/nodes.go:159:33:warning: unnecessary conversion (unconvert) manager/state/store/memory.go:692:91:warning: unnecessary conversion (unconvert) Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit 278edc28b8c4330496c01f39b72c32a054af766d) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kir Kolyshkin

commit sha 45714ba50d7c140eeb157dee912e6b71f98740f7

gometalinter: add gosimple ... and fix warnings reported by it: agent/exec/dockerapi/adapter.go:147:2:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) agent/testutils/fakes.go:143:2:warning: should use a simple channel send/receive instead of select with a single case (S1000) (gosimple) agent/testutils/fakes.go:151:2:warning: should use a simple channel send/receive instead of select with a single case (S1000) (gosimple) ca/certificates_test.go:708:2:warning: should use for range instead of for { select {} } (S1000) (gosimple) ca/config.go:630:12:warning: should use time.Until instead of t.Sub(time.Now()) (S1024) (gosimple) ca/config_test.go:790:3:warning: should use a simple channel send/receive instead of select with a single case (S1000) (gosimple) ca/external_test.go:116:3:warning: should use a simple channel send/receive instead of select with a single case (S1000) (gosimple) cmd/swarm-bench/collector.go:26:2:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) cmd/swarmctl/node/common.go:51:2:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) cmd/swarmctl/node/common.go:89:2:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) cmd/swarmctl/node/common.go:172:2:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) ioutils/ioutils_test.go:28:5:warning: should use !bytes.Equal(actual, expected) instead (S1004) (gosimple) manager/allocator/cnmallocator/networkallocator.go:818:3:warning: should merge variable declaration with assignment on next line (S1021) (gosimple) manager/constraint/constraint.go:59:7:warning: should omit comparison to bool constant, can be simplified to !matched (S1002) (gosimple) manager/constraint/constraint.go:67:7:warning: should omit comparison to bool constant, can be simplified to !matched (S1002) (gosimple) manager/dispatcher/dispatcher.go:1095:5:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) manager/dispatcher/dispatcher.go:1095:5:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) manager/dispatcher/dispatcher.go:1095:5:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) manager/dispatcher/dispatcher_test.go:2090:2:warning: redundant return statement (S1023) (gosimple) manager/manager.go:1005:4:warning: should replace loop with m.config.NetworkConfig.DefaultAddrPool = append(m.config.NetworkConfig.DefaultAddrPool, cluster.DefaultAddressPool...) (S1011) (gosimple) manager/metrics/collector.go:191:2:warning: redundant return statement (S1023) (gosimple) manager/metrics/collector.go:222:2:warning: redundant return statement (S1023) (gosimple) manager/orchestrator/replicated/update_test.go:53:3:warning: should use for range instead of for { select {} } (S1000) (gosimple) manager/orchestrator/taskinit/init.go:83:32:warning: should use time.Until instead of t.Sub(time.Now()) (S1024) (gosimple) manager/state/raft/raft.go:1185:2:warning: should use 'return <expr>' instead of 'if <expr> { return <bool> }; return <bool>' (S1008) (gosimple) manager/state/raft/raft.go:1594:2:warning: 'if err != nil { return err }; return nil' can be simplified to 'return err' (S1013) (gosimple) node/node.go:1209:2:warning: redundant return statement (S1023) (gosimple) node/node.go:1219:2:warning: redundant return statement (S1023) (gosimple) watch/sinks_test.go:42:2:warning: should merge variable declaration with assignment on next line (S1021) (gosimple) Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit 0e8bb705bab9c06e98dc1f23e8a5be123b002a8b) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Jean Rouge

commit sha 80beffd930c21a39e16bd7b198a43e195d751442

Adding a `hack/debug` script That's basically just a light wrapper around delve, to make it nicer to use (stops it after the session is done, and doesn't ignore interrupts like the vanilla version does). Also amending the containerized Makefile to add the right run options when using delve. That's controlled by the `DOCKER_SWARMKIT_USE_DELVE` env variable; it's also possible to override the port delve uses via the `DOCKER_SWARMKIT_DELVE_PORT` env variable. Signed-off-by: Jean Rouge <jer329@cornell.edu> (cherry picked from commit 0a88f50964c080e54c0ee78c4639cc53441431a9) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha eb383fd34021791f4e9b763a666f24865f37e22e

Fix make check gometalinter dropped support for gosimple, which is deprecated anyway and has been subsumed by staticcheck. This commit removes gosimple from our list of enabled linters (as it's no longer valid). It does not enable staticcheck, because staticcheck throws too many errors. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 3bfc201ae805c67d604b45c2c4f48350f5873bae) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ab656b22dec914a659f6d904a1b1603f34a8d156

gofmt files Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 54035b8d45a8315cc8cbf7b188bd54b66be2b63e

containerd-adapter: unnecessary conversion (unconvert) ``` agent/exec/containerd/adapter.go:313:26: unnecessary conversion (unconvert) timeout = time.Duration(10 * time.Second) ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 568fb3ba8046494bd8e2ef0d3a79c81fb73a6359

add golangci-lint gometalinter is deprecated, and golangci-lint is its recommended successor. This commit adds golangci-lint as the linter for swarmkit. In addition, golangci-lint found a few issues in the code that were not yet identified, and so those issues have been fixed. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 27c2d27e23e76243bb49ec5d803a6e40b3f96f7a) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Kir Kolyshkin

commit sha d0282d0d413a6f312cb95f8829f6d687777765f1

Switch to go 1.11 Whitespace changes are caused by the fact that gofmt from go-1.11 uses a different heuristic as to how to format the file, making the source code that was OK for go-1.10 causing a warning with go-1.11. NOTE this whitespace change makes the gofmt from go-1.10 complain, so please upgrade your golang. [v2: regen pb files] Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit fd2d7f2ef925282e8cd2920efa253151749102ad) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha fa7db2db12b33e933e6c6d4d5b224945ec51fef8

Merge pull request #2877 from thaJeztah/18.03_backport_golangci_lint [18.03 backport] makefile and linting changes, bump go 1.11

view details

Drew Erny

commit sha 99a65a992799f1df218818e30d00a0049950e265

Fix leaking task resources when nodes are deleted When a node is deleted, its tasks are asked to restart, which involves putting them into a desired state of Shutdown. However, the Allocator will not deallocate a task which is not in an actual state of a terminal state. Once a node is deleted, the only opportunity for its tasks to recieve updates and be moved to a terminal state is when the function moving those tasks to TaskStateOrphaned is called, 24 hours after the node enters the Down state. However, if a leadership change occurs, then that function will never be called, and the tasks will never be moved to a terminal state, leaking resources. With this change, upon node deletion, all of its tasks will be moved to TaskStateOrphaned, allowing those tasks' resources to be cleaned up. Additionally, as part of this backport, avoid using the gogo types.TimestampNow function, which does not exist in the vendored version. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 8467e6a43f9cee66f46efb457b2c22f46ae5da0b) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Drew Erny

commit sha 847a883dfd5a23bc81a7422bb73d292706660bf5

Only update non-terminal tasks on node removal. When a node is removed, its tasks are set in state ORPHANED. This does not need to be done for tasks that are already in a terminal state, and if all tasks in all states are updated, the size of the transaction may grow too large to process, and node removal becomes impossible. This changes to only set non-terminal tasks to state ORPHANED, and terminal tasks are left alone. Cherry pick does not apply cleanly, but the fix is rather simple. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit d5df26594f5b74a9de41e376206c36abffc961fe) Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

push time in 2 months

pull request commentdocker/swarmkit

Remove hardcoded IPAM config subnet value for ingress network

There's not going to be a simple way to add a test case. The code passes through too many places. I'm satisfied with the test case from Moby handling this if you are.

Otherwise, LGTM. Glad this is a simple fix.

arkodg

comment created time in 2 months

push eventdocker/stacks

Jason Schroeder

commit sha c0500a5a0f24c8e6a9dee4bbe82006f010c143c6

Aligning fake_stack tests, implementation and comments with fake_config. 84% test coverage for fake_stack_store.go.

view details

Drew Erny

commit sha 1d9e070c3d4051cbdd092f32bc094fb1e11e756b

Merge pull request #59 from undertheflowerpot/FirstTry Aligning fake_stack tests, implementation and comments

view details

push time in 2 months

PR merged docker/stacks

Aligning fake_stack tests, implementation and comments

with fake_config. 84% test coverage for fake_stack_store.go.

+516 -250

0 comment

6 changed files

undertheflowerpot

pr closed time in 2 months

push eventdocker/stacks

Jason Schroeder

commit sha de6e2d9d59970e71a5ac9803f9c46432a4788f7d

Aligning fake_service tests, implementation and comments with fake_config. 100% test coverage for fake_service_store.go

view details

Drew Erny

commit sha 576520e3ff308243efeca5dae6f56606f86b4d3f

Merge pull request #58 from undertheflowerpot/FirstTry Aligning fake_service tests, implementation and comments

view details

push time in 2 months

PR merged docker/stacks

Aligning fake_service tests, implementation and comments

with fake_config. 100% test coverage for fake_service_store.go

+317 -86

0 comment

5 changed files

undertheflowerpot

pr closed time in 2 months

push eventdocker/stacks

Jason Schroeder

commit sha 965eeb9ca7b06523e6e8613f6e26b1adbb5e264e

Improving fake_config tests, implementation and comments. These changes are mirrored in pending commits for fake_stack, fake_secret, fake_network, fake_service

view details

Jason Schroeder

commit sha 3c1f39bae387f851520d1a2b4b75c11dcdacf175

Aligning fake_secret tests, implementation and comments with fake_config. 100% test coverage for fake_secret_store.go

view details

Jason Schroeder

commit sha e1f314e406816f5fe6426c125420522e01af702a

Aligning fake_network tests, implementation and comments with fake_config. 96% test coverage for fake_network_store.go because Network is a little different from fake_config.

view details

Drew Erny

commit sha ade8718d8d846df6cb374dfaedaa7c86afdf42be

Merge pull request #57 from undertheflowerpot/FirstTry fake_config and fake_secret tests, implementations, comments

view details

push time in 2 months

PR merged docker/stacks

fake_config and fake_secret tests, implementations, comments

fake_config is exemplar implementation for the fakes. Its comments, implementation and tests were improved. fake_config_store has 100% coverage.

fake_secret has been reworked to match fake_config and fake_secret_store has 100% coverage.

+978 -174

1 comment

7 changed files

undertheflowerpot

pr closed time in 2 months

PR closed docker/swarm

Add more robust detection of node OS

Here's the deal: this in the only reasonable way to do this without writing a bunch of code to, for example, pull the image manifest and inspect it directly, which is a whole rodeo i'm not interested in getting into. It's fragile and likely to break (again) later on, but whatever.

+110 -2

4 comments

3 changed files

dperny

pr closed time in 2 months

pull request commentmoby/moby

Add support for sending down service Running and Desired task counts

@cpuguy83 it seems like everything matches up with what I'm seeing in other green PRs now.

dperny

comment created time in 2 months

Pull request review commentdocker/stacks

fake_config and fake_secret tests, implementations, comments

 func (f *FakeNetworkStore) GetNetworks(plainFilters filters.Args) ([]dockerTypes  	for _, key := range f.SortedIDs() { 		network := f.networks[key]-		// if we're filtering on stack ID, and this network doesn't match, then-		// we should skip this network++		// if we're filtering on stack ID, and this network doesn't+		// match, then we should skip this network 		if hasFilter && network.Labels[types.StackLabel] != stackID { 			continue 		} 		// otherwise, we should append this network to the set-		if err := f.causeAnError(nil, "GetNetworks", *network); err != nil {+		if err := f.maybeTriggerAnError("GetNetworks", *network); err != nil { 			return nil, err 		}-		networks = append(networks, *network)+		networks = append(networks, *CopyNetworkResource(*network))

I'm seeing, in this change set, a fair bit of derefing pointers. Just to make sure, you're aware that if the struct being derefed itself contains more pointers, those pointers will still point to the originals? I'm pretty sure you already know this, but I wanted to check before I merge.

undertheflowerpot

comment created time in 2 months

push eventdocker/swarm

Drew Erny

commit sha 8eeaf3472324affecb93a72b5727dc5d1cce12e8

Add better OS detection Adds better detection of the OS type of the engine by getting the supported platforms defined in the manifest and creating an ostype constraint using them. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 38c89321393f53deeaf3f69af596121740f06545

Merge pull request #2961 from dperny/use-image-manifest-for-os Add better OS detection

view details

push time in 2 months

PR merged docker/swarm

Add better OS detection

Adds better detection of the OS type of the engine by getting the supported platforms defined in the manifest and creating an ostype constraint using them.

This is #2959 done the right way.

+277 -1

2 comments

6 changed files

dperny

pr closed time in 2 months

push eventdocker/swarmkit

Drew Erny

commit sha 7ffcbe4336f60c63b86b5d2510f4402582cd0da0

Refactor Jobs Orchestrators It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This commit merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work. Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha 2c87b1f464d14962cae4b0dd806c13f0d2350bde

Merge pull request #2887 from dperny/refactor-jobs-orchestrators [Jobs] Refactor jobs orchestrators

view details

push time in 2 months

PR merged docker/swarmkit

[Jobs] Refactor jobs orchestrators

It became evident in the process of writing the Global Jobs orchestrator that the Orchestrators required by both Replicated and Global jobs are essentially identical. This PR merges them into one combined orchestrator, which dispatches to the appropriate Reconcilers to do the actual work.

Unlike existing services, these orchestrators can be combined because the requirements of jobs are much simpler than that of services.

Supercedes #2884

+156 -142

2 comments

10 changed files

dperny

pr closed time in 2 months

push eventdocker/swarmkit

Drew Erny

commit sha 9562ffc58d55bc9aad8445a84d4953d661cd91db

Bump vendoring to match current docker/docker master Updates vendor.conf so that the versions used all match the corresponding versions in github.com/docker/docker. Bumping the vendored version of gogo/protobuf required a few changes to the swarmkit code for generating protos. Signed-off-by: Drew Erny <drew.erny@docker.com>

view details

Drew Erny

commit sha e0f62d18ab5fcb3afc8b0ecdc6d33a020b8128c7

Merge pull request #2886 from dperny/bump-vendoring Bump vendoring to match current docker/docker master

view details

push time in 2 months

PR merged docker/swarmkit

Bump vendoring to match current docker/docker master

Updates vendor.conf so that the versions used all match the corresponding versions in github.com/docker/docker.

+126479 -29972

1 comment

523 changed files

dperny

pr closed time in 2 months

pull request commentdocker/swarm

Add better OS detection

It seems I don't actually have write access on this repo...

dperny

comment created time in 2 months

push eventdperny/docker

Staf Wagemakers

commit sha 74e3edc7d1f55b553ec099f99cc774d6d9e6548b

Updated mkimage-arch.sh * reset umask to 022 * introduced PKGREQUIRED * introduced PKGREMOVE - to be able to remove linux etc on Parabola GNU/Linux * updated PKGIGNORE - cryptsetup & device-mapper removed to not break the installation - added not required packages * force link /etc/localtime * install pacman-mirrorlist Signed-off-by: Staf Wagemakers <staf@wagemakers.be>

view details

Chow

commit sha 75a59c65880acdef589d2e6781bec83a6ef54fde

Enable DNS Lookups for CIFS Volumes This comes from an old suggestion (https://github.com/docker/cli/issues/706#issuecomment-371157691) on an issue we were having and has since popped up again. For NFS volumes, Docker will do an IP lookup on the volume name. This is not done for CIFS volumes, which forces you to add the volume via IP address instead. This change will enable the IP lookup also for CIFS volumes. Signed-off-by: Shu-Wai Chow <shu-wai.chow@seattlechildrens.org>

view details

Pavel Matěja

commit sha ee09f5a4afb4442c6cb6297ac133e52d7201cc72

Don't try to load plugin without name This can happen when you have --config-only network Such attempt will fail anyway and it will create 15s delay in container startup Signed-off-by: Pavel Matěja <pavel@verotel.cz>

view details

Sebastiaan van Stijn

commit sha 18dac2cf32faeaada3bd4e8e2bffa576ad4329fe

TestMaskSecretKeys: add more test-cases Add tests for - case-insensitive matching of fields - recursive masking Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit db5f811216e70bcb4a10e477c1558d6c68f618c5) Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Sebastiaan van Stijn

commit sha ebb542b3f88d7f5551f6b6e1d8d2774a2c166409

TestMaskSecretKeys: use subtests Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 32d70c7e21631224674cd60021d3ec908c2d888c) Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Sebastiaan van Stijn

commit sha 73db8c77bfb2d0cbdf71ce491f3d3e66c9dd5be6

DebugRequestMiddleware: unconditionally scrub data field Commit 77b8465d7e68ca102d7aae839c7b3fe0ecd28398 added a secret update endpoint to allow updating labels on existing secrets. However, when implementing the endpoint, the DebugRequestMiddleware was not updated to scrub the Data field (as is being done when creating a secret). When updating a secret (to set labels), the Data field should be either `nil` (not set), or contain the same value as the existing secret. In situations where the Data field is set, and the `dockerd` daemon is running with debugging enabled / log-level debug, the base64-encoded value of the secret is printed to the daemon logs. The docker cli does not have a `docker secret update` command, but when using `docker stack deploy`, the docker cli sends the secret data both when _creating_ a stack, and when _updating_ a stack, thus leaking the secret data if the daemon runs with debug enabled: 1. Start the daemon in debug-mode dockerd --debug 2. Initialize swarm docker swarm init 3. Create a file containing a secret echo secret > my_secret.txt 4. Create a docker-compose file using that secret cat > docker-compose.yml <<'EOF' version: "3.3" services: web: image: nginx:alpine secrets: - my_secret secrets: my_secret: file: ./my_secret.txt EOF 5. Deploy the stack docker stack deploy -c docker-compose.yml test 6. Verify that the secret is scrubbed in the daemon logs DEBU[2019-07-01T22:36:08.170617400Z] Calling POST /v1.30/secrets/create DEBU[2019-07-01T22:36:08.171364900Z] form data: {"Data":"*****","Labels":{"com.docker.stack.namespace":"test"},"Name":"test_my_secret"} 7. Re-deploy the stack to trigger an "update" docker stack deploy -c docker-compose.yml test 8. Notice that this time, the Data field is not scrubbed, and the base64-encoded secret is logged DEBU[2019-07-01T22:37:35.828819400Z] Calling POST /v1.30/secrets/w3hgvwpzl8yooq5ctnyp71v52/update?version=34 DEBU[2019-07-01T22:37:35.829993700Z] form data: {"Data":"c2VjcmV0Cg==","Labels":{"com.docker.stack.namespace":"test"},"Name":"test_my_secret"} This patch modifies `maskSecretKeys` to unconditionally scrub `Data` fields. Currently, only the `secrets` and `configs` endpoints use a field with this name, and no other POST API endpoints use a data field, so scrubbing this field unconditionally will only scrub requests for those endpoints. If a new endpoint is added in future where this field should not be scrubbed, we can re-introduce more fine-grained (path-specific) handling. This patch introduces some change in behavior: - In addition to secrets, requests to create or update _configs_ will now have their `Data` field scrubbed. Generally, the actual data should not be interesting for debugging, so likely will not be problematic. In addition, scrubbing this data for configs may actually be desirable, because (even though they are not explicitely designed for this purpose) configs may contain sensitive data (credentials inside a configuration file, e.g.). - Requests that send key/value pairs as a "map" and that contain a key named "data", will see the value of that field scrubbed. This means that (e.g.) setting a `label` named `data` on a config, will scrub/mask the value of that label. - Note that this is already the case for any label named `jointoken`, `password`, `secret`, `signingcakey`, or `unlockkey`. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit c7ce4be93ae8edd2da62a588e01c67313a4aba0c) Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Sebastiaan van Stijn

commit sha f8a0f26843bc5aff33cf9201b75bd4bdbb48a3ad

DebugRequestMiddleware: Remove path handling Path-specific rules were removed, so this is no longer used. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 530e63c1a61b105a6f7fc143c5acb9b5cd87f958) Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Akihiro Suda

commit sha 34f4729bc097f5d8fa0a8942d80889e0e8d32a67

rootless: allow exposing dockerd TCP socket easily eg. $ DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS="-p 0.0.0.0:2376:2376/tcp" \ dockerd-rootless.sh --experimental \ -H tcp://0.0.0.0:2376 \ --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem This commit bumps up RootlessKit from v0.4.1 to v0.6.0: https://github.com/rootless-containers/rootlesskit/compare/27a0c7a2483732b33d4192c1d178c83c6b9e202d...2fcff6ceae968a1d895e6205e5154b107247356f Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

view details

Sebastiaan van Stijn

commit sha 7bfe48cc00318f9d4cf388237855012aafac56b0

Bump docker-py to 4.0.2, and run tests from upstream repository This removes all the installation steps for docker-py from the Dockerfile, and instead builds the upstream Dockerfile, and runs docker-py tests in a container. To test; ``` make test-docker-py ... Removing bundles/ ---> Making bundle: dynbinary (in bundles/dynbinary) Building: bundles/dynbinary-daemon/dockerd-dev Created binary: bundles/dynbinary-daemon/dockerd-dev ---> Making bundle: test-docker-py (in bundles/test-docker-py) ---> Making bundle: .integration-daemon-start (in bundles/test-docker-py) Using test binary docker Starting dockerd INFO: Waiting for daemon to start... . INFO: Building docker-sdk-python3:3.7.0... sha256:686428ae28479e9b5c8fdad1cadc9b7a39b462e66bd13a7e35bd79c6a152a402 INFO: Starting docker-py tests... ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-4.1.0, py-1.8.0, pluggy-0.9.0 rootdir: /src, inifile: pytest.ini plugins: timeout-1.3.3, cov-2.6.1 collected 359 items tests/integration/api_build_test.py .......s.... .... ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 968345bc5c0e3817d5d271124a2da11092e7447e

Makefile: Allow passing DOCKER_TEST_HOST and TESTDEBUG to container Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha ba8f4c7994668992c27619d63e7056770ffb08ad

docker-py: don't build --quiet is TESTDEBUG is set Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 3c15cea650e356f11550e435efa79bea4be88504

docker-py: use host-network for nested build of docker-py When building this image docker-in-docker, the DNS in the environment may not be usable for the build-container, causing resolution to fail: ``` 02:35:31 W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release.gpg Temporary failure resolving 'deb.debian.org' ``` This patch detects if we're building from within a container, and if so, skips creating a networking namespace for the build by using `--network=host`. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 980f2813b435166b0fa3509a84f5e435e355f623

docker-py: skip flaky tests Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Pascal Bach

commit sha 78405559cfe5987174aa2cb6463b9b2c1b917255

Check for BRIDGE_VLAN_FILTERING in overlay section Overlay networking in docker stack does not work correctly if this option is missing, docker will output the following error: ``` enabling default vlan on bridge br0 failed open /sys/class/net/br0/bridge/default_pvdi: permission denied ``` This because `default_pvdi` does not exist without this option. Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Simon Ausserlechner <simon.ausserlechner@siemens.com>

view details

Sebastiaan van Stijn

commit sha 6aafe0fd9ed34c4d91adde62c86e47518274a5e8

WIP Move docker-py tests first again See if networking works if we run it first Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

view details

Sebastiaan van Stijn

commit sha 618be0642450e5a8c0ec34f75a2cdf635567eb36

Merge pull request #37147 from bachp/patch-1 Check for BRIDGE_VLAN_FILTERING in overlay section

view details

Andrew Hsu

commit sha c222c5ac6f906b6766e8c431a873187ccb957ead

allow running of single integration test Signed-off-by: Andrew Hsu <andrewhsu@docker.com> Signed-off-by: Tibor Vass <tibor@docker.com>

view details

Sebastiaan van Stijn

commit sha 5d04e0adce57a8909efb398e01ce194a65a364cb

Merge pull request #39540 from andrewhsu/run-single-integration allow running of single integration test

view details

Tonis Tiigi

commit sha af2e82d054a2276e5ff76fd3fb90915cad5a0a55

atomic: patch 64bit alignment on 32bit systems causes panic on armv7 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

view details

Tibor Vass

commit sha 880feeb3a814270cb47458ac53afb1182a696348

Merge pull request #39543 from tonistiigi/64align atomic: patch 64bit alignment on 32bit systems

view details

push time in 2 months

Pull request review commentdocker/swarm

Add better OS detection

 func (c *Cluster) CreateContainer(config *cluster.ContainerConfig, name string, 	return container, err } +// setOSTypeConstraint chooses an engine and leverages its /distribution+// endpoint to determine a list of compatible Platforms for the image. it then+// adds an ostype constraint for the valid OS types. If an ostype constraint+// already exists on container, then this will not overwrite it.+func (c *Cluster) setOSTypeConstraint(config *cluster.ContainerConfig, authConfig *types.AuthConfig) error {+	// first, check if there is an existing ostype constraint. if so, leave this+	// alone and take no action.+	constraints := config.Constraints()+	for _, constraint := range constraints {+		if strings.Contains(constraint, "ostype") {+			return nil+		}+	}++	// now that we know we have to set an os constraint, choose an engine at+	// random. any engine should theoretically be able to contact the registry+	engine, err := c.RANDOMENGINE()+	if err != nil {+		return err+	}++	// now call the corresponding method on this engine+	platforms, err := engine.GetImagePlatforms(config, authConfig)+	if err != nil {+		return err+	}++	// now extract the OSes. use a map to deduplicate, in case the image is+	// available on several architectures with the same platform.+	ostypes := map[string]struct{}{}+	for _, p := range platforms {+		ostypes[p.OS] = struct{}{}

I don't see where the engine reports OS version, though. Swarm relies on the output of docker info to discover information like the OS, and info does have a particular field that appears to map 1:1 to the OSVersion field in the Platform.

dperny

comment created time in 2 months

PR opened docker/swarm

Add better OS detection

Adds better detection of the OS type of the engine by getting the supported platforms defined in the manifest and creating an ostype constraint using them.

This is #2959 done the right way.

+277 -1

0 comment

6 changed files

pr created time in 2 months

more