Ooh, I missed that there were more, good catch. Updating now.

Deploy preview for brigade-docs ready!

Built with commit f40bb54158819e78314bda7c896856ee1cc944bb

https://deploy-preview-1113--brigade-docs.netlify.app

This commit updates the instructions in the installation guide to use the correct Helm 3 command for deploying Brigade - with the name passed as the first argument, rather than with the --name flag.

Signed-off-by: Radu M [email protected]

I also just tried with a new 1.16 cluster and chart versionbrigade-1.6.1. To make things easier, I set the vacuum to delete all jobs older than 1 minute, and run every 5 minutes (not recommended in a real cluster - --set vacuum.schedule="*/5 * * * *" --set vacuum.age="1m"), and the job runs fine for me as well:

$ k describe cronjob brigade-brigade-vacuum
Name:                          brigade-brigade-vacuum
Namespace:                     brigade
Labels:                        app.kubernetes.io/instance=brigade
                               app.kubernetes.io/managed-by=Helm
                               app.kubernetes.io/name=brigade-brigade
                               helm.sh/chart=brigade-1.6.1
                               role=vacuum
Annotations:                   meta.helm.sh/release-name: brigade
                               meta.helm.sh/release-namespace: brigade
Schedule:                      */5 * * * *
Concurrency Policy:            Allow
Suspend:                       False
Successful Job History Limit:  10
Failed Job History Limit:      10
Starting Deadline Seconds:     <unset>
Selector:                      <unset>
Parallelism:                   <unset>
Completions:                   <unset>
Pod Template:
  Labels:           app.kubernetes.io/name=brigade-brigade
                    role=vacuum
  Service Account:  brigade-vacuum
  Containers:
   brigade-vacuum:
    Image:      brigadecore/brigade-vacuum:v1.4.0
    Port:       <none>
    Host Port:  <none>
    Environment:
      VACUUM_AGE:                  1m
      VACUUM_MAX_BUILDS:           -1
      VACUUM_SKIP_RUNNING_BUILDS:  false
      BRIGADE_NAMESPACE:            (v1:metadata.namespace)
    Mounts:                        <none>
  Volumes:                         <none>
Last Schedule Time:                Wed, 05 Aug 2020 07:10:00 -0700
Active Jobs:                       <none>
Events:
  Type    Reason            Age    From                Message
  ----    ------            ----   ----                -------
  Normal  SuccessfulCreate  8m1s   cronjob-controller  Created job brigade-brigade-vacuum-1596636300
  Normal  SawCompletedJob   7m51s  cronjob-controller  Saw completed job: brigade-brigade-vacuum-1596636300, status: Complete
  Normal  SuccessfulCreate  3m     cronjob-controller  Created job brigade-brigade-vacuum-1596636600
  Normal  SawCompletedJob   2m50s  cronjob-controller  Saw completed job: brigade-brigade-vacuum-1596636600, status: Complete

And one job:

$ k describe job brigade-brigade-vacuum-1596636300
Name:           brigade-brigade-vacuum-1596636300
Namespace:      brigade
Selector:       controller-uid=430a14f4-0fe9-4715-b462-ce8a2a164e42
Labels:         app.kubernetes.io/name=brigade-brigade
                controller-uid=430a14f4-0fe9-4715-b462-ce8a2a164e42
                job-name=brigade-brigade-vacuum-1596636300
                role=vacuum
Annotations:    <none>
Controlled By:  CronJob/brigade-brigade-vacuum
Parallelism:    1
Completions:    1
Start Time:     Wed, 05 Aug 2020 07:05:00 -0700
Completed At:   Wed, 05 Aug 2020 07:05:05 -0700
Duration:       5s
Pods Statuses:  0 Running / 1 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/name=brigade-brigade
                    controller-uid=430a14f4-0fe9-4715-b462-ce8a2a164e42
                    job-name=brigade-brigade-vacuum-1596636300
                    role=vacuum
  Service Account:  brigade-vacuum
  Containers:
   brigade-vacuum:
    Image:      brigadecore/brigade-vacuum:v1.4.0
    Port:       <none>
    Host Port:  <none>
    Environment:
      VACUUM_AGE:                  1m
      VACUUM_MAX_BUILDS:           -1
      VACUUM_SKIP_RUNNING_BUILDS:  false
      BRIGADE_NAMESPACE:            (v1:metadata.namespace)
    Mounts:                        <none>
  Volumes:                         <none>
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  9m20s  job-controller  Created pod: brigade-brigade-vacuum-1596636300-6hwqm

Do you have RBAC enabled? What is the vacuum role configured like?

Thanks for trying the upgrade approach. Can we try the previous release of the vacuum via setting the tag to v1.3.0 in the values file here and upgrading the brigade release? It's a longshot, but could be worth the low effort to check.

All of the worker secrets/pods are in the same (probably default) namespace that the vacuum is configured with, right?

Thank you for the quick response. So I upgraded my cluster to 1.16.10 and unfortunately, it didn't help. I still can see the same log on pod created by a vacuum job but worker pods are still there.

kubectl logs brigade-brigade-vacuum-1596621600-7tfls

W0805 10:00:12.834819       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020/08/05 10:00:12 Pruning records older than 2020-08-05 09:00:12.834684911 +0000 UTC m=-3599.991204137

The updated output of kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"89d8075525967c7a619641fabcb267358d28bf08", GitTreeState:"clean", BuildDate:"2020-06-23T02:52:37Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Will appreciate any new ideas :)

Not sure if this really belongs here, but in addition to being able to restart failed pipelines it would be nice to be able to re-run successful pupelines. Having this functionality exposed via the Kashti UI might also be nice :)

I am more interested in being to easily restart entire pipelines since I expect that I can just chain pipelines to achieve some intermediate checkpoint if I want

Hi @olomakovskyi-blackbird . Thank you for the wealth of detailed information. I'm not seeing anything that looks amiss, so I'm also confused. I was unable to reproduce on my AKS v1.17.7 cluster (secrets and pods older than 1h were successfully deleted.)

My only guess is perhaps the k8s/client-go updates in 1.4.0 (libs bumped to v1.17.x) somehow affected the secret listing/deletion logic in vacuum.go. Would it be possible to either try the v1.3.0 tag for brigade-vacuum or try upgrading the underlying k8s cluster to 1.16+?

Hi, I have enabled vacuum on my cluster but it looks like it doesn't clear resources according to the settings. I haven't found similar issues here so I assume that probably something is wrong in my config. What am I missing?

kubectl describe cronjob brigade-brigade-vacuum

Namespace:                     default
Labels:                        app.kubernetes.io/instance=brigade
                               app.kubernetes.io/managed-by=Helm
                               app.kubernetes.io/name=brigade-brigade
                               helm.sh/chart=brigade-1.6.0
                               role=vacuum
Annotations:                   meta.helm.sh/release-name: brigade
                               meta.helm.sh/release-namespace: default
Schedule:                      @hourly
Concurrency Policy:            Allow
Suspend:                       False
Successful Job History Limit:  10
Failed Job History Limit:      10
Starting Deadline Seconds:     <unset>
Selector:                      <unset>
Parallelism:                   <unset>
Completions:                   <unset>
Pod Template:
  Labels:           app.kubernetes.io/name=brigade-brigade
                    role=vacuum
  Service Account:  brigade-vacuum
  Containers:
   brigade-vacuum:
    Image:      brigadecore/brigade-vacuum:v1.4.0
    Port:       <none>
    Host Port:  <none>
    Environment:
      VACUUM_AGE:                  1h
      VACUUM_MAX_BUILDS:           -1
      VACUUM_SKIP_RUNNING_BUILDS:  true
      BRIGADE_NAMESPACE:            (v1:metadata.namespace)
    Mounts:                        <none>
  Volumes:                         <none>
Last Schedule Time:                Tue, 04 Aug 2020 16:00:00 +0300
Active Jobs:                       <none>
Events:
  Type    Reason            Age    From                Message
  ----    ------            ----   ----                -------
  Normal  SuccessfulCreate  7m10s  cronjob-controller  Created job brigade-brigade-vacuum-1596546000
  Normal  SawCompletedJob   7m     cronjob-controller  Saw completed job: brigade-brigade-vacuum-1596546000, status: Complete
  Normal  SuccessfulDelete  7m     cronjob-controller  Deleted job brigade-brigade-vacuum-1596510000

kubectl get job

brigade-brigade-vacuum-1596517200   1/1           2s         8h
brigade-brigade-vacuum-1596520800   1/1           3s         7h7m
brigade-brigade-vacuum-1596524400   1/1           2s         6h7m
brigade-brigade-vacuum-1596528000   1/1           1s         5h7m
brigade-brigade-vacuum-1596531600   1/1           3s         4h7m
brigade-brigade-vacuum-1596535200   1/1           3s         3h7m
brigade-brigade-vacuum-1596538800   1/1           3s         127m
brigade-brigade-vacuum-1596542400   1/1           3s         67m
brigade-brigade-vacuum-1596546000   1/1           2s         7m45s

kubectl describe job brigade-brigade-vacuum-1596546000

Namespace:      default
Selector:       controller-uid=b537daad-0191-4c08-81fc-580a871ac0a1
Labels:         app.kubernetes.io/name=brigade-brigade
                controller-uid=b537daad-0191-4c08-81fc-580a871ac0a1
                job-name=brigade-brigade-vacuum-1596546000
                role=vacuum
Annotations:    <none>
Controlled By:  CronJob/brigade-brigade-vacuum
Parallelism:    1
Completions:    1
Start Time:     Tue, 04 Aug 2020 16:00:06 +0300
Completed At:   Tue, 04 Aug 2020 16:00:08 +0300
Duration:       2s
Pods Statuses:  0 Running / 1 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/name=brigade-brigade
                    controller-uid=b537daad-0191-4c08-81fc-580a871ac0a1
                    job-name=brigade-brigade-vacuum-1596546000
                    role=vacuum
  Service Account:  brigade-vacuum
  Containers:
   brigade-vacuum:
    Image:      brigadecore/brigade-vacuum:v1.4.0
    Port:       <none>
    Host Port:  <none>
    Environment:
      VACUUM_AGE:                  1h
      VACUUM_MAX_BUILDS:           -1
      VACUUM_SKIP_RUNNING_BUILDS:  true
      BRIGADE_NAMESPACE:            (v1:metadata.namespace)
    Mounts:                        <none>
  Volumes:                         <none>
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  8m33s  job-controller  Created pod: brigade-brigade-vacuum-1596546000-w4v2n

kubectl logs brigade-brigade-vacuum-1596546000-w4v2n

W0804 13:00:08.223420       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020/08/04 13:00:08 Pruning records older than 2020-08-04 12:00:08.223387171 +0000 UTC m=-3599.994107663

From my understanding vacuum should delete all resources related to completed builds that older than 1h but I can see all of them still present in my cluster. For example worker pods: kubectl get pods | grep worker

brigade-worker-01eev6jdmknj1s4avre56sy4z2   0/1     Completed    0          14h
brigade-worker-01eev7jz3qf14cn98s8qa24afq   0/1     Completed    0          14h
brigade-worker-01eev94gdvvrmyvnvmtz4v38m9   0/1     Completed    0          14h
brigade-worker-01eewk220c1v3gp8rha6pneamc   0/1     Completed    0          113m
brigade-worker-01eewk649ktmmw4hc8kc60m5f7   0/1     Completed    0          111m
brigade-worker-01eewkkr67aq6yd3ztvffx65tg   0/1     Completed    0          104m
brigade-worker-01eewnmnedghphvp78s6d6kc64   0/1     Completed    0          68m

Output of brig version: v1.3.0

Output of kubectl version: Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"ec831747a3a5896dbdf53f259eafea2a2595217c", GitTreeState:"clean", BuildDate:"2020-05-29T19:56:10Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): AKS

closes #294

This PR returns a sorted observable from the job service, sorted by start date. Ideally in the future, this should be performed by the API, but until that happens, we can sort client-side.

#14 #124 both addressed some variation of this issue - leaving the chart at the top of the page aside, the jobs should be sorted by their start date, and we can do this in Kashti (without changing the API server) rather easily.

Needs a rebase on top of #297

Bumps elliptic from 6.5.1 to 6.5.3. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/indutny/elliptic/commit/8647803dc3d90506aa03021737f7b061ba959ae1"><code>8647803</code></a> 6.5.3</li> <li><a href="https://github.com/indutny/elliptic/commit/856fe4d99fe7b6200556e6400b3bf585b1721bec"><code>856fe4d</code></a> signature: prevent malleability and overflows</li> <li><a href="https://github.com/indutny/elliptic/commit/60489415e545efdfd3010ae74b9726facbf08ca8"><code>6048941</code></a> 6.5.2</li> <li><a href="https://github.com/indutny/elliptic/commit/9984964457c9f8a63b91b01ea103260417eca237"><code>9984964</code></a> package: bump dependencies</li> <li><a href="https://github.com/indutny/elliptic/commit/ec735edde187a43693197f6fa3667ceade751a3a"><code>ec735ed</code></a> utils: leak less information in <code>getNAF()</code></li> <li>See full diff in <a href="https://github.com/indutny/elliptic/compare/v6.5.1...v6.5.3">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

Bumps elliptic from 6.5.1 to 6.5.3. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/indutny/elliptic/commit/8647803dc3d90506aa03021737f7b061ba959ae1"><code>8647803</code></a> 6.5.3</li> <li><a href="https://github.com/indutny/elliptic/commit/856fe4d99fe7b6200556e6400b3bf585b1721bec"><code>856fe4d</code></a> signature: prevent malleability and overflows</li> <li><a href="https://github.com/indutny/elliptic/commit/60489415e545efdfd3010ae74b9726facbf08ca8"><code>6048941</code></a> 6.5.2</li> <li><a href="https://github.com/indutny/elliptic/commit/9984964457c9f8a63b91b01ea103260417eca237"><code>9984964</code></a> package: bump dependencies</li> <li><a href="https://github.com/indutny/elliptic/commit/ec735edde187a43693197f6fa3667ceade751a3a"><code>ec735ed</code></a> utils: leak less information in <code>getNAF()</code></li> <li>See full diff in <a href="https://github.com/indutny/elliptic/compare/v6.5.1...v6.5.3">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

This PR returns a sorted observable from the job service, sorted by start date. Ideally in the future, this should be performed by the API, but until that happens, we can sort client-side.