profile
viewpoint
Sebastian J. derjust Comcast New York City

Comcast/sirius 295

A distributed system library for managing application reference data

Comcast/money 65

Dapper Style Distributed Tracing Instrumentation Libraries

derjust/bash-it 1

A community bash framework.

derjust/ansible-role-terraform 0

Ansible role for installing Hashicorp's Terraform tool.

derjust/argo-helm 0

ArgoProj Helm Charts

derjust/argo-rollouts 0

Progressive Delivery for Kubernetes

derjust/awesome-falsehood 0

:pill: Curated list of falsehoods programmers believe in.

push eventderjust/argo-rollouts

Sebastian J

commit sha 7c0b4241a78169ab15490ef033a8b5a9ec1e374b

Added documentaiton Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

push time in 3 days

push eventderjust/argo-rollouts

Andrii Perenesenko

commit sha 9d32c13c4eb469a138bc9673c08c42d73e1c01fb

fix: sending updates to dashboard when a pod terminates (#1642) * fix: sending updates to dashboard when a pod terminates Signed-off-by: Andrii Perenesenko <andrii.perenesenko@gmail.com> * Fix imports Signed-off-by: Andrii Perenesenko <andrii.perenesenko@gmail.com>

view details

Alexander Matyushentsev

commit sha 6ae3623568e8d1424f5d2d2c6f0ecfa1cd61569d

fix: use patch to update workload-generation annotation Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

view details

Sebastian J

commit sha de52ad3d7836c32ae35350893817a59c5f2ba17c

Add support for AWS ALB TargetGroupStickinessConfig Adds support for AWS ALB [TargetGroupStickinessConfig](https://aws.amazon.com/blogs/aws/new-application-load-balancer-simplifies-deployment-with-weighted-target-groups/) This is required to support sticky session on the listener level while Argo is using ALB's weighting Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha b864761518e916475354bd9501c63df58dacb1b9

Remove code smells Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha d980154568e3d444a56cf758dc2dda7668278d19

more code smells Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha ca04b578040552342daafb83302fac32bbfcc3a3

PR feedback Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha 9849b53dc2c240582b599b0d1a2b7f0db740298e

Revert Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha c1cbcddf277713da751f505cbde6a2f0c4c4e8dc

Another test round Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha 97bc37018ab82cf06923fe49f54cba9c3c211578

Force codegen GO 1.16 Forced codegen via downgrading to Go 1.16: ``` $ env|grep GO GOPATH=/Users/sebastian/go ``` ``` $ go version go version go1.16.10 darwin/amd64 ``` ``` $ echo $PATH /Users/sebastian/.sdkman/candidates/micronaut/current/bin:/Users/sebastian/.sdkman/candidates/java/current/bin:/Users/sebastian/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/bin:/Users/sebastian/go/bin ``` Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

push time in 3 days

pull request commentargoproj/argo-rollouts

fix(controller): Sticky session correction for AWS ALB. Fixes #1572

A PR for the issue in #1675 was provided. As it is already approved i assume it will be merged to master soon - so i already re-based this PR on it already.

derjust

comment created time in 3 days

issue commentargoproj/argo-rollouts

e2e TestWorkloadRef fails / LabelSelector is "null" & should be optional?

Thank you @alexmt ! I tested your branch locally with EKS and it works (as does my PR #1394.

As it is already approved & testes 🟢 - is there timeline to merge your PR?

Thanks again!

derjust

comment created time in 3 days

issue commenthashicorp/terraform-provider-aws

EKS node_group creation failure not kept in state

No, a subsequent execution of plan or apply behaved as if the node group doesn't exist (isn't part of the state) So plan shows it as creation, apply tries to create the node group and it faills as such a node group already exist in AWS

derjust

comment created time in 4 days

issue commentargoproj/argo-rollouts

e2e TestWorkloadRef fails / LabelSelector is "null" & should be optional?

Yes - though i don't know if the problem is EKS (rejecting the "null" or if minikube is just more leninent (which it shouldn't be). Leaning towards the first but I'm not sure

derjust

comment created time in 6 days

issue commentargoproj/argo-rollouts

Rollout stuck in progressing status with an old revision in progressing status

We saw this behavior and assumed it is 'expected': If the stable service is degraded, Argo doesn't touch anything as a rollback might not be possible. Thus it leaves everything as-is. Our workaround for this situation (which occurs rarely if anything) is to use promote --full to force the rollout (when we now the new version is the fix for the problem)

huikang

comment created time in 6 days

issue commentargoproj/argo-rollouts

e2e TestWorkloadRef fails / LabelSelector is "null" & should be optional?

Alright - I did another test with minikube - which lets the e2e test succeed:

# Start minikube
$  minikube start
# Install CRDs etc
$ helm repo add argo https://argoproj.github.io/argo-helm
$ helm install argo-rollouts argo/argo-rollouts
# Delete released Argo version so that only the locally running one takes action
$ kubectl delete deployment -n default       argo-rollouts
$ make start-e2e

And in a different terminal run the e2e test successfully

make test-e2e E2E_TEST_OPTIONS="-testify.m ^TestWorkloadRef$"
go test -timeout 30m -v -count 1 --tags e2e -p 4 --short ./test/e2e -testify.m ^TestWorkloadRef
=== RUN   TestAnalysisSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAnalysisSuite (0.00s)
=== RUN   TestAWSSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAWSSuite (0.00s)
=== RUN   TestBlueGreenSuite
    suite.go:190: warning: no tests to run
--- PASS: TestBlueGreenSuite (0.00s)
=== RUN   TestCanarySuite
    suite.go:190: warning: no tests to run
--- PASS: TestCanarySuite (0.00s)
=== RUN   TestExperimentSuite
    suite.go:190: warning: no tests to run
--- PASS: TestExperimentSuite (0.00s)
=== RUN   TestFunctionalSuite
time="2021-11-29T22:15:04-08:00" level=info msg="analysistemplate.argoproj.io/sleep-job created\n"
=== RUN   TestFunctionalSuite/TestWorkloadRef
time="2021-11-29T22:15:04-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
W1129 22:15:04.064097   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:04.066680   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-29T22:15:04-08:00" level=info msg="service/rollout-bluegreen-active created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment] met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:04-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Condition 'status=Healthy' met after 4s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 1' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:7c69f96b6b] met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:09-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:15-08:00" level=info msg="Condition 'status=Degraded' met after 5s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:15-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 2 after workload ref updated' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:15-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Condition 'status=Healthy' met after 1s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Updated rollout spec: {\"spec\":{\"workloadRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"non-existent-deploy\"}}}" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Updated rollout spec: {\"spec\":{\"workloadRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"rollout-ref-deployment\"}}}" rollout=rollout-ref-deployment
time="2021-11-29T22:15:16-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Condition 'status=Healthy' met after 1s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:6cd5994b44] met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Rollout expectation 'Resolved template not persisted' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment configured\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Condition 'status=Healthy' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration must be removed after switch to inline template' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
W1129 22:15:18.549226   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:18.552968   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
=== RUN   TestFunctionalSuite/TestWorkloadRefTemplate
time="2021-11-29T22:15:18-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRefTemplate"
W1129 22:15:18.638484   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:18.641610   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-29T22:15:18-08:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Rollout expectation 'error due to workload ref and template' met" rollout=rollout-ref-deployment
time="2021-11-29T22:15:18-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRefTemplate"
W1129 22:15:19.328932   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:19.377819   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-11-29T22:15:19-08:00" level=info msg="Deleting e2e-test-name"
W1129 22:15:20.527551   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 22:15:20.577157   90117 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
--- PASS: TestFunctionalSuite (17.29s)
    --- PASS: TestFunctionalSuite/TestWorkloadRef (14.57s)
    --- PASS: TestFunctionalSuite/TestWorkloadRefTemplate (1.10s)
=== RUN   TestIstioSuite
    suite.go:190: warning: no tests to run
--- PASS: TestIstioSuite (0.00s)
=== RUN   TestSMIIngressSuite
    suite.go:190: warning: no tests to run
--- PASS: TestSMIIngressSuite (0.00s)
=== RUN   TestSMISuite
    suite.go:190: warning: no tests to run
--- PASS: TestSMISuite (0.00s)
PASS
ok      github.com/argoproj/argo-rollouts/test/e2e      17.919s
derjust

comment created time in 6 days

issue commentargoproj/argo-rollouts

e2e TestWorkloadRef fails / LabelSelector is "null" & should be optional?

Thank you @huikang . I'm running of the latest main i see:

argo-rollouts git:(47d59fa9) git log

commit 47d59fa9855ab737e40e21ef5938b2b6723cb899 (HEAD, upstream/master, origin/master, origin/HEAD, master)
Author: Siddharth Maloo <92331248+smaloo-salesforce@users.noreply.github.com>
Date:   Sat Nov 20 04:12:57 2021 +0530
...

Can you add logging and see if your selector is null (and it serializes differently) or if there is a selector that somewhat is missing on my end?

I also cleared out my ~/.kube/config and have only my EKS cluster present. The cluster itself was also cleaned and it has nothing running but the AWS system pods

$ kubectl config get-contexts

CURRENT   NAME                                                   CLUSTER                                                AUTHINFO                                               NAMESPACE
*         arn:aws:eks:us-east-1:922437464650:cluster/argo-test   arn:aws:eks:us-east-1:922437464650:cluster/argo-test   arn:aws:eks:us-east-1:922437464650:cluster/argo-test

The same error as before shows up in the e2e run:

$ make test-e2e E2E_TEST_OPTIONS="-testify.m ^TestWorkloadRef$"

go test -timeout 30m -v -count 1 --tags e2e -p 4 --short ./test/e2e -testify.m ^TestWorkloadRef
=== RUN   TestAnalysisSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAnalysisSuite (0.00s)
=== RUN   TestAWSSuite
    suite.go:190: warning: no tests to run
--- PASS: TestAWSSuite (0.00s)
=== RUN   TestBlueGreenSuite
    suite.go:190: warning: no tests to run
--- PASS: TestBlueGreenSuite (0.00s)
=== RUN   TestCanarySuite
    suite.go:190: warning: no tests to run
--- PASS: TestCanarySuite (0.00s)
=== RUN   TestExperimentSuite
    suite.go:190: warning: no tests to run
--- PASS: TestExperimentSuite (0.00s)
=== RUN   TestFunctionalSuite
time="2021-11-29T21:36:43-08:00" level=info msg="analysistemplate.argoproj.io/sleep-job unchanged\n"
=== RUN   TestFunctionalSuite/TestWorkloadRef
time="2021-11-29T21:36:43-08:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
time="2021-11-29T21:36:48-08:00" level=info msg="service/rollout-bluegreen-active created\n" rollout=rollout-ref-deployment
time="2021-11-29T21:36:50-08:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T21:36:50-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-29T21:36:51-08:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-29T21:36:51-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment] met" rollout=rollout-ref-deployment
time="2021-11-29T21:36:54-08:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-29T21:36:54-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
time="2021-11-29T21:37:21-08:00" level=info msg="Condition 'status=Healthy' met after 27s" rollout=rollout-ref-deployment
time="2021-11-29T21:37:22-08:00" level=info msg="Rollout expectation 'WorkloadObservedGeneration is 1' met" rollout=rollout-ref-deployment
time="2021-11-29T21:37:22-08:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment rollouts-pod-template-hash:659f6f45f7] met" rollout=rollout-ref-deployment
time="2021-11-29T21:37:22-08:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
^CFAIL  github.com/argoproj/argo-rollouts/test/e2e      69.771s
make: *** [test-e2e] Error 1

The controller logs show the error as above but before they also complain about a missing deployment

 make start-e2e

go run ./cmd/rollouts-controller/main.go --instance-id argo-rollouts-e2e --loglevel debug
INFO[2021-11-29T21:36:21-08:00] Argo Rollouts starting                        version=v99.99.99+unknown
INFO[2021-11-29T21:36:22-08:00] Creating event broadcaster
INFO[2021-11-29T21:36:22-08:00] Setting up event handlers
INFO[2021-11-29T21:36:22-08:00] Setting up experiments event handlers
INFO[2021-11-29T21:36:22-08:00] Setting up analysis event handlers
INFO[2021-11-29T21:36:23-08:00] Waiting for controller's informer caches to sync
INFO[2021-11-29T21:36:23-08:00] Enqueueing parent of default/rollout-ref-deployment-7c69f96b6b: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:36:23-08:00] Leaderelection get id Sebastians-MacBook-Pro.local_4d8d676f-4864-4aad-b551-2eb4afc29a52
INFO[2021-11-29T21:36:23-08:00] Starting Healthz Server at 0.0.0.0:8080
I1129 21:36:23.436918   88611 leaderelection.go:243] attempting to acquire leader lease kube-system/argo-rollouts-controller-lock...
INFO[2021-11-29T21:36:23-08:00] New leader elected: Sebastians-MacBook-Pro.local_f7ee4a4e-1965-4cd1-9335-efc2ad8931ad
INFO[2021-11-29T21:36:23-08:00] Starting Secondary Metric Server at 0.0.0.0:8090
I1129 21:36:40.288217   88611 leaderelection.go:253] successfully acquired lease kube-system/argo-rollouts-controller-lock
WARN[2021-11-29T21:36:40-08:00] Shutdown Secondary Metrics Server
WARN[2021-11-29T21:36:40-08:00] Starting Secondary Metric Server: http: Server closed
INFO[2021-11-29T21:36:40-08:00] Starting Controllers
INFO[2021-11-29T21:36:40-08:00] Started controller
INFO[2021-11-29T21:36:40-08:00] Starting Metric Server at 0.0.0.0:8090
INFO[2021-11-29T21:36:40-08:00] Starting Ingress workers
INFO[2021-11-29T21:36:40-08:00] Starting Experiment workers
INFO[2021-11-29T21:36:40-08:00] Started Ingress workers
INFO[2021-11-29T21:36:40-08:00] Started Experiment workers
INFO[2021-11-29T21:36:40-08:00] Starting Rollout workers
INFO[2021-11-29T21:36:40-08:00] Started Rollout workers
WARN[2021-11-29T21:36:40-08:00] Controller is running.
INFO[2021-11-29T21:36:40-08:00] Start processing                              resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:40-08:00] Starting Service workers
INFO[2021-11-29T21:36:40-08:00] Starting analysis workers
INFO[2021-11-29T21:36:40-08:00] Started Service workers
INFO[2021-11-29T21:36:40-08:00] Started 30 analysis workers
ERRO[2021-11-29T21:36:40-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:40-08:00] syncing service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:36:40-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"True","type":"Completed"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"ReplicaSet \"rollout-ref-deployment-7c69f96b6b\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-11-30T05:36:41Z","lastUpdateTime":"2021-11-30T05:36:41Z","message":"The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","reason":"InvalidSpec","status":"True","type":"InvalidSpec"}],"message":"InvalidSpec: The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","phase":"Degraded"}}  generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment time_ms=911.8547590000001
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 4 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.202174   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"True","type":"Completed"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"ReplicaSet \"rollout-ref-deployment-7c69f96b6b\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:33:54Z","lastUpdateTime":"2021-11-30T05:33:54Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-11-30T05:36:41Z","lastUpdateTime":"2021-11-30T05:36:41Z","message":"The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","reason":"InvalidSpec","status":"True","type":"InvalidSpec"}],"message":"InvalidSpec: The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","phase":"Degraded"}}  generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917080 rollout=rollout-ref-deployment time_ms=118.99775799999999
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 6 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.321261   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.5124249999999999
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 7 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.321867   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.6034200000000001
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 8 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.358229   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.6285839999999999
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 9 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.491887   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:41-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:41-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.598853
ERRO[2021-11-29T21:36:41-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:41-08:00] rollout syncHandler queue retries: 10 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:41.753137   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:42-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:42-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:42-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=0.5721670000000001
ERRO[2021-11-29T21:36:42-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:42-08:00] rollout syncHandler queue retries: 11 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:42.269574   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:43-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:43-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917691 rollout=rollout-ref-deployment time_ms=1.030473
ERRO[2021-11-29T21:36:43-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] rollout syncHandler queue retries: 12 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:43.297678   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:43-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:43-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] rollout deleted                               namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:43-08:00] syncing service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:36:43-08:00] cleaned service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:36:50-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917742 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:36:50Z","lastUpdateTime":"2021-11-30T05:36:50Z","message":"The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","reason":"InvalidSpec","status":"True","type":"InvalidSpec"}],"message":"InvalidSpec: The Rollout \"rollout-ref-deployment\" is invalid: deployments.apps \"default/rollout-ref-deployment\" not found","observedGeneration":"1","phase":"Degraded"}}  generation=1 namespace=default resourceVersion=13917742 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917742 rollout=rollout-ref-deployment time_ms=124.02361300000001
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 3 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.788293   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.386567
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 4 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.788728   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.5878209999999999
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 5 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.793829   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.458619
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 6 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.811531   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.5054649999999999
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 7 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.845289   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:50-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:50-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.566269
ERRO[2021-11-29T21:36:50-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:50-08:00] rollout syncHandler queue retries: 8 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:50.910015   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:51-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:51-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.6207349999999999
ERRO[2021-11-29T21:36:51-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] rollout syncHandler queue retries: 9 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:51.043820   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:51-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:51-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.484018
ERRO[2021-11-29T21:36:51-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] rollout syncHandler queue retries: 10 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:51.302380   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:51-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:51-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.5353209999999999
ERRO[2021-11-29T21:36:51-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:51-08:00] rollout syncHandler queue retries: 11 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:51.818618   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
INFO[2021-11-29T21:36:52-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:52-08:00] The Rollout "rollout-ref-deployment" is invalid: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:52-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=0.44193699999999997
ERRO[2021-11-29T21:36:52-08:00] rollout syncHandler error: deployments.apps "default/rollout-ref-deployment" not found  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:52-08:00] rollout syncHandler queue retries: 12 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:52.846020   88611 controller.go:174] deployments.apps "default/rollout-ref-deployment" not found
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:36:54-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
INFO[2021-11-29T21:36:54-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:54-08:00] Updating replica set 'rollout-ref-deployment-5d466d9b5f' revision from 0 to 1  namespace=default rollout=rollout-ref-deployment
WARN[2021-11-29T21:36:55-08:00] Found a hash collision - bumped collisionCount (0->1) to resolve it  namespace=default rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:55-08:00] roCtx.reconcile err replicasets.apps "rollout-ref-deployment-5d466d9b5f" already exists  generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:55-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917743 rollout=rollout-ref-deployment time_ms=280.29145500000004
ERRO[2021-11-29T21:36:55-08:00] rollout syncHandler error: replicasets.apps "rollout-ref-deployment-5d466d9b5f" already exists  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:55-08:00] rollout syncHandler queue retries: 13 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:55.179836   88611 controller.go:174] replicasets.apps "rollout-ref-deployment-5d466d9b5f" already exists
INFO[2021-11-29T21:36:55-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:55-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917776 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] Updating replica set 'rollout-ref-deployment-7c69f96b6b' revision from 0 to 1  namespace=default rollout=rollout-ref-deployment
WARN[2021-11-29T21:36:59-08:00] Found a hash collision - bumped collisionCount (1->2) to resolve it  namespace=default rollout=rollout-ref-deployment
ERRO[2021-11-29T21:36:59-08:00] roCtx.reconcile err replicasets.apps "rollout-ref-deployment-7c69f96b6b" already exists  generation=1 namespace=default resourceVersion=13917776 rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917776 rollout=rollout-ref-deployment time_ms=271.331821
ERRO[2021-11-29T21:36:59-08:00] rollout syncHandler error: replicasets.apps "rollout-ref-deployment-7c69f96b6b" already exists  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:36:59-08:00] rollout syncHandler queue retries: 15 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:36:59.549857   88611 controller.go:174] replicasets.apps "rollout-ref-deployment-7c69f96b6b" already exists
INFO[2021-11-29T21:36:59-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:36:59-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917794 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Updating replica set 'rollout-ref-deployment-659f6f45f7' revision from 0 to 1  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Created ReplicaSet rollout-ref-deployment-659f6f45f7  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
DEBU[2021-11-29T21:37:09-08:00] No configured destinations for trigger: on-rollout-updated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Rollout updated to revision 1                 event_reason=RolloutUpdated namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917840", FieldPath:""}): type: 'Normal' reason: 'RolloutUpdated' Rollout updated to revision 1
INFO[2021-11-29T21:37:09-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917840", FieldPath:""}): type: 'Normal' reason: 'NewReplicaSetCreated' Created ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1)
DEBU[2021-11-29T21:37:09-08:00] No configured destinations for trigger: on-new-replica-set-created  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Created ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1)  event_reason=NewReplicaSetCreated namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:09-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Set rollout condition: &RolloutCondition{Type:Progressing,Status:True,LastUpdateTime:2021-11-29 21:37:09.807982 -0800 PST m=+48.190608464,LastTransitionTime:2021-11-29 21:37:09.807982 -0800 PST m=+48.190608568,Reason:NewReplicaSetCreated,Message:Created new replica set "rollout-ref-deployment-659f6f45f7",}  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:09-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:09-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:10-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment
ERRO[2021-11-29T21:37:10-08:00] roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again  generation=1 namespace=default resourceVersion=13917794 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:10-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917794 rollout=rollout-ref-deployment time_ms=494.50115300000004
ERRO[2021-11-29T21:37:10-08:00] rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:10-08:00] rollout syncHandler queue retries: 21 : key "default/rollout-ref-deployment"  namespace=default rollout=rollout-ref-deployment
E1129 21:37:10.046668   88611 controller.go:174] Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again
INFO[2021-11-29T21:37:19-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
DEBU[2021-11-29T21:37:19-08:00] No configured destinations for trigger: on-scaling-replica-set  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Scaled up ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1) from 0 to 1  event_reason=ScalingReplicaSet namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917842", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet rollout-ref-deployment-659f6f45f7 (revision 1) from 0 to 1
INFO[2021-11-29T21:37:19-08:00] New RS 'rollout-ref-deployment-659f6f45f7' is not ready to pause  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] skipping active service switch: New RS 'rollout-ref-deployment-659f6f45f7' is not fully saturated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Skipping analysis: isAborted: false, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: true  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917842", FieldPath:""}): type: 'Normal' reason: 'RolloutCompleted' Rollout completed update to revision 1 (659f6f45f7): Initial deploy
DEBU[2021-11-29T21:37:19-08:00] No configured destinations for trigger: on-rollout-completed  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Rollout completed update to revision 1 (659f6f45f7): Initial deploy  event_reason=RolloutCompleted namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Timed out (true) [last progress check: 2021-11-29 21:37:09 -0800 PST - now: 2021-11-29 21:37:19.890636 -0800 PST m=+58.273425729]  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:19-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Patched: {"status":{"conditions":[{"lastTransitionTime":"2021-11-30T05:37:19Z","lastUpdateTime":"2021-11-30T05:37:19Z","message":"ReplicaSet \"rollout-ref-deployment-659f6f45f7\" has timed out progressing.","reason":"ProgressDeadlineExceeded","status":"False","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:37:19Z","lastUpdateTime":"2021-11-30T05:37:19Z","message":"Rollout does not have minimum availability","reason":"AvailableReason","status":"False","type":"Available"}],"currentPodHash":"659f6f45f7","message":"ProgressDeadlineExceeded: ReplicaSet \"rollout-ref-deployment-659f6f45f7\" has timed out progressing.","selector":"app=rollout-ref-deployment","stableRS":"659f6f45f7","workloadObservedGeneration":"1"}}  generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917842 rollout=rollout-ref-deployment time_ms=338.75113699999997
INFO[2021-11-29T21:37:20-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:20-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] New RS 'rollout-ref-deployment-659f6f45f7' is not ready to pause  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] skipping active service switch: New RS 'rollout-ref-deployment-659f6f45f7' is not fully saturated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Patched: {"status":{"HPAReplicas":1,"conditions":[{"lastTransitionTime":"2021-11-30T05:37:19Z","lastUpdateTime":"2021-11-30T05:37:19Z","message":"Rollout does not have minimum availability","reason":"AvailableReason","status":"False","type":"Available"},{"lastTransitionTime":"2021-11-30T05:37:20Z","lastUpdateTime":"2021-11-30T05:37:20Z","message":"ReplicaSet \"rollout-ref-deployment-659f6f45f7\" is progressing.","reason":"ReplicaSetUpdated","status":"True","type":"Progressing"}],"message":"updated replicas are still becoming available","phase":"Progressing","replicas":1,"updatedReplicas":1}}  generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917890 rollout=rollout-ref-deployment time_ms=138.684145
INFO[2021-11-29T21:37:20-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:20-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] New RS 'rollout-ref-deployment-659f6f45f7' is not ready to pause  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] skipping active service switch: New RS 'rollout-ref-deployment-659f6f45f7' is not fully saturated  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Timed out (false) [last progress check: 2021-11-29 21:37:20 -0800 PST - now: 2021-11-29 21:37:20.180558 -0800 PST m=+58.563352235]  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] No status changes. Skipping patch             generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Queueing up rollout for a progress after 4s   namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:20-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment time_ms=1.222034
INFO[2021-11-29T21:37:21-08:00] Enqueueing parent of default/rollout-ref-deployment-659f6f45f7: Rollout default/rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] skipping pause: fast-tracked update           namespace=default rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] No configured destinations for trigger: on-switch-service  namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Switched selector for service 'rollout-bluegreen-active' from '' to '659f6f45f7'  event_reason=SwitchService namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"rollout-ref-deployment", UID:"20b9d61d-72b5-495c-b787-765b37568954", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"13917892", FieldPath:""}): type: 'Normal' reason: 'SwitchService' Switched selector for service 'rollout-bluegreen-active' from '' to '659f6f45f7'
INFO[2021-11-29T21:37:21-08:00] Updating active selector ( -> 659f6f45f7)     namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] syncing service                               namespace=default rollout=rollout-ref-deployment service=rollout-bluegreen-active
INFO[2021-11-29T21:37:21-08:00] Patched: {"status":{"availableReplicas":1,"blueGreen":{"activeSelector":"659f6f45f7"},"conditions":[{"lastTransitionTime":"2021-11-30T05:37:21Z","lastUpdateTime":"2021-11-30T05:37:21Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"True","type":"Completed"},{"lastTransitionTime":"2021-11-30T05:37:20Z","lastUpdateTime":"2021-11-30T05:37:21Z","message":"ReplicaSet \"rollout-ref-deployment-659f6f45f7\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2021-11-30T05:37:21Z","lastUpdateTime":"2021-11-30T05:37:21Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"}],"message":null,"phase":"Healthy","readyReplicas":1,"selector":"app=rollout-ref-deployment,rollouts-pod-template-hash=659f6f45f7"}}  generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] persisted to informer                         generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917892 rollout=rollout-ref-deployment time_ms=251.114889
INFO[2021-11-29T21:37:21-08:00] Start processing                              resource=default/rollout-ref-deployment
ERRO[2021-11-29T21:37:21-08:00] Failed to process: secret "argo-rollouts-notification-secret" not found  resource=default/rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciling stable ReplicaSet 'rollout-ref-deployment-659f6f45f7'  namespace=default rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] skipping pause: fast-tracked update           namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] No status changes. Skipping patch             generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment time_ms=0.938288
INFO[2021-11-29T21:37:21-08:00] Started syncing rollout                       generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciling stable ReplicaSet 'rollout-ref-deployment-659f6f45f7'  namespace=default rollout=rollout-ref-deployment
DEBU[2021-11-29T21:37:21-08:00] skipping pause: fast-tracked update           namespace=default rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] No status changes. Skipping patch             generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment
INFO[2021-11-29T21:37:21-08:00] Reconciliation completed                      generation=1 namespace=default resourceVersion=13917909 rollout=rollout-ref-deployment time_ms=1.3315130000000002
ERRO[2021-11-29T21:37:22-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:37:22-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-29T21:37:22-08:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
^CINFO[2021-11-29T21:37:49-08:00] Shutting down workers
make: *** [start-e2e] Error 1
  • Is Operation cannot be fulfilled on replicasets.apps "rollout-ref-deployment-659f6f45f7": the object has been modified; please apply your changes to the latest version and try again an issue or is that just noise?
  • Not sure what the secret is used for from Failed to process: secret "argo-rollouts-notification-secret" not found resource=default/rollout-ref-deployment but i created such an (empty, generic) secret though the message still remains
$ kubectl get secrets -A | grep argo-rollouts

argo-test         argo-rollouts-notification-secret                    Opaque                                0      29m
argo-test         argo-rollouts-token-hspxc                            kubernetes.io/service-account-token   3      49d
argo-test         sh.helm.release.v1.argo-rollouts.v1                  helm.sh/release.v1                    1      49d
default           argo-rollouts-notification-secret                    Opaque                                0      32m
  • The missing deployment from deployments.apps "default/rollout-ref-deployment" not found namespace=default rollout=rollout-ref-deployment is present. Not sure what to make out of that error message?
$ kubectl get deployments -A

NAMESPACE     NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
default       rollout-ref-deployment         0/0     0            0           24m
kube-system   aws-load-balancer-controller   2/2     2            2           49d
kube-system   coredns                        2/2     2            2           49d
  • All those messages 'before' 2021-11-29T21:37:22-08:00 seem to be fine as the e2e is progressing through its asserts - which leaves me with only the Cannot update the workload-ref/annotation for rollout-ref-deployment/default - which with additional logging is the "null" value shown above. I'm utterly confused what else could be different
derjust

comment created time in 6 days

issue commentargoproj/argo-helm

Upgrade to Argo-rollouts v2 fails

Yes; we did as described above:

  1. Delete just the deployment of the Argo controller - this keeps the deployment & replication set as-is. This also keeps the ingress configuration "as is"
    • We didn't change anything on the service deployment/rollout CRD
    • From what we have seen, this approach even works with in-flight rollouts. They just 'stall' and they continue later on. Though we tried to avoid this situation
  2. Run the helm upgrade
    • This re-creates the Argo control plane
  3. Argo continues where it left off
    • The rollout CRDs remained as-is

We tested this approach first and we have a pretty simple service/rollout setup w/ AWS ALB ingress. YMMV esp. with different ingress approaches (even though based on how Argo works in general I would think it should work, too)

Definitive a workaround and a lot of 'hands on'. After reading https://github.com/helm/helm/issues/7082 & linked tickets I can't image a quick solution for this TBH - but I'm not a Helm expert

derjust

comment created time in 7 days

pull request commentargoproj/argo-rollouts

fix(controller): Sticky session correction for AWS ALB. Fixes #1572

Did more investigation and posted my findings around my codegen issue - and failing e2e test - as new issue: https://github.com/argoproj/argo-rollouts/issues/1675

Hope this can shed some light?

derjust

comment created time in 8 days

issue openedargoproj/argo-rollouts

e2e TestWorkloadRef fails / LabelSelector is "null" & should be optional?

Summary

Following the docs/CONTRIBUTING.md documentation on master 47d59fa9855ab737e40e21ef5938b2b6723cb899 on OSX, theTestWorkloadRefe2e fails - usingmake test-e2e E2E_TEST_OPTIONS="-testify.m ^TestWorkloadRef$"` for all further descriptions below: (For context, i found this while working on my PR #1577)

=== RUN   TestFunctionalSuite/TestWorkloadRef
time="2021-11-28T15:51:21-05:00" level=info msg="Deleting e2e-test-name=TestWorkloadRef"
time="2021-11-28T15:51:25-05:00" level=info msg="service/rollout-bluegreen-active created\n" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="rollout.argoproj.io/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="Waiting for condition: status=Degraded" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="Condition 'status=Degraded' met after 0s" rollout=rollout-ref-deployment
time="2021-11-28T15:51:28-05:00" level=info msg="Expectation rollout-bluegreen-active selector: map[app:rollout-ref-deployment] met" rollout=rollout-ref-deployment
time="2021-11-28T15:51:30-05:00" level=info msg="deployment.apps/rollout-ref-deployment created\n" rollout=rollout-ref-deployment
time="2021-11-28T15:51:30-05:00" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-ref-deployment
FAIL  github.com/argoproj/argo-rollouts/test/e2e      22.245s

Diagnostics

What version of Argo Rollouts are you running? Master

I suspect this is an issue with my setup but i can't point to where it is:

$ go version
go version go1.16.10 darwin/amd64
$ protoc --version
libprotoc 3.17.3

Also all the other software is installed via Homebrew:

$ brew install kubectl kustomize golangci-lint protobuf swagger-codegen

Warning: kubernetes-cli 1.22.4 is already installed and up-to-date.
To reinstall 1.22.4, run:
  brew reinstall kubernetes-cli
Warning: kustomize 4.4.1 is already installed and up-to-date.
To reinstall 4.4.1, run:
  brew reinstall kustomize
Warning: golangci-lint 1.43.0 is already installed and up-to-date.
To reinstall 1.43.0, run:
  brew reinstall golangci-lint
Warning: protobuf 3.17.3 is already installed and up-to-date.
To reinstall 3.17.3, run:
  brew reinstall protobuf
Warning: swagger-codegen 3.0.29 is already installed and up-to-date.
To reinstall 3.0.29, run:
  brew reinstall swagger-codegen

What am I missing?

Two observations:

First observation

Running make codegen in master creates a diff - which i didn't expect:

$ git diff --stat 

pkg/apis/rollouts/v1alpha1/generated.pb.go      | 1520 +++++++++++++++--------
pkg/apis/rollouts/v1alpha1/generated.proto      |   25 +-
pkg/apis/rollouts/v1alpha1/openapi_generated.go |   84 +-
3 files changed, 1114 insertions(+), 515 deletions(-)

all changes are of this style:

  • Changing the index(?) of the fileDescriptor_XXXXXXXX
  • Adding XXX_Unmarshal() / XXX_Marshal() methods
diff --git a/pkg/apis/rollouts/v1alpha1/generated.pb.go b/pkg/apis/rollouts/v1alpha1/generated.pb.go
index 4af51298..2d982911 100644
--- a/pkg/apis/rollouts/v1alpha1/generated.pb.go
+++ b/pkg/apis/rollouts/v1alpha1/generated.pb.go
@@ -48,10 +48,38 @@ var _ = math.Inf
 // proto package needs to be updated.
 const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package

+func (m *ALBStatus) Reset()      { *m = ALBStatus{} }
+func (*ALBStatus) ProtoMessage() {}
+func (*ALBStatus) Descriptor() ([]byte, []int) {
+       return fileDescriptor_e0e705f843545fab, []int{0}
+}
+func (m *ALBStatus) XXX_Unmarshal(b []byte) error {
+       return m.Unmarshal(b)
+}
+func (m *ALBStatus) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+       b = b[:cap(b)]
+       n, err := m.MarshalToSizedBuffer(b)
+       if err != nil {
+               return nil, err
+       }
+       return b[:n], nil
+}
+func (m *ALBStatus) XXX_Merge(src proto.Message) {
+       xxx_messageInfo_ALBStatus.Merge(m, src)
+}
+func (m *ALBStatus) XXX_Size() int {
+       return m.Size()
+}
+func (m *ALBStatus) XXX_DiscardUnknown() {
+       xxx_messageInfo_ALBStatus.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ALBStatus proto.InternalMessageInfo
+
 func (m *ALBTrafficRouting) Reset()      { *m = ALBTrafficRouting{} }
 func (*ALBTrafficRouting) ProtoMessage() {}
 func (*ALBTrafficRouting) Descriptor() ([]byte, []int) {
-       return fileDescriptor_e0e705f843545fab, []int{0}
+       return fileDescriptor_e0e705f843545fab, []int{1}
 }
 func (m *ALBTrafficRouting) XXX_Unmarshal(b []byte) error {
        return m.Unmarshal(b)
@@ -79,7 +107,7 @@ var xxx_messageInfo_ALBTrafficRouting proto.InternalMessageInfo
 func (m *AmbassadorTrafficRouting) Reset()      { *m = AmbassadorTrafficRouting{} }
 func (*AmbassadorTrafficRouting) ProtoMessage() {}
 func (*AmbassadorTrafficRouting) Descriptor() ([]byte, []int) {
-       return fileDescriptor_e0e705f843545fab, []int{1}
+       return fileDescriptor_e0e705f843545fab, []int{2}
 }
 func (m *AmbassadorTrafficRouting) XXX_Unmarshal(b []byte) error {
        return m.Unmarshal(b)
@@ -107,7 +135,7 @@ var xxx_messageInfo_AmbassadorTrafficRouting proto.InternalMessageInfo
 func (m *AnalysisRun) Reset()      { *m = AnalysisRun{} }
 func (*AnalysisRun) ProtoMessage() {}
 func (*AnalysisRun) Descriptor() ([]byte, []int) {
-       return fileDescriptor_e0e705f843545fab, []int{2}
+       return fileDescriptor_e0e705f843545fab, []int{3}
 }

With & without this diff though I looked at the error itself:

Second observation

When the e2e test fails, the controller output is

$ make start-e2e
....
ERRO[2021-11-28T15:48:50-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-28T15:48:50-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
ERRO[2021-11-28T15:48:50-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default
....


with some additional logging from

diff --git a/rollout/temlateref.go b/rollout/temlateref.go
index 58cdf5d7..64055eca 100644
--- a/rollout/temlateref.go
+++ b/rollout/temlateref.go
@@ -250,7 +250,7 @@ func (r *informerBasedTemplateResolver) updateRolloutsReferenceAnnotation(obj in
                        ro.Spec.Template.Spec.Containers = []corev1.Container{}
                        _, err := r.argoprojclientset.ArgoprojV1alpha1().Rollouts(ro.Namespace).Update(context.TODO(), ro, v1.UpdateOptions{})
                        if err != nil {
-                               log.Errorf("Cannot update the workload-ref/annotation for %s/%s", ro.GetName(), ro.GetNamespace())
+                               log.Errorf("Cannot update the workload-ref/annotation for %s/%s: %v", ro.GetName(), ro.GetNamespace(), err)
                        }
                }
        }

The error message turns into

ERRO[2021-11-28T15:47:09-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default: Rollout.argoproj.io "rollout-ref-deployment" is invalid: spec.selector: Invalid value: "null": spec.selector in body must be of type object: "null"
ERRO[2021-11-28T15:47:09-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default: Rollout.argoproj.io "rollout-ref-deployment" is invalid: spec.selector: Invalid value: "null": spec.selector in body must be of type object: "null"
ERRO[2021-11-28T15:47:09-05:00] Cannot update the workload-ref/annotation for rollout-ref-deployment/default: Rollout.argoproj.io "rollout-ref-deployment" is invalid: spec.selector: Invalid value: "null": spec.selector in body must be of type object: "null"

I'm not sure if the selector should be set or not for the test. But my understanding is that"null"is marshaled to theRollouts().Update()` call - which is for sure not what is desired.

Altering the type to make the selector optional seems to fix the issue:

diff --git a/pkg/apis/rollouts/v1alpha1/types.go b/pkg/apis/rollouts/v1alpha1/types.go
index 244db01d..8b51aa60 100644
--- a/pkg/apis/rollouts/v1alpha1/types.go
+++ b/pkg/apis/rollouts/v1alpha1/types.go
@@ -44,7 +44,7 @@ type RolloutSpec struct {
        // selected by this will be the ones affected by this rollout.
        // It must match the pod template's labels.
        // +optional
-       Selector *metav1.LabelSelector `json:"selector" protobuf:"bytes,2,opt,name=selector"`
+       Selector *metav1.LabelSelector `json:"selector,omitempty" protobuf:"bytes,2,opt,name=selector"`
        // Template describes the pods that will be created.
        // +optional
        Template corev1.PodTemplateSpec `json:"template" protobuf:"bytes,3,opt,name=template"`

as now the all tests pass.

But that seems to be the wrong solution?! Is this even an optional field?

My kubeconfig is set to use EKS/K8s 1.19

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

All those changes are also available in branch (MasterFix)[https://github.com/derjust/argo-rollouts/pull/2/commits]


<!-- Issue Author: Don't delete this message to encourage other users to support your issue! --> Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

created time in 8 days

PR opened derjust/argo-rollouts

Master fix
+1116 -517

0 comment

5 changed files

pr created time in 8 days

PR closed derjust/argo-rollouts

Master fix
+9131 -1431

0 comment

113 changed files

derjust

pr closed time in 8 days

push eventderjust/argo-rollouts

Ryan M Smith

commit sha ff3471a2fc3ccb90dbb1f370d7e399ff3064043a

feat: add default() evaluate helper. allow empty datadog result. Fixes #1548 (#1551) Signed-off-by: Ryan M Smith <rms1000watt@gmail.com>

view details

cskh

commit sha 99c15661d836b099a8e68c24f966b249d192c441

feat: HA Leader election support on rollouts-controller (#1519) Signed-off-by: caoyang001 <caoyang001@foxmail.com> Signed-off-by: Hui Kang <hui.kang@salesforce.com> Co-authored-by: caoyang001 <caoyang001@foxmail.com> Co-authored-by: Hui Kang <hui.kang@salesforce.com> Co-authored-by: William Van Hevelingen <William.VanHevelingen@acquia.com>

view details

John Huynh

commit sha 73997ee3aeaac24b70e6d1c21a020b2fedc2c0a4

docs: example data is used in both examples (#1570) Signed-off-by: John Huynh <johnhuynh@admins-MacBook-Pro.local>

view details

Kostis (Codefresh)

commit sha f07de59893b2363b0a7d3890d588ae5acc2d3d39

docs: Added readthedocs configuration (#1528) Signed-off-by: Kostis Kapelonis <kostis@codefresh.io>

view details

cskh

commit sha 76c1b3ef3c4013e6c559cb3a2d662619828dc6f3

fix: viewcontroller gorouting leak in status and get subcommand (#1584) Signed-off-by: Hui Kang <hui.kang@salesforce.com>

view details

Andrii Perenesenko

commit sha 551f89e27ceba503532f28b287b931e88188c766

fix: release the dashboard-install.yaml (#1601) Signed-off-by: Andrii Perenesenko <andrii.perenesenko@gmail.com>

view details

Yuan Tang

commit sha d83543a8f5c2eab5abce2b8c5d962a438025e313

docs: Add Akuity to the list of users (#1598) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

view details

Kareena Hirani

commit sha 345b2d4fd9bfa00191f4e98edf213927781e2416

feat: Istio Host-level TrafficRouting with experiment step (#1569) * feat: Istio Host-level TrafficRouting with experiment step Signed-off-by: khhirani <kareena.hirani@gmail.com> * Remove comments Signed-off-by: khhirani <kareena.hirani@gmail.com>

view details

FBLGit

commit sha a398ff9cc05427dc4b7b7142246f0082188054cd

feat(controller): multiple TrafficRoutingReconciler (#1472) Signed-off-by: fblgit <info@fblnet.net>

view details

Leonardo Luz Almeida

commit sha ac155f1b263a869ad00a34c2145b641bf8ee16bd

feat: support Ingress from Networking API version (#1529) Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>

view details

Kiran Meduri

commit sha fe2c68b3364e1befbdf7908652e2dda3e591705d

fix: Handle minor version with '+' when determining ingress mode (#1529) (#1612) Signed-off-by: Kiran Meduri <meduri@amazon.com>

view details

Noam Gal

commit sha b6f0182fb697413cdaaf1ece2ad9b7453c187461

minor mis-type fix, and formatting (#1613) Signed-off-by: Noam Gal <noam.gal@codefresh.io>

view details

Noam Gal

commit sha fe87bdd584e5a5faba2a723ce24ce6ed804f8c57

feat(webmetric): Support POST/PUT content with web metrics. Fixes #371 (#1573) feat(webmetric): Support POST/PUT content with web metrics. Fixes #371 (#1573) Signed-off-by: Noam Gal <noam.gal@codefresh.io>

view details

Flávio Lemos

commit sha 9d2ccb9f1e5a617884dd0958d43aab6a4fb35fef

fix: rollout experiment template changing reference rs template labels. Fixes #1596 (#1597) * fix: rollout experiment template changing reference rs template labels Signed-off-by: Flavio Lemos <flavio.lemos@farfetch.com> * docs: Add Farfetch to USERS.md Signed-off-by: Flavio Lemos <flavio.lemos@farfetch.com>

view details

cskh

commit sha 6b2b375d503dca5bd6c9f63fa43e3e6cffcca340

feat: add healthz probe port and update the install.yaml (#1578) * feat: add healthz probe port and update the install.yaml - distinguish between exposed healthz port and metrics port - liveness probe uses healthz port readiness probe uses metrics port In HA mode, only the primary instance is ready to accept request and the metrics queries Signed-off-by: Hui Kang <hui.kang@salesforce.com> * Expose empty metrics if the controller is secondary/backup Signed-off-by: Hui Kang <hui.kang@salesforce.com> * Add unit test - Primary controller, secondary controller, NewManager test Signed-off-by: Hui Kang <hui.kang@salesforce.com> Co-authored-by: Hui Kang <hui.kang@salesforce.com>

view details

cskh

commit sha 75cdde7606cc5fdcd4254ba44960f1d61ff7c67a

fix: inconsistent status command output (#1433) Signed-off-by: Hui Kang <hui.kang@salesforce.com>

view details

Yash Mehrotra

commit sha 3ad0417c019cffd24a36ab6297a6acaf81d8d9c7

feat(manifests): Add Age column to all CRD(s). Fixes #1511 (#1527) Signed-off-by: Yash Mehrotra <yashmehrotra95@gmail.com>

view details

cskh

commit sha 2e401897d3f375ae242120f8329c0ee1e7ce0e95

fix: validate service selctor labels matching rollout template labels (#1618) Signed-off-by: Hui Kang <hui.kang@salesforce.com>

view details

cskh

commit sha dc1c11bdf1f0772c87c810a6d7ecb1ebaf799f7a

chore: update k8s pkg to resolve vulnerabilities (#1545) Signed-off-by: Hui Kang <hui.kang@salesforce.com>

view details

Rohit Agrawal

commit sha f77b2c5936f52b2d949209252e942b8b37e18631

fix(e2e): DeFlake E2E Tests #1647 (#1648) Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>

view details

push time in 8 days

push eventderjust/argo-rollouts

push time in 8 days

PR opened derjust/argo-rollouts

Master fix
+8877 -1352

0 comment

108 changed files

pr created time in 8 days

create barnchderjust/argo-rollouts

branch : MasterFix

created branch time in 8 days

push eventderjust/argo-rollouts

Sebastian J

commit sha 3218bfa017cdf9e4bb05a96ec8b1a9a57b7bc554

Force codegen GO 1.16 Forced codegen via downgrading to Go 1.16: ``` $ env|grep GO GOPATH=/Users/sebastian/go ``` ``` $ go version go version go1.16.10 darwin/amd64 ``` ``` $ echo $PATH /Users/sebastian/.sdkman/candidates/micronaut/current/bin:/Users/sebastian/.sdkman/candidates/java/current/bin:/Users/sebastian/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/bin:/Users/sebastian/go/bin ``` Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

push time in 10 days

pull request commentargoproj/argo-rollouts

fix(controller): Sticky session correction for AWS ALB. Fixes #1572

@harikrongali thank you for your continued help. Yes i did so - ran it again, too:

$ brew install go kubectl kustomize golangci-lint protobuf swagger-codegen

Warning: go 1.17.2 is already installed and up-to-date.
To reinstall 1.17.2, run:
  brew reinstall go
Warning: kubernetes-cli 1.22.4 is already installed, it's just not linked.
To link this version, run:
  brew link kubernetes-cli
Warning: kustomize 4.4.1 is already installed and up-to-date.
To reinstall 4.4.1, run:
  brew reinstall kustomize
Warning: golangci-lint 1.43.0 is already installed and up-to-date.
To reinstall 1.43.0, run:
  brew reinstall golangci-lint
Warning: protobuf 3.17.3 is already installed and up-to-date.
To reinstall 3.17.3, run:
  brew reinstall protobuf
Warning: swagger-codegen 3.0.29 is already installed and up-to-date.
To reinstall 3.0.29, run:
  brew reinstall swagger-codegen

To help it i force-use GoLang 1.16 - for some reason TestFunctionalSuite/TestWorkloadRef still fails :-(

derjust

comment created time in 10 days

push eventderjust/argo-rollouts

Sebastian J

commit sha 48b3d4fe4d886bdcd6a019dd9e6f4ff33603aa9b

Force codegen GO 1.16 Forced codegen via downgrading to Go 1.16: ``` $ env|grep GO GOPATH=/Users/sebastian/go ``` ``` $ go version go version go1.16.10 darwin/amd64 ``` ``` $ echo $PATH /Users/sebastian/.sdkman/candidates/micronaut/current/bin:/Users/sebastian/.sdkman/candidates/java/current/bin:/Users/sebastian/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/bin:/Users/sebastian/go/bin ```

view details

push time in 10 days

push eventderjust/argo-rollouts

Sebastian J

commit sha 405772932ed409a58fe175466b28800a2494b94a

Another test round Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

push time in 13 days

pull request commentargoproj/argo-rollouts

fix(controller): Sticky session correction for AWS ALB. Fixes #1572

Unit tests work locally (Note: they fail on OSX with a permission denied - which makes the list test fail as the clock (I guess) can't be reset. DCO also fixed. Thank you for the ongoing support!

derjust

comment created time in 13 days

PullRequestReviewEvent

Pull request review commentargoproj/argo-rollouts

fix(controller): Sticky session correction for AWS ALB. Fixes #1572

+//go:build !ignore_autogenerated

Thank you; undone. Does the code-gen has an issue running on Mac?

derjust

comment created time in 13 days

push eventderjust/argo-rollouts

cskh

commit sha 615bce39475c74a44bf253000037e58cadf84a9e

fix: reset the progress condition when a pod is restarted (#1649) fix: reset the progress condition when a pod is restarted (#1649) Signed-off-by: Hui Kang <hui.kang@salesforce.com> Co-authored-by: Hui Kang <hui.kang@salesforce.com> Co-authored-by: Rohit Agrawal <rohit.agrawal@databricks.com>

view details

Yuan Tang

commit sha ca3ccd01c13bf21732c484c03c6086c416e98633

docs: Add link to awesome-argo for more resources (#1622) * docs: Add link to awesome-argo for more resources Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> * Retrigger CI pipeline Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

view details

Siddharth Maloo

commit sha 47d59fa9855ab737e40e21ef5938b2b6723cb899

feat: added Argo version info in /metrics endpoint (#1662) Signed-off-by: Siddharth Maloo <smaloo@salesforce.com>

view details

Sebastian J

commit sha 78aaee34aae858b578855d40964feb0c8faa03c5

Add support for AWS ALB TargetGroupStickinessConfig Adds support for AWS ALB [TargetGroupStickinessConfig](https://aws.amazon.com/blogs/aws/new-application-load-balancer-simplifies-deployment-with-weighted-target-groups/) This is required to support sticky session on the listener level while Argo is using ALB's weighting Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha 8c15f939b52d5473598c651fa0fac50d6ab19f70

Remove code smells Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha 527ea2def2d69b9b74746a47a768d8ce10d91ab6

more code smells Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha a17a15f7bb8c10e85eeefa347dd37c731f33affa

PR feedback Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

Sebastian J

commit sha 3db8036022f1dced9b702f28c742aa2e90713b11

Revert Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

push time in 13 days

Pull request review commentargoproj/argo-rollouts

fix(controller): Sticky session correction for AWS ALB. Fixes #1572

+//go:build !windows

Figured it out: the make codegen created this change. I removed if

derjust

comment created time in 17 days

PullRequestReviewEvent

push eventderjust/argo-rollouts

Sebastian J

commit sha 5e40c54fd2872202a01d8dd24226df202ecd602f

PR feedback Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

push time in 17 days

push eventderjust/argo-rollouts

Sebastian J

commit sha 7b26322cda06f09d755cb79a3e502d4ee1fdd1cd

Lint Signed-off-by: Sebastian J <zeeman@zeeman.de>

view details

push time in 17 days

more