The script timed out due to an instance taking longer than usual to terminate.

The environment variable that needs to be set to increase the number of checks is GLOBAL_MAX_RETRY. But the documentation says this is Number of attempts of a health check. However the termination check is not a health check and this documentation should be corrected to something like Number of attempts of a health or termination check.

Fixed by #58

As discussed here https://github.com/kubernetes/kubernetes/issues/65013, while rolling the node pool, cordoning the nodes will remove all the nodes from the loadbalancer which might cause downtime. Instead, we can taint the node to make it non-schedulable instead of cordoning the node.

Update: this might not cause downtime as the scale up of nodes in the ASG is done before cordoning of the nodes.

Resolved by https://github.com/hellofresh/eks-rolling-update/pull/49

fix for https://github.com/hellofresh/eks-rolling-update/issues/48

thanks @akshaychitneni

Currently, after setting the desired capacity of the ASG, the script simply waits CLUSTER_HEALTH_WAIT seconds once (without any retries) to check if all instances come online. This works in best case scenarios, but we observe good amount of variances in AWS when it comes to time it takes for instances to come online. (In my current example that lead to this issue it took 9 minutes).

I know I can increase the CLUSTER_HEALTH_WAIT to 600s, but that means it always wait 10 minutes which is waste of time. So my request is to add a retry, so that we can have a worst case timeout without increasing the rollout time in best case scenario.

fixed by https://github.com/hellofresh/eks-rolling-update/pull/58

In order to address the Feature Request '[Feature] Timeout waiting for instance scale up #55':

I also updated the documentation to address #54

I have added the environment variable CLUSTER_HEALTH_RETRY to allow x amount of retries(specified by the user) of a cluster health check after an ASG Scaling. For some people it make take longer than 90 seconds for there instances to spin up but also by making this change it also can speed up the change for many people. So we can set the CLUSTER_HEALTH_WAIT to only 5 seconds for example and set CLUSTER_HEALTH_RETRY to 40. So the max time it will try is 200 seconds in this scenario but it's possible that it can finish within 20 second, setting it up this way can work for those times when more time is needed and other times when the scale up happens quickly. I also made sure that by default it still runs only once and still uses the wait time of 90 seconds in case people who are already using it like the way it is.

Heres an example snippet of it running and working:

2020-08-29 00:13:49,640 INFO Setting asg desired capacity from 1 to 2 and max size to 20... 2020-08-29 00:13:49,836 INFO Waiting for 5 seconds for ASG to scale before validating cluster health...

2020-08-29 00:13:54,841 INFO Describing autoscaling groups... 2020-08-29 00:13:55,390 INFO Current asg instance count in cluster is: 9. K8s node count should match this number 2020-08-29 00:13:55,390 INFO Checking asg eks-test-workers-2-spot instance count... 2020-08-29 00:13:55,504 INFO Asg eks-test-workers-2-spot does not have enough running instances to proceed 2020-08-29 00:13:55,504 INFO Actual instances: 1 Desired instances: 2 2020-08-29 00:13:55,504 INFO Validation failed for asg eks-test-workers-2-spot. Not enough instances online. 2020-08-29 00:13:55,504 INFO Waiting for 5 seconds for ASG to scale before validating cluster health...

2020-08-29 00:14:00,506 INFO Describing autoscaling groups... 2020-08-29 00:14:01,158 INFO Current asg instance count in cluster is: 9. K8s node count should match this number 2020-08-29 00:14:01,158 INFO Checking asg eks-test-workers-2-spot instance count... 2020-08-29 00:14:01,280 INFO Asg eks-test-workers-2-spot does not have enough running instances to proceed 2020-08-29 00:14:01,280 INFO Actual instances: 1 Desired instances: 2 2020-08-29 00:14:01,281 INFO Validation failed for asg eks-test-workers-2-spot. Not enough instances online. 2020-08-29 00:14:01,281 INFO Waiting for 5 seconds for ASG to scale before validating cluster health...

2020-08-29 00:14:06,285 INFO Describing autoscaling groups... 2020-08-29 00:14:06,778 INFO Current asg instance count in cluster is: 9. K8s node count should match this number 2020-08-29 00:14:06,778 INFO Checking asg eks-test-workers-2-spot instance count... 2020-08-29 00:14:06,930 INFO Asg eks-test-workers-2-spot does not have enough running instances to proceed 2020-08-29 00:14:06,930 INFO Actual instances: 1 Desired instances: 2 2020-08-29 00:14:06,931 INFO Validation failed for asg eks-test-workers-2-spot. Not enough instances online. 2020-08-29 00:14:06,931 INFO Waiting for 5 seconds for ASG to scale before validating cluster health...

2020-08-29 00:14:11,931 INFO Describing autoscaling groups... 2020-08-29 00:14:12,493 INFO Current asg instance count in cluster is: 9. K8s node count should match this number 2020-08-29 00:14:12,493 INFO Checking asg eks-test-workers-2-spot instance count... 2020-08-29 00:14:12,596 INFO Asg eks-test-workers-2-spot does not have enough running instances to proceed 2020-08-29 00:14:12,596 INFO Actual instances: 1 Desired instances: 2 2020-08-29 00:14:12,597 INFO Validation failed for asg eks-test-workers-2-spot. Not enough instances online. 2020-08-29 00:14:12,597 INFO Waiting for 5 seconds for ASG to scale before validating cluster health...

2020-08-29 00:14:17,601 INFO Describing autoscaling groups... 2020-08-29 00:14:18,145 INFO Current asg instance count in cluster is: 10. K8s node count should match this number 2020-08-29 00:14:18,145 INFO Checking asg eks-test-workers-2-spot instance count... 2020-08-29 00:14:18,249 INFO Asg eks-test-workers-2-spot scaled OK 2020-08-29 00:14:18,250 INFO Actual instances: 2 Desired instances: 2 2020-08-29 00:14:18,250 INFO Checking asg eks-test-workers-2-spot instance health... 2020-08-29 00:14:18,350 INFO Instance i-06b896811187d6cc0 - Healthy 2020-08-29 00:14:18,350 INFO Instance i-0be321879567e30aa - Healthy

Great work @crosbymichael1 and thanks @chadlwilson . I did some testing on my side as well and looks good thanks!

Describe the feature request Currently there is MeshConfig. accessLogFile which controls logging for the entire mesh. This is fine for small deployments but for larger deployments enabling logging on thousands of sidecars can cause problems for logging infrastructure. It would be ideal to be able to turn on access logging via an annotation, the Sidecar resource and via the command line.

Also for short term debugging it would be great to be able to use istioctl proxy-config log to control access logging

Previously in 1.5.x we were able to configure logging from the istio-ingressgateway via a telemetry rule, handler and instance. Without this functionality we cannot upgrade to telemetry-v2

Describe alternatives you've considered

istioctl proxy-config log EnvoyFilter - https://github.com/istio/istio/wiki/EnvoyFilter-Samples#tracing-and-access-logging

Affected product area (please put an X in all that apply)

[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [x] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster [ ] Virtual Machine [ ] Multi Control Plane

Additional context

Added .whitesource file. Configuration will now be inherited from the 'repo-config.json' file in the 'whitesource-config' repository.

+1

Enabling PodMonitor on aws-termnation-handler with the following values:

podMonitor:
  create: true
  interval: 30s
  sampleLimit: 5000
  labels:
    prometheus: kube-prometheus

STDERR:
  Error: YAML parse error on aws-node-termination-handler/templates/podmonitor.yaml: error converting YAML to JSON: yaml: line 14: did not find expected key
  Error: plugin "diff" exited with error

Enabling PodMonitor on aws-termnation-handler with the following values:

podMonitor:
  create: true
  interval: 30s
  sampleLimit: 5000
  labels:
    prometheus: kube-prometheus

STDERR:
  Error: YAML parse error on aws-node-termination-handler/templates/podmonitor.yaml: error converting YAML to JSON: yaml: line 14: did not find expected key
  Error: plugin "diff" exited with error

@bwagner5 fixed

Added PR to create a PodMonitor https://github.com/aws/aws-node-termination-handler/pull/206

Description of changes:

Since Prometheus metrics are now available, we want to be able to scrape those metrics. This PR adds a container port for the prometheus metrics and creates a PodMonitor to scrape the /metrics endpoint.

By adding the following values

podMonitor:
  # Specifies whether PodMonitor should be created
  create: true
   # The Prometheus scrape interval
  interval: 30s
  # The number of scraped samples that will be accepted
  sampleLimit: 5000
   # Additional labels to add to the metadata
  labels:
    prometheus: kube-prometheus

Generates the following PodMonitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: release-name-aws-node-termination-handler
  namespace: default
  labels:
    app.kubernetes.io/name: aws-node-termination-handler
    helm.sh/chart: aws-node-termination-handler-0.9.1
    app.kubernetes.io/instance: release-name
    k8s-app: aws-node-termination-handler
    app.kubernetes.io/version: "1.6.1"
    app.kubernetes.io/managed-by: Tiller
    prometheus: kube-prometheus
    
 spec:
  jobLabel: aws-node-termination-handler
  namespaceSelector:
    matchNames:
    - default
  podMetricsEndpoints:
  - interval: 30s
    path: /metrics
    port: http-metrics
  sampleLimit: 5000
  selector:
    matchLabels:
      app.kubernetes.io/name: aws-node-termination-handler

Ports

apiVersion: apps/v1
kind: DaemonSet
          ports:
          - containerPort: "9092"
            hostPort: "9092"
            name: http-metrics
            protocol: TCP

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

closing in favour of https://github.com/aws/aws-node-termination-handler/pull/206

Description of changes:

Since Prometheus metrics are now available, we want to be able to scrape those metrics. This PR adds a container port for the prometheus metrics and creates a PodMonitor to scrape the /metrics endpoint.

By adding the following values

podMonitor:
  # Specifies whether PodMonitor should be created
  create: true
   # The Prometheus scrape interval
  interval: 30s
  # The number of scraped samples that will be accepted
  sampleLimit: 5000
   # Additional labels to add to the metadata
  labels:
    prometheus: kube-prometheus

Generates the following PodMonitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: release-name-aws-node-termination-handler
  namespace: default
  labels:
    app.kubernetes.io/name: aws-node-termination-handler
    helm.sh/chart: aws-node-termination-handler-0.9.1
    app.kubernetes.io/instance: release-name
    k8s-app: aws-node-termination-handler
    app.kubernetes.io/version: "1.6.1"
    app.kubernetes.io/managed-by: Tiller
    prometheus: kube-prometheus
    
 spec:
  jobLabel: aws-node-termination-handler
  namespaceSelector:
    matchNames:
    - default
  podMetricsEndpoints:
  - interval: 30s
    path: /metrics
    port: http-metrics
  sampleLimit: 5000
  selector:
    matchLabels:
      app.kubernetes.io/name: aws-node-termination-handler

Ports

apiVersion: apps/v1
kind: DaemonSet
          ports:
          - containerPort: "9092"
            hostPort: "9092"
            name: http-metrics
            protocol: TCP

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue #, if available:

Description of changes: Since Prometheus metrics are now available, we want to be able to scrape those metrics. This PR adds a container port for the prometheus metrics and creates a PodMonitor to scrape the /metrics endpoint.

By adding the following values

podMonitor:
  # Specifies whether PodMonitor should be created
  create: true
   # The Prometheus scrape interval
  interval: 30s
  # The number of scraped samples that will be accepted
  sampleLimit: 5000
   # Additional labels to add to the metadata
  labels:
    prometheus: kube-prometheus

Generates the following PodMonitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: release-name-aws-node-termination-handler
  namespace: default
  labels:
    app.kubernetes.io/name: aws-node-termination-handler
    helm.sh/chart: aws-node-termination-handler-0.9.1
    app.kubernetes.io/instance: release-name
    k8s-app: aws-node-termination-handler
    app.kubernetes.io/version: "1.6.1"
    app.kubernetes.io/managed-by: Tiller
    prometheus: kube-prometheus
    
 spec:
  jobLabel: aws-node-termination-handler
  namespaceSelector:
    matchNames:
    - default
  podMetricsEndpoints:
  - interval: 30s
    path: /metrics
    port: http-metrics
  sampleLimit: 5000
  selector:
    matchLabels:
      app.kubernetes.io/name: aws-node-termination-handler

Ports

apiVersion: apps/v1
kind: DaemonSet
          ports:
          - containerPort: "9092"
            hostPort: "9092"
            name: http-metrics
            protocol: TCP

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@kornholi We updated to Istio 1.5.4 and still occasionally see this issue