profile
viewpoint

gkleiman/ace 0

Ajax.org Cloud9 Editor

gkleiman/asyncio-chatserver 0

Toy chat server implemented in Python using asyncio

gkleiman/Backbone.TableView 0

Backbone View to render collections as tables

gkleiman/cv 0

Curriculum Vitae

gkleiman/dcos-k8s-rust-skaffold-demo 0

A demo of pipelining Rust application development to Kubernetes on minikube with Skaffold.

gkleiman/envoy 0

Cloud-native high-performance edge/middle/service proxy

gkleiman/g.raphael 0

Charts for Raphaël

gkleiman/janus 0

A MacVim Distro

push eventgkleiman/envoy

Gastón Kleiman

commit sha 0a2b1fcdad5daeef1136ef7c2208f356d3fcf05f

Address feedback and cleanup BUILD file Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 7 days

push eventgkleiman/envoy

Gastón Kleiman

commit sha 4cf5f196d6ac02ac7ab7f57d31c1f03f340febdc

Perf/logging improvements Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 7 days

push eventgkleiman/envoy

Gastón Kleiman

commit sha c2304ff3554bb0e36d1de368fa17eef8a0d50994

Perf/logging improvements Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 7 days

push eventgkleiman/envoy

Gastón Kleiman

commit sha 762e61b726de4f770ac0346f2a5ae43ff4934f60

Address feedback Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

Gastón Kleiman

commit sha 76f9467c8e640ed9d57a2291204a8857f543c2e4

Perf/logging improvements Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 7 days

pull request commentenvoyproxy/envoy

Add support for making Least Requests LB behave like Round Robin in weighted hosts case

I think a better approach would be to just add some coefficient (k) that makes the weights look something like:

+1 I like this idea. Let's do that!

Good idea! I implemented this with a global runtime. If things work fine in the experiment, I'll make it a real config option inside LeastRequestLbConfig.

gkleiman

comment created time in 8 days

push eventgkleiman/envoy

Gastón Kleiman

commit sha 762e61b726de4f770ac0346f2a5ae43ff4934f60

Address feedback Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 8 days

push eventgkleiman/envoy

Gastón Kleiman

commit sha 5a5646ae3abb6c091d281bb91e54ec3856d7d9ed

Fix Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 8 days

push eventgkleiman/envoy

Gastón Kleiman

commit sha 8fc6ef5ae1b69944b3cd027f18ad17f6e5115a1f

Fixity fix Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 8 days

create barnchgkleiman/envoy

branch : alternative-lr-weights-wip

created branch time in 8 days

issue commentenvoyproxy/envoy

Add runtime flag to make Weighted Least Request LB behave like Weighted Round Robin LB in weighted mode

@tonya11en I've experimented with increasing the host weight, but it hasn't helped with services that don't get enough traffic.

Per your suggestion on https://github.com/envoyproxy/envoy/pull/11252 I'm working on making the LR LB use the following formula to calculate the host weights when not all weights are equal: new_weight = (configured_weight / outstanding_requests^k).

I'll try that change internally and then follow up here once I have the results.

gkleiman

comment created time in 8 days

PR opened envoyproxy/envoy

Add support for making Least Requests LB behave like Round Robin in weighted hosts case

Commit Message: Add support for making Least Requests LB behave like Round Robin in weighted hosts case

Additional Description: This patch makes it possible to make the Least Requests Load Balancer behave like the Round Robin load balancer if hosts have different weights. The patch will be tested at Lyft first to see whether it improves the load balancing issues during red line/squeeze tests, as described in https://github.com/envoyproxy/envoy/issues/10602.

Risk Level: S Testing: Unit test. Will use this patch to experiment at Lyft.

Runtime guard: The alternative behavior is disabled by default and can be globally turned on via the envoy.reloadable_features.alternative_least_request_weights runtime key.

If experimentation shows that this patch indeed fixes https://github.com/envoyproxy/envoy/issues/10602, I will follow up with a change to make it possible to enable the alternative LB behavior for individual clusters.

+49 -1

0 comment

5 changed files

pr created time in 9 days

push eventgkleiman/envoy

Gastón Kleiman

commit sha a1b52fd1034b8120eaad1649ece6129ce8a84722

Add support for making Least Requests LB behave like Round Robin in weighted hosts case Signed-off-by: Gastón Kleiman <gkleiman@lyft.com>

view details

push time in 9 days

create barnchgkleiman/envoy

branch : alternative-lr-weights

created branch time in 9 days

issue openedenvoyproxy/envoy

Add runtime flag to make Weighted Least Request LB behave like Weighted Round Robin LB in weighted mode

At Lyft we use Envoy to red line test/squeeze a single host of a service. Red line tests are performed by increasing the load-balancing weight of a host via EDS to redistribute the service traffic and send an increasing amount of the traffic to a particular host.

While this approach works great in many cases, we noticed a few problems:

  1. For some services there is a limit to the amount of traffic that can be redistributed to the host under test. This happens even when the service is getting a considerable larger amount of traffic.
  2. The amount of traffic that is redistributed to a particular endpoint and its load-balancing weight are not always linearly proportional.

After doing some experiments we arrived at the conclusion that these problems happen because our clusters use the Weighted Least Request (WLR) load balancer.

Currently if two or more hosts in a cluster have different load-balancing weights, the WLR load balancer shifts into a mode where it uses a weighted round robin schedule in which weights are dynamically adjusted based on the host’s request load at the time of selection (weight is divided by the current active request count).

For some services the number of active connection to a host increases as that host gets more traffic, even if the host is not overloaded. This causes both aforementioned problems.

The Weighted Round Robin (WRR) load balancer does behave as desired for red line tests and we would like to switch a cluster to the WRR load balancer while running a red line test. However this is easier said than done — a cluster’s load balancer can only be changed via CDS and doing so makes Envoy drain and reconnect all existing connection pools [1].

Since the behavior of the WLR load balancer is non-intuitive when two or more hosts in a cluster have different load-balancing weights, I propose adding a runtime flag that would make the WLR load balancer behave like the plain WRR load balancer in this scenario.

This means that if the runtime flag is enabled and there are two or more hosts in cluster that have different load-balancing weights, the WLR load balancer would not dynamically adjust the host weights based on the host’s request load. It would instead calculate the host weights using the same calculation used by the WRR load balancer: host weight / sum(host weights).

created time in 2 months

issue commentenvoyproxy/envoy

Ext Authz (gRPC): Plugin denies when failure_mode_allow set to true and Auth Server responses 5xx

According to the CheckResponse message definition what you observe is the expect behavior:

  // Status `OK` allows the request. Any other status indicates the request should be denied.
  google.rpc.Status status = 1;

CheckResponse.denied_response.status can be used to override the status that Envoy will return, but is not used to determine whether a request should be allowed or denied.

@dio I think we can probably close this issue unless we want to improve some documentation.

kikogolk

comment created time in 3 months

issue commentenvoyproxy/envoy

Ext Authz (gRPC): Plugin denies when failure_mode_allow set to true and Auth Server responses 5xx

@kikogolk are you sure that your ext authz service is responding with a 503?

I tried this using a very simple HTTP/2 server that always responds with a 503 and an empty response body and I couldn't reproduce the behavior you describe.

In my tests Envoy will allow the request and increase both the http.ingress_http.ext_authz.error and the http.ingress_http.ext_authz.failure_mode_allowed stats.

kikogolk

comment created time in 3 months

more