profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/edma2/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

edma2/asmscan 95

a port scanner written in x86 assembly

edma2/clojure-msgpack 55

MessagePack implementation in Clojure / msgpack.org[Clojure]

edma2/ntp 16

network time protocol client

edma2/legolas 2

minimum viable ELF linker

edma2/nrmp 2

NRMP residency match algorithm

edma2/asm 1

my x86 assembly projects

edma2/b64 1

base64 encode-decoder

Pull request review commentcortexproject/cortex

Federated ruler proposal

+---+title: "Ruler Tenant Federation"+linkTitle: "Ruler Tenant Federation"+weight: 1+slug: "Ruler Tenant Federation+---++- Author: [Rees Dooley](https://github.com/rdooley)+- Date: September 2021+- Status: Accepted++## Overview++This document aims to describe how to implement the ability to allow rules to cover data from more than a single Cortex tenant, here after refered to as federated rules. Since currently rules are owned by, query data from and save resulting series in the same tenant, this document aims to provide clear delineation of who owns a federated rule, what tenants the federated rule queries data from and where the federated rule saves resulting series.++## Reasoning++There are two primary use cases for allowing federated rules which query data from multiple tenants; administration of cortex and composite tenants.++In the case of the administration of cortex, when running Cortex within a large organization, there may be metrics spanning across tenants which might be desired to be monitored e.g. administrative metrics of the cortex system like `prometheus_rule_evaluation_failures_total` aggregated by `__tenant_id__`. In this case, a team e.g. `infra` may wish to be able to create a rule, owned by `infra` which queries multiple tenants `t0|t1|...|ti` and stores resulting series in `infra`. More generally a tenant `A` may wish to query data from other tenants `B|C|D` and store the data in a tenant which may be `A`, but could be `E`.++Additionally an organization may wish to treat several cortex tenants `t0|t1|...|ti` as one logical tenant due to compactor scalability. In this case a composite tenant `t0|t1|...|ti` would own a federated rule, which queries each subtenant `t0` thru `ti`. The resulting data could be sent to a specific tenant e.g. `admin` or as the composite tenant treats each subtenant uniformly, a random, but consistent subtenant can be chosen to store the resulting series in, in this case `tj` where 0 <= j <= i.

clarification: the resulting series are spread approximately equally among the tenants, or all go to a single tenant?

rdooley

comment created time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentcortexproject/cortex

Federated ruler proposal

+---+title: "Ruler Tenant Federation"+linkTitle: "Ruler Tenant Federation"+weight: 1+slug: "Ruler Tenant Federation+---++- Author: [Rees Dooley](https://github.com/rdooley)+- Date: September 2021+- Status: Accepted++## Overview++This document aims to describe how to implement the ability to allow rules to cover data from more than a single Cortex tenant, here after refered to as federated rules. Since currently rules are owned by, query data from and save resulting series in the same tenant, this document aims to provide clear delineation of who owns a federated rule, what tenants the federated rule queries data from and where the federated rule saves resulting series.++## Reasoning++There are two primary use cases for allowing federated rules which query data from multiple tenants; administration of cortex and composite tenants.++In the case of the administration of cortex, when running Cortex within a large organization, there may be metrics spanning across tenants which might be desired to be monitored e.g. administrative metrics of the cortex system like `prometheus_rule_evaluation_failures_total` aggregated by `__tenant_id__`. In this case, a team e.g. `infra` may wish to be able to create a rule, owned by `infra` which queries multiple tenants `t0|t1|...|ti` and stores resulting series in `infra`. More generally a tenant `A` may wish to query data from other tenants `B|C|D` and store the data in a tenant which may be `A`, but could be `E`.++Additionally an organization may wish to treat several cortex tenants `t0|t1|...|ti` as one logical tenant due to compactor scalability. In this case a composite tenant `t0|t1|...|ti` would own a federated rule, which queries each subtenant `t0` thru `ti`. The resulting data could be sent to a specific tenant e.g. `admin` or as the composite tenant treats each subtenant uniformly, a random, but consistent subtenant can be chosen to store the resulting series in, in this case `tj` where 0 <= j <= i.++## Challenges++### Allow federated rules behind feature flag++#### Challenge++Federated tenant rules and alerts will not be a good fit for organization and should be behind a feature flag.++#### Proposal++For federated rules owned by a single tenant, creation of federated rules (those sourcing data from multiple tenants) should be blocked behind the feature flag `tenant-federation.enabled`.++To support composite tenants, if tenant federation is enabled for ruler and alertmanager via `tenant-federation.enabled`, then ruler use a `mergeQueryable` to aggregate the results of querying multiple tenants. The ruler and alertmanager APIs should be updated to always call `tenant.GetTenantIDs` instead of `tenant.GetTenantID`, which will use the MultiTenantResolver when tenant federation is enabled.++### Where to store resulting series of federated rule++#### Challenge++A single tenant rule always stores produced series in the tenant where the rule exists. This 1 -> 1 mapping becomes a many -> 1 mapping for federated rules.++#### Proposal++As some use cases will demand that a specific federated rule, querying tenants `B` and `C`, is stored in the owning team's tenant `A`, an option to allow explicit assignment of destination tenant for a federated rule is needed. Some use cases where a set of tenants `A|B|C|...|Z` are being treated as a single logical tenant when querying this explicit assignment of destination tenant is not needed.++To allow both of these, we suggest an additional field to the rules proto `dest_tenant_id` which determines which tenant to send the series produced by the rule to, used if and only if the rule is a federated rule. If this field is not given, for composite tenants a random but consistent subtenant of the multiple tenants owning the rule is chosen using a hashmod of the series label to determine the subtenant and for single tenants owning a federated rule the resulting series is saved in the tenant which owns the rule.++### Which tenants to query from for federated rules++#### Challenge++A single tenant rule always queries the tenant which owns the rule. This 1 -> 1 mapping becomes a 1 -> many mapping for federated rules.++#### Proposal++As some use cases will demand that a specific federated rule, querying tenant B and C, is stored in the owning teams tenant A, an option to allow explicit assignment of source tenants for a federated rule is needed. In the case of a composite tenant where a set of tenants `A|B|C|...|Z` are being treated as a single logical tenant when querying this explicit assignment of destination tenant is explicitly called for, but could prove useful.

when querying this explicit assignment of destination tenant is explicitly called for

what does this mean?

rdooley

comment created time in 3 days

pull request commentcortexproject/cortex

Federated ruler

I don't have any issue with making the destination tenant for a recording rule more explicit, I'm just trying to think thru where to store that info in the rules file structure of rules/${OrgID}/${namespace}. A convenience of random consistent subtenant for federated rules is that I did not have to deal with that. 🤔

Can Cortex support and parse an additional field in the Prometheus rule format, or is compatibility a concern?

rdooley

comment created time in 8 days

Pull request review commentcortexproject/cortex

Multitenant ruler

 import ( const ( 	defaultTenantLabel   = "__tenant_id__" 	retainExistingPrefix = "original_"-	maxConcurrency       = 16+	maxConcurrency       = 60

how about making this one a flag? we can use this param could to tune the maximum concurrent fanout when executing cross-tenant queries or rules.

rdooley

comment created time in 8 days

PullRequestReviewEvent

pull request commentcortexproject/cortex

Multitenant ruler

Hi @rdooley, thanks for taking on this feature. For my use-case, I'm not sure that writing to a random (albeit consistent) tenant would work well for Ruler tenant federation. It would be ideal from my perspective to give the user control over the "source" and "destination" tenants.

IMO, multitenant federation in the ruler is a special case of the more general problem of accessing another tenant's data when defining recording rules/alerts (in the multitenant case, this means multiple tenants). When running Cortex within a single large organization split by team (where tenant == team), for example, team A may want to consume team B's metrics within team A's recording rules and alerts. Or, in the case of multitenant federation, team A may want to consume teams B and C metrics in a single rule.

One way to achieve this would be supporting extra metadata in the rule/alert which specifies the tenant ID (defaults to the tenant in which the rule is created), similar to how you pass a composite X-Scope-OrgID header to the query frontend. I'm curious what you think about this idea and whether it aligns with your original vision.

rdooley

comment created time in 8 days

pull request commentcortexproject/cortex

Allow disabling of ring heartbeats by setting relevant options to zero.

Out of curiosity, what is the motivation behind this PR (and https://github.com/cortexproject/cortex/pull/4342)? I couldn't find much docs on this feature. Does this improve availability by ignoring heartbeat failures?

stevesg

comment created time in 2 months

push eventedma2/cortex-tenant

Eugene Ma

commit sha a4dfc5331a5ac36220e1094d05949b23caf7ac0b

set default Concurrency to 512

view details

Eugene Ma

commit sha c51c4351aca09d1349f62d5f8b725561d6c47d37

add test

view details

push time in 3 months

pull request commentblind-oracle/cortex-tenant

add Concurrency flag to fasthttp server

Sure, let's use a more reasonable default. FWIW I've set this to 50 for my workload (starts OOMing around 70 with my benchmarks).

edma2

comment created time in 3 months

PR opened blind-oracle/cortex-tenant

add Concurrency flag to fasthttp server

We are seeing the proxy crash due to OOM errors when there is a surge in requests. Pass through a Concurrency flag to fasthttp server options to shed load (503s) when server is overloaded. When unset, the fasthttp default is 256 * 1024.

From fasthttp docs, the flag is defined as:

The maximum number of concurrent connections the server may serve.

+4 -0

0 comment

3 changed files

pr created time in 3 months

create barnchedma2/cortex-tenant

branch : add-concurrency-flag

created branch time in 3 months

fork edma2/cortex-tenant

Prometheus remote write proxy that adds Cortex tenant ID based on metric labels

fork in 3 months