profile
viewpoint
Mads Hartmann mads-hartmann @gitpod-com Denmark https://mads-hartmann.com When I’m not rummaging around in production I spend my energy making it easier to understand, and eventually more reliable.

gitpod-io/gitpod 6098

Gitpod automates the provisioning of ready-to-code development environments.

bash-lsp/bash-language-server 1142

A language server for Bash

mads-hartmann/ensime.tmbundle 37

Textmate port of ensime using the ensime backend.

mads-hartmann/lt-ocaml 18

LightTable plugin for OCaml

bash-lsp/ide-bash 16

Bash language support for Atom-IDE

mads-hartmann/Functional-Dictionary 10

A Lift based web application. It's a dictionary of Functional Programming terms.

mads-hartmann/dotfiles 9

I ❤️ dotfiles

mads-hartmann/mugs 9

Mugs is a immutable collections library for JavaScript written in Coffeescript

mads-hartmann/opa-chess 9

A multiplayer (no AI) chess game implemented using the Opa Programming Language

mads-hartmann/ocaml-utop-emacs-example 7

A small example to accompany a blog post on using utop in Emacs with your own OCaml projects

Pull request review commentgitpod-io/gitpod

[server, protocol] Further fix and improve API tracing

 export namespace TraceContext {             return;         } -        ctx.span.addTags({+        addNestedTags(ctx, {             rpc: {                 system: "jsonrpc",                 method,                 jsonrpc: {-                    parameters: args.slice(),+                    // parameters are added below, because we don't want them be nested, so that we have the possiblity to redact them.                 },             },         });++        for (let i = 0; i < args.length; i++) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, args[i]);+        }+    }++    /**+     * Offers a way to opt-out of tracing for a particular JsonRPC parameter. Required if it contains PII, for instance.+     * @param ctx+     * @param indexes+     * @returns+     */+    export function redactJsonRPCParameters(ctx: TraceContext, indexes: number[]) {+        if (!ctx.span) {+            return;+        }++        for (const i of indexes) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, "REDACTED");+        }+    }++    /**+     * Does what one would expect from `span.addTags`: Calls `span.addTag` for all keys in map, recursively for objects.+     * Example:+     * ```+     * TraceContext.addNestedTags(ctx, {+     *    rpc: {+     *       system: "jsonrpc",+     *       jsonrpc: {+     *          version: "1.0",+     *          method: "test",+     *          parameters: ["abc", "def"],+     *       },+     *    },+     * });+     * ```+     * gives+     * rpc.system = "jsonrpc"+     * rpc.jsonrpc.version = "1.0"+     * rpc.jsonrpc.method = "test"+     * rpc.jsonrpc.parameters.0 = "abc"+     * rpc.jsonrpc.parameters.1 = "def"+     * @param ctx+     * @param keyValueMap+     * @returns+     */+    export function addNestedTags(ctx: TraceContext, keyValueMap: { [key: string]: any }, _namespace?: string) {+        if (!ctx.span) {+            return;+        }+        const namespace = _namespace ? `${_namespace}.` : '';++        for (const k of Object.keys(keyValueMap)) {+            const v = keyValueMap[k];+            if (typeof v === 'object') {+                addNestedTags(ctx, v, `${namespace}${k}`);+            } else {+                ctx.span.setTag(`${namespace}${k}`, v);+            }+        }+    }++    export function setOWI(ctx: TraceContext, owi: LogContext) {+        if (!ctx.span) {+            return;+        }+        addNestedTags(ctx, {+            context: owi,

For now, go with whatever is most consistent with what you have already. We'll then together across all teams work towards standardising on namespaces, but right now only worry about namespaces when you're introducing new attributes. I'm writing up the initial document now so we'll have something to iterate on ☺️

geropl

comment created time in 3 days

PullRequestReviewEvent

pull request commentgitpod-io/gitpod

Made tracing optional for preview environments

/assign @princerachit

Per quest by roboquat

wulfthimm

comment created time in 3 days

Pull request review commentgitpod-io/gitpod

[server, protocol] Further fix and improve API tracing

 export namespace TraceContext {             return;         } -        ctx.span.addTags({+        addNestedTags(ctx, {             rpc: {                 system: "jsonrpc",                 method,                 jsonrpc: {-                    parameters: args.slice(),+                    // parameters are added below, because we don't want them be nested, so that we have the possiblity to redact them.                 },             },         });++        for (let i = 0; i < args.length; i++) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, args[i]);+        }+    }++    /**+     * Offers a way to opt-out of tracing for a particular JsonRPC parameter. Required if it contains PII, for instance.+     * @param ctx+     * @param indexes+     * @returns+     */+    export function redactJsonRPCParameters(ctx: TraceContext, indexes: number[]) {+        if (!ctx.span) {+            return;+        }++        for (const i of indexes) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, "REDACTED");+        }+    }

After having read through the rest of the comments here I agree with @jankeromnes that redacting based on the position of the argument is fragile and I change my recommendation to be that you add a line where you opt-in to adding the arguments, which also gives you the opportunity to give them names, as you suggested @geropl ☺️

For the future, perhaps we can find a way to use named arguments with JSON RPC so we can get the best of both worlds.

geropl

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgitpod-io/gitpod

[server, protocol] Further fix and improve API tracing

 export namespace TraceContext {             return;         } -        ctx.span.addTags({+        addNestedTags(ctx, {             rpc: {                 system: "jsonrpc",                 method,                 jsonrpc: {-                    parameters: args.slice(),+                    // parameters are added below, because we don't want them be nested, so that we have the possiblity to redact them.                 },             },         });++        for (let i = 0; i < args.length; i++) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, args[i]);+        }+    }++    /**+     * Offers a way to opt-out of tracing for a particular JsonRPC parameter. Required if it contains PII, for instance.+     * @param ctx+     * @param indexes+     * @returns+     */+    export function redactJsonRPCParameters(ctx: TraceContext, indexes: number[]) {+        if (!ctx.span) {+            return;+        }++        for (const i of indexes) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, "REDACTED");+        }+    }

I can extend a bit about the motivations behind adding all parameters automatically, if you don't mind a little detour.

There isn't a single agreed-upon definition of observability. Back in august 2019 I had the following take (sorry for quoting myself 😂) based on what I had read and my experience at the time.

Observability is all about being able to ask questions of your system and get answers based on the existing telemetry it produces; if you have to re-configure or modify the service to get answers to your questions you haven’t achieved observability yet

I think it's still a good guiding definition to consider when you instrument your services. You want to strive towards having all the information you need available to you when you're debugging, and sometimes you can't know what that information is ahead of time. With metrics this is impossible as each label you add will introduce more time series which explodes based on the number of labels and their cardinality. With traces that's not a concern, so you're free to add as many attributes as you like. So I'd always default to add things automatically assuming that you can namespace it correctly.

So in that sense, having to specify exactly which parameter to add to the traces means you have to think about what information you might need during debugging ahead of time, and you will inevitably overlook something important.

The PII concern is important though. Honeycomb has a retention of 60 so the data is gone after that, but it would still have reached their servers of course.

It's a shame that JSON RPC arguments are positional, otherwise we might have been able to add some processing in the OpenTelemetry Collector to redact any parameter name that matched "email" or "address" etc.

I'd lean towards the opt-out approach as the vast majority of your parameters (I'm assuming) isn't PII. And as you already have to be careful about PII I don't think it's too much additional load to think about redacting it from telemetry.

But this is up to you. Having a single line in each method also allows you to name the parameters as @geropl highlighted, so that has benefits too. Either way this is a huge improvement and I'm happy to see it.

geropl

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgitpod-io/gitpod

[server, protocol] Further fix and improve API tracing

 export namespace TraceContext {             return;         } -        ctx.span.addTags({+        addNestedTags(ctx, {             rpc: {                 system: "jsonrpc",                 method,                 jsonrpc: {-                    parameters: args.slice(),+                    // parameters are added below, because we don't want them be nested, so that we have the possiblity to redact them.                 },             },         });++        for (let i = 0; i < args.length; i++) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, args[i]);+        }+    }++    /**+     * Offers a way to opt-out of tracing for a particular JsonRPC parameter. Required if it contains PII, for instance.+     * @param ctx+     * @param indexes+     * @returns+     */+    export function redactJsonRPCParameters(ctx: TraceContext, indexes: number[]) {+        if (!ctx.span) {+            return;+        }++        for (const i of indexes) {+            ctx.span.setTag(`rpc.jsonrpc.parameters.${i}`, "REDACTED");+        }+    }++    /**+     * Does what one would expect from `span.addTags`: Calls `span.addTag` for all keys in map, recursively for objects.+     * Example:+     * ```+     * TraceContext.addNestedTags(ctx, {+     *    rpc: {+     *       system: "jsonrpc",+     *       jsonrpc: {+     *          version: "1.0",+     *          method: "test",+     *          parameters: ["abc", "def"],+     *       },+     *    },+     * });+     * ```+     * gives+     * rpc.system = "jsonrpc"+     * rpc.jsonrpc.version = "1.0"+     * rpc.jsonrpc.method = "test"+     * rpc.jsonrpc.parameters.0 = "abc"+     * rpc.jsonrpc.parameters.1 = "def"+     * @param ctx+     * @param keyValueMap+     * @returns+     */+    export function addNestedTags(ctx: TraceContext, keyValueMap: { [key: string]: any }, _namespace?: string) {+        if (!ctx.span) {+            return;+        }+        const namespace = _namespace ? `${_namespace}.` : '';++        for (const k of Object.keys(keyValueMap)) {+            const v = keyValueMap[k];+            if (typeof v === 'object') {+                addNestedTags(ctx, v, `${namespace}${k}`);+            } else {+                ctx.span.setTag(`${namespace}${k}`, v);+            }+        }+    }++    export function setOWI(ctx: TraceContext, owi: LogContext) {+        if (!ctx.span) {+            return;+        }+        addNestedTags(ctx, {+            context: owi,

So the way I'd usually go would multiple levels of namespacing. E.g. workspace.owner.instance that way you have something quite discoverable in your tool - you start writing workspace. and you'll get all attributes related to workspace.

geropl

comment created time in 3 days

PullRequestReviewEvent

issue commentgitpod-io/observability

Find a way to introduce new extVars in a backwards compatible way

@meysholdt You suggested the config option, would you be able extend a bit on your suggestion here? ☺️

mads-hartmann

comment created time in 3 days

pull request commentgitpod-io/gitpod

[tracing] Add version to all traces

Can we configure the opentelemetry-collector somehow? E.g. run one per preview environment and make that one add metadata such as the "name" of the environment?

Yeah that's possible, the OpenTelemetry Collector can be configured to add/remove/modify attributes of all spans it processes (docs). The only caveat is that we have to take the architecture of our OpenTelemetry deployment into consideration. In our case where the OpenTelemetry Collector is co-located in the same cluster as the services it processes spans for we can definitely add cluster specific information to the spans.

We had discarded the idea originally (see comment) as that wouldn't work for version information as a single collector will processes spans for multiple versions of the same service. However, now that we're separating the concerns we should be able to configure it just fine. @ArthurSens I will create a separate ticket for adding preview environment information to the spans through the OpenTelemetry Collector.

ArthurSens

comment created time in 3 days

pull request commentgitpod-io/gitpod

[tracing] Add version to all traces

@ArthurSens Thanks a lot for driving this forward!

I suspect that this means we won't have an as easy way to see what preview environment is sending the trace. If that's the case that's okay, the original assumption was that the preview environment name was encoded in the version so we'd fix two things in one go: making version available in production, and making it easy to find all traces from a preview environment.

Developers will still be able to find their traces form preview environments, they'll just have to look up the relevant git sha first, right? If so, that's okay for now. Once we have individual clusters we might be able to add the k8s cluster name to the spans and then that would be an easier way for people to find all traces from their preview environments.

ArthurSens

comment created time in 4 days

Pull request review commentgitpod-io/gitpod

[tracing] Add version to all traces

 func Init(serviceName string, opts ...Option) io.Closer { 		return nil 	} +	version := os.Getenv("VERSION")

How would this work for the TypeScript services @csweichel?

ArthurSens

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgitpod-io/gitpod

[tracing] Add version to all traces

 export class TracingManager {             },             serviceName         }+        let version = process.env.VERSION;+        if (!version) {+            version = ''+        }         const t = initTracerFromEnv(config, {-            logger: console+            logger: console,+            tags: {+                'service.version': version

Haha amazing, I would suggest using process.env.VERSION || '' then but I also don't want to stall this PR more and as you say, we expect VERSION to be set.

ArthurSens

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentgitpod-io/gitpod

[tracing] Add version to all traces

 export class TracingManager {             },             serviceName         }+        let version = process.env.VERSION;+        if (!version) {+            version = ''+        }         const t = initTracerFromEnv(config, {-            logger: console+            logger: console,+            tags: {+                'service.version': version

That's fair - in the interest of keeping it consistent between Go and TypeScript I still vote we don't fallback to empty string :)

ArthurSens

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentgitpod-io/gitpod

[tracing] Add version to all traces

 export class TracingManager {             },             serviceName         }+        let version = process.env.VERSION;+        if (!version) {+            version = ''+        }         const t = initTracerFromEnv(config, {-            logger: console+            logger: console,+            tags: {+                'service.version': version

I'd change this to 'service.version': process.env.VERSION and get rid of the version variable - I only introduced the variable in my POC to debug if the VERSION environment variable was set at all :)

ArthurSens

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentgitpod-io/gitpod

[tracing] Add version to all traces

/werft run with-observability

ArthurSens

comment created time in 4 days

Pull request review commentgitpod-io/gitpod

Made tracing optional for preview environments

 env: {{- $gp := .gp -}} {{- $comp := .comp -}} {{- $tracing := $comp.tracing | default $gp.tracing -}}-{{- if $tracing }}

I think this might be a breaking change. I believe that all users of the Helm chart that previous has tracing enabled by setting tracing.endpoint would now have to update their values files to also have tracing.enabled

If so I would rather no go down this route and suggested in https://github.com/gitpod-io/ops/issues/334 and just use additional values files for preview environments.

wulfthimm

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentgitpod-io/gitpod

[tracing] Add version to all traces

@ArthurSens Let me know if you're blocked ☺️

ArthurSens

comment created time in 4 days

Pull request review commentgitpod-io/gitpod

[server] Add generic tracing for API calls

 export namespace TraceContext {             "error": err.message,             "stacktrace": err.stack         })-        ctx.span.setTag("error", true)+        ctx.span.setTag("error", true);+    }++    export function logJsonRPCError(ctx: TraceContext, method: string, err: ResponseError<any>) {+        if (!ctx.span) {+            return;+        }+        logError(ctx, err);++        // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/rpc.md#json-rpc+        ctx.span.addTags({+            rpc: {+                system: "jsonrpc",+                method,+                jsonrpc: {+                    error_code: err.code,+                    error_message: err.message,+                },+            },+        });+    }++    export function addJsonRPCParameters(ctx: TraceContext, method: string, args: any[]) {+        if (!ctx.span) {+            return;+        }++        ctx.span.addTags({+            rpc: {

@geropl Looks like it isn't well defined in the JS SDK for opentracing how to deal with the values (see docs) and it decided to deal with it poorly (see example trace) and just serialise the value for the rpc key as a string 😅

So either we have to change this to be multiple calls to addTag

ctx.span.addTag('rpc.system', jsonrpc')
ctx.span.addTag('rpc.method',method)
// and so on

Or create a helper function like the one below (it would need cleaning and tests of course)

function addNestedTags(tags: any, namespace?: string) {
  namespace = namespace ? `${namespace}.` : ''
  Object.entries(tags).forEach((arr) => {
    const [k,v] = arr
    // Arrays are objects, this is fine.
    if (typeof v == 'object') {
      addNestedTags(v, `${namespace}${k}`)
    }
    else {
      // delegate to span.addTag and let it deal with the rest of the cases ()
      console.log(`${namespace}${k} = ${v}`)
    }
  })
}

addNestedTags({
  rpc: {
    system: 'jsonrpc',
    method: 'foobar',
    parameters: ['test', {
      'more': 42
    }]
  }
})

// rpc.system = jsonrpc
// rpc.method = foobar
// rpc.parameters.0 = test
// rpc.parameters.1.more = 42
geropl

comment created time in 5 days

PullRequestReviewEvent
more