profile
viewpoint
Eliza Weisman hawkw @BuoyantIO @linkerd SF/Oakland elizas.website systems witch

an-cabal/an-rope 11

an rope data structure

BuoyantIO/flossy 6

a tool for testing standard compliance of HTTP proxies

ArcticLight/JRPNMachine 2

Java Reverse Polish Notation Machine

ArcticLight/ScalaTween 2

A functional animations library built for Scala

hawkw/AMWA 1

Atmospheric Monitoring with Arduino

hawkw/autodeck 1

Elgato Stream Deck based automation daemon

hawkw/3_clor 0

A quick PlatformIO library for controlling RGB LEDs on Arduino

hawkw/5mof 0

Boilerplate project for getting started with Spectacle Core

startedmmastrac/stylus

started time in a day

push eventtokio-rs/tracing

Eliza Weisman

commit sha 85995a5b5363dd0fe1830514b3e621bf137098e2

whoops it needs to be public for it to work Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in a day

PR opened tokio-rs/tracing

[DRAFT] make `Span::in_scope` work with async blocks as well as closures

This isn't quite done yet, and would need to wait until tracing 0.2 to release, but here's a rough sketch of a version of Span::in_scope that works transparently with async blocks or closures.

+81 -3

0 comment

1 changed file

pr created time in a day

push eventtokio-rs/tracing

Eliza Weisman

commit sha 763a2963be5d304c65bab49488a7b81ff54c2fac

WIP draft of a more generic `in_scope` Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in a day

create barnchtokio-rs/tracing

branch : eliza/async-in-scope

created branch time in a day

issue openedtokio-rs/tracing

opentelemetry: support `tracing-log` special values

Feature Request

Crates

tracing-opentelemetry

Motivation

Currently, when converting log::Records to tracing::Events using tracing-log, data from log's metadata is converted to special fields prefixed with log., rather than being recorded as tracing metadata. In order to properly display this data from log, the tracing-log crate provides the NormalizeMetadata trait. tracing-opentelemetry does not currently use this, however, and location data from log events is treated as a regular field rather than a source code location in OpenTelemetry logs.

For example, note the difference between the logs from tower-buffer (generated from native tracing events) and the logs from h2 (converted from log records via tracing-log): image

Proposal

We should update tracing-opentelemetry's on_event implementation to do something similar to what tracing-subscriber::fmt does here: https://github.com/tokio-rs/tracing/blob/954b1b8ab8b7fc577763aac0237c86b10313ae14/tracing-subscriber/src/fmt/format/mod.rs#L376-L381

We may also want to feature-flag the tracing-log dependency so that users can opt out.

created time in a day

Pull request review commenttokio-rs/tracing

opentelemetry: Add dynamic span name field

 impl<'a> field::Visit for SpanAttributeVisitor<'a> {     ///     /// [`Span`]: https://docs.rs/opentelemetry/latest/opentelemetry/api/trace/span/trait.Span.html     fn record_debug(&mut self, field: &field::Field, value: &dyn fmt::Debug) {-        let attribute = api::Key::new(field.name()).string(format!("{:?}", value));-        if let Some(attributes) = &mut self.0.attributes {-            attributes.push(attribute);+        if field.name() == SPAN_NAME_FIELD {+            self.0.name = format!("{:?}", value);         } else {

It looks like SpanNameVisitor handles this case in both record_debug and record_str, while here, we only handle it in record_debug? That means that if a string value is used as the name in a call to Span::record after the span is created, we will format it using fmt::Debug. The Debug formatter for strings includes quotes and escapes special characters, which may not be the right behavior for the span name?

jtescher

comment created time in a day

Pull request review commenttokio-rs/tracing

opentelemetry: Add dynamic span name field

 where         let span = ctx.span(id).expect("Span not found, this is a bug");         let mut extensions = span.extensions_mut(); -        let mut builder = self-            .tracer-            .span_builder(attrs.metadata().name())-            .with_start_time(SystemTime::now())-            // Eagerly assign span id so children have stable parent id-            .with_span_id(self.id_generator.new_span_id());+        let mut span_name = None;+        attrs.record(&mut SpanNameVisitor(&mut span_name));

Instead of visiting all the fields once to find the name override, and then again to record the otel attributes, would it make more sense to just visit everything in one pass? Could we construct the builder with the name from the metadata, and then, if we find an otel.name field while recording the fields, replace it with that value?

I suppose the downside to this would be that we might allocate an additional string that we end up throwing away, but we wouldn't need to iterate over the attributes twice. I don't know off the top of my head which is more efficient...

jtescher

comment created time in a day

Pull request review commenttokio-rs/tracing

opentelemetry: Add dynamic span name field

 //! [OpenTelemetry]: https://opentelemetry.io //! [`tracing`]: https://github.com/tokio-rs/tracing //!+//! ### Special Fields+//!+//! These fields have specific meaning for `tracing-opentelemetry` and will be+//! treated as ordinary fields by other layers:

Should we maybe also note something about reserving the otel. prefix for special fields for OpenTelemetry?

jtescher

comment created time in a day

Pull request review commenttokio-rs/tracing

opentelemetry: Add dynamic span name field

 //!     let root = span!(tracing::Level::TRACE, "app_start", work_units = 2); //!     let _enter = root.enter(); //!+//!     // Optionally set a dynamic span name for otel to export+//!     let operation = "GET http://example.com".to_string();+//!     span!(tracing::Level::TRACE, "http_request", otel.name=operation.as_str());

Hmm, that makes sense. I'm fine with leaving it as-is, then. Or, we could add multiple examples with string constants, string slices, etc?

jtescher

comment created time in a day

push eventtokio-rs/tracing

Luca Palmieri

commit sha 954b1b8ab8b7fc577763aac0237c86b10313ae14

docs: clarify `record` behaviour for new fields (#731) ## Motivation When working with `tracing` I was caught by surprise when trying to record a new field on a span to later find out that the field was completely ignored. ## Solution Add a more explicit reference to this behaviour in the documentation of `Span::record` as well as an example.

view details

push time in a day

PR merged tokio-rs/tracing

[Documentation] Clarify `record` behaviour for new fields

Motivation

When working with tracing I was caught by surprise when trying to record a new field on a span to later find out that the field was completely ignored.

Solution

Add a more explicit reference to this behaviour in the documentation of Span::record as well as an example.

+26 -0

0 comment

1 changed file

LukeMathWalker

pr closed time in a day

CommitCommentEvent

push eventtokio-rs/tracing

Eliza Weisman

commit sha d24a43aaee201bcddbaf1cd8bf1d1cff72f2de12

attributes: support arbitrary field expressions (#672) ## Motivation Fields on spans generated by `#[instrument]` consist of the parameters to the `instrument`ed function, and a list of optional fields with literal values passed in the `fields` argument of the attribute. The current API for additional user-defined fields is pretty limited, since it only accepts literals (and doesn't accept dotted field names). It would be nice to have an API for including additional fields with values set by calling methods on `self` or on parameters, or by accessing values from parameters. ## Solution This branch extends the currently supported `fields` argument to allow providing arbitrary expressions which are invoked within the function's scope. This is a superset of the previous support for literal values only. In addition, field names passed to `fields` may now contain dotted Rust identifiers. Implementing this required rewriting how arguments to the `#[instrument]` attribute are parsed. Previously, we used `syn`'s `AttributeArgs` type, which parses a comma-separated of "nested meta" items, consisting of `$ident($nested meta)`, `$ident = $nested_meta`, paths, and literals. By replacing the use of `AttributeArgs` with our own types implementing `syn::parse::Parse`, we can accept more expressions in the attribute arguments position. This also lets us reject more invalid inputs at parse-time, which improves syntax error reporting a bit. One thing that's worth noting is that the current implementation will generate an _empty_ field when a field name is provided without an `=` and a value. This makes sense when only simple fields with literal values are permitted. However, if we accept arbitrary expressions in the macro, this is not the ideal behavior --- we would prefer to use the same local variable shorthand as the function-like `tracing` macros. However, changing this now is a breaking change. Any code which uses a name that doesn't exist in the current scope to declare an empty field would fail to compile, because it attempts to reference a name that doesn't exist. Instead, I left a comment noting that this is not the ideal behavior and it should be changed next time we're ready to break the proc macros. Fixes: #650 Signed-off-by: Eliza Weisman <eliza@buoyant.io> Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 2 days

PR merged tokio-rs/tracing

Reviewers
attributes: support arbitrary field expressions crate/attributes kind/feature

Motivation

Fields on spans generated by #[instrument] consist of the parameters to the instrumented function, and a list of optional fields with literal values passed in the fields argument of the attribute. The current API for additional user-defined fields is pretty limited, since it only accepts literals (and doesn't accept dotted field names). It would be nice to have an API for including additional fields with values set by calling methods on self or on parameters, or by accessing values from parameters.

Solution

This branch extends the currently supported fields argument to allow providing arbitrary expressions which are invoked within the function's scope. This is a superset of the previous support for literal values only. In addition, field names passed to fields may now contain dotted Rust identifiers.

Implementing this required rewriting how arguments to the #[instrument] attribute are parsed. Previously, we used syn's AttributeArgs type, which parses a comma-separated of "nested meta" items, consisting of $ident($nested meta), $ident = $nested_meta, paths, and literals. By replacing the use of AttributeArgs with our own types implementing syn::parse::Parse, we can accept more expressions in the attribute arguments position. This also lets us reject more invalid inputs at parse-time, which improves syntax error reporting a bit.

One thing that's worth noting is that the current implementation will generate an empty field when a field name is provided without an = and a value. This makes sense when only simple fields with literal values are permitted. However, if we accept arbitrary expressions in the macro, this is not the ideal behavior --- we would prefer to use the same local variable shorthand as the function-like tracing macros. However, changing this now is a breaking change. Any code which uses a name that doesn't exist in the current scope to declare an empty field would fail to compile, because it attempts to reference a name that doesn't exist. Instead, I left a comment noting that this is not the ideal behavior and it should be changed next time we're ready to break the proc macros.

Fixes: #650

Signed-off-by: Eliza Weisman eliza@buoyant.io

+441 -268

3 comments

4 changed files

hawkw

pr closed time in 2 days

issue closedtokio-rs/tracing

instrument macro: provide way to include additional fields

It would be nice to have an API for including additional fields with values set by calling methods on self.

For example (making up an API that probably isn't possible)

struct Foo {
    conn: TcpStream,
}

impl Foo {
    #[instrument(fields(local_addr = self.local_addr))]
    fn run(&self) {
    }

    fn local_addr(&self) -> SocketAddr {
        self.conn.local_addr()
    }
}

closed time in 2 days

carllerche

Pull request review commenttokio-rs/tracing

opentelemetry: Add dynamic span name field

 //!     let root = span!(tracing::Level::TRACE, "app_start", work_units = 2); //!     let _enter = root.enter(); //!+//!     // Optionally set a dynamic span name for otel to export+//!     let operation = "GET http://example.com".to_string();+//!     span!(tracing::Level::TRACE, "http_request", otel.name=operation.as_str());

nit: i think the to_string/as_str can be removed?

//!     let operation = "GET http://example.com";
//!     span!(tracing::Level::TRACE, "http_request", otel.name = operation);
jtescher

comment created time in 2 days

Pull request review commenttokio-rs/tracing

opentelemetry: Add dynamic span name field

 //!     let root = span!(tracing::Level::TRACE, "app_start", work_units = 2); //!     let _enter = root.enter(); //!+//!     // Optionally set a dynamic span name for otel to export+//!     let operation = "GET http://example.com".to_string();+//!     span!(tracing::Level::TRACE, "http_request", otel.name=operation.as_str());

If we expect there to ever be other fields with special meaning to OpenTelemetry namespaced under otel., should we consider having a master list in the docs, as well?

jtescher

comment created time in 2 days

pull request commenttokio-rs/tracing

opentelemetry: Add dynamic span name field

CI failure is unrelated...we really need to fix the flaky appender tests.

jtescher

comment created time in 2 days

push eventtokio-rs/tracing

Luca Palmieri

commit sha d090db7c79de27fce49bef68f9b1eddded20c4a5

docs: add tracing-bunyan-formatter to the list of related crates. (#730) Adding the crate to the list of community-related crates for `tracing`. Happy to provide additional details if needed. * Add tracing-bunyan-formatter to the list of related crates. * Add tracing-bunyan-formatter to the module documentation in tracing/lib.rs

view details

push time in 2 days

PR merged tokio-rs/tracing

Add tracing-bunyan-formatter to the list of related crates.

Adding the crate to the list of community-related crates for tracing. Happy to provide additional details if needed.

+7 -1

1 comment

2 changed files

LukeMathWalker

pr closed time in 2 days

Pull request review commenttokio-rs/tracing

[Documentation] Clarify `record` behaviour for new fields

 impl Span {     ///     }     /// }     /// ```+    ///+    /// Beware though!+    /// The fields associated with a span are part of its [`Metadata`].+    /// The [`Metadata`] describing a particular span is constructed statically when the span is+    /// created and cannot be extended later to add new fields.+    /// Hence you cannot record a value for a field that was not specified at the moment+    /// of span creation:+    /// ```+    /// use tracing::{trace_span, field};+    ///+    /// // Create a span with two fields: `greeting`, with the value "hello world", and+    /// // `parting`, without a value.+    /// let span = trace_span!("my_span", greeting = "hello world", parting = field::Empty);+    ///+    /// // ...+    ///+    /// // Now, you try to record a value for a new field, `new_field`, which was not+    /// // declared as `Empty` or populated when you created `span`.+    /// // You won't get any error, but the assignment will have no effect!+    /// span.record("new_field", &"interesting_value_you_really_need");+    /// ```

I think a little repetition is just fine.

LukeMathWalker

comment created time in 2 days

pull request commenttokio-rs/tracing

attributes: support arbitrary field expressions

Okay, merging #711 into this was a bit of a mess, but I've got all the tests passing again so it should be good.

@davidbarsky, do you want to give it another look? I did change a few things while merging.

hawkw

comment created time in 2 days

push eventtokio-rs/tracing

Eliza Weisman

commit sha bcb7326e59c3efb91af1624df32ed21fca9edc90

chore: remove outdated example (#671) This example is outdated and should have been removed when `tracing-fmt` was merged into `tracing-subscriber`. There is another example in the `examples` crate, [`fmt-custom-fields`][1], which demonstrates the same usage, but uses the current APIs. Therefore, we can just delete the old example. [1]: https://github.com/tokio-rs/tracing/blob/master/examples/examples/fmt-custom-field.rs Fixes: #444 Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Zeki Sherif

commit sha 6540c0557f9959509eb0c8904d8ad99d3861516c

appender: introduce a non blocking file appender (#673) ## Motivation A service we manage at work is currently migrating from using log4rs to Tracing as it provides many features we feel will greatly help our ability to investigate and resolve issues that may arise. Our services generally log to a file stored on disk and are rotated hourly. Tracing today doesn't provide us out of the box support for logging to a file and we've had to work around this by implementing it ourself. ## Solution Our solution was to create a `FileWriter` which holds a reference to a crossbeam channel (sender side) . This `FileWriter` is what we pass to `.make_writer` on a subscriber. Tracing events will be sent through this channel and on the receiving end a worker thread will be responsible for logging the event to disk. This PR creates a new `tracing-appender` crate, which contains a `rolling` type that implements the rolling log file behavior, and a `non_blocking` type, which wraps a type implementing `Write` and spawns a worker thread responsible for writing to that writer. Other threads can log to the writer using a `NonBlocking` type, which implements `MakeWriter`, as described above. These types can be used independently of eachother, or composed together to create a non-blocking rolling file appender. ## Notes No tests are being included in this PR as from my understanding after talking with @davidbarsky is that we will be looking primarily for an interface here and any tests that will be added here will be different than what we have written internally. Co-authored-by: Zeki Sherif <zekshi@amazon.com> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

view details

Sam Schlegel

commit sha dac59b2a2d760cb850b1163de4e3218263a513c9

subscriber: remove trailing space in ChronoUtc format_time (#677) This removes the trailing space from the end of the ChronoUtc FormatTime impl. I'm pretty sure this is accidental? None of the other FormatTime impls have this there. Not sure if this is considered a breaking change. Hopefully not.

view details

Adam Perry

commit sha aeca93883ffcc482e9b8f26251877092f3c66290

subscriber: Bump sharded-slab dependency

view details

Jane Lusby

commit sha 280b664034fe18fc38accb96d6c4391bac81dc20

Add `tracing-flame` crate for generating flamegraphs/flamecharts (#631)

view details

David Barsky

commit sha d531f2a6bf573f2fa305c87b266319e7577cfd00

futures: prepare to release 0.2.4 (#683) ### Fixed - docs.rs build failures (#618) - Spelling in documentation skins -> sinks (#643) Co-authored-by: Eliza Weisman <eliza@buoyant.io>

view details

David Barsky

commit sha dfa92ac225b60daf679b775f89ab3ae907770b03

subscriber: prepare 0.2.5 (#684) subscriber: prepare 0.2.5 (#684) ### Changed - **fmt**: Bump sharded-slab dependency (#679) ### Fixed - **fmt**: remove trailing space in `ChronoUtc` `format_time` (#677)

view details

Bastian Köcher

commit sha a2e2a9452931930a6068650545f956dbae12714a

Add `must_use` attribute to `set_default` (#685) This makes the user aware of if the guard is dropped immediately and thus, the tracing does not work.

view details

Zeki Sherif

commit sha 0c01c0d3f527d131dce1c0f39f69bc71df34ea37

appender: add documentation to tracing appender crate (#676) ## Motivation Adds documentation for tracing-appender that was added in PR #673 Co-authored-by: Zeki Sherif <zekshi@amazon.com> Co-authored-by: Eliza Weisman <eliza@buoyant.io> Co-authored-by: David Barsky <me@davidbarsky.com>

view details

Eliza Weisman

commit sha fbcd27e9bf05034e1346096927530743f4770937

chore: remove redundant imports (#694) ## Motivation It looks like Rust 1.43's version of clippy added a new lint for redundant `use` statements with just a crate name. These are unnecessary because the root module of an external crate is always available without imports. Apparently, `tracing-log`, `tracing-subscriber`, `tracing`, and `tracing-tower` all had some redundant `use` statements, which clippy now warns us about. ## Solution This PR removes them. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Bastian Köcher

commit sha 0ef28d131c751f367fba559503da7c812faacc78

core: Add `must_use` to `dispatcher::set_default` (#686) Follow up of: #685

view details

Eliza Weisman

commit sha df8530d094a1fa34018524e17a5688ae2e0453f4

chore: Add preliminary CODEOWNERS file (#699) This branch adds a CODEOWNERS file to configure Github to automatically assign reviewers for pull requests. In general, when a crate or large subsystem of a crate was contributed by an individual person, we should use this file to ensure that that person is assigned to review future changes to that code. The @tokio-rs/tracing team should still be tagged to approve all pull requests, in addition to the person who originally contributed the code, so the CODEOWNERS file ensures this as well. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Zeki Sherif

commit sha 34fb9104b269ed99d470920cabc68fe420104766

appender: add top level docs (#692) Closes #687 Co-authored-by: Zeki Sherif <zekshi@amazon.com> Co-authored-by: Eliza Weisman <eliza@buoyant.io> Co-authored-by: David Barsky <me@davidbarsky.com>

view details

Zeki Sherif

commit sha f93ba2ee26bd7f66757cab0287e716abf06d3086

Appender: Use channel to signal shutdown of worker thread (#701) ## Motivation Fixes a race condition which occurs on dropping of `WorkerGuard`. If the worker thread missed seeing the shutdown signal in time, it would end up blocking on trying to call `recv()` on the crossbeam channel and block indefinitely. This bug was identified in #678 ## Solution Signal that the worker should stop by sending a `ShutDown` message through the crossbeam channel. Co-authored-by: Zeki Sherif <zekshi@amazon.com>

view details

Julian Tescher

commit sha 5a971befacc233612ec6577c9bae6188c79926ad

Import tracing-opentelemetry (#700) Import `tracing-opentelemetry` from https://github.com/jtescher/tracing-opentelemetry Co-authored-by: Eliza Weisman <eliza@buoyant.io>

view details

Inanna Malick

commit sha 89132933922d878d223b9d6c8c01f1da8eb576e6

docs: update related crates to include tracing-distributed (#702) update to add/fix links for my related crates per https://twitter.com/mycoliza/status/1255672523426951170

view details

Jane Lusby

commit sha 4e47811988c4d21090ee22e3cfd0bee44044657d

update readmes for flame and error (#695)

view details

Zeki Sherif

commit sha 750b031d0302f7703cf3aaea162643ee4067eb93

appender: add tests (#678) ## Motivation Adding tests for PR #673 Co-authored-by: Zeki Sherif <zekshi@amazon.com> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 361a8678c488b0747ef1414330a3a4fe9c3ad04a

core: add `fmt::Debug` impl to `dyn Value`s (#696) ## Motivation Currently, the only way to interact with `Value`s is to record them with a visitor. In the most common case, where typed data is not needed, `Value`s are recorded with their `fmt::Debug` implementations — a visitor which does not implement the `record_${TYPE}` traits will automatically fall back to recording all primitive value types with `Debug`. However, implementing a visitor requires a bit of boilerplate: an entire trait implementation has to be written, and a visitor object passed to `record`. ## Solution This branch hopes to simplify this common case by adding a `Debug` implementation to _all_ `dyn Value` trait objects. This is equivalent to a visitor that only implements `record_debug`, but requiring less boilerplate. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Zeki Sherif

commit sha f8b52bdf104399b810da65dc396ba29cef835ef0

appender: prepare tracing-appender for release (#705) Following the release process for https://github.com/tokio-rs/tracing/blob/master/CONTRIBUTING.md and these changes are the only things left to commit before release. I'd like to hopefully get this released today so we can start using this crate internally. Note: I need to get PR #703 and PR #678 merged before release. Co-authored-by: Zeki Sherif <zekshi@amazon.com> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 2 days

push eventhawkw/dotfiles

Eliza Weisman

commit sha 3740ff06dcead1d97643b7b2ad5753cd10756658

hello nix Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha afef64e277bd2581c8c493ffc2992c30a37f8c84

more nix Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 061d6c790b3efee05286aa4f8e86ef7858af2c67

manage starship with nix! Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha ac8f0b9793183f10024395d539ab0c3d6bd3bee8

put nix stuff in a place where everything will soure it Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 2 days

Pull request review commentlinkerd/linkerd2-proxy

[WIP] Add json log format as an option.

 pub fn init_log_compat() -> Result<(), Error> {     tracing_log::LogTracer::init().map_err(Error::from) } -pub fn with_filter(filter: impl AsRef<str>) -> (Dispatch, LevelHandle) {+pub fn with_filter_and_format(+    filter: impl AsRef<str>,+    format: impl AsRef<str>,+) -> (Dispatch, LevelHandle) {     let filter = filter.as_ref();+    let format = format.as_ref();      // Set up the subscriber     let start_time = clock::now();-    let builder = FmtSubscriber::builder()-        .with_timer(Uptime { start_time })-        .with_env_filter(filter)-        .with_filter_reloading()-        .with_ansi(cfg!(test));-    let handle = LevelHandle {-        inner: builder.reload_handle(),-    };-    let dispatch = Dispatch::new(builder.finish());--    (dispatch, handle)++    match format {+        "JSON" => {+            let builder = FmtSubscriber::builder()+                .json()

Oh, I think with_filter_reloading needs to be called after json, because json changes the type of the formatter. So, you could still factor out the with_timer and with_filter calls, if you like (but it's not a big deal regardless).

naseemkullah

comment created time in 2 days

push eventtokio-rs/tracing

Jane Lusby

commit sha 02b7c4c028f9cf0edbf0904a4643b1562ca7145e

docs: add tracing-error, tracing-flame, and eyre to related crates (#728) * Add related crates for tracing-error * boop * boop

view details

push time in 2 days

PR merged tokio-rs/tracing

docs: add tracing-error to related crates
+30 -0

1 comment

2 changed files

yaahc

pr closed time in 2 days

issue commenttokio-rs/tracing

Document how to log synchronously to a `RollingFileAppender`

Unfortunately, this doesn't work yet: MakeWriter implementations are expected to return a writer, while the rolling file appender is a writer.

Since a single MakeWriter is used by all threads in the program, the MakeWriter::make_writer method takes &self; however, writing to a Write implementation takes &mut self. This means there would need to be a locking strategy to coordinate access. Ideally, we should have an implementation of MakeWriter for Mutex<T> where T: Write, but this isn't currently possible due to a limitation of the MakeWriter trait (see #675). This limitation is easy to fix but is unfortunately a breaking change — it will be fixed in tracing-subscriber 0.3.

sourcefrog

comment created time in 2 days

issue commenttokio-rs/tracing

tracing-subscriber Filtering Confusion

Continued:

Intuition behind directives and filtering | Names of fields between spans matter

I don't have a clear mental model for how directives are applied to filter spans. Specifically for field directives: I thought it worked as follows: "The directive is checked against a span to decide whether to enter that span or not".

The following examples show my confusion:

> RUST_LOG=[{id}] cargo run
  INFO Thread{id=Id<0>}: hello from logger
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Creating Chan
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Sending Message!
  INFO Child Thread{id=Id<1>}:Channel{id=Id<3>}: Sending Message!

This makes sense based on my filter since all spans have an id field.

> RUST_LOG=[{id=Id<3>}] cargo run
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Creating Chan
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Sending Message!
  INFO Child Thread{id=Id<1>}:Channel{id=Id<3>}: Sending Message!

This is a bit surprising as the Thread span id field has the wrong value (!= Id<3>) so I'm surprised it is entered (Note: this is actually the behaviour I want though since I want to see the "context", that is, all the parent spans).

But for some reason this seems to happen only because both fields are named id if I change the name of the field for the Channel from id to cid:

- span!(Level::INFO, "Channel", id=?Id::new(3)).in_scope(||{
+ span!(Level::INFO, "Channel", cid=?Id::new(3)).in_scope(||{

This no longer follow my intuition:

> RUST_LOG=[{cid=Id<3>}] cargo run
  INFO Channel{cid=Id<3>}: Creating Chan
  INFO Channel{cid=Id<3>}: Sending Message!
  INFO Channel{cid=Id<3>}: Sending Message!

In this example all context is lost. Which is a reasonable implementation by itself. But given the example above where the context was kept since both fields happened to be named id seems wrong.

> RUST_LOG=[{cid}] cargo run
  INFO Thread{id=Id<0>}:Channel{cid=Id<3>}: Creating Chan
  INFO Thread{id=Id<0>}:Channel{cid=Id<3>}: Sending Message!
  INFO Child Thread{id=Id<1>}:Channel{cid=Id<3>}: Sending Message!

Similarly why is the context kept in this case but not the one above?

This is kind of a consequence of an interaction between a few of the more counterintuitive aspects of how span filtering works.

The filter's enabled method is called before a span is recorded; this is to ensure that we don't have to record the field values for spans that are disabled, which might be costly to evaluate. However, this means that when filtering on the values of a span's fields, the EnvFilter has to enable all spans which have the field name it cares about, regardless of their values. The span itself will be enabled so that the field's value can be recorded, and then, when the value is recorded, it's matched against the filter. If the value matches, the span will then enable all of its children, otherwise, they will only be enabled if another filter enables them. This is why the Thread span with Id(0) is enabled even though the filter expects Id(3); if the Channel span inside that span did not have Id(3), none of those events would have been enabled. On the other hand, when you change the field name to cid, none of the Thread spans are enabled, since they don't have a field by that name.

Does that make sense? I realize this behavior is somewhat surprising, and while I don't think there's a lot we can do to change the behavior here, it looks like the docs don't really explain this — that, at least, I think we can fix! :)

Using primitives in Debug Instances

I can happily filter based on my Id:

> RUST_LOG=[{id=Id<3>}] cargo run
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Creating Chan
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Sending Message!
  INFO Child Thread{id=Id<1>}:Channel{id=Id<3>}: Sending Message!

However if I change my Debug impl for ID:

-        write!(fmt, "Id<{}>", self.id)
+        write!(fmt, "{}", self.id)

So that now it looks like:

> RUST_LOG=trace cargo run
  INFO Thread{id=0}: hello from logger
  INFO Thread{id=0}:Channel{id=3}: Creating Chan
  INFO Thread{id=0}:Channel{id=3}: Sending Message!
  INFO Created child thread
  INFO Child Thread{id=1}:Channel{id=3}: Sending Message!

Looks good... but then filtering fails:

gatowololo@thinkpad ~/R/tracing_example (master)> RUST_LOG=[{id=3}] cargo run
# No output

It seems we cannot have our Debug instances "look like" primitive types? I know tracking-subscriber treats certain numeric types, books, and strings special.

Ah, yup, this is because the filtering DSL interprets those values as u64 primtives, not as fmt::Debug values, which creates a filter that only matches primitive values:

https://github.com/tokio-rs/tracing/blob/18764d149ff4c47ef875af3d7ec881d5f610533b/tracing-subscriber/src/filter/env/field.rs#L40-L45

Since the values are recorded using fmt::Debug, they go directly to the record_debug implementation: https://github.com/tokio-rs/tracing/blob/18764d149ff4c47ef875af3d7ec881d5f610533b/tracing-subscriber/src/filter/env/field.rs#L315-L321 and therefore don't match the filter, even though they probably should.

This behavior is...not great. I think we can fix it by not special-casing primitives here, and just performing all field matching with debug patterns in record_debug. If we removed the record_u64, record_bool, etc implementations from MatchVisitor, everything would be recorded with record_debug, and that pattern would match the u64 value 3 or a value whose debug implementation produces the string "3".

This should, at least, be pretty simple to fix.

> #### Field names cannot be reserved words?
> 
> Again I can happily filter by the id field:
> 
> ```
> > RUST_LOG=[{id}] cargo run
>   INFO Thread{id=Id<0>}: hello from logger
>   INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Creating Chan
>   INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Sending Message!
>   INFO Child Thread{id=Id<1>}:Channel{id=Id<3>}: Sending Message!
> ```
> 
> Looks good. But if my field is called type:
> 
> ```diff
> -        span!(Level::INFO, "Channel", id=?Id::new(3)).in_scope(||{
> +        span!(Level::INFO, "Channel", r#type=?Id::new(3)).in_scope(||{
> ```
> 
> Looks like:
> 
> ```
> > RUST_LOG=[] cargo run
>   INFO Thread{id=Id<0>}: hello from logger
>   INFO Thread{id=Id<0>}:Channel{type=Id<3>}: Creating Chan
>   INFO Thread{id=Id<0>}:Channel{type=Id<3>}: Sending Message!
>   INFO Created child thread
>   INFO Child Thread{id=Id<1>}:Channel{type=Id<3>}: Sending Message!
> ```
> 
> I don't seem to be able to filter by the field name anymore:
> 
> ```
> > RUST_LOG=[{type}] cargo run
> # no output
> ```
>

So, when you create a field named `r#type`, the field name string that's generated is `r#type`, _not_ `type`. `tracing-subscriber::fmt` special-cases field names beginning with `r#` when _formatting_:
https://github.com/tokio-rs/tracing/blob/18764d149ff4c47ef875af3d7ec881d5f610533b/tracing-subscriber/src/fmt/format/mod.rs#L548
and outputs `type`.

However, `EnvFilter` doesn't currently do this. If you had written `RUST_LOG=[{r#type}]`, it would have worked. 

We should make this behavior more consistent, by changing `EnvFilter` to also ignore a leading `r#`. That, too, shouldn't be too difficult.

> ### Conclusion
> 
> I realize this is a lot of questions for one issue! Sorry! If anyone could confirm any of the observations/bugs above I'd be happy to dig deeper. I really want filtering for my project. I think the idea is extremely powerful. Thank you for making `tracing`!

That's quite alright, thanks for opening the issue! I think you are probably trying to use a lot more of the advanced filtering functionality than most people tend to, and so you've hit a bunch of edge cases. Sorry about that — the filtering code is actually one of the most complex part of `tracing`, and it really hasn't gotten the attention it  deserves. Fortunately, I think we can fix a lot of the weird or incorrect behavior you've hit, and improve the documentation for some of the things that are confusing.

Are you interested in helping out with this? If so, please let me know! I'd be happy to provide additional guidance on how to fix some of these problems. Otherwise, no pressure! I appreciate the work you've done in reporting these issues. Thanks!
gatoWololo

comment created time in 2 days

issue commenttokio-rs/tracing

tracing-subscriber Filtering Confusion

@gatoWololo Sorry I didn't see this issue earlier! Let me try and answer your questions:

Spans don't live through thread boundaries

Even though span Thread is in scope it seems spans don't live through thread boundaries as shown by the span-less INFO Created child thread is this correct?

That's correct: entering a span is per-thread; spawning a new thread does not automatically propagate the current span. This is stated in the docs here, but it might not be sufficiently clear?

Entering a span is per-thread by design, because in many cases each thread in a program is executing a separate logical unit of work, and it wouldn't make sense to propagate them by default. However, if you want a spawned thread to also enter a span, you can do this:

let span = tracing::info_span!(...);
let _enter = span.enter();

// ...

let span2 = span.clone();
let thread = std::thread::spawn(move || {
    let _enter = span2.enter();
    // ...
});

Or, if the span is not in the same scope as where the thread is spawned, you can propagate the current span to the spawned thread, like this:

let span = Span::current();
let thread = std::thread::spawn(move || {
    let _enter = span.enter();
    // ...
});

Env filter does not accept parentheses, braces, or curly braces as field values.

I can filter by id like:

> env RUST_LOG=[{id=Id<3>}] cargo run
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Creating Chan
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Sending Message!
  INFO Child Thread{id=Id<1>}:Channel{id=Id<3>}: Sending Message!

Great! But it seems I cannot use [] or () or {}:

-        write!(fmt, "Id<{}>", self.id)
+        write!(fmt, "Id({})", self.id)

Which looks like:

> RUST_LOG=trace cargo run
  INFO Thread{id=Id(0)>}: hello from logger
  INFO Thread{id=Id(0)>}:Channel{id=Id(3)>}: Creating Chan
  INFO Thread{id=Id(0)>}:Channel{id=Id(3)>}: Sending Message!
  INFO Created child thread
  INFO Child Thread{id=Id(1)>}:Channel{id=Id(3)>}: Sending Message!

This sorta makes sense since it clashes with regex syntax. Using parens seems to fail silently:

> env RUST_LOG="[{id=Id(3)}]" cargo run
# No output

This would be useful as Rust's auto derive for debug uses curly braces and I want to filter by small arrays which use square brackets.

If memory serves, you should be able to escape parens using \( and \)? The regex parser should interpret that as a comma character rather than as a group.

Not being able to use commas, square brackets, and curly brackets is a current limitation of the env-filter parser, since these are part of the filter syntax — we should definitely support them as well. The parser for the filtering syntax is currently quite simple, so it may require a bit of a rewrite. Square and curly brackets would also need to be escaped, as they also have a meaning in the regex syntax, but if we stopped interpreting them as part of the filter syntax when inside a field, that should just work.

This could be worth opening a separate feature-request issue, as I'd really like to get it working!

Span Ordering and Commas

I'm unsure what the difference is between the following filters. They all seem to produce some output:

> env RUST_LOG="[Thread Channel]" cargo run
  INFO Thread{id=Id<0>}: hello from logger
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Creating Chan
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Sending Message!
> env RUST_LOG="[Channel Thread]" cargo run
  INFO Channel{id=Id<3>}: Creating Chan
  INFO Channel{id=Id<3>}: Sending Message!
  INFO Channel{id=Id<3>}: Sending Message!
> env RUST_LOG="[Thread][Channel]" cargo run
  INFO Channel{id=Id<3>}: Creating Chan
  INFO Channel{id=Id<3>}: Sending Message!
  INFO Channel{id=Id<3>}: Sending Message!
> env RUST_LOG="[Thread],[Channel]" cargo run
  INFO Thread{id=Id<0>}: hello from logger
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Creating Chan
  INFO Thread{id=Id<0>}:Channel{id=Id<3>}: Sending Message!
  INFO Channel{id=Id<3>}: Sending Message!

It seems order matters when applying filters: RUST_LOG="[Thread Channel]" vs RUST_LOG="[Channel Thread]"? I assume it tries to apply the directives in order from left to rights?

I'm not sure what the difference is between: RUST_LOG="[Thread Channel]" vs env RUST_LOG="[Thread][Channel]"?

Adding the comma produces different results between: RUST_LOG="[Thread][Channel]" vs RUST_LOG="[Thread],[Channel]"

I'm not sure if some of this syntax is documented.

Directives in the filter syntax are separated by commas. So [Thread],[Channel] creates two separate directives, one which enables everything occurring in a span named Thread, and a separate one which enables everything occurring in a span named Channel. On the other hand, [Thread][Channel] creates a single directive. I'm actually surprised that that parses, since the filter parser was written with the expectation that there would only be a single [...] span filter per directive.

If that form was supported, I would expect that to create a filter that enables everything inside a span named Channel which is itself inside a span named Thread — it appears that it is not enabling the Thread span, which is not quite right. We should either fix this, or make that a parse error.

On the other hand, when there is a single set of brackets, like [Channel Thread] or [Thread Channel], that's interpreted as one span name pattern. I'm actually surprised that matches anything; it should only match a span whose name is "Channel Thread" or "Thread Channel" respectively: https://github.com/tokio-rs/tracing/blob/18764d149ff4c47ef875af3d7ec881d5f610533b/tracing-subscriber/src/filter/env/directive.rs#L154 It's possible that's a parser bug!

I'm going to continue replying to your questions in a separate comment; this one has gotten quite long. But, it seems like you've hit a lot of edge cases in the filter implementation — I think most folks don't use a lot of the more advanced filtering options, so I think nobody else has found some of these issues. Sorry about that!

gatoWololo

comment created time in 2 days

pull request commenttokio-rs/tracing

Add on_follows_from to OpenTelemetryLayer

Let's go ahead and have it create one-directional links, especially if that's what other OpenTelemetry implementations are doing.

bodymindarts

comment created time in 2 days

pull request commenttokio-rs/tracing

Add related crates for tracing-error

The test that keeps failing, tracing_appender::logs_dropped_if_lossy, unfortunately seems to be flaky. I restarted CI.

yaahc

comment created time in 2 days

Pull request review commenttokio-rs/tracing

[Documentation] Clarify `record` behaviour for new fields

 impl Span {     ///     }     /// }     /// ```+    ///+    /// Beware though!+    /// The fields associated with a span are part of its [`Metadata`].+    /// The [`Metadata`] describing a particular span is constructed statically when the span is+    /// created and cannot be extended later to add new fields.+    /// Hence you cannot record a value for a field that was not specified at the moment+    /// of span creation:

How about:

    /// Therefore, you cannot record a value for a field that was not specified when the span
    /// was created:
LukeMathWalker

comment created time in 2 days

Pull request review commenttokio-rs/tracing

[Documentation] Clarify `record` behaviour for new fields

 impl Span {     ///     }     /// }     /// ```+    ///+    /// Beware though!+    /// The fields associated with a span are part of its [`Metadata`].+    /// The [`Metadata`] describing a particular span is constructed statically when the span is+    /// created and cannot be extended later to add new fields.+    /// Hence you cannot record a value for a field that was not specified at the moment+    /// of span creation:+    /// ```+    /// use tracing::{trace_span, field};+    ///+    /// // Create a span with two fields: `greeting`, with the value "hello world", and+    /// // `parting`, without a value.+    /// let span = trace_span!("my_span", greeting = "hello world", parting = field::Empty);+    ///+    /// // ...+    ///+    /// // Now, you try to record a value for a new field, `new_field`, which was not+    /// // declared as `Empty` or populated when you created `span`.+    /// // You won't get any error, but the assignment will have no effect!+    /// span.record("new_field", &"interesting_value_you_really_need");+    /// ```

In general, with this kind of note, I like to conclude by re-stating what users should do instead to avoid problems, even though this is already explained in the docs before the note. So, I think it might be good to end with something like "Instead, all fields that may be recorded should be declared up front, using field::Empty when a value is not known".

What do you think?

LukeMathWalker

comment created time in 2 days

Pull request review commenttokio-rs/tracing

[Documentation] Clarify `record` behaviour for new fields

 impl Span {     ///     }     /// }     /// ```+    ///+    /// Beware though!+    /// The fields associated with a span are part of its [`Metadata`].

How about

    /// **Note**: The fields associated with a span are part of its [`Metadata`].
LukeMathWalker

comment created time in 2 days

Pull request review commentlinkerd/linkerd2-proxy

[WIP] Add json log format as an option.

 pub fn init_log_compat() -> Result<(), Error> {     tracing_log::LogTracer::init().map_err(Error::from) } -pub fn with_filter(filter: impl AsRef<str>) -> (Dispatch, LevelHandle) {+pub fn with_filter_and_format(+    filter: impl AsRef<str>,+    format: impl AsRef<str>,+) -> (Dispatch, LevelHandle) {     let filter = filter.as_ref();+    let format = format.as_ref();      // Set up the subscriber     let start_time = clock::now();-    let builder = FmtSubscriber::builder()-        .with_timer(Uptime { start_time })-        .with_env_filter(filter)-        .with_filter_reloading()-        .with_ansi(cfg!(test));-    let handle = LevelHandle {-        inner: builder.reload_handle(),-    };-    let dispatch = Dispatch::new(builder.finish());--    (dispatch, handle)++    match format {+        "JSON" => {+            let builder = FmtSubscriber::builder()+                .json()+                .with_timer(Uptime { start_time })+                .with_env_filter(filter)+                .with_filter_reloading()+                .with_ansi(cfg!(test));

IIRC the .with_ansi option doesn't actually do anything with the JSON formatter --- it doesn't support ANSI colors.

naseemkullah

comment created time in 2 days

Pull request review commentlinkerd/linkerd2-proxy

[WIP] Add json log format as an option.

 pub fn init_log_compat() -> Result<(), Error> {     tracing_log::LogTracer::init().map_err(Error::from) } -pub fn with_filter(filter: impl AsRef<str>) -> (Dispatch, LevelHandle) {+pub fn with_filter_and_format(+    filter: impl AsRef<str>,+    format: impl AsRef<str>,+) -> (Dispatch, LevelHandle) {     let filter = filter.as_ref();+    let format = format.as_ref();      // Set up the subscriber     let start_time = clock::now();-    let builder = FmtSubscriber::builder()-        .with_timer(Uptime { start_time })-        .with_env_filter(filter)-        .with_filter_reloading()-        .with_ansi(cfg!(test));-    let handle = LevelHandle {-        inner: builder.reload_handle(),-    };-    let dispatch = Dispatch::new(builder.finish());--    (dispatch, handle)++    match format {+        "JSON" => {+            let builder = FmtSubscriber::builder()+                .json()

I think that if we call the json method on the builder last, we can factor out most of the shared configuration:

let builder = FmtSubscriber::builder()
    .with_timer(Uptime { start_time })
    .with_filter(filter)
    .with_filter_reloading();

match format {
    "JSON" => {
        let builder = builder.json();
        // ...
    }
    // ...
}
naseemkullah

comment created time in 2 days

Pull request review commentlinkerd/linkerd2-proxy

[WIP] Add json log format as an option.

 pub fn init_log_compat() -> Result<(), Error> {     tracing_log::LogTracer::init().map_err(Error::from) } -pub fn with_filter(filter: impl AsRef<str>) -> (Dispatch, LevelHandle) {+pub fn with_filter_and_format(+    filter: impl AsRef<str>,+    format: impl AsRef<str>,+) -> (Dispatch, LevelHandle) {     let filter = filter.as_ref();+    let format = format.as_ref();      // Set up the subscriber     let start_time = clock::now();-    let builder = FmtSubscriber::builder()-        .with_timer(Uptime { start_time })-        .with_env_filter(filter)-        .with_filter_reloading()-        .with_ansi(cfg!(test));-    let handle = LevelHandle {-        inner: builder.reload_handle(),-    };-    let dispatch = Dispatch::new(builder.finish());--    (dispatch, handle)++    match format {+        "JSON" => {

I think we probably want these to be case-insensitive — "json" should probably not get the plain format.

naseemkullah

comment created time in 2 days

Pull request review commentlinkerd/linkerd2-proxy

[WIP] Add json log format as an option.

 use tracing_subscriber::{     reload, EnvFilter, FmtSubscriber, }; -const ENV_LOG: &str = "LINKERD2_PROXY_LOG";--type Subscriber = Formatter<format::DefaultFields, format::Format<format::Full, Uptime>>;+const ENV_LOG_LEVEL: &str = "LINKERD2_PROXY_LOG";+const ENV_LOG_FORMAT: &str = "LINKERD2_PROXY_LOG_FORMAT";  #[derive(Clone)]-pub struct LevelHandle {-    inner: reload::Handle<EnvFilter, Subscriber>,+pub enum LevelHandle {+    Json {+        inner: reload::Handle<+            EnvFilter,+            Formatter<format::JsonFields, format::Format<format::Json, Uptime>>,

Take it or leave it: I might consider adding JsonFormatter and PlainFormatter type aliases for these, because this is a bit wordy; not a blocker though.

naseemkullah

comment created time in 2 days

Pull request review commentlinkerd/linkerd2-proxy

[WIP] Add json log format as an option.

 use tracing_subscriber::{     reload, EnvFilter, FmtSubscriber, }; -const ENV_LOG: &str = "LINKERD2_PROXY_LOG";--type Subscriber = Formatter<format::DefaultFields, format::Format<format::Full, Uptime>>;+const ENV_LOG_LEVEL: &str = "LINKERD2_PROXY_LOG";+const ENV_LOG_FORMAT: &str = "LINKERD2_PROXY_LOG_FORMAT";

These should probably go in env.rs, where we define all the other environment variables.

naseemkullah

comment created time in 2 days

pull request commentlinkerd/linkerd2-proxy

[WIP] Add json log format as an option.

Hi @hawkw, there is probably room for stylistic improvement, refactoring and deduplicating so please let me know. But I think this is ready for a first review.

  • Could you please advise as to what would be the most straightforward way to test this locally? I guess I build and run the proxy standalone locally?

The proxy integration tests use the same tracing configuration as the proxy binary: https://github.com/linkerd/linkerd2-proxy/blob/cb161a13d504b963f10ed386d08352b17b4fe501/linkerd/app/integration/src/lib.rs#L40-L51 (note the call to app::core::trace::with_filter)

So, you should be able to run the integration tests with LINKERD2_PROXY_LOG=debug,LINKERD2_PROXY_LOG_FORMAT=json for a quick spot check. You can also run a proxy locally with cargo run, but you'll need to set all the required environment variables, so running the integration tests is probably quicker and easier.

  • Where would the fallback/default value of PLAIN go for LINKERD2_LOG_FORMAT?

If you look in linkerd/app/src/env.rs, you'll see that that module is where we define constants for all the environment variables that configure the proxy, and their default values.

  • Would --proxy-log-format CLI flag and config.linkerd.io/proxy-log-format annotation be implemented in the linkerd2 repo?

Yup, that's correct. I believe it would be necessary to add it to the proxy-injector control plane component, and to the CLI. Both of those live in the linkerd2 repo.

I'll take a closer look at the code changes here soon, but it's great that everything is working!

naseemkullah

comment created time in 2 days

Pull request review commentlinkerd/linkerd2-proxy

Update strip_header to std::future

 pub mod response {      impl<F, H, B> Future for ResponseFuture<F, H>     where-        F: Future<Item = http::Response<B>>,+        F: TryFuture<Ok = http::Response<B>>,         H: AsHeaderName + Clone,     {-        type Item = F::Item;-        type Error = F::Error;+        type Output = Result<F::Ok, F::Error>; -        fn poll(&mut self) -> Poll<Self::Item, Self::Error> {-            let mut res = try_ready!(self.inner.poll());-            res.headers_mut().remove(self.header.clone());-            Ok(res.into())+        fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {+            let this = self.project();+            let mut res = ready!(this.inner.try_poll(cx))?;+            res.headers_mut().remove(this.header.clone());+            Poll::Ready(Ok(res.into()))

The Into conversion here was to turn the inner value into an Async::Ready; it's no longer necessary and we should remove it

            Poll::Ready(Ok(res))
kleimkuhler

comment created time in 3 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 726d3d0eed06bb95a65c64d6f42811d20a7a272f

rustfmt all the things Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha e307121a2e73958568b7e475682695821bae5bea

use tonic without rustfmt, fixing CI Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 3075ebab8d6faddf4863cf2e29a454ca8efd8daf

fixup recover Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

push eventlinkerd/linkerd2-proxy-api

Eliza Weisman

commit sha c2dba642c7301b5b562c75fd6e2bd0a12ec763e8

turn rustfmt back off again it's nice, but the proxy docker build shouldn't depend on rustfmt Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

push eventlinkerd/linkerd2-proxy-api

Eliza Weisman

commit sha c9567c455c030111e7c8096cc8b1dfe7c962d703

turn rustfmt back off again it's nice, but the proxy docker build shouldn't depend on rustfmt Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

Pull request review commentlinkerd/linkerd2-proxy

turn outbound back into a proxy

 enum State<F, R: resolve::Resolution, B> {  // === impl Resolve === -impl<E, R> Resolve<E, R> {+impl<E, R> Resolve<E, R> +where+R: Clone,+E: Recover + Clone,+E::Backoff: Unpin,

agh, i thought i deleted this

hawkw

comment created time in 3 days

PR opened linkerd/linkerd2-proxy

update protocol upgrades to std::future

This branch updates the l5d-orig-proto upgrade/downgrade code to use std::future, and puts it back in the stack.

It turns out that the integration tests that exercise this don't actually assert that an upgrade is performed when the hint is sent, just that sending the hint doesn't break proxy-to-proxy traffic (which is a bummer, but it's hard to assert that in an integration test). So, those tests were passing before this branch.

However, I validated that the upgrade is performed by checking the logs:

[     0.73767491s] DEBUG proxy{test=main:proxy}:outbound:accept{peer.addr=127.0.0.1:60420}:source{target.addr=127.0.0.1:34913}:logical{addr=transparency.test.svc.cluster.local:80}:balance{addr=transparency.test.svc.cluster.local:80}:endpoint{peer.addr=127.0.0.1:34885}: linkerd2_proxy_transport::connect: Connected local.addr=127.0.0.1:50732 keepalive=None
[     0.73871743s] DEBUG proxy{test=main:proxy}:outbound:accept{peer.addr=127.0.0.1:60420}:source{target.addr=127.0.0.1:34913}:logical{addr=transparency.test.svc.cluster.local:80}:balance{addr=transparency.test.svc.cluster.local:80}:endpoint{peer.addr=127.0.0.1:34885}: linkerd2_proxy_transport::metrics: client connection open
[     0.74886384s] DEBUG proxy{test=main:proxy}:outbound:accept{peer.addr=127.0.0.1:60420}:source{target.addr=127.0.0.1:34913}:logical{addr=transparency.test.svc.cluster.local:80}:balance{addr=transparency.test.svc.cluster.local:80}: linkerd2_timeout::failfast: Recovered
[     0.75516939s] DEBUG proxy{test=main:proxy}:outbound:accept{peer.addr=127.0.0.1:60420}:source{target.addr=127.0.0.1:34913}:logical{addr=transparency.test.svc.cluster.local:80}:balance{addr=transparency.test.svc.cluster.local:80}:endpoint{peer.addr=127.0.0.1:34885}: linkerd2_proxy_http::orig_proto: Upgrading request to HTTP2 from HTTP/1.1
[     0.75671201s] DEBUG proxy{test=main:proxy}:outbound:accept{peer.addr=127.0.0.1:60420}:source{target.addr=127.0.0.1:34913}:logical{addr=transparency.test.svc.cluster.local:80}:balance{addr=transparency.test.svc.cluster.local:80}:endpoint{peer.addr=127.0.0.1:34885}: linkerd2_proxy_http::client: method=POST uri=/ version=HTTP/2.0 headers={"l5d-orig-proto": "HTTP/1.1", "host": "transparency.test.svc.cluster.local"}
[     0.77214679s] DEBUG proxy{test=main:proxy}:outbound:accept{peer.addr=127.0.0.1:60420}:source{target.addr=127.0.0.1:34913}:logical{addr=transparency.test.svc.cluster.local:80}:balance{addr=transparency.test.svc.cluster.local:80}: linkerd2_timeout::failfast: Recovered
[     0.143665402s] DEBUG proxy{test=main:proxy}:inbound:accept{peer.addr=127.0.0.1:50732}:source{target.addr=127.0.0.1:39975}:linkerd2_proxy_http::orig_proto: translating HTTP2 to orig-proto: "HTTP/1.1"

Depends on #526

+99 -86

0 comment

7 changed files

pr created time in 3 days

Pull request review commentlinkerd/linkerd2-proxy

Add loop detection to inbound & TCP forwarding

+use super::endpoint::{Target, TcpEndpoint};+use linkerd2_app_core::admit;++/// A connection policy that drops+#[derive(Copy, Clone, Debug)]+pub struct PreventLoop {+    port: u16,+}++#[derive(Copy, Clone, Debug)]+pub struct LoopPrevented {+    port: u16,+}++impl PreventLoop {+    pub fn new(port: u16) -> Self {+        Self { port }+    }+}++impl admit::Admit<Target> for PreventLoop {+    type Error = LoopPrevented;++    fn admit(&mut self, ep: &Target) -> Result<(), Self::Error> {+        tracing::debug!(addr = %ep.addr, self.port);+        if ep.addr.port() == self.port {+            return Err(LoopPrevented { port: self.port });+        }++        Ok(())+    }+}++impl admit::Admit<TcpEndpoint> for PreventLoop {+    type Error = LoopPrevented;++    fn admit(&mut self, ep: &TcpEndpoint) -> Result<(), Self::Error> {+        tracing::debug!(port = %ep.port, self.port);+        if ep.port == self.port {+            return Err(LoopPrevented { port: self.port });+        }++        Ok(())+    }+}

nit/tioli: it seems like there could be some kinda trait for "things that have a port" that we could use to generalize this? but, it's not repetitive enough for it to matter all that much, esp. since i can't immediately think of a way to generalize between the inbound and outbound logic...

olix0r

comment created time in 3 days

Pull request review commentlinkerd/linkerd2-proxy

Add loop detection to inbound & TCP forwarding

+use super::endpoint::{Target, TcpEndpoint};+use linkerd2_app_core::admit;++/// A connection policy that drops+#[derive(Copy, Clone, Debug)]+pub struct PreventLoop {+    port: u16,+}++#[derive(Copy, Clone, Debug)]+pub struct LoopPrevented {+    port: u16,+}++impl PreventLoop {+    pub fn new(port: u16) -> Self {+        Self { port }+    }+}++impl admit::Admit<Target> for PreventLoop {+    type Error = LoopPrevented;++    fn admit(&mut self, ep: &Target) -> Result<(), Self::Error> {+        tracing::debug!(addr = %ep.addr, self.port);+        if ep.addr.port() == self.port {+            return Err(LoopPrevented { port: self.port });+        }++        Ok(())+    }+}++impl admit::Admit<TcpEndpoint> for PreventLoop {+    type Error = LoopPrevented;++    fn admit(&mut self, ep: &TcpEndpoint) -> Result<(), Self::Error> {+        tracing::debug!(port = %ep.port, self.port);+        if ep.port == self.port {+            return Err(LoopPrevented { port: self.port });+        }++        Ok(())+    }+}++impl std::fmt::Display for LoopPrevented {+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+        write!(+            f,+            "inboudn requests must not target localhost:{}",

typo:

            "inbound requests must not target localhost:{}",
olix0r

comment created time in 3 days

create barnchlinkerd/linkerd2-proxy

branch : eliza/0.2-upgrades

created branch time in 3 days

issue commenttokio-rs/tokio

Support something like smol::Async

Maybe something we can do on Tokio's side to help prevent similar confusion in the future is highlighting the use of PollEvented to add new evented primitives to Tokio's event loop more clearly in the documentation?

yihuang

comment created time in 3 days

PR opened linkerd/linkerd2-proxy

Reviewers
turn outbound back into a proxy

This branch turns the outbound side of the proxy back into a real proxy, similarly to what #516 did for the inbound side. Getting outbound to work again was somewhat more complex, since the outbound side also requires the load balancer (and, therefore, discovery). In order to get this all working, it was also necessary to update the linkerd2-request-filter crate (which was quite trivial), as it's necessary for the resolver to work, and to make some changes in the resolver so that all trait bounds were satisfied.

In addition, now that both inbound and outbound work, we're now able to re-enable a large amount of the integration tests in the transparency and discovery modules. Some of these tests are still disabled, as they either depend on service profiles, HTTP upgrades and orig-proto, or header maniuplation, all of which have yet to be updated.

Signed-off-by: Eliza Weisman eliza@buoyant.io

+501 -254

0 comment

16 changed files

pr created time in 3 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 92338dddb61efd5f6600edf8a538b582f3129f02

update linkerd2-proxy-http::balance to std::future (#524) This branch updates the `balance` module in `linkerd2-proxy-http` to use Tower 0.3 rather than 0.1. This change is pretty trivial — it's just updating the dependencies in Cargo.toml and changing imports around to track code that moved around upstream. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 5f4d6763da4c4d3e331742ed588548f90d63dae9

update linkerd2-proxy-discover to std::future (#523) This branch updates the `linkerd2-proxy-discover` crate to use `std::future` and Tower 0.3. Overall, this is a fairly mechanical change. The main upstream API difference is that `tower-discover` is now a trait alias for `Stream`, rather than a trait that user code can directly implement. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 75f6d4faaeca5c0ce9090afbc19b7586ac4eb671

put back accdientally deleted code Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha a0378cc11425e8b5c967052e2b016d8d800ee29b

put back some endpoint stuff Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 3be137131c57105899aace3f1c5ca8baf5fdde89

wip Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha fc89b3997646d21566fc58b2c5ea15b126f3e21c

whoops canonicalize doesn't work yet Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha b558d7fb6ce762df779bea43b743f2a98b647d82

wip Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 059661f3d63b843a38d449f9143bd153351c557b

WORKY OUTBOUND Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 0f2f235699918f695f3034a146a2e8ff00ea28a5

put back dst resolver Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha a393d1ac3a5bd45f88da2e3d4b1a033a620d3d0a

update request-filter to tower 0.3 Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 2ef8a2c2ac098867fdf16549d70f9ca1e7d34e51

WHOA IT ALL ACTUALLY WORKS NOW Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 79c6ae894982aad30b126069739d94f2da3c5bd3

enable transparency tests for working features Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha a6151936fe804da00b034eb003554998c177ae8f

re-enable discovery tests that work now Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha acc3a7cebe75ef0ed1b56ca9d20729ac4612d808

enable transparency tests for working features Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 3410b45435a8994b3dedd5a3950b771cfe4517b4

re-enable discovery tests that work now Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha c9b0c4989d0dc6d0554c8c532be91f6e5417a7b6

WHOA IT ALL ACTUALLY WORKS NOW Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

push eventlinkerd/linkerd2-proxy-api

Eliza Weisman

commit sha 8b400dee750671b41cb24dcea23e5cceb6188301

fix rustfmt feature flag Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

push eventlinkerd/linkerd2-proxy-api

Kevin Leimkuhler

commit sha 59d91c1d8787f907fbab2de5ca844c25e7aeb9f7

Use GH actions for CI (#40) This change switches CI from Travis to GitHub Actions `make check-go` depended on `dep` which Go diverged from after officially adopting modules. This workflow relied a lot on previous assumptions about Go projects (such as being in `$GOPATH`) and was not easy to run locally. Go modules are now used for builing this project (`go build ./...`) instead of the previous dependence on `dep`. Closes linkerd/linkerd2#4207

view details

Eliza Weisman

commit sha 2ab62257ea3a077c2035d5c608d4bef1a9f1117a

turn tonic rustfmt feature back on turns out reading the generated sources without this is *really bad* Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 4cec14b69c656f999f3e4ce3727a99b3999d3201

Merge remote-tracking branch 'origin/master' into eliza/tonic Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha bade72971007103aa810305aa046aa8a0ce08183

update CI action to install rustfmt Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 3 days

issue commenttokio-rs/tokio

Support something like smol::Async

And this is my experiment result with smol's Async: erickt/rust-zmq@b33e08e. You can see that it's trivial to integrate the existing zmq library into async world with this approach.

Does the linked code actually work? My understanding is that smol::Async::new requires the wrapped type implement the standard library's AsRawFd trait, which (as far as I can tell) zmq::Socket does not implement. Am I missing something?

As a side note, tokio already provides the PollEvented type for associating an I/O resource with the Tokio event loop. Using PollEvented in conjunction with mio's EventedFd is similar to what smol::Async does (as I understand it). Would this work in your use-case?

yihuang

comment created time in 3 days

pull request commenttokio-rs/tracing

Add on_follows_from to OpenTelemetryLayer

Actually I'm not sure if linking should be bi-directional by default. From reading the open telemetry specs and samples it seems that links usually point only in 1 direction. Eg here.

Please lmk what you think and I'll update the code as appropriate.

Hmm, I'm not the most familiar with what's considered idiomatic in OpenTelemetry, so I'd defer to @jatescher on that.

bodymindarts

comment created time in 3 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 168217bcce41a866385b95f4b6043eedee530575

put back dst resolver Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha b793376a62d93f66053ff27402ca21b7dc7c740e

update request-filter to tower 0.3 Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 4 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha bf102b26a0c1c331e4d379536530440d6310e9bd

WORKY OUTBOUND Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 4 days

issue commentSergioBenitez/Rocket

Improve Logging

:wave: Hi, I'm the primary author of tracing. Based on my understanding of Rocket's requirements, it does seem like tracing was designed pretty much for this exact use-case, especially in the upcoming async version of Rocket.. If you all are interested, I'd be happy to help out with a move to tracing, either by opening a PR or by providing guidance and answering questions.

Rocket could probably provide its own tracing Subscriber that implements formatting similar to what Rocket provides now (potentially enriched with the additional data available in tracing). However, given that tracing is capable of more than just logging (e.g. integrating with distributed tracing such as OpenTelemetry or Honeycomb; generating flamegraphs and histograms, etc), I think users will want to be able to override the default subscriber provided by Rocket to layer in additional functionality. This would also allow customizing the format if users prefer something different.

SergioBenitez

comment created time in 4 days

issue commenttokio-rs/tracing

#[instrument] doesn't get send to jaeger/opentelemetry when used on rocket.rs request handlers

It could be helpful to try to isolate the problem, since this seems to involve rocket, multiple parts of tracing, and OpenTelemetry.

What happens if you use tracing_subscriber::fmt as the subscriber, rather than tracing_opentelemetry? If you put an event in the handler function, does tracing_subscriber::fmt log that event annotated with the the instrument-generated span, or not?

MTRNord

comment created time in 4 days

create barnchlinkerd/linkerd2-proxy

branch : eliza/0.2-outbound

created branch time in 4 days

PR merged linkerd/linkerd2-proxy

update linkerd2-proxy-discover to std::future

This branch updates the linkerd2-proxy-discover crate to use std::future and Tower 0.3.

Overall, this is a fairly mechanical change. The main upstream API difference is that tower-discover is now a trait alias for Stream, rather than a trait that user code can directly implement.

Signed-off-by: Eliza Weisman eliza@buoyant.io

+515 -461

0 comment

6 changed files

hawkw

pr closed time in 4 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 5f4d6763da4c4d3e331742ed588548f90d63dae9

update linkerd2-proxy-discover to std::future (#523) This branch updates the `linkerd2-proxy-discover` crate to use `std::future` and Tower 0.3. Overall, this is a fairly mechanical change. The main upstream API difference is that `tower-discover` is now a trait alias for `Stream`, rather than a trait that user code can directly implement. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 4 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 92338dddb61efd5f6600edf8a538b582f3129f02

update linkerd2-proxy-http::balance to std::future (#524) This branch updates the `balance` module in `linkerd2-proxy-http` to use Tower 0.3 rather than 0.1. This change is pretty trivial — it's just updating the dependencies in Cargo.toml and changing imports around to track code that moved around upstream. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 4 days

PR merged linkerd/linkerd2-proxy

update linkerd2-proxy-http::balance to std::future

This branch updates the balance module in linkerd2-proxy-http to use Tower 0.3 rather than 0.1.

This change is pretty trivial — it's just updating the dependencies in Cargo.toml and changing imports around to track code that moved around upstream.

Signed-off-by: Eliza Weisman eliza@buoyant.io

+15 -15

0 comment

4 changed files

hawkw

pr closed time in 4 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 021175395f19a6aa7a04d6944ad5aadf51bee2e8

fix warning Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 4 days

PR opened linkerd/linkerd2-proxy

Reviewers
update linkerd2-proxy-http::balance to std::future

This branch updates the balance module in linkerd2-proxy-http to use Tower 0.3 rather than 0.1.

This change is pretty trivial — it's just updating the dependencies in Cargo.toml and changing imports around to track code that moved around upstream.

Signed-off-by: Eliza Weisman eliza@buoyant.io

+15 -15

0 comment

4 changed files

pr created time in 4 days

create barnchlinkerd/linkerd2-proxy

branch : eliza/0.2-http-balance

created branch time in 4 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 02906d5f148fd8389cbffd51c1232fa1ac49cc35

update controller client to std::future (#522) This branch updates the proxy's control-plane client to use `std::future` and Tonic. This should unblock updating the outbound proxy (as it needs service discovery) and bringing back service profiles. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 440c09fd16fe6cde9362882254f5fcbc8c7cb014

update discover::buffer to std::future Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha adb311b8faa4c6353d74f43d6abbf1c970d64afd

update discover::from_resolve to std::future Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

Eliza Weisman

commit sha 1d61a2896eb5f3ac4bbef11e8788589a63299f8a

blahhhhhh

view details

Eliza Weisman

commit sha 8e83b6283b8bc1a2b9d60d1e229e92003e40fd36

update discover::make_endpoint and lib to std::future Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 4 days

PR opened linkerd/linkerd2-proxy

Reviewers
update linkerd2-proxy-discover to std::future

This branch updates the linkerd2-proxy-discover crate to use std::future and Tower 0.3.

Overall, this is a fairly mechanical change. The main upstream API difference is that tower-discover is now a trait alias for Stream, rather than a trait that user code can directly implement.

Signed-off-by: Eliza Weisman eliza@buoyant.io

+515 -461

0 comment

6 changed files

pr created time in 4 days

create barnchlinkerd/linkerd2-proxy

branch : eliza/0.2-discover

created branch time in 4 days

issue commenttokio-rs/tracing

#[instrument(name = "")] not documented

I think this was an oversight on my part, thanks for catching it! Will make sure it gets fixed.

jonhoo

comment created time in 4 days

issue commenttokio-rs/tracing

#[instrument] does not warn on unrecognized options

#672 will make instrument error on unrecognized syntax. I'm not sure if we can easily emit warnings rather than errrors from proc macros, but I can take a look at it.

Separately, wouldn't it be kind of nice to support this short-hand syntax? :p

Sure, we could probably add something like that!

jonhoo

comment created time in 4 days

push eventlinkerd/linkerd2-proxy

Eliza Weisman

commit sha 02906d5f148fd8389cbffd51c1232fa1ac49cc35

update controller client to std::future (#522) This branch updates the proxy's control-plane client to use `std::future` and Tonic. This should unblock updating the outbound proxy (as it needs service discovery) and bringing back service profiles. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

view details

push time in 5 days

PR merged linkerd/linkerd2-proxy

update controller client to std::future

This branch updates the proxy's control-plane client to use std::future and Tonic. This should unblock updating the outbound proxy (as it needs service discovery) and bringing back service profiles.

+499 -445

0 comment

13 changed files

hawkw

pr closed time in 5 days

startedrancher/k3d

started time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {             ssh_key_name: Default::default(),             private_key_path: Some(                 tempfile::NamedTempFile::new()-                    .context("failed to create temporary file for keypair")?,+                    .wrap_err("failed to create temporary file for keypair")?,             ),             outstanding_spot_request_ids: Default::default(),             instances: Default::default(),             client: Some(ec2),-            log: Some(log),         })     }      /// Region-specific instance setup.     ///     /// Make spot instance requests, wait for the instances, and then call the     /// instance setup functions.-    pub async fn launch(+    #[instrument(debug, skip(self, max_instance_duration_hours, max_wait))]+    pub async fn launch<M>(         &mut self,         max_instance_duration_hours: usize,         max_wait: Option<time::Duration>,-        machines: impl IntoIterator<Item = (String, Setup)>,-    ) -> Result<(), Error> {+        machines: M,+    ) -> Result<(), Report>+    where+        M: IntoIterator<Item = (String, Setup)> + std::fmt::Debug,+    {         self.make_spot_instance_requests(             max_instance_duration_hours * 60, // 60 mins/hr             machines,         )-        .await?;+        .await+        .wrap_err("failed to make spot instance requests")?;          let start = time::Instant::now();-        self.wait_for_spot_instance_requests(max_wait).await?;+        self.wait_for_spot_instance_requests(max_wait)+            .await+            .wrap_err("failed while waiting for spot instances fulfilment")?;         if let Some(mut d) = max_wait {             d -= time::Instant::now().duration_since(start);         } -        self.wait_for_instances(max_wait).await?;+        self.wait_for_instances(max_wait)+            .await+            .wrap_err("failed while waiting for instances to come up")?;         Ok(())     } -    async fn make_security_group(mut self, use_open_ports: bool) -> Result<Self, Error> {-        let log = self.log.as_ref().expect("RegionLauncher uninitialized");+    #[instrument(trace, skip(self))]
    #[instrument(level = "trace", skip(self))]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 where         })     } +    #[instrument(debug)]

I think you meant

    #[instrument(level = "debug")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl std::str::FromStr for Region {             r if r == Region::SouthAfricaNorth.as_ref() => Region::SouthAfricaNorth,             r if r == Region::UaeNorth.as_ref() => Region::UaeNorth,             r if r == Region::GermanyWestCentral.as_ref() => Region::GermanyWestCentral,-            u => bail!("Unknown azure region {}. Valid regions: eastus, eastus2, westus, centralus, northcentralus, southcentralus, northeurope, westeurope, eastasia, southeastasia, japaneast, japanwest, australiaeast, australiasoutheast, australiacentral, brazilsouth, southindia, centralindia, westindia, canadacentral, canadaeast, westus2, westcentralus, uksouth, ukwest, koreacentral, koreasouth, francecentral, southafricanorth, uaenorth, germanywestcentral", u),+            u => eyre::bail!("Unknown azure region {}. Valid regions: eastus, eastus2, westus, centralus, northcentralus, southcentralus, northeurope, westeurope, eastasia, southeastasia, japaneast, japanwest, australiaeast, australiasoutheast, australiacentral, brazilsouth, southindia, centralindia, westindia, canadacentral, canadaeast, westus2, westcentralus, uksouth, ukwest, koreacentral, koreasouth, francecentral, southafricanorth, uaenorth, germanywestcentral", u),         })     } }  mod azcmd {-    use super::Error;     use super::IpInfo;     use super::Region;-    use failure::ResultExt;+    use super::*;     use serde::{Deserialize, Serialize};     use tokio::process::Command; -    pub(crate) async fn check_az() -> Result<(), Error> {-        ensure!(-            Command::new("az").arg("account").arg("show").status().await?.success(), +    pub(crate) async fn check_az() -> Result<(), Report> {+        eyre::ensure!(+            Command::new("az").arg("account").arg("show").status().await.wrap_err("az account show")?.success(),              "Azure CLI not found. See https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest for installation, then run `az login`.",         );         Ok(())     } -    pub(crate) async fn create_resource_group(r: Region, name: &str) -> Result<(), Error> {+    #[instrument(trace)]
    #[instrument(level = "trace")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 mod azcmd {                 vm_name,             ])             .output()-            .await?;-        if !out.status.success() {-            return Err(format_err!("Failed to open ports for {}", vm_name))-                .context(String::from_utf8(out.stderr).unwrap())?;-        }+            .await+            .wrap_err("az vm open-port")?;++        eyre::ensure!(+            out.status.success(),+            "failed to open ports: {}",+            String::from_utf8_lossy(&out.stderr)+        );          Ok(())     } -    pub(crate) async fn delete_resource_group(rg: &str) -> Result<(), Error> {+    #[instrument(trace)]
    #[instrument(level = "trace")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 mod azcmd {                 &r.to_string(),             ])             .status()-            .await?;--        if !out.success() {-            bail!("Failed to create resource group {} in region {:?}", name, r)-        }+            .await+            .context("az group create")?; +        eyre::ensure!(out.success(), "failed to create resource group");         Ok(())     } +    #[instrument(trace)]
    #[instrument(level = "trace")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 pub struct Machine { impl super::Launcher for Machine {     type MachineDescriptor = Setup; +    #[instrument(debug, skip(self))]
    #[instrument(level = "debug", skip(self))]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl super::Launcher for Machine {         })     } +    #[instrument(debug)]
    #[instrument(level = "debug")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 mod azcmd {                 "--generate-ssh-keys",             ])             .output()-            .await?;+            .await+            .wrap_err("az vm create")?; -        if !out.status.success() {-            return Err(format_err!("Failed to create vm {}", name))-                .context(String::from_utf8(out.stderr).unwrap())?;-        }+        eyre::ensure!(+            out.status.success(),+            "failed to create vm: {}",+            String::from_utf8_lossy(&out.stderr)+        );          let vm: VmCreateOut = serde_json::from_slice(&out.stdout)?;-        ensure!(vm.powerState == "VM running", "VM power state incorrect");-        ensure!(vm.resourceGroup == rg, "VM resource group incorrect");+        eyre::ensure!(vm.powerState == "VM running", "VM power state incorrect");+        eyre::ensure!(vm.resourceGroup == rg, "VM resource group incorrect");         Ok(IpInfo {             public_ip: vm.publicIpAddress,             private_ip: vm.privateIpAddress,         })     } -    pub(crate) async fn open_ports(rg: &str, vm_name: &str) -> Result<(), Error> {+    #[instrument(trace)]
    #[instrument(level = "trace")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl super::MachineSetup for Setup {  impl Setup {     /// Create a new instance of Setup.-    pub fn new(-        addr: impl std::net::ToSocketAddrs,+    #[instrument(debug)]
    #[instrument(level = "debug")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher { impl super::Launcher for RegionLauncher {     type MachineDescriptor = Setup; +    #[instrument(debug, skip(self))]     fn launch<'l>(         &'l mut self,         l: super::LaunchDescriptor<Self::MachineDescriptor>,-    ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'l>> {+    ) -> Pin<Box<dyn Future<Output = Result<(), Report>> + Send + 'l>> {         Box::pin(async move {-            self.log = Some(l.log);-            let log = self.log.as_ref().unwrap();             let max_wait = l.max_wait;-            self.machines = futures_util::future::join_all(l.machines.into_iter().map(-                |(nickname, desc)| async {-                    let vm_name = super::rand_name_sep("vm", "-");-                    debug!(log, "setting up azure instance";-                        "nickname" => &nickname,-                        "vm_name" => &vm_name,-                    );-                    let ipinfo = azcmd::create_vm(-                        &self.resource_group_name,-                        &vm_name,-                        &desc.instance_type,-                        &desc.image,-                        &desc.username,-                    )-                    .await?;-                    azcmd::open_ports(&self.resource_group_name, &vm_name).await?;--                    if let Setup {-                        ref username,-                        setup_fn: Some(ref f),-                        ..-                    } = desc-                    {-                        super::setup_machine(-                            log,-                            &nickname,-                            &ipinfo.public_ip,-                            &username,-                            max_wait,-                            None,-                            f.as_ref(),+            self.machines =+                futures_util::future::join_all(l.machines.into_iter().map(|(nickname, desc)| {+                    let machine_span = tracing::debug_span!("machine", %nickname, ?desc);+                    async {+                        let vm_name = super::rand_name_sep("vm", "-");+                        tracing::debug!(%vm_name, "setting up instance");++                        let ipinfo = azcmd::create_vm(+                            &self.resource_group_name,+                            &vm_name,+                            &desc.instance_type,+                            &desc.image,+                            &desc.username,                         )                         .await?;+                        azcmd::open_ports(&self.resource_group_name, &vm_name).await?;++                        if let Setup {+                            ref username,+                            setup_fn: Some(ref f),+                            ..+                        } = desc+                        {+                            super::setup_machine(+                                &nickname,+                                &ipinfo.public_ip,+                                &username,+                                max_wait,+                                None,+                                f.as_ref(),+                            )+                            .await?;+                        }++                        Ok::<_, Report>(Descriptor {+                            name: nickname,+                            username: desc.username,+                            ip: ipinfo,+                        })                     }--                    Ok::<_, Error>(Descriptor {-                        name: nickname,-                        username: desc.username,-                        ip: ipinfo,-                    })-                },-            ))-            .await-            .into_iter()-            .collect::<Result<Vec<_>, Error>>()?;+                    .instrument(machine_span)+                }))+                .await+                .into_iter()+                .collect::<Result<Vec<_>, Report>>()?;              Ok(())         })     } +    #[instrument(debug)]
    #[instrument(level = "debug")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl super::Launcher for RegionLauncher {                 };                  async move {-                    m.connect_ssh(log, username, None, None, 22).await?;-                    Ok::<_, Error>((name.clone(), m))+                    m.connect_ssh(username, None, None, 22).await?;+                    Ok::<_, Report>((name.clone(), m))                 }+                .instrument(machine_span)             }))             .await             .into_iter()-            .collect::<Result<HashMap<_, _>, Error>>()+            .collect::<Result<HashMap<_, _>, Report>>()         })     } -    fn terminate_all(self) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send>> {-        debug!(self.log.as_ref().unwrap(), "Cleaning up resource group");+    #[instrument(debug)]
    #[instrument(level = "debug")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher { impl super::Launcher for RegionLauncher {     type MachineDescriptor = Setup; +    #[instrument(debug, skip(self))]
    #[instrument(level = "debug", skip(self))]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {     ///     /// Additionally deletes ephemeral keys and security groups. Note: it is a known issue that     /// security groups often will not be deleted, due to timing quirks in the AWS api.+    #[instrument(debug)]     pub async fn shutdown(&mut self) {         let client = self.client.as_ref().unwrap();-        let log = self.log.as_ref().expect("RegionLauncher uninitialized");         // terminate instances         if !self.instances.is_empty() {-            info!(log, "terminating instances");+            tracing::info!("terminating instances");             let instances = self.instances.keys().cloned().collect();             self.instances.clear();             let mut termination_req = rusoto_ec2::TerminateInstancesRequest::default();             termination_req.instance_ids = instances;             while let Err(e) = client.terminate_instances(termination_req.clone()).await {-                let msg = format!("{}", e);+                let msg = e.to_string();                 if msg.contains("Pooled stream disconnected") || msg.contains("broken pipe") {-                    trace!(log, "retrying instance termination");+                    tracing::trace!("retrying instance termination");                     continue;                 } else {-                    warn!(log, "failed to terminate tsunami instances: {:?}", e);+                    tracing::warn!("failed to terminate tsunami instances: {}", e);                     break;                 }             }         } -        debug!(log, "cleaning up temporary resources");+        tracing::debug!("cleaning up temporary resources");         if !self.security_group_id.trim().is_empty() {-            trace!(log, "cleaning up temporary security group"; "name" => self.security_group_id.clone());+            let group_span = tracing::trace_span!("security_group", id = %self.security_group_id);+            let _guard = group_span.enter();++            tracing::trace!("removing security group");             // clean up security groups and keys             // TODO need a retry loop for the security group. Currently, this fails             // because AWS takes some time to allow the security group to be deleted.             let mut req = rusoto_ec2::DeleteSecurityGroupRequest::default();             req.group_id = Some(self.security_group_id.clone());             if let Err(e) = client.delete_security_group(req).await {-                warn!(log, "failed to clean up temporary security group";-                    "group_id" => &self.security_group_id,-                    "error" => ?e,-                )+                tracing::warn!("failed to clean up temporary security group: {}", e);             }         }          if !self.ssh_key_name.trim().is_empty() {-            trace!(log, "cleaning up temporary keypair"; "name" => self.ssh_key_name.clone());+            let key_span = tracing::trace_span!("key", name = %self.ssh_key_name);+            let _guard = key_span.enter();

It looks like these span entered guards are held across await points, which is probably going to result in malformed traces. You should use async blocks and .instrument instead. See https://github.com/tokio-rs/tracing/#in-asynchronous-code

jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {     ///     /// Additionally deletes ephemeral keys and security groups. Note: it is a known issue that     /// security groups often will not be deleted, due to timing quirks in the AWS api.+    #[instrument(debug)]
    #[instrument(level = "debug")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {      /// Establish SSH connections to the machines. The `Ok` value is a `HashMap` associating the     /// friendly name for each `Setup` with the corresponding SSH connection.-    pub async fn connect_all<'l>(&'l self) -> Result<HashMap<String, crate::Machine<'l>>, Error> {-        let log = self.log.as_ref().unwrap();+    #[instrument(debug)]
    #[instrument(level = "debug")]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {                 client                     .cancel_spot_instance_requests(cancel)                     .await-                    .context("failed to cancel spot instances")-                    .map_err(|e| {-                        warn!(log, "failed to cancel spot instance request: {:?}", e);-                        e-                    })?;--                trace!(-                    log,-                    "spot instances cancelled -- gathering remaining instances"-                );+                    .wrap_err("failed to cancel spot instances")?;++                tracing::trace!("spot instances cancelled -- gathering remaining instances");                 // wait for a little while for the cancelled spot requests to settle                 // and any that were *just* made active to be associated with their instances                 thread::sleep(time::Duration::from_secs(1));                  let sirs = client                     .describe_spot_instance_requests(req)-                    .await?+                    .await+                    .wrap_err("failed to inspect canceled spot requests")?                     .spot_instance_requests                     .unwrap_or_else(Vec::new);                 for sir in sirs {+                    let req_id = sir.spot_instance_request_id.unwrap();                     match sir.instance_id {                         Some(instance_id) => {-                            trace!(log, "spot request cancelled";-                                "req_id" => sir.spot_instance_request_id,-                                "iid" => &instance_id,+                            tracing::trace!(+                                %req_id,+                                iid = %instance_id,+                                "spot request cancelled"                             );                         }                         _ => {-                            error!(-                                log,-                                "spot request failed: {:?}", &sir.status;-                                "req_id" => sir.spot_instance_request_id,+                            tracing::error!(+                                %req_id,+                                "spot request failed: {:?}", &sir.status,                             );                         }                     }                 }-                bail!("wait limit reached");+                eyre::bail!("wait limit reached");             }         }          Ok(())     }      /// Poll AWS until `max_wait` (if not `None`) or the instances are ready to SSH to.-    async fn wait_for_instances(&mut self, max_wait: Option<time::Duration>) -> Result<(), Error> {+    #[instrument(trace, skip(self, max_wait))]
    #[instrument(level = "trace", skip(self, max_wait))]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {     ///     /// To wait for the instances to be ready, call     /// [`wait_for_instances`](RegionLauncher::wait_for_instances).+    #[instrument(trace, skip(self, max_wait))]
    #[instrument(level = "trace", skip(self, max_wait))]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {     ///     /// Will *not* wait for the spot instance requests to complete. To wait, call     /// [`wait_for_spot_instance_requests`](RegionLauncher::wait_for_spot_instance_requests).-    async fn make_spot_instance_requests(+    #[instrument(trace, skip(self, max_duration))]
    #[instrument(level = "trace", skip(self, max_duration))]
jonhoo

comment created time in 5 days

Pull request review commentjonhoo/tsunami

Move everything to color_eyre + tracing

 impl RegionLauncher {             ssh_key_name: Default::default(),             private_key_path: Some(                 tempfile::NamedTempFile::new()-                    .context("failed to create temporary file for keypair")?,+                    .wrap_err("failed to create temporary file for keypair")?,             ),             outstanding_spot_request_ids: Default::default(),             instances: Default::default(),             client: Some(ec2),-            log: Some(log),         })     }      /// Region-specific instance setup.     ///     /// Make spot instance requests, wait for the instances, and then call the     /// instance setup functions.-    pub async fn launch(+    #[instrument(debug, skip(self, max_instance_duration_hours, max_wait))]
    #[instrument(level = "debug", skip(self, max_instance_duration_hours, max_wait))]
jonhoo

comment created time in 5 days

more