profile
viewpoint

bobrik/collectd-docker 152

Collect docker container resource usage

bobrik/ansible-grafana 52

Ansible role for grafana deployment

bobrik/bay 32

p2p distribution for docker images

bobrik/collectd-mesos-tasks 32

Collect task resource usage in mesos

bobrik/ceph-collectd-graphite 13

Collect ceph metrics to graphite

bobrik/buzzer 7

Dialer for Go that does retries on different IPs

bobrik/ansible-kibana 4

Ansible role for kibana deployment

bobrik/callisto 4

Callisto is my own Gentoo overlay

bobrik/bsuir 1

university labs etc.

bobrik/cloudflare-access-tab-auto-close 1

Google Chrome plugin to automatically close Cloudflare Access tabs on successful authentication

push eventbobrik/jaeger

Ivan Babrou

commit sha fe508a190eae18db64b2a5c2daa96a66911e2d3a

Clickhouse storage support

view details

push time in 6 days

push eventbobrik/jaeger

Ivan Babrou

commit sha b3158ef61dc503dd2f2cb52d6fd3f7d9fee9fd44

Clickhouse storage support

view details

push time in 12 days

issue commentjaegertracing/jaeger

ClickHouse as a storage backend

Turns out that having a sparse tags with nested fields is not great for searches when you have a lot of spans, so went back to array of key=value strings with a bloom filter index on top.

I've tried combining span storage and indexing into one table, but that proved to be very detrimental to performance of lookup by trace id. It compressed somewhat better, though.

If anyone wants to give it a spin, please be my guest:

  • https://github.com/bobrik/jaeger/tree/ivan/clickhouse/plugin/storage/clickhouse

It feels a lot snappier than Elasticsearch, and it's at least 2.5x more compact (4.4x if you compare to Elasticsearch behaviour out of the box).

sboisson

comment created time in 12 days

push eventbobrik/jaeger

Ivan Babrou

commit sha 7c112a1d6c22e8998dc7cf99c0985d65a2866d71

Clickhouse storage support

view details

push time in 12 days

push eventbobrik/jaeger

Ivan Babrou

commit sha 4c7cef4d2941ae50bf515a7dc357234e4002601c

Clickhouse storage support

view details

push time in 12 days

push eventbobrik/jaeger

Ivan Babrou

commit sha 328489121fd45c81bb17afe986b0725e2181b065

Clickhouse storage support

view details

push time in 12 days

push eventbobrik/jaeger

Ivan Babrou

commit sha 2e83d3265eeee364af1ac36c28482fa56efa9d6b

Clickhouse storage support

view details

push time in 12 days

issue commentjaegertracing/jaeger

ClickHouse as a storage backend

I think I might be more confused about tag queries that I initially realized, please take my explanation with a big grain of salt.

sboisson

comment created time in 15 days

issue commentjaegertracing/jaeger

ClickHouse as a storage backend

@yurishkuro retrieving by trace ID is pretty very fast, since you're doing a primary key lookup.

https://github.com/ClickHouse/ClickHouse/issues/4074#issuecomment-455189310 says this about LowCardinality:

Rule of thumb: it should make benefits if the number of distinct values is less that few millions.

That said, the schema is in no way final.

I'm confused why this would be the case. Doesn't CH evaluate the query in full against each row (i.e. each span)? Or is this because how your plugin interacts with CH?

The row is not span, it's span-key-value combination. That's the key.

Take a look at the output of this query, which is an equivalent of what I do:

SELECT *
FROM jaeger_index
ARRAY JOIN tags
ORDER BY timestamp DESC
LIMIT 20
FORMAT PrettyCompactMonoBlock
┌──────────────────timestamp─┬─traceID──────────┬─service──────┬─operation─────┬─durationUs─┬─tags.key─────────────┬─tags.valueString─┬─tags.valueBool─┬─tags.valueInt─┬─tags.valueFloat─┐
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ num_trace_ids        │                  │              0 │            20 │               0 │
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ weird                │                  │              1 │             0 │               0 │
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ π                    │                  │              0 │             0 │            3.14 │
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ internal.span.format │ proto            │              0 │             0 │               0 │
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ jaeger.version       │ Go-2.22.1        │              0 │             0 │               0 │
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ hostname             │ C02TV431HV2Q     │              0 │             0 │               0 │
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ ip                   │ 192.168.1.43     │              0 │             0 │               0 │
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces     │    3414327 │ client-uuid          │ 3f9574079594605c │              0 │             0 │               0 │
│ 2020-05-11 04:53:45.723921 │ 700a1bff0bdf3141 │ jaeger-query │ GetOperations │    2268055 │ internal.span.format │ proto            │              0 │             0 │               0 │
│ 2020-05-11 04:53:45.723921 │ 700a1bff0bdf3141 │ jaeger-query │ GetOperations │    2268055 │ jaeger.version       │ Go-2.22.1        │              0 │             0 │               0 │
└────────────────────────────┴──────────────────┴──────────────┴───────────────┴────────────┴──────────────────────┴──────────────────┴────────────────┴───────────────┴─────────────────┘

Here we joint traceID with nested tags, repeating every tag key value with corresponding trace.

Operation getTraces does not happen multiple times in trace 3adb641936b21d98, but our query makes a separate row for it, so that cross-span tags naturally work out of the box

Compare this to how it looks like at span level (this is what Elasticsearch works with):

SELECT *
FROM jaeger_index
WHERE (traceID = '3adb641936b21d98') AND (service = 'jaeger-query') AND (operation = 'getTraces')
┌──────────────────timestamp─┬─traceID──────────┬─service──────┬─operation─┬─durationUs─┬─tags.key────────────────────────────────────────────────────────────────────────────────────────────┬─tags.valueString────────────────────────────────────────────────────────────────┬─tags.valueBool────┬─tags.valueInt──────┬─tags.valueFloat──────┐
│ 2020-05-11 04:53:54.299067 │ 3adb641936b21d98 │ jaeger-query │ getTraces │    3414327 │ ['num_trace_ids','weird','π','internal.span.format','jaeger.version','hostname','ip','client-uuid'] │ ['','','','proto','Go-2.22.1','C02TV431HV2Q','192.168.1.43','3f9574079594605c'] │ [0,1,0,0,0,0,0,0] │ [20,0,0,0,0,0,0,0] │ [0,0,3.14,0,0,0,0,0] │
└────────────────────────────┴──────────────────┴──────────────┴───────────┴────────────┴───────────────────────────────────────────────

Clickhouse doesn't allow direct lookups against nested fields like tags.ip=192.168.1.43, instead you have to do a JOIN ARRAY which results in this new property.

sboisson

comment created time in 15 days

issue commentjaegertracing/jaeger

ClickHouse as a storage backend

I took a stab at it (very early WIP):

  • https://github.com/jaegertracing/jaeger/compare/master...bobrik:ivan/clickhouse

This is the schema I used:

  • Index table for fast searches (haven't measured if indices are useful yet)
CREATE TABLE jaeger_index (
  timestamp DateTime64(6),
  traceID FixedString(16),
  service LowCardinality(String),
  operation LowCardinality(String),
  durationUs UInt64,
  tags Nested(
    key LowCardinality(String),
    valueString LowCardinality(String),
    valueBool UInt8,
    valueInt Int64,
    valueFloat Float64
  ),
  INDEX tags_strings (tags.key, tags.valueString) TYPE set(0) GRANULARITY 64,
  INDEX tags_ints (tags.key, tags.valueInt) TYPE set(0) GRANULARITY 64
) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (timestamp, service, operation);
  • Data table for span storage and quick retrieval by traceID
CREATE TABLE jaeger_spans (
  timestamp DateTime64(6),
  traceID FixedString(16),
  model String
) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY traceID;

You probably need Clickhouse 20.x for DateTime64, I used 20.1.11.73.

Index table looks like this:

SELECT *
FROM jaeger_index
ARRAY JOIN tags
ORDER BY timestamp DESC
LIMIT 20
FORMAT PrettyCompactMonoBlock
┌──────────────────timestamp─┬─traceID──────────┬─service──────┬─operation────┬─durationUs─┬─tags.key─────────────┬─tags.valueString─┬─tags.valueBool─┬─tags.valueInt─┬─tags.valueFloat─┐
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ num_trace_ids        │                  │              0 │            13 │               0 │
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ weird                │                  │              1 │             0 │               0 │
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ π                    │                  │              0 │             0 │            3.14 │
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ internal.span.format │ proto            │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ jaeger.version       │ Go-2.22.1        │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ hostname             │ C02TV431HV2Q     │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ ip                   │ 192.168.1.43     │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.248314 │ 212e5c616f4b9c2f │ jaeger-query │ getTraces    │     131605 │ client-uuid          │ 7fc8f98ddbcd358c │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065577 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraceIDs │     182728 │ internal.span.format │ proto            │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065577 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraceIDs │     182728 │ jaeger.version       │ Go-2.22.1        │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065577 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraceIDs │     182728 │ hostname             │ C02TV431HV2Q     │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065577 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraceIDs │     182728 │ ip                   │ 192.168.1.43     │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065577 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraceIDs │     182728 │ client-uuid          │ 7fc8f98ddbcd358c │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065574 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraces   │     314349 │ internal.span.format │ proto            │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065574 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraces   │     314349 │ jaeger.version       │ Go-2.22.1        │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065574 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraces   │     314349 │ hostname             │ C02TV431HV2Q     │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065574 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraces   │     314349 │ ip                   │ 192.168.1.43     │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065574 │ 212e5c616f4b9c2f │ jaeger-query │ FindTraces   │     314349 │ client-uuid          │ 7fc8f98ddbcd358c │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065535 │ 212e5c616f4b9c2f │ jaeger-query │ /api/traces  │     315554 │ sampler.type         │ const            │              0 │             0 │               0 │
│ 2020-05-10 20:43:23.065535 │ 212e5c616f4b9c2f │ jaeger-query │ /api/traces  │     315554 │ sampler.param        │                  │              1 │             0 │               0 │
└────────────────────────────┴──────────────────┴──────────────┴──────────────┴────────────┴──────────────────────┴──────────────────┴────────────────┴───────────────┴─────────────────┘

Tags are stored in their original types, so with enough SQL-fu you can find all spans with response size between X and Y bytes, for example.

The layout of the query is different from Elasticsearch, since now you have all tags for the trace laid out on a single view. This means that if you search for all operations of some service, you will get a cross-span result where one tag can match one span, and another tag can match another span. Consider the following trace:

+ upstream (tags: {"host": "foo.bar"})
++ upstream_ttfb (tags: {"status": 200})
++ upstream_download (tags: {"error": true})

You can search for host=foo.bar status=200 across all operations and this trace will be found, even though no since span has both tags. This seems like a really nice upside.

There's support for both JSON and Protobuf storage. The former allows out-of-band queries, since Clickhouse supports JSON functions. The latter is much more compact.

I pushed 100K spans from tracegen through this with a local Clickhouse in a Docker container with stock settings, and here's how storage looks like:

SELECT
    table,
    sum(marks) AS marks,
    sum(rows) AS rows,
    sum(bytes_on_disk) AS bytes_on_disk,
    sum(data_compressed_bytes) AS data_compressed_bytes,
    sum(data_uncompressed_bytes) AS data_uncompressed_bytes,
    toDecimal64(data_uncompressed_bytes / data_compressed_bytes, 2) AS compression_ratio,
    toDecimal64(data_compressed_bytes / rows, 2) AS compressed_bytes_per_row
FROM system.parts
WHERE table LIKE 'jaeger_%'
GROUP BY table
ORDER BY table ASC
┌─table────────┬─marks─┬────rows─┬─bytes_on_disk─┬─data_compressed_bytes─┬─data_uncompressed_bytes─┬─compression_ratio─┬─compressed_bytes_per_row─┐
│ jaeger_index │   434 │ 1320800 │      26752641 │              26455785 │               280802430 │             10.61 │                    20.03 │
│ jaeger_spans │   351 │  646835 │      32997215 │              32963094 │               224993472 │              6.82 │                    50.96 │
└──────────────┴───────┴─────────┴───────────────┴───────────────────────┴─────────────────────────┴───────────────────┴──────────────────────────┘
SELECT
    table,
    column,
    type,
    sum(column_data_compressed_bytes) AS compressed,
    sum(column_data_uncompressed_bytes) AS uncompressed,
    toDecimal64(uncompressed / compressed, 2) AS compression_ratio,
    sum(rows) AS rows,
    toDecimal64(compressed / rows, 2) AS bytes_per_row
FROM system.parts_columns
WHERE (table LIKE 'jaeger_%') AND active
GROUP BY
    table,
    column,
    type
ORDER BY
    table ASC,
    column ASC
┌─table────────┬─column───────────┬─type──────────────────────────┬─compressed─┬─uncompressed─┬─compression_ratio─┬───rows─┬─bytes_per_row─┐
│ jaeger_index │ durationUs       │ UInt64                        │     248303 │       853336 │              3.43 │ 106667 │          2.32 │
│ jaeger_index │ operation        │ LowCardinality(String)        │       5893 │       107267 │             18.20 │ 106667 │          0.05 │
│ jaeger_index │ service          │ LowCardinality(String)        │        977 │       107086 │            109.60 │ 106667 │          0.00 │
│ jaeger_index │ tags.key         │ Array(LowCardinality(String)) │      29727 │      1811980 │             60.95 │ 106667 │          0.27 │
│ jaeger_index │ tags.valueBool   │ Array(UInt8)                  │      29063 │      1810904 │             62.30 │ 106667 │          0.27 │
│ jaeger_index │ tags.valueFloat  │ Array(Float64)                │      44762 │      8513880 │            190.20 │ 106667 │          0.41 │
│ jaeger_index │ tags.valueInt    │ Array(Int64)                  │     284393 │      8513880 │             29.93 │ 106667 │          2.66 │
│ jaeger_index │ tags.valueString │ Array(LowCardinality(String)) │      31695 │      1814416 │             57.24 │ 106667 │          0.29 │
│ jaeger_index │ timestamp        │ DateTime64(6)                 │     431835 │       853336 │              1.97 │ 106667 │          4.04 │
│ jaeger_index │ traceID          │ FixedString(16)               │    1063375 │      1706672 │              1.60 │ 106667 │          9.96 │
│ jaeger_spans │ model            │ String                        │    4264180 │     34552264 │              8.10 │ 106667 │         39.97 │
│ jaeger_spans │ timestamp        │ DateTime64(6)                 │     463444 │       853336 │              1.84 │ 106667 │          4.34 │
│ jaeger_spans │ traceID          │ FixedString(16)               │     905193 │      1706672 │              1.88 │ 106667 │          8.48 │
└──────────────┴──────────────────┴───────────────────────────────┴────────────┴──────────────┴───────────────────┴────────┴───────────────┘

We have around 74B daily docs in our production Elasticsearch storage. My plan is to switch that to fields-as-tags, remove indexing of non-queried fields (logs, nested tags, references), then switch to a sorted index and then see how Clickhouse compares to that for the same spans.

sboisson

comment created time in 15 days

create barnchbobrik/jaeger

branch : ivan/clickhouse

created branch time in 15 days

push eventbobrik/FlameGraph

Ivan Babrou

commit sha 4727f3e39578cecea1d44d4d7876ef4d30b6541b

Do not depend on the exact kernel release The script does not necessarily run on a machine that produced `perf.data`, so kernel versions may mismatch.

view details

push time in 16 days

push eventbobrik/FlameGraph

Ivan Babrou

commit sha de8db1f6083454d95eae951443df0c43a15dba06

Do not depend on the exact kernel release The script does not necessarily run on a machine that produced `perf.data`, so kernel versions may mismatch.

view details

push time in 16 days

issue commentsaltstack/salt

salt 2018.3.2 py3 on Ubuntu 18.04 takes a long time to run test.true

I opened #57062 to fix this for good.

kartiksubbarao

comment created time in 22 days

PR opened saltstack/salt

Only inspect one frame in depends decorator

Some very good background can be found on StackOverflow:

  • https://stackoverflow.com/questions/17407119/python-inspect-stack-is-slow

I filed an issue about slow inspect.stack() in Python some time ago:

  • https://bugs.python.org/issue39643

Slow inspect.stack() is a problem, but it's not the problem in this case. The problem is that Salt asks Python to get a full blown stack with all the frames, but then discards everything but the second frame from the top. This is very wasteful, and it's much more economic to ask for the current frame and walk back once.

Here's how long it takes to run test.ping with existing code:

$ time sudo salt-call --local test.ping
local:
    True

real	0m12.166s
user	0m11.679s
sys	0m0.499s

And this is how much better it gets with the proposed change:

$ time sudo salt-call --local test.ping
local:
    True

real	0m2.092s
user	0m1.772s
sys	0m0.336s

If you follow the advice from #48773 and disable vmware module, which is responsible for the most calls into inspect module, you won't get much better than that:

$ cat /etc/salt/minion
disable_grains:
- esxi
disable_modules:
- vsphere

$ time sudo salt-call --local test.ping
local:
    True

real	0m2.006s
user	0m1.671s
sys	0m0.353s

Closes #48773.

+1 -1

0 comment

1 changed file

pr created time in 22 days

issue commentsaltstack/salt

salt 2018.3.2 py3 on Ubuntu 18.04 takes a long time to run test.true

I don't think this should've been closed. The issue still exists in the latest release.

kartiksubbarao

comment created time in 22 days

startedeiginn/nftrace

started time in 25 days

pull request commentbrendangregg/FlameGraph

Support kernel annotations for versioned vmlinux

Added another commit to detect kernel modules by .ko file extension.

We see the following stack traces:

swapper     0 [012] 2281339.604067:   20834821 cycles:
        ffffffffc09ace93 bpf_prog_2b956549c660136a_uni_l4lb+0xb80 (bpf_prog_2b956549c660136a_uni_l4lb)
        ffffffffc087cbc9 mlx5e_xdp_handle+0xa9 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko)
        ffffffffc0878d87 mlx5e_skb_from_cqe_linear+0xc7 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko)
        ffffffffc087a254 mlx5e_handle_rx_cqe+0x64 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko)
        ffffffffc087bb48 mlx5e_poll_rx_cq+0x808 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko)
        ffffffffc087beee mlx5e_napi_poll+0xde (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko)
        ffffffffb8f03eaa net_rx_action+0x13a (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb94000e0 __softirqentry_text_start+0xe0 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb886c8c0 irq_exit+0xa0 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb9201908 do_IRQ+0x58 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb9200a0f common_interrupt+0xf (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb8ebc022 cpuidle_enter_state+0xb2 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb8ebc3c9 cpuidle_enter+0x29 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb88960f8 do_idle+0x1b8 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb88962c9 cpu_startup_entry+0x19 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb8841383 start_secondary+0x143 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb88000d4 [unknown] (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)

And without this change mlx5e_core kernel module is not being recognized as a kernel side code.

bobrik

comment created time in 2 months

push eventbobrik/FlameGraph

Ivan Babrou

commit sha 057b185c82a3ca9e50c9db6662641b72e1fc3649

Detect kernel modules by .ko file extension We see the following stack traces: ``` swapper 0 [012] 2281339.604067: 20834821 cycles: ffffffffc09ace93 bpf_prog_2b956549c660136a_uni_l4lb+0xb80 (bpf_prog_2b956549c660136a_uni_l4lb) ffffffffc087cbc9 mlx5e_xdp_handle+0xa9 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko) ffffffffc0878d87 mlx5e_skb_from_cqe_linear+0xc7 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko) ffffffffc087a254 mlx5e_handle_rx_cqe+0x64 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko) ffffffffc087bb48 mlx5e_poll_rx_cq+0x808 (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko) ffffffffc087beee mlx5e_napi_poll+0xde (/lib/modules/5.4.14-cloudflare-2020.1.11/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko) ffffffffb8f03eaa net_rx_action+0x13a (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb94000e0 __softirqentry_text_start+0xe0 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb886c8c0 irq_exit+0xa0 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb9201908 do_IRQ+0x58 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb9200a0f common_interrupt+0xf (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb8ebc022 cpuidle_enter_state+0xb2 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb8ebc3c9 cpuidle_enter+0x29 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb88960f8 do_idle+0x1b8 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb88962c9 cpu_startup_entry+0x19 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb8841383 start_secondary+0x143 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ffffffffb88000d4 [unknown] (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11) ``` And without this change `mlx5e_core` kernel module is not being recognized as a kernel side code.

view details

push time in 2 months

PR opened brendangregg/FlameGraph

Support kernel annotations for versioned vmlinux

We use Debian and vanilla kernel packaging capabilities, and those produce versiond vmlinux files with frames like these:

        ffffffffb8f6abf0 nf_hook_slow+0x40 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb8f74051 ip_local_deliver+0xd1 (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)
        ffffffffb8f7412c ip_rcv+0xbc (/usr/lib/debug/boot/vmlinux-5.4.14-cloudflare-2020.1.11)

This change allows matching for those as kernel functions.

+5 -1

0 comment

1 changed file

pr created time in 2 months

create barnchbobrik/FlameGraph

branch : ivan/versioned-vmlinux

created branch time in 2 months

issue closedcloudflare/ebpf_exporter

Cannot attach softevent

There is a coding error in the PerfEvent struct:

diff --git a/config/config.go b/config/config.go
index 9c2a7a9..38ca87c 100644
--- a/config/config.go
+++ b/config/config.go
@@ -20,7 +20,7 @@ type Program struct {

 // PerfEvent describes perf_event to attach to
 type PerfEvent struct {
-       Type            int    `yaml:"string"`
+       Type            int    `yaml:"type"`
        Name            int    `yaml:"name"`
        Target          string `yaml:"target"`
        SamplePeriod    int    `yaml:"sample_period"`

closed time in 2 months

rexrock

issue commentcloudflare/ebpf_exporter

Cannot attach softevent

Fixed in #73.

rexrock

comment created time in 2 months

pull request commentcloudflare/ebpf_exporter

Fix bug for Perf_event struct.

Thanks!

rexrock

comment created time in 2 months

push eventcloudflare/ebpf_exporter

root

commit sha 139fb406fe57f29b044dd4e388d0c6b083c6eb2d

Fix bug for Perf_event struct. There is a code error in Perf_event struct, it causes only the HARDWARE event to work properly

view details

Ivan Babrou

commit sha d8a470d7672d4a5ffc58e697507d90f0a8c9f7e7

Merge pull request #73 from rexrock/rexrock/master Fix bug for Perf_event struct.

view details

push time in 2 months

PR merged cloudflare/ebpf_exporter

Fix bug for Perf_event struct.

There is a code error in Perf_event struct, it causes only the HARDWARE event to work properly

+1 -1

0 comment

1 changed file

rexrock

pr closed time in 2 months

issue commentcloudflare/ebpf_exporter

Cannot attach softevent

Indeed! In the examples we only use HARDWARE type, which conveniently happens to be zero, so it works as expected there.

Do you want to make a PR yourself to correct this?

rexrock

comment created time in 2 months

issue commentjaegertracing/jaeger-ui

Allow setting lookback in UTC

Timezone selector is another option, but I'm not sure if it's common to need more than "UTC" and "Local time" options.

bobrik

comment created time in 2 months

issue openedjaegertracing/jaeger-ui

Allow setting lookback in UTC

Requirement - what kind of business use case are you trying to solve?

I'm trying to correlate logs and graphs in UTC with traces in Jaeger UI.

Problem - what in Jaeger blocks you from solving the requirement?

Time in Jaeger UI is in local timezone (Pacific time for me), but logs and graphs come with UTC timestamps. I have to manually convert UTC to local time and then use that to look up some traces.

I am very bad at timezone conversion.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Let's have a timezone selector for lookback. I'm not sure what's the most ergonomic way of doing it, my best suggestion is to have Local timezone checkbox that is enabled by default to preserve current behavior. If a user wants to use UTC, they can uncheck that.

created time in 2 months

push eventbobrik/jaeger-client-go

Ivan Babrou

commit sha ed640637c2068cbee40b9313cb13ce4f2f95e207

Fix typo: span.finish() -> span.Finish() Signed-off-by: Ivan Babrou <github@ivan.computer>

view details

push time in 2 months

push eventbobrik/jaeger-client-go

Ivan Babrou

commit sha 4cfec1572d30da497353921644caf8100b4fd321

Fix typo: span.finish() -> span.Finish()

view details

push time in 2 months

pull request commentprometheus/prometheus

Port isolation from old TSDB PR

I think this could fix #4580 as well.

beorn7

comment created time in 2 months

issue commentcloudflare/ebpf_exporter

runqlat needs an update for newer kernels

Thanks for pointing this out. Lack of stable interface is the fact of life with kprobe, that's why we have both kprobe and raw_tracepoint variants of the timers example.

PR is welcome to migrate runqlat example, so I'll leave this open in case anyone wants to try.

vedranf

comment created time in 2 months

startedMellanox/mstflint

started time in 2 months

push eventbobrik/bobrik.github.io

Ivan Babrou

commit sha e6079c34b8d7783c72166ccba7779ba8eaa772b1

Remove ok.txt, not sure why it was needed

view details

Ivan Babrou

commit sha bc924362bd5008b7880f8963d109d75132a3e0d8

Remove keybase stuff

view details

Ivan Babrou

commit sha c4e5645b4179ca94293b3fcc709558e878f3f8f6

Point to ivan.computer

view details

push time in 3 months

more