profile
viewpoint
Robert Milan robert-milan Grafana Labs (@grafana) The World

grafana/carbon-relay-ng 412

Fast carbon relay+aggregator with admin interfaces for making changes online - production ready

grafana/go-generic-object-store 11

go-generic-object-store is a small, fast, light-weight, in-memory, off-heap library designed specifically for use with string interning and other similar redundancy reduction concepts.

replay/go-generic-object-store 4

go-generic-object-store is a small, fast, light-weight, in-memory, off-heap library designed specifically for use with string interning and other similar redundancy reduction concepts.

robert-milan/bitset 0

Go package implementing bitsets

robert-milan/go-object-interning 0

object interning built on top of github.com/replay/go-generic-object-store

robert-milan/gocql 0

Package gocql implements a fast and robust Cassandra client for the Go programming language.

push eventgrafana/metrictank

Robert Milan

commit sha 2ed9833fadd4bc209592b9657047b57476669167

update tests and add more comments

view details

push time in 18 hours

push eventgrafana/metrictank

Robert Milan

commit sha b36ab3fe50d38caad5c5a84a4ef52f737aa708bd

update metrics.md for invalid utf8

view details

push time in a day

pull request commentgrafana/metrictank

add utf8 validation checks to incoming metrics

I would still like to add some more unit tests, but I feel pretty confident with this approach

robert-milan

comment created time in a day

push eventgrafana/metrictank

Robert Milan

commit sha d7efdcf9afb996d126695c3d1b7f7a098b3009f6

use less allocations in checks

view details

push time in a day

Pull request review commentgrafana/metrictank

add utf8 validation checks to incoming metrics

 func (m *MetricData) Validate() error { 	if m.Mtype == "" || (m.Mtype != "gauge" && m.Mtype != "rate" && m.Mtype != "count" && m.Mtype != "counter" && m.Mtype != "timestamp") { 		return ErrInvalidMtype 	}+	if !utf8.ValidString(m.Name) {

Yeah, I'll rework this a bit

robert-milan

comment created time in 2 days

pull request commentgrafana/metrictank

update zoom link for community call

Updated comment ^^

robert-milan

comment created time in 5 days

create barnchgrafana/metrictank

branch : update-community-docs

created branch time in 5 days

PR opened grafana/metrictank

add utf8 validation checks to incoming metrics

This PR adds the option to reject invalid UTF8 data for incoming metrics. This is turned off by default, to enable you just need to change the config setting reject-invalid-utf8 to true.

Closes: #1728

+41 -1

0 comment

11 changed files

pr created time in 6 days

create barnchgrafana/metrictank

branch : reject-invalid-utf8

created branch time in 6 days

IssuesEvent

issue closedgrafana/metrictank

panic in master

version : v0.13.1-655-gd950ad98 looking at our internal metrictank-render-request-proxying dashboard for worldping with timerange "last 30days" I get this:

runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:513 (0x42c0b8)
/usr/local/go/src/runtime/panic.go:82 (0x42b20d)
/usr/local/go/src/runtime/signal_unix.go:390 (0x442031)
/go/src/github.com/grafana/metrictank/consolidation/consolidate.go:60 (0xa0140f)
/go/src/github.com/grafana/metrictank/expr/normalize.go:92 (0xb6de42)
/go/src/github.com/grafana/metrictank/expr/normalize.go:30 (0xb6d6fb)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xb490d9)
/go/src/github.com/grafana/metrictank/expr/func_alias.go:28 (0xb4a9f3)
/go/src/github.com/grafana/metrictank/expr/plan.go:309 (0xb732c8)
/go/src/github.com/grafana/metrictank/api/graphite.go:837 (0xbe3677)
/go/src/github.com/grafana/metrictank/api/graphite.go:270 (0xbddb56)
/go/src/github.com/grafana/metrictank/api/routes.go:65 (0xbfd894)
/usr/local/go/src/runtime/asm_amd64.s:525 (0x45aa74)
/usr/local/go/src/reflect/value.go:447 (0x4b7b53)
/usr/local/go/src/reflect/value.go:308 (0x4b75e3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xa73553)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xa72eb9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 (0xbccdab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8f020)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xa9f50a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xa92343)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 (0xaf52bf)
/usr/local/go/src/runtime/asm_amd64.s:522 (0x45a8aa) /usr/local/go/src/reflect/value.go:447
(0x4b7b53) /usr/local/go/src/reflect/value.go:308 (0x4b75e3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xa73553)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xa72eb9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/api/middleware/tracer.go:70 (0xaf6b15)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8f020)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/api/middleware/stats.go:76 (0xaf6268)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8f020)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xa9f50a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xa92343)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:187 (0xaa0717)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:294 (0xa9aa76)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:220 (0xa93661)
/usr/local/go/src/net/http/server.go:2741 (0x6f49ca)
/usr/local/go/src/net/http/server.go:1847 (0x6f0d65)
/usr/local/go/src/runtime/asm_amd64.s:1333 (0x45c440)
message:""
message:"Query error: 500 "

closed time in 7 days

Dieterbe

issue commentgrafana/metrictank

panic in master

Thanks @shanson7

Fixed by: #1813

Dieterbe

comment created time in 7 days

issue commentgrafana/metrictank

panic in master

I cannot reproduce this

Dieterbe

comment created time in 7 days

PR opened grafana/metrictank

add details format and update docs

This PR adds a new format called details to the /metrics/find endpoint. You can use this to return detailed information on a series, including all of its archives (along with their MKeys and Intervals, etc..). This partly addresses some of the comments in #1115. Will need another PR to add retrieval (in Json format) by MKey.

See also: #1527, #1565

+111 -3

0 comment

4 changed files

pr created time in 8 days

create barnchgrafana/metrictank

branch : add-options-to-metrics-find

created branch time in 8 days

issue openedgrafana/metrictank

Add more specific documentation about runtime consolidation and the effects of Max Data Points

We need to ensure everyone is very aware of how runtime consolidation works, including how setting MDP may affect the returned results as it relates to timestamps, aggregated buckets, etc...

See also: #983

created time in 8 days

issue commentgrafana/metrictank

Timestamps returned with small MaxDataPoints value are completely wrong

After much discussion and research we have decided to keep things as they are. We will update the documentation and add a note about this behavior.

shanson7

comment created time in 8 days

issue closedgrafana/metrictank

Timestamps returned with small MaxDataPoints value are completely wrong

The basic issue here is that when MaxDataPoints is a small number, the time ranges that come back are completely off. This is very easy to miss when using the pie chart in grafana (which sets the MDP to 3 by default...) but users noticed when comparing graphs to pie charts.

Metrictank

(well, raintank tsdb on the play.grafana.org site) Snapshot Example response:

{
  "xhrStatus": "complete",
  "request": {
    "method": "POST",
    "url": "api/datasources/proxy/14/render",
    "data": "target=worldping.~google_com_demo.new-york.ping.median&from=1533614400&until=1533700859&format=json&maxDataPoints=3"
  },
  "response": [
    {
      "target": "worldping.~google_com_demo.new-york.ping.median",
      "tags": {
        "name": "worldping.~google_com_demo.new-york.ping.median"
      },
      "datapoints": [
        [
          1.9022073917418467,
          1533656300
        ],
        [
          1.8690466450381682,
          1533685120
        ],
        [
          1.8561943401144299,
          1533713940
        ]
      ]
    }
  ]
}

Graphite

Snapshot Example response:

{
  "xhrStatus": "complete",
  "request": {
    "method": "POST",
    "url": "api/datasources/proxy/1/render",
    "data": "target=apps.fakesite.web_server_01.counters.requests.count&from=1533614400&until=1533700859&format=json&maxDataPoints=3"
  },
  "response": [
    {
      "target": "apps.fakesite.web_server_01.counters.requests.count",
      "datapoints": [
        [
          266.5722481176049,
          1533627480
        ],
        [
          267.38693467336685,
          1533656300
        ],
        [
          258.3019243530192,
          1533685120
        ]
      ]
    }
  ]
}

closed time in 8 days

shanson7

delete branch grafana/metrictank

delete branch : fix-mt-fakemetrics

delete time in 12 days

PR closed grafana/metrictank

Reviewers
fix datafeed metrics loop

This PR fixes a problem in mt-fakemetrics tool inside the loop that sends data. In certain cases where the --speedup was set to a high value it would cause mt-fakemetrics to send data with future timestamps.

+6 -0

2 comments

1 changed file

robert-milan

pr closed time in 12 days

pull request commentgrafana/metrictank

fix datafeed metrics loop

Closed in favor of #1825

robert-milan

comment created time in 12 days

Pull request review commentgrafana/metrictank

fix datafeed metrics loop

 times %4d orgs: each %s, flushing %d metrics so rate of %d Hz. (%d total unique 			// respecting where a previous flush left off, we need to start from 			// the point after it. 			var m int+			ts = startingTs

I didn't extensively test it, only for the case with a high speedup. I think you might be right though, in the case where the speedup is low that could cause a problem. I will need to think about this some more.

robert-milan

comment created time in 12 days

push eventgrafana/metrictank

Robert Milan

commit sha 752dbc8f80bc3529c01c99dd0f9558b2756f1f38

remove log line

view details

push time in 13 days

PR opened grafana/metrictank

fix datafeed metrics loop

This PR fixes a problem in mt-fakemetrics tool inside the loop that sends data. In certain cases where the --speedup was set to a high value it would cause mt-fakemetrics to send data with future timestamps.

+7 -0

0 comment

1 changed file

pr created time in 13 days

create barnchgrafana/metrictank

branch : fix-mt-fakemetrics

created branch time in 13 days

push eventgrafana/metrictank

Dieter Plaetinck

commit sha 213158650f2a38e99b2e3cf7a2cd16cfcaf3e0f1

alignment: better, more comprehensive testing

view details

Dieter Plaetinck

commit sha 88073aaf17fe107608452dfd63423912261122da

simpler align.{Backward,Forward}IfNotAligned: no need to branch

view details

Robert Milan

commit sha 2a7ac28f9044333282b81c685705a5e1e44daa86

Merge pull request #1803 from grafana/better-alignment-testing Faster, simpler alignment methods, better testing

view details

push time in a month

delete branch grafana/metrictank

delete branch : better-alignment-testing

delete time in a month

PR merged grafana/metrictank

Faster, simpler alignment methods, better testing

might be easiest to review by looking at the tests first

+62 -27

0 comment

2 changed files

Dieterbe

pr closed time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 There are 3 levels of support: * Stable : 100% compatible with graphite and vetted * Unstable: not fully compatible yet or not vetted enough -When you request functions that metrictank cannot provide, it will automatically proxy requests to graphite for a seamless failover.+When you request functions that metrictank cannot provide, it will automatically, seamlessly proxy requests to graphite.
When you request functions that Metrictank cannot provide, it will automatically, seamlessly proxy requests to graphite.
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 There are 3 levels of support: * Stable : 100% compatible with graphite and vetted * Unstable: not fully compatible yet or not vetted enough -When you request functions that metrictank cannot provide, it will automatically proxy requests to graphite for a seamless failover.+When you request functions that metrictank cannot provide, it will automatically, seamlessly proxy requests to graphite.
When you request functions that Metrictank cannot provide, it will automatically, seamlessly proxy requests to graphite.
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 # Graphite -Metrictank aims to be a drop-in replacement for Graphite, but also to address a few of Graphite's shortcomings.-Here are some important functional differences to keep in mind:-(we specifically do not go into subjective things like performance or scalability here)+For a general overview of how Metrictank relates and compares to Graphite, please see the [Readme](../README.md) -* currently no support for rewriting old data; for a given key and timestamp first write wins, not last. We aim to fix this.-* timeseries can change resolution (interval) over time, they will be merged seamlessly at read time.-* multiple rollup functions are supported and can be selected via consolidateBy() at query time. (except when using functions which change the nature of the data such as perSecond() etc)-* xFilesfactor is currently not supported-* will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)-* graphite timezone defaults to Chicago, we default to server time-* many functions are not implemented yet in metrictank itself, but it autodetects this and will proxy requests it cannot handle to graphite-web-  (which then uses metrictank as a simple backend).  See below for details+## Caveats +There are some small behavioral and functional differences with Graphite:++* Currently no support for rewriting old data; There is a reorder-buffer to support out-of-order writes to an extent.  Full archived data rewriting is on the roadmap.+* Will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)+* Graphite timezone defaults to Chicago, we default to server time+* xFilesfactor is currently not supported for rollups. Fairly easy to address, but haven't had a need for it yet.+* Graphite supports render formats csv, json, dygraph, msgpack, pickle, png, pdf, raw, rickshaw, svg.+  Metrictank only implements json, msgp, msgpack, pickle. Grafana only uses json. In particular, Metrictank does not render images, because Grafana renders great.+* Some less commonly used functions are not implemented yet in metrictank itself, but Metrictank can seamlessly proxy those to graphite-web (see below for details)+  At Grafana Labs, 90 to 95 % of requests get handled by metrictank without involving Graphite.
  At Grafana Labs, 90 to 95 % of requests get handled by Metrictank without involving Graphite.
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 # Graphite -Metrictank aims to be a drop-in replacement for Graphite, but also to address a few of Graphite's shortcomings.-Here are some important functional differences to keep in mind:-(we specifically do not go into subjective things like performance or scalability here)+For a general overview of how Metrictank relates and compares to Graphite, please see the [Readme](../README.md) -* currently no support for rewriting old data; for a given key and timestamp first write wins, not last. We aim to fix this.-* timeseries can change resolution (interval) over time, they will be merged seamlessly at read time.-* multiple rollup functions are supported and can be selected via consolidateBy() at query time. (except when using functions which change the nature of the data such as perSecond() etc)-* xFilesfactor is currently not supported-* will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)-* graphite timezone defaults to Chicago, we default to server time-* many functions are not implemented yet in metrictank itself, but it autodetects this and will proxy requests it cannot handle to graphite-web-  (which then uses metrictank as a simple backend).  See below for details+## Caveats +There are some small behavioral and functional differences with Graphite:++* Currently no support for rewriting old data; There is a reorder-buffer to support out-of-order writes to an extent.  Full archived data rewriting is on the roadmap.+* Will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)+* Graphite timezone defaults to Chicago, we default to server time+* xFilesfactor is currently not supported for rollups. Fairly easy to address, but haven't had a need for it yet.+* Graphite supports render formats csv, json, dygraph, msgpack, pickle, png, pdf, raw, rickshaw, svg.+  Metrictank only implements json, msgp, msgpack, pickle. Grafana only uses json. In particular, Metrictank does not render images, because Grafana renders great.+* Some less commonly used functions are not implemented yet in metrictank itself, but Metrictank can seamlessly proxy those to graphite-web (see below for details)
* Some less commonly used functions are not implemented yet in Metrictank itself, but Metrictank can seamlessly proxy those to graphite-web (see below for details)
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 # Graphite -Metrictank aims to be a drop-in replacement for Graphite, but also to address a few of Graphite's shortcomings.-Here are some important functional differences to keep in mind:-(we specifically do not go into subjective things like performance or scalability here)+For a general overview of how Metrictank relates and compares to Graphite, please see the [Readme](../README.md) -* currently no support for rewriting old data; for a given key and timestamp first write wins, not last. We aim to fix this.-* timeseries can change resolution (interval) over time, they will be merged seamlessly at read time.-* multiple rollup functions are supported and can be selected via consolidateBy() at query time. (except when using functions which change the nature of the data such as perSecond() etc)-* xFilesfactor is currently not supported-* will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)-* graphite timezone defaults to Chicago, we default to server time-* many functions are not implemented yet in metrictank itself, but it autodetects this and will proxy requests it cannot handle to graphite-web-  (which then uses metrictank as a simple backend).  See below for details+## Caveats +There are some small behavioral and functional differences with Graphite:++* Currently no support for rewriting old data; There is a reorder-buffer to support out-of-order writes to an extent.  Full archived data rewriting is on the roadmap.+* Will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)+* Graphite timezone defaults to Chicago, we default to server time+* xFilesfactor is currently not supported for rollups. Fairly easy to address, but haven't had a need for it yet.
* xFilesfactor is currently not supported for rollups. It is fairly easy to address, but we haven't had a need for it yet.
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 # Graphite -Metrictank aims to be a drop-in replacement for Graphite, but also to address a few of Graphite's shortcomings.-Here are some important functional differences to keep in mind:-(we specifically do not go into subjective things like performance or scalability here)+For a general overview of how Metrictank relates and compares to Graphite, please see the [Readme](../README.md) -* currently no support for rewriting old data; for a given key and timestamp first write wins, not last. We aim to fix this.-* timeseries can change resolution (interval) over time, they will be merged seamlessly at read time.-* multiple rollup functions are supported and can be selected via consolidateBy() at query time. (except when using functions which change the nature of the data such as perSecond() etc)-* xFilesfactor is currently not supported-* will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)-* graphite timezone defaults to Chicago, we default to server time-* many functions are not implemented yet in metrictank itself, but it autodetects this and will proxy requests it cannot handle to graphite-web-  (which then uses metrictank as a simple backend).  See below for details+## Caveats +There are some small behavioral and functional differences with Graphite:++* Currently no support for rewriting old data; There is a reorder-buffer to support out-of-order writes to an extent.  Full archived data rewriting is on the roadmap.+* Will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)+* Graphite timezone defaults to Chicago, we default to server time+* xFilesfactor is currently not supported for rollups. Fairly easy to address, but haven't had a need for it yet.+* Graphite supports render formats csv, json, dygraph, msgpack, pickle, png, pdf, raw, rickshaw, svg.
* Graphite supports the following render formats: csv, json, dygraph, msgpack, pickle, png, pdf, raw, rickshaw, and svg.
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 # Graphite -Metrictank aims to be a drop-in replacement for Graphite, but also to address a few of Graphite's shortcomings.-Here are some important functional differences to keep in mind:-(we specifically do not go into subjective things like performance or scalability here)+For a general overview of how Metrictank relates and compares to Graphite, please see the [Readme](../README.md) -* currently no support for rewriting old data; for a given key and timestamp first write wins, not last. We aim to fix this.-* timeseries can change resolution (interval) over time, they will be merged seamlessly at read time.-* multiple rollup functions are supported and can be selected via consolidateBy() at query time. (except when using functions which change the nature of the data such as perSecond() etc)-* xFilesfactor is currently not supported-* will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)-* graphite timezone defaults to Chicago, we default to server time-* many functions are not implemented yet in metrictank itself, but it autodetects this and will proxy requests it cannot handle to graphite-web-  (which then uses metrictank as a simple backend).  See below for details+## Caveats +There are some small behavioral and functional differences with Graphite:++* Currently no support for rewriting old data; There is a reorder-buffer to support out-of-order writes to an extent.  Full archived data rewriting is on the roadmap.+* Will never move observations into the past (e.g. consolidation and rollups will only cause data to get an equal or higher timestamp)+* Graphite timezone defaults to Chicago, we default to server time+* xFilesfactor is currently not supported for rollups. Fairly easy to address, but haven't had a need for it yet.+* Graphite supports render formats csv, json, dygraph, msgpack, pickle, png, pdf, raw, rickshaw, svg.+  Metrictank only implements json, msgp, msgpack, pickle. Grafana only uses json. In particular, Metrictank does not render images, because Grafana renders great.
  Metrictank only implements json, msgp, msgpack, and pickle. Grafana only uses json. In particular, Metrictank does not render images, because Grafana renders great.
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

 that makes this process much easier. ## Features  * 100% open source-* Inspired by the [Facebook gorilla paper](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf).-Most notably, the heavily compressed chunks dramatically lower cpu, memory and storage requirements.-* Writeback RAM cache, serving most data out of memory.-* Graphite is a first class citizen. As of graphite-1.0.1, metrictank can be used as a graphite CLUSTER_SERVER.-* Can also act as a Graphite server itself, though the functions processing library is only partially implemented, metrictank proxies requests to Graphite if it can't handle the required processing (for those requests it will degrade to just being the backend storage)-* Accurate, flexible rollups by storing min/max/sum/count (which also gives us average).+* Heavily compressed chunks (inspired by the [Facebook gorilla paper](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf)) dramatically lower cpu, memory and storage requirements and get much greater performance out of Cassandra than other Cassandra based solutions.
* Heavily compressed chunks (inspired by the [Facebook gorilla paper](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf)) dramatically lower cpu, memory, and storage requirements and get much greater performance out of Cassandra than other solutions.
Dieterbe

comment created time in a month

Pull request review commentgrafana/metrictank

Rework graphite docs

  ## Introduction -Metrictank is a multi-tenant timeseries engine for Graphite and friends.+Metrictank is a multi-tenant timeseries platform, that can be used as a backend or replacement for Graphite.
Metrictank is a multi-tenant timeseries platform that can be used as a backend or replacement for Graphite.
Dieterbe

comment created time in a month

push eventgrafana/metrictank

Robert Milan

commit sha 07176fd209ae7e1e3f54e196d766a96e68827c27

re-add recording language to community call

view details

Robert Milan

commit sha 4e160a945cf7eefd3e12edcdd6f9f100a3c5fef2

Merge pull request #1790 from grafana/update-call-md re-add recording language to community call

view details

push time in a month

delete branch grafana/metrictank

delete branch : update-call-md

delete time in a month

create barnchgrafana/metrictank

branch : update-call-md

created branch time in a month

push eventgrafana/metrictank

Robert Milan

commit sha 56b1f33e602d7fb95ceb2412474a65943a181b46

add values policy to mt-fakemetrics

view details

Robert Milan

commit sha ad7fab68dae38ac9b8cf85a4e2c60b6943f6f15c

fix tools documentation

view details

Robert Milan

commit sha 03b0bc1d69479c5c46bf20ee35d1f8120e3cea8f

update fakemetrics readme

view details

Robert Milan

commit sha eec25a8985c12520ae3b8311adf4c470076ed449

use interface for policies

view details

Robert Milan

commit sha 69eacc55766b9b0b8e056f643af111455987ce59

minor cleanup

view details

Robert Milan

commit sha 9b56c02992fec8558ec5e77b04ef4d7cf9d063f2

minor fixes

view details

Robert Milan

commit sha 91e1c3df45619bc3de4c4ef0afecac47d0d18422

change None policy to Random

view details

Robert Milan

commit sha 3ae28ab08ea1f0f8ef029e0eca4fc785203c36b1

Merge pull request #1773 from grafana/use-policies-in-fakemetrics add values policy to mt-fakemetrics

view details

push time in a month

delete branch grafana/metrictank

delete branch : use-policies-in-fakemetrics

delete time in a month

PR merged grafana/metrictank

Reviewers
add values policy to mt-fakemetrics

This PR adds a value-policy option to mt-fakemetrics. It is explained in greater detail in the tool's README file.

It adds 3 options initially:

  1. single
  2. multiple
  3. timestamp

If no policy is specified the default behavior of generating a random float is still used.

+188 -10

0 comment

7 changed files

robert-milan

pr closed time in a month

push eventgrafana/metrictank

Robert Milan

commit sha 91e1c3df45619bc3de4c4ef0afecac47d0d18422

change None policy to Random

view details

push time in a month

Pull request review commentgrafana/metrictank

add values policy to mt-fakemetrics

+package policy++import (+	"fmt"+	"math/rand"+	"strconv"+	"strings"+)++type ValuePolicy interface {+	Value(ts int64) float64+}++type ValuePolicyNone struct {

Yes, that makes sense

robert-milan

comment created time in a month

push eventgrafana/metrictank

Robert Milan

commit sha 9b56c02992fec8558ec5e77b04ef4d7cf9d063f2

minor fixes

view details

push time in a month

push eventgrafana/metrictank

Robert Milan

commit sha 69eacc55766b9b0b8e056f643af111455987ce59

minor cleanup

view details

push time in a month

push eventgrafana/metrictank

Robert Milan

commit sha eec25a8985c12520ae3b8311adf4c470076ed449

use interface for policies

view details

push time in a month

Pull request review commentgrafana/metrictank

add values policy to mt-fakemetrics

 func init() { 	rootCmd.PersistentFlags().StringSliceVar(&customTags, "custom-tags", []string{}, "A list of comma separated tags (i.e. \"tag1=value1,tag2=value2\")(default empty) conflicts with add-tags") 	rootCmd.PersistentFlags().IntVar(&numUniqueCustomTags, "num-unique-custom-tags", 0, "a number between 0 and the length of custom-tags. when using custom-tags this will make the tags unique (default 0)") +	rootCmd.PersistentFlags().StringVar(&valuePolicy, "value-policy", "", "a value policy (i.e. \"single:1\" \"multiple:1,2,3,4,5\" \"timestamp\")")

So, the tags flags in root should also be moved to feed and backfill?

robert-milan

comment created time in a month

Pull request review commentgrafana/metrictank

add values policy to mt-fakemetrics

 The speed is 100x what it would be if it were realtime (so a rate of 4x100x100=4 mt-fakemetrics backfill --kafka-mdm-addr localhost:9092 --offset 5h --period 10s --speedup 100 --orgs 4 --mpo 100 ``` -# Outputs+## Value Policies++You can specify a Value Policy with the `--value-policy` flag. Currently the following Value Policies are supported:++* single+* multiple+* timestamp

The default behavior with no policy specified is still the same, it will generate a random float. I am updating the documentation to reflect this.

robert-milan

comment created time in a month

Pull request review commentgrafana/metrictank

add values policy to mt-fakemetrics

 func dataFeed(outs []out.Out, orgs, mpo, period, flush, offset, speedup int, sto 		panic(fmt.Sprintf("num-unique-custom-tags must be a value between 0 and %d, you entered %d", len(customTags), numUniqueCustomTags)) 	} +	vp, err := policy.ParseValuePolicy(valuePolicy)

It's a global/root flag, that's why it is in root and doesn't get parsed in the caller functions. If you look at feed or backfill you will see that they duplicate code / settings individually. Since this settings applies to everything, I have it broken out this way. Otherwise we have to parse it in each of those other places which may or may not use it. The better question is why feed and backfill duplicate all those settings instead of just setting them in root.

robert-milan

comment created time in a month

Pull request review commentgrafana/metrictank

add values policy to mt-fakemetrics

 times %4d orgs: each %s, flushing %d metrics so rate of %d Hz. (%d total unique 				// we already sent, so we must increase the timestamp 				if m == 0 { 					ts += mp+					if vp.Policy == policy.Multiple {+						if (idxValues + 1) >= lenValues {+							idxValues = 0+						} else {+							idxValues+++						}+					} 				} 				metricData.Time = ts-				metricData.Value = rand.Float64() * float64(m+1)++				switch vp.Policy {+				case policy.None:

I thought about doing that initially. I don't have any issues with it.

robert-milan

comment created time in a month

push eventgrafana/metrictank

Robert Milan

commit sha 03b0bc1d69479c5c46bf20ee35d1f8120e3cea8f

update fakemetrics readme

view details

push time in a month

push eventgrafana/metrictank

Robert Milan

commit sha ad7fab68dae38ac9b8cf85a4e2c60b6943f6f15c

fix tools documentation

view details

push time in a month

PR opened grafana/metrictank

add values policy to mt-fakemetrics

This PR adds a value-policy option to mt-fakemetrics. It is explained in greater detail in the tool's README file.

It adds 3 options initially:

  1. single
  2. multiple
  3. timestamp
+166 -5

0 comment

4 changed files

pr created time in a month

create barnchgrafana/metrictank

branch : use-policies-in-fakemetrics

created branch time in a month

PR closed grafana/metrictank

Reviewers
add options to use custom values with fakemetrics

This PR adds the ability to use fixed custom values to the fakemetrics tool. This should make it easier to run certain types of tests.

+54 -1

3 comments

3 changed files

robert-milan

pr closed time in a month

pull request commentgrafana/metrictank

add options to use custom values with fakemetrics

Will update as discussed in comments and open a different PR

robert-milan

comment created time in a month

pull request commentgrafana/metrictank

add options to use custom values with fakemetrics

this is already many flags to support only a small number of possible value types. i'ld like us to be able to add more types in the future, which would only make this worse.

how about --values <valuepolicy> where policy can be a floating point value (for fixed values), a list of floating point values, something like random:<lower>,<upper>, etc?

I'm ok with changing it, but we do need the following options at a minimum:

  1. Use a single fixed value for all points
  2. Use a list of fixed values that repeats for all points
  3. Use the timestamp as the value for each point
robert-milan

comment created time in a month

push eventgrafana/metrictank

Robert Milan

commit sha 59a79d9de2e9490bdd2706b9c3d0d444db69cdd6

add more checks to fixed-values

view details

push time in a month

PR opened grafana/metrictank

Reviewers
add options to use custom values with fakemetrics

This PR adds the ability to use fixed custom values to the fakemetrics tool. This should make it easier to run certain types of tests.

+50 -1

0 comment

3 changed files

pr created time in a month

create barnchgrafana/metrictank

branch : add-custom-values-to-fakemetrics

created branch time in a month

push eventgrafana/piechart-panel

Robert Milan

commit sha b8921deaf74356293196616241c57747293f4089

Update default maxDataPoints to 1 instead of 3

view details

Robert Milan

commit sha 304a575b226621ec25448572a868e41b479e7313

Merge pull request #221 from grafana/change-max-data-points Update default maxDataPoints to 1 instead of 3

view details

push time in 2 months

PR merged grafana/piechart-panel

Update default maxDataPoints to 1 instead of 3

This PR updates maxDataPoints default to 1. A few related issues:

https://github.com/grafana/piechart-panel/issues/42 https://github.com/grafana/metrictank/issues/983

Basically it's a problem with how runtime consolidation in Metrictank / Graphite deals with a small maxDataPoints. Setting it to 1 as default should, in most cases, be a better option and give more accurate and reproducible query results as a time window shifts to the future.

+1 -1

1 comment

1 changed file

robert-milan

pr closed time in 2 months

pull request commentgrafana/metrictank

stop double closing the index

The interesting thing about this PR is that this should fix a panic (ugly although harmless) on shutdown, correct @robert-milan ?

Yes this should fix the panic on shutdown

robert-milan

comment created time in 2 months

push eventgrafana/metrictank

Robert Milan

commit sha f7f1eb9b0ba4a0cafd54e4655d9cac9b8fc98f3d

change where metricIndex is stopped

view details

push time in 2 months

PR opened grafana/metrictank

stop double closing the index

The index is already closed when apiServer.Stop() is called. Removing the second call to Stop() the index to avoid a panic during shutdown.

+0 -2

0 comment

1 changed file

pr created time in 2 months

create barnchgrafana/metrictank

branch : fix-double-close-channel-panic

created branch time in 2 months

push eventgrafana/metrictank

Robert Milan

commit sha fdaacec0ab314a2021b071dddde206ee0605fb48

add probot-stale configuration

view details

Robert Milan

commit sha 18a3644032c920be884742342d27f608100c44b9

change label to stale

view details

Robert Milan

commit sha d1c01c22b3fde8c78286bf0f663a08c4a0b5b675

Merge pull request #1741 from grafana/add-stale-bot-config add probot-stale configuration

view details

push time in 2 months

delete branch grafana/metrictank

delete branch : add-stale-bot-config

delete time in 2 months

push eventgrafana/metrictank

Robert Milan

commit sha 18a3644032c920be884742342d27f608100c44b9

change label to stale

view details

push time in 2 months

Pull request review commentgrafana/metrictank

add probot-stale configuration

+# Number of days of inactivity before an issue becomes stale+daysUntilStale: 90++# Number of days of inactivity before a stale issue is closed+daysUntilClose: 7++# Issues with these labels will never be considered stale+exemptLabels:+  - pinned+  - security+  - Epic++  # Set to true to ignore issues in a milestone (defaults to false)+exemptMilestones: true++# Set to true to ignore issues with an assignee (defaults to false)+exemptAssignees: false++# Label to use when marking an issue as stale+staleLabel: wontfix

fair enough

robert-milan

comment created time in 2 months

pull request commentgrafana/metrictank

update names

I disagree. Consistency is important. There were a bunch of places where sometimes Metrictank and Graphite were capitalized, sometimes not. Since we are changing / fixing everything, we should just apply it everywhere, as it is a proper name. Even if we were to merge this PR as is, there are still a few places that need to be updated (because of the docs scripts).

I'm open to only applying this to public facing documentation I guess, but I don't think the comparison to Loki makes much sense. They may suffer from the same problems we do, i.e. sometimes we just forget or write it differently.

robert-milan

comment created time in 2 months

Pull request review commentgrafana/metrictank

update names

 Otherwise data loss of current chunks will be incurred.  See [operations guide]( License ======= -Copyright 2016-2019 Grafana Labs+Copyright 2016-2020 Grafana Labs

Also not sure about this change

robert-milan

comment created time in 2 months

Pull request review commentgrafana/metrictank

update names

-Copyright 2016-2018 Dieter Plaetinck, Anthony Woods, Jeremy Bingham, Damian Gryski, raintank inc+Copyright 2016-2020 Dieter Plaetinck, Anthony Woods, Jeremy Bingham, Damian Gryski, raintank inc

Wasn't completely sure about this change.

robert-milan

comment created time in 2 months

push eventgrafana/metrictank

Robert Milan

commit sha 276acd9e006aeea6b6f782f942fad928a4b2b323

update tools

view details

push time in 2 months

push eventgrafana/metrictank

Robert Milan

commit sha 255dac764707d8264fa03f1b02fdd305877ac1aa

update docs

view details

push time in 2 months

push eventgrafana/metrictank

agao48

commit sha 31e90cc0d4b7535fe2c7df8ff345f49153ecb4ca

Add constant line function (#1734) * Remove check for reqs.count == 0 * Add constantLine function * Format string differently * Add initial simple test for constantLine * Clean up simple test * Make constant line test more modular * Flesh out constantLIne tests * constantLine to stable, update docs * Fix test naming * Use time package in constantLine testing * go format files * Handle edge cases for constantLine * Change constantLine from/to -> first/last. Add comment explaning constantLine timestamps. * Gofmt

view details

Robert Milan

commit sha b520e914957063c6be72d251d92aa6dd1eb97c83

update names

view details

push time in 2 months

PR opened grafana/metrictank

update names

This PR is the first step to changing Metrictank to Grafana Metrictank.

Metrictank is a Grafana Labs project and should be more closely associated with it.

+371 -370

0 comment

66 changed files

pr created time in 2 months

create barnchgrafana/metrictank

branch : change-name

created branch time in 2 months

create barnchgrafana/metrictank

branch : update-community-docs-to-include-call

created branch time in 2 months

PR opened grafana/metrictank

add probot-stale configuration
+29 -0

0 comment

1 changed file

pr created time in 2 months

create barnchgrafana/metrictank

branch : add-stale-bot-config

created branch time in 2 months

Pull request review commentgrafana/metrictank

Use new archive type for cleaner non-standard archinterval handling

 func (r ReqMap) Dump() string { 	return out } -// Requests indexed by their retention ID-type ReqsByRet [][]models.Req+// Requests indexed by their ArchivesID+type ReqsByArchives [][]models.Req -// OutInterval returns the outinterval of the ReqsByRet+func (rba *ReqsByArchives) Add(idx int, r models.Req) {

Is there a reason you chose to modify the slice by pointer here instead of the normal pattern of just returning the new slice?

Dieterbe

comment created time in 2 months

push eventgrafana/metrictank

Robert Milan

commit sha 0e664b3840cf0c4a9b6021a68c8df25c20146133

fix typo in calendar link

view details

Robert Milan

commit sha 0ddb6f55ef6c9e64011a21ad533ea397dd83af39

Merge pull request #1737 from grafana/fix-typo-in-call-markdown fix typo in calendar link

view details

push time in 2 months

delete branch grafana/metrictank

delete branch : fix-typo-in-call-markdown

delete time in 2 months

PR opened grafana/metrictank

fix typo in calendar link
+1 -1

0 comment

1 changed file

pr created time in 2 months

create barnchgrafana/metrictank

branch : fix-typo-in-call-markdown

created branch time in 2 months

PR opened grafana/piechart-panel

Update default maxDataPoints to 1 instead of 3

This PR updates maxDataPoints default to 1. A few related issues:

https://github.com/grafana/piechart-panel/issues/42 https://github.com/grafana/metrictank/issues/983

Basically it's a problem with how runtime consolidation in Metrictank / Graphite deals with a small maxDataPoints. Setting it to 1 as default should, in most cases, be a better option and give more accurate and reproducible query results as a time window shifts to the future.

+1 -1

0 comment

1 changed file

pr created time in 2 months

create barnchgrafana/piechart-panel

branch : change-max-data-points

created branch time in 2 months

delete branch grafana/metrictank

delete branch : add-community-call-guidelines

delete time in 2 months

push eventgrafana/metrictank

Robert Milan

commit sha 6cd1e0d04360463151b69bfc011026f9191a3806

add public community call information

view details

Robert Milan

commit sha d7c2c7215beb178f7ac778c25817d81f163ff287

Update community/CALL.md Co-Authored-By: Sean Hanson <shanson7@bloomberg.net>

view details

Robert Milan

commit sha aa324297b2c17bd661fd5c673dbb7506f6679a7b

Update community/CALL.md Co-Authored-By: Sean Hanson <shanson7@bloomberg.net>

view details

Robert Milan

commit sha 8302e1ba37c91a14210f3383e7dcd7692bd5dda6

Merge pull request #1732 from grafana/add-community-call-guidelines add public community call information

view details

push time in 2 months

PR opened grafana/metrictank

clarify new team member voting rights

Adds a note to clarify new team member voting rights

+2 -0

0 comment

1 changed file

pr created time in 2 months

more