profile
viewpoint
Aliaksandr Valialkin valyala @VictoriaMetrics Kyiv https://victoriametrics.com Working on @VictoriaMetrics

valyala/fasthttp 11520

Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

valyala/fastjson 881

Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

valyala/gorpc 577

Simple, fast and scalable golang rpc library for high load

valyala/fasttemplate 349

Simple and fast template engine for Go

valyala/bytebufferpool 286

Anti-memory-waste byte buffer pool

valyala/goloris 232

Slowloris for nginx DoS. Written in go

valyala/gozstd 224

go wrapper for zstd

valyala/gheap 92

Fast generalized heap tree algorithms in C++ and C. Provides simultaneous support for D-heap and B-heap.

valyala/fastrand 76

Fast and scalable pseudorandom generator for Go

valyala/fastrpc 67

Building blocks for fast rpc systems

issue commentVictoriaMetrics/VictoriaMetrics

Label rewrites/metric rewrite configs

There is no a dedicated ticket for vmagent component yet. There is only a comment at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/260#issuecomment-565014679 , which describes its functionality.

AeroNotix

comment created time in 9 hours

issue commentVictoriaMetrics/VictoriaMetrics

vminsert 1.33.1 performance regression

@johnatannvmd , thanks for the bug report!

Could you try other releases between v1.30.4 and v1.31.1 in order to detect the first release where the issue has been introduced?

I'd also recommend returning back -memory.allowedPercent to default value unless you have important reason to increase it. Default value for -memory.allowedPercent should work well in the majority of cases. See troubleshooting section for more info.

johnatannvmd

comment created time in 11 hours

issue commentVictoriaMetrics/VictoriaMetrics

Metrics Values ​​differ between Promtheus and VictoriaMetrics (sum(...) by)

@govargo , could you provide Prometheus and VictoriaMetrics graphs for the following query:

stackdriver_spanner_instance_spanner_googleapis_com_instance_cpu_utilization_by_priority{instance_id="$spanner_env",is_system="false"}

These graphs should show the difference between time series before applying sum to them. Please remove user - {{ priority }} string from the Legend field and leave blank the field when performing the query above, so label sets for the returned time series could be compared between Prometheus and VictoriaMetrics.

govargo

comment created time in 11 hours

issue commentVictoriaMetrics/VictoriaMetrics

Label rewrites/metric rewrite configs

@AeroNotix , thanks for the feature request! It looks useful.

This task will be solved by the upcoming vmagent component. This component will accept data via all the ingestion protocols supported by VictoriaMetrics and proxy it to VictoriaMetrics. The component will support metrics relabeling with relabel_configs from Prometheus.

AeroNotix

comment created time in 11 hours

pull request commentVictoriaMetrics/VictoriaMetrics

basic vmalert backbone

Thanks!

kreedom

comment created time in 2 days

delete branch VictoriaMetrics/VictoriaMetrics

delete branch : vmalert/backbone

delete time in 2 days

push eventVictoriaMetrics/VictoriaMetrics

kreedom

commit sha 3c06179184d13ec22b463778541fe3b82e842c16

basic vmalert backbone (#317) * basic vmalert backbone * Resolve code review comments for vmalert backbone * Second review fixes for vmalert backbone

view details

push time in 2 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 32e153e83432fe3e5932dbfcdb834db4f1555bce

lib/storage: prevent from clobbering nin-nil lastError in Storage.add

view details

Aliaksandr Valialkin

commit sha ce15cecae4f8ad7001cf26bef24f5f49ad956215

lib/storage: typo fix

view details

Aliaksandr Valialkin

commit sha e7ba18b0d995431180a0cfd4403adf6cfdd184da

vendor: `make vendor-udpate`

view details

Aliaksandr Valialkin

commit sha 71a52f5f9062a1538f4146d72be98b3b4f79d35d

lib/protoparser/prometheus: skip leading whitespace from tag names

view details

push time in 2 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha ad4cb9f3cad42818ea39b1551a658d82bc069773

lib/storage: prevent from clobbering nin-nil lastError in Storage.add

view details

Aliaksandr Valialkin

commit sha a2b81b71b980d30945be2a92b3fec46bd9c5e69f

lib/storage: typo fix

view details

Aliaksandr Valialkin

commit sha d687e5518d7cb6ce7ab45ad47b325f4882ad1954

vendor: `make vendor-udpate`

view details

Aliaksandr Valialkin

commit sha fb5848f536f94672f647918a06b6c05fc396d2b7

Makefile: add missing `vmbackup` and `vmrestore` to `all` and `all-pure` targets

view details

Aliaksandr Valialkin

commit sha 9d279e26a7745d2976cde6c608240ee2ff9f3e0e

lib/protoparser/prometheus: skip leading whitespace from tag names

view details

push time in 2 days

Pull request review commentVictoriaMetrics/VictoriaMetrics

basic vmalert backbone

+package main++import (+	"flag"+	"net/http"++	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/config"+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"+)++var (+	configPath     = flag.String("config", "config.yaml", "Path to alert configuration file")+	httpListenAddr = flag.String("httpListenAddr", ":8880", "Address to listen for http connections")+)++func main() {+	buildinfo.Init()

missing envflag.Parse() call. Otherwise flag values won't be read from command line and environment vars.

kreedom

comment created time in 3 days

issue commentVictoriaMetrics/VictoriaMetrics

【too big data size】vmstorage can't process specific query

I published docker images for v1.26.1-cluster, so you can upgrade to this release in order to detect the original error message.

govargo

comment created time in 4 days

issue commentVictoriaMetrics/VictoriaMetrics

Metrics Values ​​differ between Promtheus alone and aggregated VictoriaMetrics

@govargo , thanks for the detailed bug report!

There was a fix for histogram_quantile function after v1.26.0 in the commit 19dfe52254378b359dd4e14acbb3771c04536c35 . A few additional commits touched the histogram_quantile function as well after v1.26.0:

  • c4287b3c8663945ff59e88e71b718143df9c6ba4
  • 7753c8c0a1a08fa428d26040801084a6a47c141c
  • 8582b503603f5ba487de5cdcc36f1c31250b58c4
  • 73b2a3d4b7afa2f5eafe91bb19516a5cb300cb28

So it would be great checking for the issue in the latest release.

Additionally, using irate inside histogram_quantile could result in jumpy graphs after each refresh. It is better using either rate or increase instead. See this article for details. Could you try substituting irate in the queries above with rate or increase and see whether the discrepancy between Prometheus and VictoriaMetrics remains?

If the discrepancy remains, then it would be great attaching graphs for the inner query inside histogram_quantile for both Prometheus and VictoriaMetrics, i.e.

sum(rate(nginx_http_request_duration_seconds_bucket{kubernetes_namespace=~"$namespace",host!=""}[2m])) by (app,le)

These graphs should help determining the root cause of the issue.

govargo

comment created time in 4 days

issue closedVictoriaMetrics/VictoriaMetrics

Constant growth of goroutines number at vmstorage

Describe the bug

Running VictoriaMetrics cluster v1.33.0 with 3 vmstorage instances. Observing constant increase in a number of goroutines at vmstorage components. Had same behavior with v1.32.6.

To Reproduce

These issues just started happening after upgrade from 1.31.5 to 1.32.6.

Screenshots

Drops to zero are caused by vmstorage restarts.

vmstorage-goroutines

Version

vmstorage-20200204-220434-tags-v1.33.0-cluster-0-g8e77b548

Used command-line flags

vmstorage flags:

flag{name="bigMergeConcurrency", value="0"} 1
flag{name="dedup.minScrapeInterval", value="0s"} 1
flag{name="enableTCP6", value="false"} 1
flag{name="fs.disableMmap", value="false"} 1
flag{name="http.disableResponseCompression", value="false"} 1
flag{name="httpListenAddr", value=":8482"} 1
flag{name="loggerFormat", value="default"} 1
flag{name="loggerLevel", value="INFO"} 1
flag{name="loggerOutput", value="stderr"} 1
flag{name="memory.allowedPercent", value="60"} 1
flag{name="precisionBits", value="64"} 1
flag{name="retentionPeriod", value="6"} 1
flag{name="rpc.disableCompression", value="false"} 1
flag{name="search.maxTagKeys", value="secret"} 1
flag{name="search.maxTagValues", value="100000"} 1
flag{name="search.maxUniqueTimeseries", value="300000"} 1
flag{name="smallMergeConcurrency", value="0"} 1
flag{name="snapshotAuthKey", value="secret"} 1
flag{name="storageDataPath", value="/vmstorage-data"} 1
flag{name="version", value="false"} 1

Additional context

Also attaching /debug/pprof/goroutine from every vmstorage instance:

vm-pprof.zip

closed time in 4 days

pavdmyt

issue commentVictoriaMetrics/VictoriaMetrics

Constant growth of goroutines number at vmstorage

@pavdmyt , thanks for the confirmation! Closing the issue as fixed.

pavdmyt

comment created time in 4 days

issue commentVictoriaMetrics/VictoriaMetrics

【too big data size】vmstorage can't process specific query

The commit that limits the maximum error message size, which can be sent from vmstorage to vminsert, has been included into v1.33.1 release. @govargo , could you check either v1.26.1 as described above or try v1.33.1?

govargo

comment created time in 5 days

issue closedVictoriaMetrics/VictoriaMetrics

Add support to environment variables in addition to flags

Is your feature request related to a problem? Please describe. Currently the only way to configure VM is thru launch flags, while this is not itself an issue it could be a bit problematic for some options, let's say -snapshotAuthKey, as it leaks in the processes table the auth key.

Describe the solution you'd like Supporting environment files would allow to pass the auth key in a more "stealthy" manner. For example, by supplying an env file to a systemd unit file, the authKey could be passed on vmstorage without the need to be a flag.

Additional context Supporting env file could bring more than just protect the authKey: it is the most standard way to configure docker containers for example and would simplify a lot starting VM with systemd. For example, we actually start our storage nodes this way:

[Unit]
Description=Victoria Metrics - cluster version: storage module
After=network-ready.target

[Service]
Type=simple
User=vmc-storage
EnvironmentFile=/etc/default/vmc-storage
ExecStart=/usr/bin/vmc-storage -bigMergeConcurrency "$BigMergeConcurrency" -dedup.minScrapeInterval "$MinScrapeInterval" -httpListenAddr "$HTTPListenAddr" -loggerLevel "$LoggerLevel" -memory.allowedPercent "$MemoryAllowedPercent" -precisionBits "$PrecisionBits" -retentionPeriod "$RetentionPeriod" -search.maxTagKeys "$SearchMaxTagKeys" -search.maxTagValues "$SearchMaxTagValues" -search.maxUniqueTimeseries "$SearchMaxUniqueTimeseries" -smallMergeConcurrency "$SmallMergeConcurrency" -snapshotAuthKey "$SnapshotAuthKey" -storageDataPath "$StorageDataPath" -vminsertAddr "$VMInsertAddr" -vmselectAddr "$VMSelectAddr" $Others
WorkingDirectory=/var/lib/vmc-storage
Restart=on-failure

[Install]
WantedBy=multi-user.target

With /etc/default/vmc-storage declaring all of theses variables (and thus acting as a config file). With env var support, the ExecStart would be much cleaner/nicer (just the binary with no flags) and adding an option/value to the env file would not require to edit the unit file (or use an ugly hack as the $Others var we use) as all vars declared in this file would automatically be passed to the binary.

Just as reference, some Golang cli libs support for a given option both launch flags and env vars: https://github.com/urfave/cli/blob/master/docs/v2/manual.md#values-from-the-environment This allow the customize the default value of flags from env but still allow overwrite thru flag.

I think that kind of approach would be ideal :)

closed time in 5 days

hekmon

issue commentVictoriaMetrics/VictoriaMetrics

Add support to environment variables in addition to flags

Flags can be read from env vars starting from v1.33.1. Closing this feature request as implemented.

hekmon

comment created time in 5 days

issue closedVictoriaMetrics/VictoriaMetrics

rollup_candlestick() results

Version: 1.32.7

I have been testing the rollup_candlestick() function to get OHLC values. I've written some data to VictoriaMetrics using the Influx line protocol (raw CURL requests attached). Measurement name: asset. Random price values every 4min.

1580802480 => price=9111.5 1580802720 => price=9341.5 1580802960 => price=9436.5 1580803200 => price=9384.5 (02/04/2020 @ 8:00am) 1580803440 => price=9300.4 1580803680 => price=9287.6 1580803920 => price=9316.2 1580804160 => price=9423.5 1580804400 => price=9574.1 1580804640 => price=9600.9 1580804880 => price=9579.4 1580805120 => price=9624.2 1580805360 => price=9740.5 1580805600 => price=9642.1 1580805840 => price=9500.1 1580806080 => price=9450.2 1580806320 => price=9400.3 1580806560 => price=9350.4 1580806800 => price=9300.5 (02/04/2020 @ 9:00am) 1580807040 => price=9250.6 1580807280 => price=9150.8 1580807520 => price=9100.9 1580807760 => price=9240.5 1580808000 => price=9564.7 1580808240 => price=9532.7 1580808480 => price=9421.7 1580808720 => price=9325.7 1580808960 => price=9514.7 1580809200 => price=9647.7 1580809440 => price=9874.7 1580809680 => price=9742.7 1580809920 => price=9532.7 1580810160 => price=9412.7 1580810400 => price=9117.7 (02/04/2020 @ 10:00am)

I've got OHLC values over the interval from 8:00AM to 10:00AM with resolution of 1hour using the request:

http://127.0.0.1:8428/api/v1/query_range?query=rollup_candlestick(asset_price)&start=1580803200&end=1580810400&step=3600

And I've got the response:

{"status":"success","data":{"resultType":"matrix","result": [ {"metric":{"rollup":"open","tag":"p"}, "values":[[1580803200,"9436.5"],[1580806800,"9300.4"],[1580810400,"9250.6"]]}, {"metric":{"rollup":"high","tag":"p"}, "values":[[1580803200,"9436.5"],[1580806800,"9740.5"],[1580810400,"9874.7"]]}, {"metric":{"rollup":"low","tag":"p"}, "values":[[1580803200,"9384.5"],[1580806800,"9287.6"],[1580810400,"9100.9"]]}, {"metric":{"rollup":"close","tag":"p"}, "values":[[1580803200,"9384.5"],[1580806800,"9300.5"],[1580810400,"9117.7"]]} ] }}

In other words, I've received 3 OHLC candles: [9436.5 9436.5 9384.5 9384.5] for 1580803200 [9300.4 9740.5 9287.6 9300.5] for 1580806800 [9250.6 9874.7 9100.9 9117.7] for 1580810400

The expected result should be in my opinion 2 OHLC candles: [9384.5 9740.5 9287.6 9350.4] for 1580803200 [9300.5 9874.7 9100.9 9412.7] for 1580806800

It seems that VictoriaMetrics has divided data in the following intervals:

1580802960 => price=9436.5 (because of maxSilenceInterval constant) 1580803200 => price=9384.5 (02/04/2020 @ 8:00am)

1580803440 => price=9300.4 1580803680 => price=9287.6 1580803920 => price=9316.2 1580804160 => price=9423.5 1580804400 => price=9574.1 1580804640 => price=9600.9 1580804880 => price=9579.4 1580805120 => price=9624.2 1580805360 => price=9740.5 1580805600 => price=9642.1 1580805840 => price=9500.1 1580806080 => price=9450.2 1580806320 => price=9400.3 1580806560 => price=9350.4 1580806800 => price=9300.5

1580807040 => price=9250.6 1580807280 => price=9150.8 1580807520 => price=9100.9 1580807760 => price=9240.5 1580808000 => price=9564.7 1580808240 => price=9532.7 1580808480 => price=9421.7 1580808720 => price=9325.7 1580808960 => price=9514.7 1580809200 => price=9647.7 1580809440 => price=9874.7 1580809680 => price=9742.7 1580809920 => price=9532.7 1580810160 => price=9412.7 1580810400 => price=9117.7 (02/04/2020 @ 10:00am)

And all the functions (rollupFirst, rollupLast, rollupMax, rollupMin) operate on these intervals.

Is my vision of the rollup_candlestick() result is wrong? How should I use the rollup_candlestick() function to get OHLC values on the specified interval (from start to end values)?

test_requests.txt

closed time in 5 days

torwig

issue commentVictoriaMetrics/VictoriaMetrics

rollup_candlestick() results

The bugfix is available starting from v1.33.1. Closing the issue as fixed.

torwig

comment created time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

Constant growth of goroutines number at vmstorage

@pavdmyt , the bugfix is available v1.33.1, so you can try it and verify whether it fixes the issue.

pavdmyt

comment created time in 5 days

created tagVictoriaMetrics/VictoriaMetrics

tagv1.33.1

VictoriaMetrics - fast, cost-effective and scalable time series database, long-term remote storage for Prometheus

created time in 5 days

created tagVictoriaMetrics/VictoriaMetrics

tagv1.33.1-cluster

VictoriaMetrics - fast, cost-effective and scalable time series database, long-term remote storage for Prometheus

created time in 5 days

push eventVictoriaMetrics/helm-charts

Aliaksandr Valialkin

commit sha e024aed056d5f52c0defaa2c9784d4f67fc860e4

Update docker image tags from v1.33.0 to v1.33.1

view details

push time in 5 days

push eventVictoriaMetrics/helm-charts

Aliaksandr Valialkin

commit sha 0d32768966050afed51d382ed731a0f427698ea8

Update docker image tags from v1.33.0 to v1.33.1

view details

Aliaksandr Valialkin

commit sha 6bbfa59bcda98a8f8dd964ffaa115f270821a8e4

`make package merge`

view details

push time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

container_cpu_usage_seconds_total return strange value

The provided configs and screenshots from the official dashboard for VictoriaMetrics look OK. Could you answer the following additional questions?

  • Does the issue affect only container_cpu_usage_seconds_total metric or other metrics too?
  • Does the issue affect only metrics obtained from kubernetes-cadvisor job or from other jobs too?
  • Does the issue affect only metrics obtained from the second Prometheus which scrapes kubernetes-cadvisor job or it affects metrics from the first Prometheus too?
  • Do you use relabeling in Prometheus configs?
  • Do you set honor_labels in any scrape_config inside Prometheus configs?
vissible

comment created time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

VM losts metrics until i scale timeline

There is no need in deleting data collected with the previous version of VictoriaMetrics. Just stop VictoriaMetrics, delete storage/cache folder and then start VictoriaMetrics again. Then report here whether this helped resolving the issue.

turbotankist

comment created time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

【too big data size】vmstorage can't process specific query

It is likely the original issue is already fixed in v1.33.*

govargo

comment created time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

【too big data size】vmstorage can't process specific query

It looks like vmstorage tries sending too big error message to vmselect. I added handling for this case in the commit afecb3449159a5a2256b7b7ace594f39e0ae677e. This commit is located far from v1.26.0-cluster , i.e. it will go into v1.33.1-cluster, so I backported the commit to v1.26.1-cluster. You can build VictoriaMetrics cluster components from this tag in order to obtain the original error message, which vmstorage failed to send to vmselect. Build steps for building Docker images for v1.26.1-cluster are below:

git clone https://github.com/VictoriaMetrics/VictoriaMetrics
cd VictoriaMetrics
git checkout v1.26.1-cluster
make package

This should build the following local docker images:

victoriametrics/vminsert:v1.26.1-cluster
victoriametrics/vmselect:v1.26.1-cluster
victoriametrics/vmstorage:v1.26.1-cluster

Another option is to wait for the v1.33.1-cluster release and upgrade to it.

govargo

comment created time in 5 days

created tagVictoriaMetrics/VictoriaMetrics

tagv1.26.1-cluster

VictoriaMetrics - fast, cost-effective and scalable time series database, long-term remote storage for Prometheus

created time in 5 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha afecb3449159a5a2256b7b7ace594f39e0ae677e

app/vmstorage: limit the maximum error message size before sending it to client Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/315

view details

push time in 5 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 846d7fa7e9ded3bff1e73a583003897e292be9bf

app/vmselect: add `sort_by_label(q, label)` and `sort_by_label_desc(q, label)` functions This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.

view details

push time in 5 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 7b1c7051a37316ce2ebe10733416d0b92edf8822

app/vmselect: add `sort_by_label(q, label)` and `sort_by_label_desc(q, label)` functions This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.

view details

push time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

Constant growth of goroutines number at vmstorage

The issue must be reproduced by creating new time series at constant pace. These time series would add new entries to inverted index. These entries will be put into new inmemory parts, which should trigger the issue with constantly growing goroutines.

pavdmyt

comment created time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

Feature: create a tool for migrating historical data from Prometheus to VictoriaMetrics

@syepes , Influx write url for cluster version should be http://<any_vminsert_host>:8480/insert/0/influx/write . See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#url-format for more details.

valyala

comment created time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

Constant growth of goroutines number at vmstorage

Oops, found yet another bug, which could result in stuck goroutines after v1.32.3. The bug should be fixed in the following commits:

  • single-node version - 7836ad89078847083ae8f9da92cafabe60a0b274
  • cluster version - e3b18ca1ab69082ec46e82acd5909f1311e39cc8

@pavdmyt , @freeseacher , could you build VictoriaMetrics from these commit and verify that they fix the goroutine leak?

pavdmyt

comment created time in 5 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 7836ad89078847083ae8f9da92cafabe60a0b274

lib/mergeset: skip createing temporary part objects when merging source inmemory parts This should reduce CPU usage when adding new entries to inverted index. This should alos prevent from creating stalled cleaner goroutines for the created temporary parts, since they were never closed. This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .

view details

push time in 5 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha e3b18ca1ab69082ec46e82acd5909f1311e39cc8

lib/mergeset: skip createing temporary part objects when merging source inmemory parts This should reduce CPU usage when adding new entries to inverted index. This should alos prevent from creating stalled cleaner goroutines for the created temporary parts, since they were never closed. This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .

view details

push time in 5 days

issue commentVictoriaMetrics/VictoriaMetrics

Constant growth of goroutines number at vmstorage

Confirmed the issue. It looks like the issue has been introduced in v1.32.3 . It is related to erroneous use of time.Timer instead of time.Ticker in cleaner code for index and block caches. Each new block creates new cleaner goroutine. This goroutine never finishes due to the issue mentioned above. So the number of goroutines constantly grows. The bug should be fixed in the following commits:

  • single-node version - eceaf13e5e1eb5463770fff3ee19b0f3b51d9713
  • cluster version - 347aaba79d44df3b21bef1f8f107352a133a635c

@pavdmyt , @freeseacher , could you build VictoriaMetrics from these commits and verify whether the issue with growing number of goroutines is gone for your workloads? See instructions on how to build VictoriaMetrics from sources:

pavdmyt

comment created time in 5 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 347aaba79d44df3b21bef1f8f107352a133a635c

lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate It has been appeared that time.Timer was used in places where time.Ticker must be used instead. This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .

view details

push time in 5 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha eceaf13e5e1eb5463770fff3ee19b0f3b51d9713

lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate It has been appeared that time.Timer was used in places where time.Ticker must be used instead. This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .

view details

push time in 5 days

startedprofefe/profefe

started time in 6 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 6e0013ca393982e2d792f00d6ca27d4f53732834

app/vmselect/prometheus: typo fix in `-latencyOffset` description

view details

push time in 6 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha ebf7785d79cc17ae0bc491f7982d1312ef434fe2

vendor: update github.com/VictoriaMetrics/metrics from v1.9.3 to v1.10.1

view details

Aliaksandr Valialkin

commit sha 22ede83146f7c54e1b438f9feb85f5dc68f61142

`make vendor-update`

view details

push time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 848d5da0be665800162eba19b23d102a62c08dcb

vendor: update github.com/VictoriaMetrics/metrics from v1.9.3 to v1.10.1

view details

Aliaksandr Valialkin

commit sha 8162d58dbdf69c0057a2ccc09cfcfb3a533aa5c8

`make vendor-update`

view details

push time in 8 days

created tagVictoriaMetrics/metrics

tagv1.10.1

Lightweight alternative to github.com/prometheus/client_golang

created time in 8 days

push eventVictoriaMetrics/metrics

Aliaksandr Valialkin

commit sha 0ee6a1fa32c0388e4926e902806881addeb4fd11

Update url to MetricsQL in docs to Histogram

view details

push time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha e7d1037210063cd2aadcefd1ab5df50bd7aef309

docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent

view details

push time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 02e852854a3951281ae013c513b1d2e249ddaa9d

README.md: refer to the article about data deletion via relabeling

view details

Aliaksandr Valialkin

commit sha a801a1a6e79f193437f75165f230abeec9f1ff26

.github/ISSUE_TEMPLATE: ask for command-line flags and Prometheus logs

view details

Aliaksandr Valialkin

commit sha 4cc0163c7c14ae417f7a310d819e7fcf25afbed9

docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent

view details

push time in 8 days

issue commentVictoriaMetrics/VictoriaMetrics

Add support to environment variables in addition to flags

Let's leave env vars without vm_ prefix. IMHO, this is more clear and more user-friendly than applying various transformations to env var names :) If VictoriaMetrics-specific env var clashes with third-party env vars, then the conflict may be resolved by setting the corresponding flag via command line. Additionally, by default reading flag values from env vars is disabled, so clashing env vars shouldn't bother users, who don't use env vars.

hekmon

comment created time in 8 days

pull request commentVictoriaMetrics/VictoriaMetrics

Cluster - prometheus metrics fix

@hekmon , thanks for the fix!

hekmon

comment created time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Edouard Hur

commit sha e8f92a4ee8635bbeae6096c83a3915ee3e5ee81f

Cluster - prometheus metrics fix (#314) * add missing '/{}' in prom query range requests * fix missing leading '/' on prom lavelValuesErrors path

view details

push time in 8 days

PR merged VictoriaMetrics/VictoriaMetrics

Cluster - prometheus metrics fix

Hello,

I was wandering thru VM cluster metrics and I think I found a small typo. image

query_range path for the vm_http_requests_total metric is missing the {} tenant placeholder.

While fixing it, I found an even smaller typo: all paths have a leading / in the path tag, all but vm_http_request_errors_total. While this could be intended, it seems very unlikely as it counterpart vm_http_requests_total does have it.

TL;DR: check the diff, it will make more sense

+2 -2

0 comment

1 changed file

hekmon

pr closed time in 8 days

issue commentVictoriaMetrics/VictoriaMetrics

VM losts metrics until i scale timeline

I don't understand that you mean

VictoriaMetrics contains response cache, which may work incorrectly when writing data with timestamps older than 5 minutes from the current time. Read here for more information. If Prometheus delays writing data to VictoriaMetrics for periods longer than 5 minutes, this may result in incomplete graphs as in your case. See the related issue.

@turbotankist , could you provide additional info by answering the following questions?

  • Did you modify -search.latencyOffset command-line flag value?
  • Could you provide the command line used for starting VictoriaMetrics?
  • Did you upgrade from version lower than v1.29 to newer version? The release v1.29.0 introduced per-day inverted index, which can sometimes return invalid results. The following issue could be related - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/290 .
turbotankist

comment created time in 8 days

issue commentVictoriaMetrics/VictoriaMetrics

Failed to get some metrics after update

@pavdmyt , thanks for the information. The issue is still under investigation. If somebody encounters something similar again, then it would be great posting here with some details. It looks like the issue is related to VictoriaMetrics upgrades. VictoriaMetrics introduced per-day inverted index in v1.29.0, which could result in missing time series on the upgrade day if the upgrade is performed from the version below v1.29.0 to any newer version. The issue should automatically disappear on the next day after the upgrade.

soffokl

comment created time in 8 days

issue commentVictoriaMetrics/VictoriaMetrics

container_cpu_usage_seconds_total return strange value

@vissible , thanks for the bug report!

Unfortunately I cannot reproduce it. Could you answer the following questions in order to better understand the issue?

  • How many Prometheus instances write data to VictoriaMetrics?
  • Could you provide remote_write section from Prometheus configs?
  • Could you provide the command line used for starting VictoriaMetrics? (remove sensitive information with passwords before posting it)
  • Could you provide screenshots from the official Grafana dashboard for VictoriaMetrics? See more information about this here.
  • Could you check for error logs for both Prometheus and VictoriaMetrics?
vissible

comment created time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 9e6e2319b98f9ad393c75a1d88a13d9d320a94d9

README.md: mention that flags may be read from env vars if `-envflag.enable` command-line flag is set

view details

push time in 8 days

issue commentVictoriaMetrics/VictoriaMetrics

Add support to environment variables in addition to flags

Added -envflag.enable command-line flag, which must be explicitly set in order to enable reading flags from environment vars. This should protect from reading unexpected flag values from env vars if the -enflag.enable flag isn't set.

hekmon

comment created time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 9c5db9400c03b4a90d40a788b5a86c9f990b4f42

lib/envflag: add `-envflag.enable` command-line flag for enabling reading flags from environment vars By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311

view details

Aliaksandr Valialkin

commit sha fcdd95a6ef40201a748c718778442ea20567d0af

lib/envflag: check for incorrect flag values read from environment vars

view details

push time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 5d207b2025325487f9dd8b340a5561dc5f2a4fa8

lib/envflag: add `-envflag.enable` command-line flag for enabling reading flags from environment vars By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311

view details

Aliaksandr Valialkin

commit sha 025297f15dc52fb7429299e25694553e515e1ed2

lib/envflag: check for incorrect flag values read from environment vars

view details

push time in 8 days

issue commentVictoriaMetrics/VictoriaMetrics

Add support to environment variables in addition to flags

@hekmon , support for setting flags via environment vars has been added in the following commits:

  • Single-node version - 8466ab003423cc3a3cc7755eb9561bbaa4a49b87
  • Cluster version - 1010a57882cd8ec97fadfcd3bd831f7f4679eab9

These commits will be included in the next release.

In the mean time you can build VictoriaMetrics from these commits and verify whether it works as expected. See instructions on how to build VictoriaMetrics:

hekmon

comment created time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 8466ab003423cc3a3cc7755eb9561bbaa4a49b87

all: allow setting flags via environment vars Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311

view details

push time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 1010a57882cd8ec97fadfcd3bd831f7f4679eab9

all: allow setting flags via environment vars Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311

view details

push time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha e210cd9da1b996d8f7812a3f735675864e9297f1

lib/storage: move `-dedup.minScrapeInterval` flag outside lib/storage, so it doesnt show up in `vminsert` in cluster version

view details

push time in 8 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha ea66212c9350c7b1e67b7cf27085b515f17ba0d3

lib/storage: move `-dedup.minScrapeInterval` flag outside lib/storage, so it doesnt show up in `vminsert` in cluster version

view details

push time in 8 days

issue commentVictoriaMetrics/VictoriaMetrics

Unable to filter on missing/empty label

@vissible , could you provide more details about the issue you encounter? It looks like the original issue isn't related your issue, since in your case VictoriaMetrics correctly returns the same time series as Prometheus. This time series doesn't contain image, systemd_service_name and namespace labels as requested by the query. The time series returned by VictoriaMetrics contains an additional label - "prom_name":"prometheus-core2" . It is likely the label is from external_labels section inside Prometheus configuration.

nkilzer

comment created time in 12 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 07c067697e8815412a2a0d685c286990105b7536

docs/Single-server-VictoriaMetrics.md: sync with README.md

view details

push time in 12 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 6db573470cb948b7c1551cbc607b6bb036b69f83

docs/Single-server-VictoriaMetrics.md: sync with README.md

view details

push time in 12 days

issue commentVictoriaMetrics/VictoriaMetrics

Add support to environment variables in addition to flags

Thanks for the detailed proposal, @hekmon !

While env vars have a disadvantage comparing to explicitly set command-line flags, since evn vars are implicitly applied to the app, I think their usability outweigh this disadvantage. So I'm going to add environment variables support into VictoriaMetrics in the near future according to your proposal.

The proposal is somewhat related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/138 .

hekmon

comment created time in 12 days

issue commentVictoriaMetrics/VictoriaMetrics

rollup_candlestick() results

Great! Then the fix will be included in the next release.

torwig

comment created time in 12 days

startedswarmstack/victoria-metrics

started time in 12 days

pull request commentVictoriaMetrics/VictoriaMetrics

Fix a typo in README (selfScrapeInterval)

Thanks, @ryotarai !

ryotarai

comment created time in 12 days

push eventVictoriaMetrics/VictoriaMetrics

Ryota Arai

commit sha fffe5d4ba49e885e4bff1f4c6fe5c5903c89bb17

Fix a typo in README (selfScrapeInterval) (#310)

view details

push time in 12 days

PR merged VictoriaMetrics/VictoriaMetrics

Fix a typo in README (selfScrapeInterval)

-scrapeInterval option does not exist. It seems to be a typo of -selfScrapeInterval.

+1 -1

0 comment

1 changed file

ryotarai

pr closed time in 12 days

issue closedVertamedia/chproxy

Releases schedule

Can you tell me about the release schedule? You have some release schedule? When is the next release planned?

Thanks!

closed time in 13 days

gemini1389

issue commentVertamedia/chproxy

Releases schedule

Released v1.14.0.

gemini1389

comment created time in 13 days

release Vertamedia/chproxy

v1.14.0

released time in 13 days

push eventVertamedia/chproxy

Aliaksandr Valialkin

commit sha 1758e7399fe57c97aeec8e55dd13c6300399969b

go fmt

view details

push time in 13 days

created tagVertamedia/chproxy

tagv1.14.0

ClickHouse http proxy and load balancer

created time in 13 days

issue commentVictoriaMetrics/VictoriaMetrics

rollup_candlestick() results

@torwig , could you try building VictoriaMetrics from the latest commit and verifying whether the issue has been fixed? Note that now VictoriaMetrics should return the expected OHLC results for timestamp ts on the range [ts ... ts+step), so yo don't need to adjust the resulting timestamps with offset -step or something similar. See how to build VictoriaMetrics from sources.

torwig

comment created time in 13 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 78b62dee872398d2ea6ad69894f40848b6b41e35

app/vmselect/promql: properly adjust time range for data to select Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309

view details

Aliaksandr Valialkin

commit sha a6c6a2debc501e5c4316b5fd2e3fc64094fe4d1b

app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8e52cdbb171649b6aa46caafd8a2a97

view details

push time in 13 days

push eventVictoriaMetrics/VictoriaMetrics

Aliaksandr Valialkin

commit sha 4a1de7fee971b1f3a010da062fb9ece270eb325b

app/vmselect/promql: properly adjust time range for data to select Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309

view details

Aliaksandr Valialkin

commit sha e6d9ea3094753588126befc29e058824aa6c73bf

app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8e52cdbb171649b6aa46caafd8a2a97

view details

push time in 13 days

issue closedryotarai/prometheus-tsdb-dump

Add ability to set time range for data importing

The issue

Sometimes users start replicating data to VictoriaMetrics from Prometheus, while continuing storing data to Prometheus. Then they need importing historical data from Prometheus into VictoriaMetrics, but don't won importing duplicate data, which already exists in VictoriaMetrics after it has been added as remote storage to Prometheus.

The solution

To add ability to set time range (start and end timestamps) for data to import from Prometheus storage to VictoriaMetrics. For instance, -startTimestamp and -endTimestamp command-line flags. Default value for -startTimestamp should be 0, while default value for -endTimestamp should be +Inf.

closed time in 14 days

valyala

issue commentvalyala/gozstd

Add support for custom window sizes (long distance matching)

Wow! These are good improvements in compression ratio, @arl . Will take a look into adding support for custom window sizes soon.

pushshift

comment created time in 14 days

issue commentVictoriaMetrics/VictoriaMetrics

Remote_read from Prometheus via 'vmselect'

@xkfen , it looks like the issue has been already resolved in Slack. Could you post details on how the issue has been resolved here, so other users could find this info?

NikolayShama

comment created time in 14 days

startedanthonyshull/victoria-metrics-operator

started time in 14 days

issue commentVictoriaMetrics/VictoriaMetrics

Improve panel hints for dashboards

@hagen1778 , dashboard improvements have been merged into v1.33.0.

hagen1778

comment created time in 14 days

issue commentVictoriaMetrics/VictoriaMetrics

rollup_candlestick() results

Fixes mentioned above are available in v1.33.0. @torwig , could you verify whether VictoriaMetrics v1.33.0 returns expected results for rollup_candlestick function?

torwig

comment created time in 14 days

issue closedVictoriaMetrics/VictoriaMetrics

Victoria metrics cluster should write logs to stdout instead of stderr

Describe the bug Given following systemd file

[Service]
Type=simple
LimitNOFILE=8192
User=victoriametrics
Group=victoriametrics
ExecStart=/opt/victoria-metrics/vminsert-prod \
...
    -httpListenAddr=":8480" \
    -loggerLevel="INFO" 


StandardOutput=append:/var/log/vminsert.log
StandardError=append:/var/log/vminsert-error.log

Expected that logs should be in vminsert.log but got all logs in vminsert-error.log

Version The line returned when passing --version command line flag to binary. For example:

/opt/victoria-metrics/vminsert-prod --version 
vminsert-20200128-195548-tags-v1.32.8-cluster-0-gb28c9a39

closed time in 14 days

freeseacher

issue commentVictoriaMetrics/VictoriaMetrics

Victoria metrics cluster should write logs to stdout instead of stderr

The -loggerOutput command-line flag has been added in v1.33.0. Closing the issue as fixed.

freeseacher

comment created time in 14 days

issue commentVictoriaMetrics/VictoriaMetrics

Add metrics deduplication functionality in `vmselect`

FYI, the -dedup.minScrapeInterval flag has been added in VictoriaMetrics v1.33.0. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#deduplication for more details.

Janesee3

comment created time in 14 days

issue commentVictoriaMetrics/VictoriaMetrics

Add data deduplication from HA Prometheus pair based on `--query.replica-label` arg similar to Thanos Query

FYI, the -dedup.minScrapeInterval flag has been added in VictoriaMetrics v1.33.0. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#deduplication for more details.

stormobile

comment created time in 14 days

created tagVictoriaMetrics/VictoriaMetrics

tagv1.33.0

VictoriaMetrics - fast, cost-effective and scalable time series database, long-term remote storage for Prometheus

created time in 14 days

created tagVictoriaMetrics/VictoriaMetrics

tagv1.33.0-cluster

VictoriaMetrics - fast, cost-effective and scalable time series database, long-term remote storage for Prometheus

created time in 14 days

push eventVictoriaMetrics/helm-charts

Aliaksandr Valialkin

commit sha c39ff329a9885c0157012566b83964e984d0260e

Update docker image tags from v1.32.8 to v1.33.0

view details

push time in 14 days

push eventVictoriaMetrics/helm-charts

Aliaksandr Valialkin

commit sha c2a10e234917a5ca359cce5025818cf752160a48

Update docker image tags from v1.32.8 to v1.33.0

view details

Aliaksandr Valialkin

commit sha b18e9987012f6beefd4218403283a674ab292087

`make package merge`

view details

push time in 14 days

more