use the vendor/golang.org/x/sys at the release version or (probably better) use the OS release version DISTNAME = go-sys-20190904 ? which is the go-sys package in OpenBSD

I'm unsure that using OS release version of golang.org/x/sys or other third-party Go packages is better than using vendored packages from vendor folder. This requires setting up thrid-party dependencies before building VictoriaMetrics. Currently VictoriaMetrics can be built without the need to configure any third-party dependencies except of Go itself. It would be better upstreaming OpenBSD-related changes to from OS release versions to golang.org/x/sys, so they can be incorporated into vendor directory with make vendor-update command.

FYI, support for OpenBSD has been added in the following commits:

• single-node version - cc90a548b1822f536988cf4cfc3f4f52c34afcf5
• cluster version - 658a05ef0f6d93ab00b68194ee3e71e134caa776

The following line must be used for building VictoriaMetrics binary for OpenBSD: GOOS=openbsd gmake victoria-metrics-pure, as @GhioRodolphe mentioned above. victoria-metrics-pure can be substituted with vmagent-pure, vmalert-pure, etc. for building other vm* binaries.

These commits will be included in the next release.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785

removed fadvise for openbsd, added freespace implemenation for openbsd

Describe the bug During backup with

        {{ victoriametrics_root }}/vmbackup-prod
        -dst gcs://{{ vmbackup_bucket_path }}
        -snapshotName {{ snapshot_name }}
        -storageDataPath {{ victoriametrics_data_dir }}
        -origin gcs://{{ vmbackup_bucket_path }}
        -maxBytesPerSecond {{ vmbackup_max_upload_speed }}
        -loggerOutput {{ vmbackup_logging }}
        -concurrency {{ vmbackup_concurency }}
        -credsFilePath {{ vmbackup_creds_file }}

daily backup failed. On all three storage nodes with googleapi: Error 503: Backend Error

To Reproduce no idea. got once. Probably we should retry here

Expected behavior Backup done correctly

Screenshots If applicable, add screenshots to help explain your problem.

Version vmbackup-20200303-194040-tags-v1.34.2-0-g0b1e877a

$ ./victoria-metrics-prod --version
victoria-metrics-20190730-121249-heads-single-node-0-g671d9e55

Thanks for the update! Then closing this issue in the hope it has been fixed at Google cloud SDK side.

@freeseacher , could you check whether vmbackup from new releases still emits googleapi: Error 503: Backend Error?

The best way to migrate big amounts of data between VictoriaMetrics instances is to export the data via /api/v1/export from one instance and then import it to another instance /api/v1/import. This approach works for migrating data from single-node to cluster instance and vice versa. Note that cluster version of VictoriaMetrics has slightly different urls comparing to single-node version - see these docs for details.

As you noted above, this approach may work slowly or may use big amounts of RAM during export when match[] query arg matches all the time series, i.e. {__name__!=""}. The solution is to try splitting time series into multiple groups by certain label filters and export each such group individually. For instance, if the database contains a million of unique time series with label deployment_id with label values starting from random digit, then it is possible to split such time series into 10 groups by 100K time series per group with the following match[] filter: {deployment_id=~"0.*"}, {deployment_id=~"1.*"}, ... , {deployment_id=~"9.*"}. Then each such group can be migrated independently of each other. I'd suggest investigating the output of /api/v1/status/tsdb page in order to determine labels, which could be used for even data splitting.

Can we just copy over /storage data directly? I assume it is not going to work?

Unfortunately this doesn't work, because cluster version and single-node version of VictoriaMetrics have different data formats - cluster version additionally stores accountID and projectID (aka tenants).

It is clear that the approach outlined above may be slow and awkward to execute. So let's re-purpose this issue to feature request on fast data migration between VictoriaMetrics instances.

Describe the bug When using Grafana Instant queries and plot them as a table (for e.g. to plot label values), the table shows multiple series, instead of a single series. This keeps changing on refresh (more clear in screenshots below)

To Reproduce Plot labels of two queries onto a Grafana Table by using instant queries. Only happens with there are two queries that are bing plotted.

Expected behavior This is the expected output:

But on refresh it'll change to:

Version

$ ./victoria-metrics-prod --version
victoriametrics02 : victoria-metrics-20200911-214525-tags-v1.41.0-0-gba74d0c14

Used command-line flags

--retentionPeriod=3 --memory.allowedPercent=60

Additional Context

• Metrics are scraped via VMagent (1min scrape interval)
• Parallel Prometheus cluster doesn't exhibit this issue (same interval as VMagent)
• Seems to be related to the Timestamps that don't match between the two queries, but why does that not happen in Prometheus

Closing the issue as duplicate for #720 .

This looks like a duplicate for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720 .

I must say unless i m mistaken, isn't that a bit strange to embed third party libs like that ?https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/vendor/cloud.google.com/go

This is called vendoring in Go. VictoriaMetrics uses third-party libraries (aka Go packages). All the source code for these packages is placed under vendor directory. This provides the following benefits:

• This makes possible to build VictoriaMetrics from sources in an environment without access to Internet and without the need to manually install dependencies before building VictoriaMetrics, since vendor folder contains all the source code for all third-party dependencies.
• This makes easy grep'ing across all the source code including all thrid-party dependencies.
• This makes easy investigating changes to third-party dependencies via git history.

Go provides simple mechanism for determining why the given third-party dependency is needed. For example:

$ go mod why cloud.google.com/go
# cloud.google.com/go
github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/gcsremote
cloud.google.com/go/storage
cloud.google.com/go

The output for go mod why says that cloud.google.com/go is transitively used by github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/gcsremote.

The bug must be fixed in the commit c584aece388fa46cd2eecd6080fdd9e70d84325e . @propertone , could you build VictoriaMetrics from this commit according to these docs and verify whether -search.maxStalenessInterval command-line flag works there as expected?

@Muxa1L , thanks for the spot! It must be fixed in the following commits:

• Single-node VictoriaMetrics: 3ba507000ca78b35cbe2db8106131c8376450ba6
• Cluster VictoriaMetrics: 1fce79518a58956c8cc8d62654907edbefe53a04

Unfortunately these commits weren't included in v1.41.1, but they will be included in the next release.

VictoriaMetrics should properly emit errors when it cannot parse timestamps and/or values in Graphite line starting from v1.41.1. Closing this issue as fixed.

I have VM cluster setup. I'm trying to send metrics in Graphite format. And I found that if the tag value contains a space, that next uniq metrics with same first tag will be broken.

It's hard to explain, let me show you a code:

>>> echo 'metric;tag1=aaa1;tag2=bbb2;tag3=ccc3 2' | nc 127.0.0.1 2003
>>> curl 'http://localhost:8481/select/0/prometheus/api/v1/export?match[]=metric'

{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bbb2","tag3":"ccc3"},"values":[2,2],"timestamps":[1563200403000,1563200407000]}

Everething is fine.

Then I send metric with a space symbol in tags (tag2 and tag3)

>>> echo 'metric;tag1=aaa1;tag2=bb b2;tag3=cc c3 2' | nc 127.0.0.1 2003
>>> curl 'http://localhost:8481/select/0/prometheus/api/v1/export?match[]=metric'

{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bb"},"values":[0],"timestamps":[1563200424000]}
{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bbb2","tag3":"ccc3"},"values":[2,2],"timestamps":[1563200403000,1563200407000]}

I recieved the second metric poorly parsed. It's bad, but not the main issue.

If I trying to send the next metrics with different and correct tag2 and tag3 I didn't receive anything

>>> echo 'metric;tag1=aaa1;tag2=bb_b2;tag3=ccc3 2' | nc 127.0.0.1 2003
>>> curl 'http://localhost:8481/select/0/prometheus/api/v1/export?match[]=metric'

{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bb"},"values":[0],"timestamps":[1563200424000]}
{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bbb2","tag3":"ccc3"},"values":[2,2],"timestamps":[1563200403000,1563200407000]}

And if I change metrics next to tag1 to any others VM still refuse to save them. It works only if I change tag1 value to any other

>>> echo 'metric;tag1=aa_a1;tag2=bb_b2;tag3=ccc3 2' | nc 127.0.0.1 2003
>>> curl 'http://localhost:8481/select/0/prometheus/api/v1/export?match[]=metric'

{"metric":{"__name__":"metric","tag1":"aa_a1","tag2":"bb_b2","tag3":"ccc3"},"values":[2],"timestamps":[1563200569000]}
{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bb"},"values":[0],"timestamps":[1563200424000]}
{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bbb2","tag3":"ccc3"},"values":[2,2],"timestamps":[1563200403000,1563200407000]}


# As you can see if I add tag4 and tag5 instead of tag2 and tag3 it didn't change anything
>>> echo 'metric;tag1=aaa1;tag4=bbb2;tag5=ccc3 2' | nc 127.0.0.1 2003
>>> curl 'http://localhost:8481/select/0/prometheus/api/v1/export?match[]=metric'

{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bb"},"values":[0],"timestamps":[1563200424000]}
{"metric":{"__name__":"metric","tag1":"aaa1","tag2":"bbb2","tag3":"ccc3"},"values":[2,2,2],"timestamps":[1563200403000,1563200407000,1563201198000]}
{"metric":{"__name__":"metric","tag1":"aa_a1","tag2":"bb_b2","tag3":"ccc3"},"values":[2],"timestamps":[1563200569000]}

It's a very interesting mistake how the single broken metric can affect other didn't broken metrics.

FYI, VictoriaMetrics should properly parse +Inf and should properly emit errors when it cannot parse timestamps and/or values in all the supported text-based data ingestion protocols starting from release v1.41.1.

Describe the bug VictoriaMetrics leaving not completely removed files

lsof | grep victoria | grep -c deleted
142359
lsof | grep deleted | awk '{print $10;}'
/data/victoriametrics/data/small/2020_08/12509375_11558539_20200807065635.791_20200807065717.993_1626FBD2B11EAAC2/values.bin
/data/victoriametrics/data/small/2020_08/12509375_11558539_20200807065635.791_20200807065717.993_1626FBD2B11EAAC2/index.bin
/data/victoriametrics/data/small/2020_08/5491622_5298795_20200810064325.700_20200810064358.140_1626FBD2B128ED58/timestamps.bin
/data/victoriametrics/data/small/2020_08/5491622_5298795_20200810064325.700_20200810064358.140_1626FBD2B128ED58/values.bin
/data/victoriametrics/data/small/2020_08/377844286_30473412_20200810062718.795_20200810063744.222_1626FBD2B128E9BE/timestamps.bin
/data/victoriametrics/data/small/2020_08/377844286_30473412_20200810062718.795_20200810063744.222_1626FBD2B128E9BE/values.bin
/data/victoriametrics/data/small/2020_08/377844286_30473412_20200810062718.795_20200810063744.222_1626FBD2B128E9BE/index.bin
[...]

it's not a NFS - just regular setup with SSD disks there are no errors in logs. single-node

To Reproduce

• run VictoriaMetrics
• wait few days
• run lsof | grep deleted | grep victoria

Expected behavior files should be removed properly

Version The line returned when passing --version command line flag to binary. For example:

victoria-metrics-20200714-141946-tags-v1.38.1-0-gb442a42d8

single-node

Used command-line flags -httpListenAddr=:8428 -storageDataPath=/data/victoriametrics/ -memory.allowedPercent=80 -retentionPeriod=2 -search.maxUniqueTimeseries=10000000000 -search.maxConcurrentRequests=16 -loggerFormat=json

FYI, the bugfix has been included in release v1.41.1. Closing the issue as fixed.

@xor-eax-ebx , could you upgrade VictoriaMetrics to v1.41.1 and verify whether the issue is gone in your case? If it still exists, then please re-open the issue.

FYI, the commit https://github.com/VictoriaMetrics/VictoriaMetrics/commit/89d652b583cb04e9cf8a8c6090ed7f20d68daa3e has been included in v1.41.1.

Is your feature request related to a problem? Please describe. When victoria-metrics runs on kubernetes it is redundant to put timestamp into log records because kubernetes does this as well. The same happens in case of VM runs as a systemd unit. Journald already adds timestamp in logs.

Describe the solution you'd like it would be nice to have an argument for command line which disables timestamp in log records

FYI, the -loggerDisableTimestamps command-line flag has been added in v1.41.1. Closing this feature request as done.

Is your feature request related to a problem? Please describe.

In Our usage, We use vmAgent as a scraper, It will scrape target metrics, and then push data to vmstorage, However , When The data is very huge , It will takes some time.

And I do some recording rule in vmAlert, Some time the recording rule calc less data than vmagent scrape because the latency write.

Describe the solution you'd like And a flag like --query.backoff flag to wait the write end.

FYI, the -datasource.lookback command-line flag has been added to vmalert starting from v1.41.1. Closing this feature request as done.

Describe the solution you'd like Hide password in logs in case if user want's to use basic auth credentials as part of remoteWrite.url e.g. --remoteWrite.url=https://username:[email protected]/api/v1/write

Additional context If username and password is set this way they will be showed up in logs and will be partially masked in error logs:

2020-09-16T15:54:37.079Z    info    VictoriaMetrics/lib/logger/flag.go:11   build version: vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3
2020-09-16T15:54:37.079Z    info    VictoriaMetrics/lib/logger/flag.go:12   command line flags
2020-09-16T15:54:37.079Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "csvTrimTimestamp" = "1ms"
2020-09-16T15:54:37.079Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "dryRun" = "false"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "enableTCP6" = "false"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "envflag.enable" = "false"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "envflag.prefix" = ""
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "fs.disableMmap" = "false"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "graphiteListenAddr" = ""
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "graphiteTrimTimestamp" = "1s"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "http.disableResponseCompression" = "false"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "http.maxGracefulShutdownDuration" = "7s"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "http.pathPrefix" = ""
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "http.shutdownDelay" = "0s"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "httpAuth.password" = "secret"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "httpAuth.username" = ""
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "httpListenAddr" = ":8429"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "import.maxLineLen" = "104857600"
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "influxListenAddr" = ""
2020-09-16T15:54:37.080Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "influxMeasurementFieldSeparator" = "_"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "influxSkipMeasurement" = "false"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "influxSkipSingleField" = "false"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "influxTrimTimestamp" = "1ms"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "insert.maxQueueDuration" = "1m0s"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "loggerErrorsPerSecondLimit" = "10"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "loggerFormat" = "default"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "loggerLevel" = "INFO"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "loggerOutput" = "stderr"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "maxConcurrentInserts" = "16"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "maxInsertRequestSize" = "33554432"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "memory.allowedBytes" = "0"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "memory.allowedPercent" = "60"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "metricsAuthKey" = "secret"
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "opentsdbHTTPListenAddr" = ""
2020-09-16T15:54:37.081Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "opentsdbListenAddr" = ""
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "opentsdbTrimTimestamp" = "1s"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "opentsdbhttp.maxInsertRequestSize" = "33554432"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "opentsdbhttpTrimTimestamp" = "1ms"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "pprofAuthKey" = "secret"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.config" = "/config/agent.yaml"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.config.dryRun" = "false"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.config.strictParse" = "true"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.configCheckInterval" = "0s"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.consulSDCheckInterval" = "30s"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.disableCompression" = "false"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.disableKeepAlive" = "false"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.discovery.concurrency" = "100"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.discovery.concurrentWaitTime" = "1m0s"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.dnsSDCheckInterval" = "30s"
2020-09-16T15:54:37.082Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.ec2SDCheckInterval" = "1m0s"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.fileSDCheckInterval" = "30s"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.gceSDCheckInterval" = "1m0s"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.kubernetesSDCheckInterval" = "30s"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.maxScrapeSize" = "16777216"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "promscrape.suppressScrapeErrors" = "false"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.basicAuth.password" = "secret"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.basicAuth.username" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.bearerToken" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.decimalPlaces" = "0"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.flushInterval" = "1s"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.label" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.maxBlockSize" = "33554432"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.maxDiskUsagePerURL" = "0"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.proxyURL" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.queues" = "20"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.relabelConfig" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.sendTimeout" = "1m0s"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.showURL" = "false"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.tlsCAFile" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.tlsCertFile" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.tlsInsecureSkipVerify" = "false"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.tlsKeyFile" = "secret"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.tlsServerName" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.tmpDataPath" = "vmagent-remotewrite-data"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.url" = "https://agent:[email protected]/api/v1/write"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "remoteWrite.urlRelabelConfig" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "tls" = "false"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "tlsCertFile" = ""
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "tlsKeyFile" = "secret"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/logger/flag.go:20   flag "version" = "false"
2020-09-16T15:54:37.083Z    info    VictoriaMetrics/lib/cgroup/cpu.go:26    updating GOMAXPROCS to 1 according to cgroup CPU quota
2020-09-16T15:54:37.084Z    info    VictoriaMetrics/app/vmagent/main.go:80  starting vmagent at ":8429"...
2020-09-16T15:54:37.084Z    info    VictoriaMetrics/lib/memory/memory.go:45 limiting caches to 2576980377 bytes, leaving 1717986919 bytes to the OS according to -memory.allowedPercent=60.000000 and -memory.allowedBytes=0
2020-09-16T15:54:37.087Z    info    VictoriaMetrics/lib/persistentqueue/fastqueue.go:46 opened fast persistent queue at "vmagent-remotewrite-data/persistent-queue/AF03C634A296195F" with maxInmemoryBlocks=200
2020-09-16T15:54:37.087Z    info    VictoriaMetrics/app/vmagent/remotewrite/client.go:125   initialized client for -remoteWrite.url="https://agent:[email protected]/api/v1/write"
2020-09-16T15:54:37.087Z    info    VictoriaMetrics/app/vmagent/main.go:102 started vmagent in 0.004 seconds
2020-09-16T15:54:37.087Z    info    VictoriaMetrics/lib/httpserver/httpserver.go:77 starting http server at http://:8429/
2020-09-16T15:54:37.087Z    info    VictoriaMetrics/lib/httpserver/httpserver.go:78 pprof handlers are exposed at http://:8429/debug/pprof/
2020-09-16T15:54:37.088Z    info    VictoriaMetrics/lib/promscrape/scraper.go:78    reading Prometheus configs from "/config/agent.yaml"
2020-09-16T15:54:37.093Z    info    VictoriaMetrics/lib/promscrape/scraper.go:303   static_configs: added targets: 1, removed targets: 0; total targets: 1
2020-09-16T15:54:37.363Z    info    VictoriaMetrics/lib/promscrape/scraper.go:303   kubernetes_sd_configs: added targets: 62, removed targets: 0; total targets: 62
2020-09-16T16:30:29.135Z    error   VictoriaMetrics/app/vmagent/remotewrite/client.go:212   couldn't send a block with size 151051 bytes to "https://agent:[email protected]/api/v1/write": Post "https://username:***@remote-storage.com/api/v1/write": EOF; re-sending the block in 2.000 seconds

FYI, all the occurrences of -remoteWrite.url in logs and metrics are substituted by secret-url text by default starting from v1.41.1. If you need -remoteWrite.url in logs and metrics, then pass -remoteWrite.showURL command-line flag to vmagent.

Closing the issue as resolved. @ofen , feel free re-opening it if you still see plain -remoteWrite.url in logs / metrics.

Is your feature request related to a problem? Please describe. Metrics which are stored at the vmagent persistent queue while there is no connection with the server, are lost if the host powered off or vmagent process has been killed.

Describe the solution you'd like After restart vmagent keeps all metrics that were in a queue before power-off or vmagent killed.

Describe alternatives you've considered A precision may not be ideal, but it will be good if just a few minutes of metrics history are lost after the incident.

Additional context We want to use vmagent as a part of fail-safe pipeline on POS Terminals, Scales and other Linux devices which are using in retail and may have no connection with Data Center and get powered off by unqualified personnel.

FYI, persistent queues in vmagent must become more durable starting from v1.41.1. Closing this issue as done.

@alpi-ua , feel free re-opening it if you still see significant data loss after unclean shutdown for vmagent v1.41.1 or newer.

Describe the bug

vmagent doesn't support all prometheus ec2_sd_config features.

With prometheus ec2_sd_config is possible to use following auth methods in addition to AWS_KEY and AWS_SECRET vars:

• role_arn - sets explicit at ec2_sd_config
• instance iam role - sets implicit, when iam role assigned to ec2 instance and AWS_KEY with AWS_SECRET configuration isn't set.

To Reproduce based on this article Assign iam role to instance and create following configuration:

scrape_configs:
  - job_name: 'node'
    ec2_sd_configs:
      - region: eu-west-1
    # Discover instances in account 111111111111
    - port: 9100
      role_arn: arn:aws:iam::111111111111:role/prometheus-assume-role-PrometheusAssumeRole-KJ4TK9KJBPU7 

with prometheus it works, vmagent returns error: missing `access_key` in AWS_ACCESS_KEY_ID env var; probably, `access_key` must be set in `ec2_sd_config`?

Expected behavior

ec2_sd_config must return instances for scraping.

Version 1.41.0

FYI, support for role_arn and EC2 instance tokens has been included in v1.41.1. Closing this bug as fixed.

FYI, enhancements mentioned above have been included in v1.41.1 . This should improve performance for queries with complex regexp filters.

It looks like the root cause of the issue has been identified and fixed in the commit 09b0f7c202f2c0e9a6b5879372fdb62d47f4a0fd . Previously VictoriaMetrics wasn't closing files on query timeout errors. Such files were eventually deleted after merging into bigger files, but they remained opened until VictoriaMetrics restart.

The commit will be included in the next release of VictoriaMetrics.

@YuriGrigorov , could you verify whether cluster components of VictoriaMetrics properly determine the number of available CPU cores starting from the commit 89d652b583cb04e9cf8a8c6090ed7f20d68daa3e ?

since 1.41.0 introduces a change that is not backward compatible with prior versions (data incompatibility) shouldn't it ideally be 2.0.0?

It is safe to upgrade from older releases to v1.41.0 and newer releases. It is impossible to downgrade to releases prior v1.41.0 though. The 2.0.0 version will be released only if it won't support seamless upgrade from older releases.

hi I want to directly push the data of Prometheus structure to VictoriaMetrics instead of remote_write_api, what should I do?

Closing the question as answered. @jinlongwang , feel free re-opening it if you have additional questions related to this one.

In my tests this produced results that didn't have anything to do with what I'm looking for. When I divide the result bei number of hours in my time range, the result becomes more realistic but still almost 100 kWh off:

There is no need to wrap integrate(m[$__range]) into range_sum(). The original query should return an integral for m over the selected time range in Grafana.

I think it is better limiting the size of the cache, so it could be saved to disk in reasonable time. I see little sense in partially saved cache.

But now isn't getMetricIDsForDateTagFilter lookup also divided into these two steps ?

Yes, but metricNames lookup is performed only when the number of metricIDs is relatively small.

Is it because a single index lookup has a higher overhead ?

Yes. Performance difference between sequential index scan and index lookup is multiple orders of magnitude (i.e. 1000x and more). So it is better to scan 100M of metricIDs than to look up 10K metricNames.

Note that too big values for search.maxConcurrentRequests may result in vmstorage overload and trashing. So this config must be adjusted carefully.

It looks like you hit the limit for concurrently executed queries, which can be adjusted with -search.maxConcurrentRequests command-line flag. If the limit is reached, then vmselect puts incoming requests in a wait queue until free slots are available for query processing. Requests may wait in the queue for up to -search.maxQueueDuration. After that vmselect logs the cannot handle more than ... concurrent search requests warning. Default value for -search.maxQueueDuration is 10 seconds.

Why requests may wait for 10 seconds in the queue if every request duration is limited by -search.storageTimeout=1s? Let's look at the following sequence:

• -search.maxConcurrentRequests=16 concurrent requests hit vmselect. All of them wait for -search.storageTimeout=1s due to the slow vmstorage node.
• Additional requests arrived while the first 16 requests are waiting for response from slow vmstorage node. They are put in the wait queue. If these requests arrive at rate higher than search.maxConcurrentRequests / search.storageTimeout, i.e. 16 req/s for the case above, then the queue will constantly grow. This means that the majority of incoming requests will never reach vmstorage because the will be timed out in the wait queue.

How to deal with the issue for the given constant requests rate? Either increase search.maxConcurrentRequests or decrease search.storageTimeout.

The commit 69eb9783e6f5e7398faa1a5daad1765f9d5ed769 should partially address the issue by limiting max wait time in the queue to min(search.maxQueryTimeout, search.storageTimeout), so now all the requests must be finished in -search.storageTimeout.

We can use the index of MetricIDToMetricName to find all the metrics and then cycle through the collection for the rest of the TagFilter matches, which should be more efficient.

Unfortunately this doesn't work, since matching by metricName requires the following steps:

1. locating and unmarshaling all the metricName values from the inverted index for the given metricIDs;
2. matching found metricNames against all the tag filters.

These steps are usually much slower than locating and intersecting metricIDs by tag filters.

Thanks for great idea and implementation! This should help with the performance for regexp label filters.

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781

The code probably looks like this, but I don't know how to test the performance of this strategy in a real-world environment.

There is a typo in your example which was hard to spot: regexp should be regex.

Thanks for spotting the typo! Fixed it in the original comment.

The other (related) issue is that range queries do not appear to respect the -search.maxStalenessInterval option, while the instant query does.

Could you file a separate issue regarding this bug?

I couldn't tell from the commit. Will it be possible to query on Inf, like count(foo == Inf)?

It seems to be working in the same way as in Prometheus. But, to be honest, I'm surprised with this, since, according to math rules, the result of inf == inf is undetermined.

Try upgrading Loki to the latest release - probably this will help fixing the issue.

FYI, it looks like the source of the issue has been identified and fixed at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720 . The bugfix will be included in the next release of VictoriaMetrics.

The issue must be fixed in the following commits:

• Single-node VictoriaMetrics - 2eb72e09ab7bd3208cfbe08a08938f043ea4e876
• Cluster VictoriaMetrics - 07c622633476d61e9ceef646a41e9e3f06fd7fd4

The bugfix rounds default time value to seconds when the query to /api/v1/query doesn't contain time query arg. This is a workaround, which reduces the probability of the original issue. The proper fix should be applied on Grafana side - it must pass time query arg with each query to /api/v1/query.

It looks like time series churn rate (the first graph) and the number of unique time series (the second graph) has been increased in the middle of August. But there is nothing special at the beginning of July when new Prometheus version has been installed according to the initial bug report. Probably, newer versions of Prometheus introduced instability for the real interval between scrapes? This may result in not so good compression ratio for timestamps data.

I just think it would be ideal to have a configuration option that allowed you to ignore those errors. Then you wouldn't have to worry about making it so that you only got one target. You could just set your labels up so you only got one and the duplicates would be eliminated by victoria without having to do anything else.

VictoriaMetrics already leaves only a single target if it sees duplicate targets with identical set of labels. I don't think it is good idea to suppress such errors, since they point to improper configuration for relabeling, which must be investigated and fixed instead of silently ignored.

This approach looks promising! @faceair , could you prepare a pull request for further investigation? Make sure it properly handles both positive and negative matches. I.e. foo{bar!="baz"} and foo{bar!~"baz.*"} shouldn't break after the change.

The -loggerDisableTimestamps command-line flag has been added in the following commits:

• Single-node version: 29108cc53e42d29a41bc3ff6e70b36a8c5788db2
• Cluster version: d32c3f747c383dcc12ce303ad1c5b18a72a8917b

@varuzam , could you build VictoriaMetrics from these commits and verify whether it works as expected? See build instructions for single-node VictoriaMetrics and build instructions for cluster version.

It looks like the bottleneck is at vminsert side. Every vminsert node establishes a single connection to every vmstorage node from -storageNode command-line flag list. Data transfer rate for each such vminsert->vmstorage connection is limited by a single CPU core on both sides - vminsert and vmstorage. So, in order to increase the overall bandwidth for incoming data, the following actions might be taken:

• Increasing the number of vminsert nodes and evenly spreading incoming load among them.
• Increasing the number of vmstorage nodes.

Both these actions increase the number of vminsert->vmstorage connections, so this increases the max data ingestion bandwidth for the cluster. See also capacity planning docs for VictoriaMetrics cluster.

@saclive , could you verify that all the vminsert nodes in the cluster have the same set of -storageNode command-line flags and these flags have identical order? If this isn't the case, then the data distribution among vmstorage nodes may become uneven.

we find a function named updatemetrics() but it is never be called

This function is called inside registerStorageMetrics() function in app/vmstorage/main.go file in cluster branch - see https://github.com/VictoriaMetrics/VictoriaMetrics/blob/8cd89cb8477dc7fcf003292bcba107bce4208ba7/app/vmstorage/main.go#L222

The /api/v1/series/count counts the number of unique time series by scanning the inverted index containing records for every time series. That's why it may be slow if the number of time series in the database exceeds tens of millions. There is vm_cache_entries{type="storage/hour_metric_ids"} metric, which shows the number of active time series, i.e. time series with recently added samples - see monitoring docs for details. If no new data is ingested for the time series during long periods of time (i.e. more than a hour), then such time series isn't counted by vm_cache_entries{type="storage/hour_metric_ids"} metric.

The total number of time series in the database can be much bigger than the number of active time series if there is high churn rate, i.e. if old time series are constantly substituted by new time series. Time series are uniquely identified by its name plus a set of its' labels. If at least a single label value changes, then new time series is created. For example, foo{bar="baz1"} and foo{bar="baz2"} are distinct time series.

Closing this question as answered.