profile
viewpoint

Ask questionsConstant growth of goroutines number at vmstorage

Describe the bug

Running VictoriaMetrics cluster v1.33.0 with 3 vmstorage instances. Observing constant increase in a number of goroutines at vmstorage components. Had same behavior with v1.32.6.

To Reproduce

These issues just started happening after upgrade from 1.31.5 to 1.32.6.

Screenshots

Drops to zero are caused by vmstorage restarts.

vmstorage-goroutines

Version

vmstorage-20200204-220434-tags-v1.33.0-cluster-0-g8e77b548

Used command-line flags

vmstorage flags:

flag{name="bigMergeConcurrency", value="0"} 1
flag{name="dedup.minScrapeInterval", value="0s"} 1
flag{name="enableTCP6", value="false"} 1
flag{name="fs.disableMmap", value="false"} 1
flag{name="http.disableResponseCompression", value="false"} 1
flag{name="httpListenAddr", value=":8482"} 1
flag{name="loggerFormat", value="default"} 1
flag{name="loggerLevel", value="INFO"} 1
flag{name="loggerOutput", value="stderr"} 1
flag{name="memory.allowedPercent", value="60"} 1
flag{name="precisionBits", value="64"} 1
flag{name="retentionPeriod", value="6"} 1
flag{name="rpc.disableCompression", value="false"} 1
flag{name="search.maxTagKeys", value="secret"} 1
flag{name="search.maxTagValues", value="100000"} 1
flag{name="search.maxUniqueTimeseries", value="300000"} 1
flag{name="smallMergeConcurrency", value="0"} 1
flag{name="snapshotAuthKey", value="secret"} 1
flag{name="storageDataPath", value="/vmstorage-data"} 1
flag{name="version", value="false"} 1

Additional context

Also attaching /debug/pprof/goroutine from every vmstorage instance:

vm-pprof.zip

VictoriaMetrics/VictoriaMetrics

Answer questions valyala

@pavdmyt , the bugfix is available v1.33.1, so you can try it and verify whether it fixes the issue.

useful!

Related questions

Remote write for Prometheus not working hot 1
Graph missing some points hot 1
How to write data to VictoriaMetrics individually in batches? hot 1
Add data deduplication from HA Prometheus pair based on `--query.replica-label` arg similar to Thanos Query hot 1
Can't start VictoriaMetrics use docker-compose and volume mapping on Windows hot 1
While under heavy load getting error panic hot 1
While under heavy load getting error panic hot 1
Failure to deploy with `helm template` in ArgoCD hot 1
While under heavy load getting error panic hot 1
Windows build hot 1
Increase default value for -search.maxPointsPerTimeseries hot 1
changing precisionBits doesn't result in change in precision of stored value hot 1
Drop-in replacement prometheus not work with promxy (Prometheus HA) hot 1
Github User Rank List