profile
viewpoint

Ask questionsruntime: crash with "invalid pc-encoded table"

What version of Go are you using (go version)?

go 1.11

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/users/kai.hayashi/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build387775402=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

I am running a fairly run-of-the-mill production api server. I have observed one crash so far on this code since upgrading to go1.11 with fatal error: invalid runtime symbol table. I am running a library in this server code that periodically captures a cpu, block, and mutex profile every few minutes. The sample rates on the mutex profile is 1/10 and 1 block profile every second.

What did you expect to see?

No crashes

What did you see instead?

runtime: invalid pc-encoded table [...]
fatal error: invalid runtime symbol table
goroutine 0 [idle]:
runtime.throw(0x10501de, 0x1c)
      /usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc00064f258 sp=0xc00064f228 pc=0x42c2b2
runtime.pcvalue(0x179ed90, 0x19f6800, 0xXXXXXXXX005ea0e7, 0xaaf94c, 0xc00064f480, 0xXXXXXXXXXXXXXX01, 0xXXXXXXXX00000000)
      /usr/local/go/src/runtime/symtab.go:791 +0x50a fp=0xc00064f300 sp=0xc00064f258 pc=0x449d3a
runtime.funcspdelta(0x179ed90, 0x19f6800, 0xaaf94c, 0xc00064f480, 0xXXXXXXXX00000000)
     /usr/local/go/src/runtime/symtab.go:843 +0x5f fp=0xc00064f370 sp=0xc00064f300 pc=0x44a1ef
runtime.gentraceback(0x45d5d5, 0xc0005f3e00, 0x0, 0xc0004d1080, 0x0, 0xc00064f6d8, 0x40, 0x0, 0x0, 0x6, ...)
      /usr/local/go/src/runtime/traceback.go:208 +0x1828 fp=0xc00064f678 sp=0xc00064f370 pc=0x450ca8
runtime.sigprof(0x45d5d5, 0xc0005f3e00, 0x0, 0xc0004d1080, 0xc000244380)
      /usr/local/go/src/runtime/proc.go:3823 +0x4b6 fp=0xc00064f928 sp=0xc00064f678 pc=0x436cd6
runtime.sighandler(0xc00000001b, 0xc00064fbf0, 0xc00064fac0, 0xc0004d1080)
      /usr/local/go/src/runtime/signal_sighandler.go:38 +0x73c fp=0xc00064f9b8 sp=0xc00064f928 pc=0x440dec

This crash is reminiscent of a bug I was hoping would be fixed with the 1.11 release related to https://github.com/golang/go/issues/24925. The crashes do seem less frequent than when I was running the same code under 1.10.

CC @heschik and @rhysh

golang/go

Answer questions rkuska

FYI; Still present with 1.12.0, same story as others, service with profiling (cloud.google.com/go/profiler)

Some tracebacks from the failure if it helps:

runtime.throw(0xee6853, 0x1c)
	/usr/local/go/src/runtime/panic.go:617 +0x72
runtime.pcvalue(0x13e4228, 0x1919980, 0x33984f, 0x85abc6, 0xc000c2b4d0, 0x1123701, 0x0)
	/usr/local/go/src/runtime/symtab.go:704 +0x53f
runtime.funcspdelta(0x13e4228, 0x1919980, 0x85abc6, 0xc000c2b4d0, 0xc000000000)
	/usr/local/go/src/runtime/symtab.go:756 +0x5f
runtime.gentraceback(0x45ddb6, 0x7f8ff17f9b90, 0x0, 0xc000483500, 0x0, 0xc000c2b728, 0x40, 0x0, 0x0, 0x6, ...)
	/usr/local/go/src/runtime/traceback.go:222 +0x1811
runtime.sigprof(0x45ddb6, 0x7f8ff17f9b90, 0x0, 0xc000483500, 0xc000ca4000)
	/usr/local/go/src/runtime/proc.go:3754 +0x425
runtime.sighandler(0xc00000001b, 0xc000c2bbf0, 0xc000c2bac0, 0xc000483500)
	/usr/local/go/src/runtime/signal_sighandler.go:38 +0x6e7
runtime.sigtrampgo(0x1b, 0xc000c2bbf0, 0xc000c2bac0)
	/usr/local/go/src/runtime/signal_unix.go:351 +0x1f2
runtime: unexpected return pc for runtime.sigtramp called from 0xc5a5e0
runtime.sigtramp(0x7, 0x0, 0xc000c24000, 0xc000000000, 0x8000, 0x136a7c658dead1, 0x7fff2a5b6080, 0x1, 0xbc9fb4be, 0x2, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:357 +0x43

goroutine 1177223 [GC worker (idle)]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc005591760 sp=0xc005591758 pc=0x45a260
runtime.gcBgMarkWorker(0xc000054000)
	/usr/local/go/src/runtime/mgc.go:1890 +0x1be fp=0xc0055917d8 sp=0xc005591760 pc=0x41d69e
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0055917e0 sp=0xc0055917d8 pc=0x45c331
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1784 +0x77

goroutine 1 [chan receive, 1884 minutes]:
main.main()
	/conveyance-store/cmd/storage/conveyance-api.go:274 +0x1173

goroutine 19 [syscall, 1884 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 367539921 [IO wait, 110 minutes]:
internal/poll.runtime_pollWait(0x7f906813c858, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc00ae38a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00ae38a80, 0xc00d4bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00ae38a80, 0xc00d4bb000, 0x1000, 0x1000, 0xc000048500, 0xc0089ebb08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000844440, 0xc00d4bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc00f5884b0, 0xc00d4bb000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc00e496960)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc00e496960, 0x4, 0x0, 0x0, 0x0, 0x0, 0x4bceba)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00a088820, 0x104e480, 0xc00bc74f80, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc00a088820, 0x104e480, 0xc00bc74f80)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 378316985 [IO wait, 51 minutes]:
internal/poll.runtime_pollWait(0x7f9058038d18, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc005612998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005612980, 0xc004a86000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc005612980, 0xc004a86000, 0x1000, 0x1000, 0xc00005af00, 0xc012171b08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000a3e2b8, 0xc004a86000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc0065b0ba0, 0xc004a86000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc004e38000)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc004e38000, 0x4, 0x0, 0x0, 0x0, 0x0, 0xfa)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00a6c68c0, 0x104e480, 0xc00b83cd00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc00a6c68c0, 0x104e480, 0xc00b83cd00)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 370175441 [IO wait, 97 minutes]:
internal/poll.runtime_pollWait(0x7f903c47d2b8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc00778b298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00778b280, 0xc010efe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00778b280, 0xc010efe000, 0x1000, 0x1000, 0xc000064500, 0xc014bbbb08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000d0c440, 0xc010efe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc00e1fdd10, 0xc010efe000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc003cca540)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc003cca540, 0x4, 0x0, 0x0, 0x0, 0x0, 0x4bceba)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00c221cc0, 0x104e480, 0xc00555c300, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc00c221cc0, 0x104e480, 0xc00555c300)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 387659619 [IO wait]:
internal/poll.runtime_pollWait(0x7f9068245180, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc011601b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc011601b00, 0xc01339f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc011601b00, 0xc01339f000, 0x1000, 0x1000, 0xc000066a00, 0xc013789b08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc019cd4838, 0xc01339f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc00716e6f0, 0xc01339f000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc0045981e0)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc0045981e0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x4bceba)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc0040c43c0, 0x104e480, 0xc00829ce80, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc0040c43c0, 0x104e480, 0xc00829ce80)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 375849207 [IO wait, 65 minutes]:
internal/poll.runtime_pollWait(0x7f9068413c30, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc00dc83918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00dc83900, 0xc01371c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00dc83900, 0xc01371c000, 0x1000, 0x1000, 0xc000056500, 0xc006a9bb08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0007d4250, 0xc01371c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc00c9c4b70, 0xc01371c000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc011a97c80)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc011a97c80, 0x4, 0x0, 0x0, 0x0, 0x0, 0x4bceba)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00566a0a0, 0x104e480, 0xc00103ee80, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc00566a0a0, 0x104e480, 0xc00103ee80)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 377139601 [IO wait, 58 minutes]:
internal/poll.runtime_pollWait(0x7f90682031d8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc01184c298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc01184c280, 0xc00d694000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc01184c280, 0xc00d694000, 0x1000, 0x1000, 0xc000080500, 0xc012221b08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000106e8, 0xc00d694000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc001a828a0, 0xc00d694000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc0043db380)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc0043db380, 0x4, 0x0, 0x0, 0x0, 0x0, 0x4bceba)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00b25f540, 0x104e480, 0xc007b37d00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc00b25f540, 0x104e480, 0xc007b37d00)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 375323721 [IO wait, 68 minutes]:
internal/poll.runtime_pollWait(0x7f9058039058, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc008c1c018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc008c1c000, 0xc01d0a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc008c1c000, 0xc01d0a2000, 0x1000, 0x1000, 0xc000064500, 0xc00a22bb08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000101e0, 0xc01d0a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc00632c090, 0xc01d0a2000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc01c9ba000)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc01c9ba000, 0x4, 0x0, 0x0, 0x0, 0x0, 0x4bceba)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc007a61040, 0x104e480, 0xc0155d4240, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc007a61040, 0x104e480, 0xc0155d4240)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 371375030 [IO wait, 91 minutes]:
internal/poll.runtime_pollWait(0x7f9068063400, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc01485de98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc01485de80, 0xc017f21000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc01485de80, 0xc017f21000, 0x1000, 0x1000, 0xc000058a00, 0xc0145e9b08, 0x6b05ca)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0008ce370, 0xc017f21000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).Read(0xc001a83f20, 0xc017f21000, 0x1000, 0x1000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc01486a120)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc01486a120, 0x4, 0x0, 0x0, 0x0, 0x0, 0x4bceba)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc002328820, 0x104e480, 0xc0053474c0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:964 +0xb03
net/http.(*conn).serve(0xc002328820, 0x104e480, 0xc0053474c0)
	/usr/local/go/src/net/http/server.go:1819 +0x6a8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 14 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000509000)
	/go/pkg/mod/go.opencensus.io@v0.14.0/stats/view/worker.go:146 +0xdd
created by go.opencensus.io/stats/view.init.0
	/go/pkg/mod/go.opencensus.io@v0.14.0/stats/view/worker.go:29 +0x57

goroutine 79 [IO wait]:
internal/poll.runtime_pollWait(0x7f9070054578, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000476098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000476080, 0xc000856000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000476080, 0xc000856000, 0x1000, 0x1000, 0x43f5bd, 0xc0003f94f8, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000010110, 0xc000856000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
bufio.(*Reader).Read(0xc000403140, 0xc000103c68, 0x1, 0x9, 0x2, 0x3, 0x80)
	/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x1033de0, 0xc000403140, 0xc000103c68, 0x1, 0x9, 0x1, 0x38, 0xc00942d740, 0x0)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/gocql/gocql.readHeader(0x1033de0, 0xc000403140, 0xc000103c68, 0x9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/frame.go:446 +0xab
github.com/gocql/gocql.(*Conn).recv(0xc000103c20, 0x0, 0x0)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:526 +0xf8
github.com/gocql/gocql.(*Conn).serve(0xc000103c20)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:490 +0x31
created by github.com/gocql/gocql.(*Session).dial
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:259 +0x749

goroutine 86 [select, 49 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000509a80)
	/go/pkg/mod/google.golang.org/grpc@v1.18.0/balancer_conn_wrappers.go:122 +0x110
created by google.golang.org/grpc.newCCBalancerWrapper
	/go/pkg/mod/google.golang.org/grpc@v1.18.0/balancer_conn_wrappers.go:113 +0x14f

goroutine 4602 [IO wait, 1884 minutes]:
internal/poll.runtime_pollWait(0x7f90682db498, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000685998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000685980, 0xc001b0a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000685980, 0xc001b0a000, 0x1000, 0x1000, 0x43f5bd, 0xc0009b40b8, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0008440c8, 0xc001b0a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
bufio.(*Reader).Read(0xc000b89080, 0xc000dddb48, 0x1, 0x9, 0x2, 0x3, 0x80)
	/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x1033de0, 0xc000b89080, 0xc000dddb48, 0x1, 0x9, 0x1, 0x38, 0xc000a0a2c0, 0x0)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/gocql/gocql.readHeader(0x1033de0, 0xc000b89080, 0xc000dddb48, 0x9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/frame.go:446 +0xab
github.com/gocql/gocql.(*Conn).recv(0xc000dddb00, 0x0, 0x0)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:526 +0xf8
github.com/gocql/gocql.(*Conn).serve(0xc000dddb00)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:490 +0x31
created by github.com/gocql/gocql.(*Session).dial
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:259 +0x749

goroutine 88 [select]:
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*tracer).worker(0xc0000f5860)
	/go/pkg/mod/gopkg.in/!data!dog/dd-trace-go.v1@v1.9.0/ddtrace/tracer/tracer.go:149 +0x1a8
created by gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.newTracer
	/go/pkg/mod/gopkg.in/!data!dog/dd-trace-go.v1@v1.9.0/ddtrace/tracer/tracer.go:136 +0x30f

goroutine 4987 [IO wait, 1884 minutes]:
internal/poll.runtime_pollWait(0x7f907009d010, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000685898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000685880, 0xc001aac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000685880, 0xc001aac000, 0x1000, 0x1000, 0x43f5bd, 0xc00024baf8, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000ce00c0, 0xc001aac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
bufio.(*Reader).Read(0xc00173c840, 0xc001a12168, 0x1, 0x9, 0x2, 0x3, 0x80)
	/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x1033de0, 0xc00173c840, 0xc001a12168, 0x1, 0x9, 0x1, 0x38, 0xc000b3c740, 0x0)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/gocql/gocql.readHeader(0x1033de0, 0xc00173c840, 0xc001a12168, 0x9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/frame.go:446 +0xab
github.com/gocql/gocql.(*Conn).recv(0xc001a12120, 0x0, 0x0)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:526 +0xf8
github.com/gocql/gocql.(*Conn).serve(0xc001a12120)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:490 +0x31
created by github.com/gocql/gocql.(*Session).dial
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:259 +0x749

goroutine 4447 [IO wait, 1884 minutes]:
internal/poll.runtime_pollWait(0x7f90700fbfe8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000685798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000685780, 0xc001a0e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000685780, 0xc001a0e000, 0x1000, 0x1000, 0x43f5bd, 0xc0000d0b38, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0007d4158, 0xc001a0e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
bufio.(*Reader).Read(0xc001118ae0, 0xc001918288, 0x1, 0x9, 0x7f906855b6d8, 0x41, 0x80)
	/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x1033de0, 0xc001118ae0, 0xc001918288, 0x1, 0x9, 0x1, 0x38, 0xc000b1b040, 0x0)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/gocql/gocql.readHeader(0x1033de0, 0xc001118ae0, 0xc001918288, 0x9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/frame.go:446 +0xab
github.com/gocql/gocql.(*Conn).recv(0xc001918240, 0x0, 0x0)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:526 +0xf8
github.com/gocql/gocql.(*Conn).serve(0xc001918240)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:490 +0x31
created by github.com/gocql/gocql.(*Session).dial
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/conn.go:259 +0x749

goroutine 87 [select]:
github.com/googleapis/gax-go.Sleep(0x104e540, 0xc00024d200, 0x2540be400, 0x0, 0x0)
	/go/pkg/mod/github.com/googleapis/gax-go@v2.0.0+incompatible/invoke.go:55 +0xd2
cloud.google.com/go/profiler.(*agent).profileAndUpload(0xc000509ac0, 0x104e540, 0xc00024d200, 0xc00945a1c0)
	/go/pkg/mod/cloud.google.com/go@v0.34.0/profiler/profiler.go:311 +0x5b5
cloud.google.com/go/profiler.pollProfilerService(0x104e540, 0xc00024d200, 0xc000509ac0)
	/go/pkg/mod/cloud.google.com/go@v0.34.0/profiler/profiler.go:527 +0x6f
created by cloud.google.com/go/profiler.start
	/go/pkg/mod/cloud.google.com/go@v0.34.0/profiler/profiler.go:204 +0x491

goroutine 30 [chan receive, 49 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0002d0000)
	/go/pkg/mod/google.golang.org/grpc@v1.18.0/clientconn.go:1070 +0x9e5
created by google.golang.org/grpc.(*addrConn).connect
	/go/pkg/mod/google.golang.org/grpc@v1.18.0/clientconn.go:684 +0xb6

goroutine 1818 [IO wait]:
internal/poll.runtime_pollWait(0x7f9068499008, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000150718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000150700, 0xc001068000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000150700, 0xc001068000, 0x1000, 0x1000, 0x43f5bd, 0xc000168118, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000010020, 0xc001068000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
bufio.(*Reader).Read(0xc000946540, 0xc000ddc288, 0x1, 0x9, 0x2, 0x3, 0x80)
	/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x1033de0, 0xc000946540, 0xc000ddc288, 0x1, 0x9, 0x1, 0x38, 0xc0157a5440, 0x0)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/gocql/gocql.readHeader(0x1033de0, 0xc000946540, 0xc000ddc288, 0x9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/scylladb/gocql@v1.0.1/frame.go:446 +0xab
github.com/gocql/gocql.(*Conn).recv(0xc000ddc240, 0x0, 0x0)
useful!

Related questions

cmd/link: segmentation fault during mach-o linking
cmd/go: cannot find module providing package error stops `go get` processing hot 3
vendor/golang.org/x/xerrors/adaptor_go1_13.go:16:14: undefined: errors.Frame ... hot 2
cmd/vet: potential false positive in the "suspect or" check hot 2
cmd/go: needs a better error than "missing dot in first path element" when GOROOT is set incorrectly hot 2
x/xerrors: fails to compile on tip hot 1
cmd/go: `go clean <package>` downloads modules hot 1
cmd/cgo error: runtime: unknown pc 0x7fff5c805b86 hot 1
cmd/link: showing many ld warnings of "building for macOS, but linking in object file" hot 1
runtime: go program crach, it seems fall into infinite loop hot 1
cmd/go: major version without preceding tag must be v0, not v1 - breaks build of github.com/go-check hot 1
gollvm: Using External Go Packages with gollvm hot 1
runtime: macOS Sierra builders spinning hot 1
cmd/go: Problem using go modules hot 1
cmd/go: "unrecognized import path" for local packages after updating to go1.13 hot 1
Github User Rank List