profile
viewpoint

hhy5861/distribution 1

The Docker toolset to pack, ship, store, and deliver content

hhy5861/Actions-OpenWrt 0

Build OpenWrt using GitHub Actions | 使用 GitHub Actions 编译 OpenWrt

hhy5861/agollo 0

🚀Go Client for Apollo

hhy5861/alioss 0

the SDK for aliyun oss with golang

hhy5861/analyzer-solr 0

analyzer adapter for solr 5, we support Jieba, and stranford in the future

hhy5861/api-auth 0

RESTful Auth for Yii2

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861 Also, can you post the picture of your cpu profile displaying the full stacks of the functions where cpu samples were collected please? (e.g. by typing "svg" in the go profiler)

I have now rolled back the version. The impact on me is relatively large, I suggest that this can be reproduced through stress testing.

hhy5861

comment created time in 6 days

create barnchhhy5861/chat

branch : feature/push

created branch time in 6 days

push eventhhy5861/chat

Mike

commit sha 8793c7f4b03e6979c53f446bfb5da12d87b0506c

remove desc

view details

push time in 6 days

delete branch hhy5861/chat

delete branch : debug-fix

delete time in 7 days

create barnchhhy5861/chat

branch : Alpha

created branch time in 7 days

create barnchhhy5861/chat

branch : debug-fix

created branch time in 7 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@or-else @aforge This modified version of the cpu has a big problem, I suggest you stress the sessions of 100,000 users

hhy5861

comment created time in 7 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861 If I am reading it correctly, it appears your chat server instances are consuming ~1 cpu each. The rest doesn't look that bad at a quick glance. Are your instances being starved of cpu? What happens if you increase their cpu budget?

@hhy5861 But is the single instance (node) maxing out all the 32 cores that are at its disposal?

Is it a bit wasteful to use a 1core cpu for each chat room? The monitoring chart above is fully loaded with 32core.

hhy5861

comment created time in 7 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861 If I am reading it correctly, it appears your chat server instances are consuming ~1 cpu each. The rest doesn't look that bad at a quick glance. Are your instances being starved of cpu? What happens if you increase their cpu budget?

@aforge Now there are 20 nodes, each node is 32 Core

hhy5861

comment created time in 7 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861

Can you grab a cpu profile please?

Saved profile in /Users/mike/pprof/pprof.spider-linux.samples.cpu.007.pb.gz
File: spider-linux
Type: cpu
Time: Oct 25, 2020 at 4:00am (CST)
Duration: 30.17s, Total samples = 5mins (994.31%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 178.39s, 59.46% of 300.01s total
Dropped 734 nodes (cum <= 1.50s)
Showing top 10 nodes out of 96
      flat  flat%   sum%        cum   cum%
    30.24s 10.08% 10.08%     30.25s 10.08%  google.golang.org/grpc/internal/backoff.Exponential.Backoff
    28.95s  9.65% 19.73%     77.05s 25.68%  runtime.selectgo
    22.80s  7.60% 27.33%     79.44s 26.48%  runtime.mallocgc
    18.67s  6.22% 33.55%     22.40s  7.47%  runtime.heapBitsSetType
    18.01s  6.00% 39.56%     18.49s  6.16%  runtime.lock
    16.16s  5.39% 44.94%     16.24s  5.41%  runtime.unlock
    15.54s  5.18% 50.12%     21.13s  7.04%  runtime.scanobject
    12.65s  4.22% 54.34%    177.33s 59.11%  reflect.Select
     7.96s  2.65% 56.99%      7.96s  2.65%  runtime.nextFreeFast
     7.41s  2.47% 59.46%      7.41s  2.47%  runtime.memclrNoHeapPointers
(pprof) top 20
Showing nodes accounting for 3.64mins, 72.71% of 5mins total
Dropped 734 nodes (cum <= 0.03mins)
Showing top 20 nodes out of 96
      flat  flat%   sum%        cum   cum%
  0.50mins 10.08% 10.08%   0.50mins 10.08%  google.golang.org/grpc/internal/backoff.Exponential.Backoff
  0.48mins  9.65% 19.73%   1.28mins 25.68%  runtime.selectgo
  0.38mins  7.60% 27.33%   1.32mins 26.48%  runtime.mallocgc
  0.31mins  6.22% 33.55%   0.37mins  7.47%  runtime.heapBitsSetType
  0.30mins  6.00% 39.56%   0.31mins  6.16%  runtime.lock
  0.27mins  5.39% 44.94%   0.27mins  5.41%  runtime.unlock
  0.26mins  5.18% 50.12%   0.35mins  7.04%  runtime.scanobject
  0.21mins  4.22% 54.34%   2.96mins 59.11%  reflect.Select
  0.13mins  2.65% 56.99%   0.13mins  2.65%  runtime.nextFreeFast
  0.12mins  2.47% 59.46%   0.12mins  2.47%  runtime.memclrNoHeapPointers
  0.09mins  1.76% 61.22%   1.87mins 37.49%  reflect.rselect
  0.09mins  1.73% 62.95%   0.11mins  2.14%  runtime.findObject
  0.08mins  1.66% 64.61%   0.09mins  1.78%  runtime.nanotime
  0.08mins  1.60% 66.20%   0.08mins  1.60%  runtime.casgstatus
  0.06mins  1.28% 67.48%   0.06mins  1.28%  runtime.acquirem
  0.06mins  1.16% 68.64%   0.08mins  1.60%  runtime.releaseSudog
  0.05mins  1.08% 69.72%   0.08mins  1.54%  runtime.acquireSudog
  0.05mins  1.02% 70.74%   0.25mins  5.07%  runtime.chansend
  0.05mins  1.00% 71.74%   0.05mins  1.00%  runtime.(*hchan).sortkey
  0.05mins  0.97% 72.71%   0.24mins  4.85%  runtime.sellock

heap

File: spider-linux
Type: inuse_space
Time: Oct 25, 2020 at 4:02am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 10
Showing nodes accounting for 4770.88MB, 97.68% of 4883.97MB total
Dropped 203 nodes (cum <= 24.42MB)
Showing top 10 nodes out of 19
      flat  flat%   sum%        cum   cum%
 3854.36MB 78.92% 78.92%  3855.36MB 78.94%  main.(*SessionStore).NewSession
  390.15MB  7.99% 86.91%   390.15MB  7.99%  reflect.rselect
  361.59MB  7.40% 94.31%   751.74MB 15.39%  reflect.Select
  113.67MB  2.33% 96.64%   117.17MB  2.40%  main.(*Hub).run
      30MB  0.61% 97.25%   781.74MB 16.01%  main.(*Topic).clusterSelectProxyEvent
   20.60MB  0.42% 97.67%    26.61MB  0.54%  github.com/gorilla/websocket.newConn
    0.50MB  0.01% 97.68%  3333.88MB 68.26%  main.(*Cluster).TopicMaster
         0     0% 97.68%    26.61MB  0.54%  github.com/gorilla/websocket.(*Upgrader).Upgrade
         0     0% 97.68%   781.74MB 16.01%  main.(*Topic).clusterWriteLoop
         0     0% 97.68%    25.51MB  0.52%  main.(*Topic).run
hhy5861

comment created time in 7 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861

Can you grab a cpu profile please?

heap profile: 6010: 941128296 [4016301: 2211623256] @ heap/1048576
3639: 775077888 [3706: 789348352] @ 0xd019cf 0xcbf3f5 0x49cf4b 0x49c454 0x96b04f 0x467aa1
#	0xd019ce	main.(*SessionStore).NewSession+0x18e	/Users/mike/golang/poizon/duinfr/dewu-chat/server/sessionstore.go:72
#	0xcbf3f4	main.(*Cluster).TopicMaster+0xd64	/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:462
#	0x49cf4a	reflect.Value.call+0x8aa		/Users/mike/.gvm/gos/go1.14.9/src/reflect/value.go:460
#	0x49c453	reflect.Value.Call+0xb3			/Users/mike/.gvm/gos/go1.14.9/src/reflect/value.go:321
#	0x96b04e	net/rpc.(*service).call+0x17e		/Users/mike/.gvm/gos/go1.14.9/src/net/rpc/server.go:377

770: 164003840 [2182: 464748544] @ 0xd019cf 0xcce0d2 0x7f89e4 0x7fa825 0x7fbd73 0x7f77dc 0x467aa1
#	0xd019ce	main.(*SessionStore).NewSession+0x18e	/Users/mike/golang/poizon/duinfr/dewu-chat/server/sessionstore.go:72
#	0xcce0d1	main.serveWebSocket+0x521		/Users/mike/golang/poizon/duinfr/dewu-chat/server/hdl_websock.go:170
#	0x7f89e3	net/http.HandlerFunc.ServeHTTP+0x43	/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:2041
#	0x7fa824	net/http.(*ServeMux).ServeHTTP+0x1a4	/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:2416
#	0x7fbd72	net/http.serverHandler.ServeHTTP+0xa2	/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:2836
#	0x7f77db	net/http.(*conn).serve+0x86b		/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:1924

1: 524288 [1: 524288] @ 0xccf5b8 0xcdcfac 0x4370aa 0x467aa1
#	0xccf5b7	main.newHub+0x47	/Users/mike/golang/poizon/duinfr/dewu-chat/server/hub.go:123
#	0xcdcfab	main.main+0x13cb	/Users/mike/golang/poizon/duinfr/dewu-chat/server/main.go:470
#	0x4370a9	runtime.main+0x1f9	/Users/mike/.gvm/gos/go1.14.9/src/runtime/proc.go:203

1: 458752 [1: 458752] @ 0x40df03 0x40dc9f 0x473b0e 0x472967 0xcd12f2 0xcd1296 0x467aa1
#	0x473b0d	sync.(*Map).dirtyLocked+0xad	/Users/mike/.gvm/gos/go1.14.9/src/sync/map.go:355
#	0x472966	sync.(*Map).Store+0x476		/Users/mike/.gvm/gos/go1.14.9/src/sync/map.go:157
#	0xcd12f1	main.(*Hub).topicPut+0x1401	/Users/mike/golang/poizon/duinfr/dewu-chat/server/hub.go:112
#	0xcd1295	main.(*Hub).run+0x13a5		/Users/mike/golang/poizon/duinfr/dewu-chat/server/hub.go:208

25: 134400 [55: 295680] @ 0xa658c3 0xa6bba1 0xccdfb3 0x7f89e4 0x7fa825 0x7fbd73 0x7f77dc 0x467aa1
#	0xa658c2	github.com/gorilla/websocket.newConn+0x2c2		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:302
#	0xa6bba0	github.com/gorilla/websocket.(*Upgrader).Upgrade+0x530	/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/server.go:202
#	0xccdfb2	main.serveWebSocket+0x402				/Users/mike/golang/poizon/duinfr/dewu-chat/server/hdl_websock.go:161
#	0x7f89e3	net/http.HandlerFunc.ServeHTTP+0x43			/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:2041
#	0x7fa824	net/http.(*ServeMux).ServeHTTP+0x1a4			/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:2416
#	0x7fbd72	net/http.serverHandler.ServeHTTP+0xa2			/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:2836
#	0x7f77db	net/http.(*conn).serve+0x86b				/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:1924

49: 100352 [64: 131072] @ 0xcd1091 0x467aa1
#	0xcd1090	main.(*Hub).run+0x11a0	/Users/mike/golang/poizon/duinfr/dewu-chat/server/hub.go:190

20: 81920 [51: 208896] @ 0x731cff 0x7352e1 0x73567a 0x74e803 0x74ca3f 0x74c5f1 0x7376c1 0x7f710d 0x467aa1
#	0x731cfe	crypto/tls.(*halfConn).encrypt+0x11e				/Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:447
#	0x7352e0	crypto/tls.(*Conn).writeRecordLocked+0x240			/Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:940
#	0x735679	crypto/tls.(*Conn).writeRecord+0xb9				/Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:966
#	0x74e802	crypto/tls.(*serverHandshakeState).doFullHandshake+0x282	/Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/handshake_server.go:459
#	0x74ca3e	crypto/tls.(*serverHandshakeState).handshake+0x31e		/Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/handshake_server.go:105
#	0x74c5f0	crypto/tls.(*Conn).serverHandshake+0x100			/Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/handshake_server.go:60
#	0x7376c0	crypto/tls.(*Conn).Handshake+0x1e0				/Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:1342
#	0x7f710c	net/http.(*conn).serve+0x19c					/Users/mike/.gvm/gos/go1.14.9/src/net/http/server.go:1816

40: 81920 [63: 129024] @ 0xcd1045 0x467aa1
#	0xcd1044	main.(*Hub).run+0x1154	/Users/mike/golang/poizon/duinfr/dewu-chat/server/hub.go:188

38: 77824 [61: 124928] @ 0xcd106b 0x467aa1
#	0xcd106a	main.(*Hub).run+0x117a	/Users/mike/golang/poizon/duinfr/dewu-chat/server/hub.go:189

1: 65536 [1: 65536] @ 0x87802c 0x877fa5 0x861629 0x8e2d65 0x8e2c3f 0x8e25ed 0x8e1b67 0x467aa1
#	0x87802b	google.golang.org/grpc/internal/transport.newBufWriter+0xeb		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/http_util.go:542
#	0x877fa4	google.golang.org/grpc/internal/transport.newFramer+0x64		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/http_util.go:595
#	0x861628	google.golang.org/grpc/internal/transport.newHTTP2Client+0xa88		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/http2_client.go:259
#	0x8e2d64	google.golang.org/grpc/internal/transport.NewClientTransport+0x414	/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/transport.go:577
#	0x8e2c3e	google.golang.org/grpc.(*addrConn).createTransport+0x2ee		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/clientconn.go:1282
#	0x8e25ec	google.golang.org/grpc.(*addrConn).tryAllAddrs+0x39c			/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/clientconn.go:1212
#	0x8e1b66	google.golang.org/grpc.(*addrConn).resetTransport+0x236			/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/clientconn.go:1127

224: 64512 [429302: 123638976] @ 0x447fda 0x4a4fe9 0xcc3297 0xcc3822 0x467aa1
#	0x447fd9	reflect.rselect+0x59				/Users/mike/.gvm/gos/go1.14.9/src/runtime/select.go:527
#	0x4a4fe8	reflect.Select+0x178				/Users/mike/.gvm/gos/go1.14.9/src/reflect/value.go:2236
#	0xcc3296	main.(*Topic).clusterSelectProxyEvent+0xe6	/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1127
#	0xcc3821	main.(*Topic).clusterWriteLoop+0x101		/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1170

19: 34048 [14758: 26446336] @ 0x447fda 0x4a4fe9 0xcc3297 0xcc3822 0x467aa1
#	0x447fd9	reflect.rselect+0x59				/Users/mike/.gvm/gos/go1.14.9/src/runtime/select.go:527
#	0x4a4fe8	reflect.Select+0x178				/Users/mike/.gvm/gos/go1.14.9/src/reflect/value.go:2236
#	0xcc3296	main.(*Topic).clusterSelectProxyEvent+0xe6	/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1127
#	0xcc3821	main.(*Topic).clusterWriteLoop+0x101		/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1170

1: 32768 [1: 32768] @ 0x878293 0x878356 0x861629 0x8e2d65 0x8e2c3f 0x8e25ed 0x8e1b67 0x467aa1
#	0x878292	bufio.NewReaderSize+0x352						/Users/mike/.gvm/gos/go1.14.9/src/bufio/bufio.go:56
#	0x878355	google.golang.org/grpc/internal/transport.newFramer+0x415		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/http_util.go:593
#	0x861628	google.golang.org/grpc/internal/transport.newHTTP2Client+0xa88		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/http2_client.go:259
#	0x8e2d64	google.golang.org/grpc/internal/transport.NewClientTransport+0x414	/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/transport.go:577
#	0x8e2c3e	google.golang.org/grpc.(*addrConn).createTransport+0x2ee		/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/clientconn.go:1282
#	0x8e25ec	google.golang.org/grpc.(*addrConn).tryAllAddrs+0x39c			/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/clientconn.go:1212
#	0x8e1b66	google.golang.org/grpc.(*addrConn).resetTransport+0x236			/Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/google.golang.org/grpc@v1.31.0/clientconn.go:1127

146: 32704 [333708: 74750592] @ 0x4a4ec4 0xcc3297 0xcc3822 0x467aa1
#	0x4a4ec3	reflect.Select+0x53				/Users/mike/.gvm/gos/go1.14.9/src/reflect/value.go:2170
#	0xcc3296	main.(*Topic).clusterSelectProxyEvent+0xe6	/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1127
#	0xcc3821	main.(*Topic).clusterWriteLoop+0x101		/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1170

image

hhy5861

comment created time in 7 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861 I've sent #559 for review. It should make the situation better.

image

Updated your fix commit cpu full load

hhy5861

comment created time in 7 days

delete branch hhy5861/chat

delete branch : topic-proxy

delete time in 7 days

create barnchhhy5861/chat

branch : MR-DEVEL

created branch time in 7 days

delete branch hhy5861/chat

delete branch : devel

delete time in 7 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

··· goroutine profile: total 142928 124108 @ 0x437470 0x447833 0x44830a 0x4a4fe9 0xcc2708 0x467aa1

0x448309 reflect.rselect+0x389 /Users/mike/.gvm/gos/go1.14.9/src/runtime/select.go:544

0x4a4fe8 reflect.Select+0x178 /Users/mike/.gvm/gos/go1.14.9/src/reflect/value.go:2236

0xcc2707 main.(*Topic).clusterWriteLoop+0x117 /Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1131

7753 @ 0x437470 0x447833 0xd1fdd8 0xd0149b 0x467aa1

0xd1fdd7 main.(*Topic).runProxy+0x257 /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic_proxy.go:22

0xd0149a main.(*Topic).run+0x3a /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:212

2861 @ 0x437470 0x447833 0xd01c0d 0xd014b8 0x467aa1

0xd01c0c main.(*Topic).runLocal+0x2bc /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:305

0xd014b7 main.(*Topic).run+0x57 /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:210

2568 @ 0x437470 0x43042a 0x42f995 0x4dec15 0x4dfaab 0x4dfa8d 0x645d2f 0x65874e 0x734640 0x47f711 0x73488c 0x732e45 0x736f6b 0x736f76 0x59aa73 0x59abdf 0xa65470 0xa679cc 0xa68a5f 0xa691df 0xccc222 0x467aa1

0x42f994 internal/poll.runtime_pollWait+0x54 /Users/mike/.gvm/gos/go1.14.9/src/runtime/netpoll.go:203

0x4dec14 internal/poll.(*pollDesc).wait+0x44 /Users/mike/.gvm/gos/go1.14.9/src/internal/poll/fd_poll_runtime.go:87

0x4dfaaa internal/poll.(*pollDesc).waitRead+0x19a /Users/mike/.gvm/gos/go1.14.9/src/internal/poll/fd_poll_runtime.go:92

0x4dfa8c internal/poll.(*FD).Read+0x17c /Users/mike/.gvm/gos/go1.14.9/src/internal/poll/fd_unix.go:169

0x645d2e net.(*netFD).Read+0x4e /Users/mike/.gvm/gos/go1.14.9/src/net/fd_unix.go:202

0x65874d net.(*conn).Read+0x8d /Users/mike/.gvm/gos/go1.14.9/src/net/net.go:184

0x73463f crypto/tls.(*atLeastReader).Read+0x5f /Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:760

0x47f710 bytes.(*Buffer).ReadFrom+0xb0 /Users/mike/.gvm/gos/go1.14.9/src/bytes/buffer.go:204

0x73488b crypto/tls.(*Conn).readFromUntil+0xeb /Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:782

0x732e44 crypto/tls.(*Conn).readRecordOrCCS+0x114 /Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:589

0x736f6a crypto/tls.(*Conn).readRecord+0x15a /Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:557

0x736f75 crypto/tls.(*Conn).Read+0x165 /Users/mike/.gvm/gos/go1.14.9/src/crypto/tls/conn.go:1233

0x59aa72 bufio.(*Reader).fill+0x102 /Users/mike/.gvm/gos/go1.14.9/src/bufio/bufio.go:100

0x59abde bufio.(*Reader).Peek+0x4e /Users/mike/.gvm/gos/go1.14.9/src/bufio/bufio.go:138

0xa6546f github.com/gorilla/websocket.(*Conn).read+0x3f /Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:370

0xa679cb github.com/gorilla/websocket.(*Conn).advanceFrame+0x5b /Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:798

0xa68a5e github.com/gorilla/websocket.(*Conn).NextReader+0x8e /Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:980

0xa691de github.com/gorilla/websocket.(*Conn).ReadMessage+0x2e /Users/mike/.gvm/pkgsets/go1.14.9/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:1064

0xccc221 main.(*Session).readLoop+0x191 /Users/mike/golang/poizon/duinfr/dewu-chat/server/hdl_websock.go:53

2568 @ 0x437470 0x447833 0xccc598 0x467aa1

0xccc597 main.(*Session).writeLoop+0x1c7 /Users/mike/golang/poizon/duinfr/dewu-chat/server/hdl_websock.go:76

1463 @ 0x437470 0x405f17 0x405bdb 0xd29f7e 0x467aa1

0xd29f7d main.(*ClusterNode).callAsync.func1+0x3d /Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:342

1171 @ 0x437470 0x44834f 0x448358 0x4a4fe9 0xcc2708 0x467aa1

0x448357 reflect.rselect+0x3d7 /Users/mike/.gvm/gos/go1.14.9/src/runtime/select.go:525

0x4a4fe8 reflect.Select+0x178 /Users/mike/.gvm/gos/go1.14.9/src/reflect/value.go:2236

0xcc2707 main.(*Topic).clusterWriteLoop+0x117 /Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1131

126 @ 0x437470 0x4052fd 0x4050c5 0xd1ed04 0xd1f746 0xd037a9 0xd02aab 0xd014b8 0x467aa1

0xd1ed03 main.(*Topic).remProxiedSession+0x283 /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:3176

0xd1f745 main.(*Topic).remSession+0x3c5 /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:3233

0xd037a8 main.(*Topic).handleLeaveRequest+0xd8 /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:557

0xd02aaa main.(*Topic).runLocal+0x115a /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:338

0xd014b7 main.(*Topic).run+0x57 /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:210

59 @ 0x437470 0x448770 0x44875b 0x4483c2 0x4758f4 0xcf64fb 0xd2a656 0xccc2c1 0x467aa1

0x4483c1 sync.runtime_Semacquire+0x41 /Users/mike/.gvm/gos/go1.14.9/src/runtime/sema.go:56

0x4758f3 sync.(*WaitGroup).Wait+0x63 /Users/mike/.gvm/gos/go1.14.9/src/sync/waitgroup.go:130

0xcf64fa main.(*Session).cleanUp+0x4a /Users/mike/golang/poizon/duinfr/dewu-chat/server/session.go:369

0xd2a655 main.(*Session).readLoop.func1+0x35 /Users/mike/golang/poizon/duinfr/dewu-chat/server/hdl_websock.go:41

0xccc2c0 main.(*Session).readLoop+0x230 /Users/mike/golang/poizon/duinfr/dewu-chat/server/hdl_websock.go:59

49 @ 0x437470 0x405f17 0x405c1b 0xcbca9f 0x467aa1

0xcbca9e main.(*ClusterNode).asyncRpcLoop+0x6e /Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:91

49 @ 0x437470 0x43042a 0x42f995 0x4dec15 0x4dfaab 0x4dfa8d 0x645d2f 0x65874e 0x59b12f 0x475db7 0x94a33f 0x94a2f2 0x954997 0x954eec 0x9551fb 0x95507d 0x9683a5 0x967675 0x467aa1

0x42f994 internal/poll.runtime_pollWait+0x54 /Users/mike/.gvm/gos/go1.14.9/src/runtime/netpoll.go:203

0x4dec14 internal/poll.(*pollDesc).wait+0x44 /Users/mike/.gvm/gos/go1.14.9/src/internal/poll/fd_poll_runtime.go:87

0x4dfaaa internal/poll.(*pollDesc).waitRead+0x19a /Users/mike/.gvm/gos/go1.14.9/src/internal/poll/fd_poll_runtime.go:92

0x4dfa8c internal/poll.(*FD).Read+0x17c /Users/mike/.gvm/gos/go1.14.9/src/internal/poll/fd_unix.go:169

0x645d2e net.(*netFD).Read+0x4e /Users/mike/.gvm/gos/go1.14.9/src/net/fd_unix.go:202

0x65874d net.(*conn).Read+0x8d /Users/mike/.gvm/gos/go1.14.9/src/net/net.go:184

0x59b12e bufio.(*Reader).Read+0x24e /Users/mike/.gvm/gos/go1.14.9/src/bufio/bufio.go:226

0x475db6 io.ReadAtLeast+0x86 /Users/mike/.gvm/gos/go1.14.9/src/io/io.go:310

0x94a33e io.ReadFull+0x6e /Users/mike/.gvm/gos/go1.14.9/src/io/io.go:329

0x94a2f1 encoding/gob.decodeUintReader+0x21 /Users/mike/.gvm/gos/go1.14.9/src/encoding/gob/decode.go:120

0x954996 encoding/gob.(*Decoder).recvMessage+0x56 /Users/mike/.gvm/gos/go1.14.9/src/encoding/gob/decoder.go:81

0x954eeb encoding/gob.(*Decoder).decodeTypeSequence+0x10b /Users/mike/.gvm/gos/go1.14.9/src/encoding/gob/decoder.go:143

0x9551fa encoding/gob.(*Decoder).DecodeValue+0xda /Users/mike/.gvm/gos/go1.14.9/src/encoding/gob/decoder.go:211

0x95507c encoding/gob.(*Decoder).Decode+0x16c /Users/mike/.gvm/gos/go1.14.9/src/encoding/gob/decoder.go:188

0x9683a4 net/rpc.(*gobClientCodec).ReadResponseHeader+0x44 /Users/mike/.gvm/gos/go1.14.9/src/net/rpc/client.go:228

0x967674 net/rpc.(*Client).input+0xa4 /Users/mike/.gvm/gos/go1.14.9/src/net/rpc/client.go:109

···

hhy5861

comment created time in 8 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

File: spider-linux
Type: inuse_space
Time: Oct 24, 2020 at 2:35am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 10
Showing nodes accounting for 16221.35MB, 97.38% of 16657.33MB total
Dropped 292 nodes (cum <= 83.29MB)
Showing top 10 nodes out of 13
      flat  flat%   sum%        cum   cum%
15762.05MB 94.63% 94.63% 15782.56MB 94.75%  main.(*SessionStore).NewSession
  454.72MB  2.73% 97.36%   459.28MB  2.76%  main.newHub.func1
    4.58MB 0.027% 97.38% 14148.08MB 84.94%  main.(*Cluster).TopicMaster
         0     0% 97.38%    92.02MB  0.55%  main.(*Topic).run
         0     0% 97.38%  1702.31MB 10.22%  main.serveWebSocket
         0     0% 97.38%    94.08MB  0.56%  main.topicInit
         0     0% 97.38%  1702.31MB 10.22%  net/http.(*ServeMux).ServeHTTP
         0     0% 97.38%  1741.42MB 10.45%  net/http.(*conn).serve
         0     0% 97.38%  1702.31MB 10.22%  net/http.HandlerFunc.ServeHTTP
         0     0% 97.38%  1702.31MB 10.22%  net/http.serverHandler.ServeHTTP
hhy5861

comment created time in 8 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861 : my apologies for late response (I was traveling, just arrived at my destination). Here's a temporary fix: 67225b9 I have identified the source of the problem. I'll make a proper fix tomorrow.

@aforge @or-else This problem is very serious, the response message will be lost, and a memory leak will also occur.

hhy5861

comment created time in 8 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861 : my apologies for late response (I was traveling, just arrived at my destination). Here's a temporary fix: 67225b9 I have identified the source of the problem. I'll make a proper fix tomorrow.

Now there is a serious problem, sub did not return ctrl

hhy5861

comment created time in 9 days

create barnchhhy5861/chat

branch : RC

created branch time in 11 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

2020/10/21 02:59:49 cluster: stop msg received - multi sid grpWcnHgjfpNU0-node-08
2020/10/21 02:59:49 cluster: session proxy closed grpWcnHgjfpNU0-node-08
2020/10/21 02:59:49 cluster: session proxy closed grpWcnHgjfpNU0-node-15
2020/10/21 02:59:49 cluster: session proxy closed grpWcnHgjfpNU0-node-04
panic: runtime error: index out of range [2] with length 2

goroutine 617505 [running]:
main.(*Topic).clusterWriteLoop(0xc015624000)
	/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1144 +0xb60
created by main.(*Topic).addProxiedSession
	/Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:3130 +0xb03

image

@aforge @or-else

  • this problem still exists
hhy5861

comment created time in 11 days

issue commenttinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

@hhy5861 You may also want to patch this one: 1344186

ok, thx

hhy5861

comment created time in 16 days

issue openedtinode/chat

Emergency clusterWriteLoop addProxiedSession panic error

2020/10/14 21:55:15 cluster: stop msg received - multi sid usrF6DZPtYyirM-node-02
2020/10/14 21:55:15 cluster: session proxy closed usrF6DZPtYyirM-node-02
panic: runtime error: index out of range [0] with length 0

goroutine 229 [running]:
main.(*Topic).clusterWriteLoop(0xc0005d1d40)
	/Users/mike/golang/poizon/duinfr/dewu-chat/server/cluster.go:1137 +0xa62
created by main.(*Topic).addProxiedSession
	/Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:3125 +0xb03

image

image

created time in 17 days

issue commenttinode/chat

Cluster communication is upgraded to grpc

@aforge @or-else

Is there any way for me to send messages to the specified topic without logging in or subscribing?

hhy5861

comment created time in 20 days

push eventhhy5861/goim

Mike

commit sha bbcff261fe51d7673c98f286f6530e97a20e596a

reset

view details

push time in 24 days

create barnchhhy5861/goim

branch : Alpha

created branch time in 24 days

PR opened tinode/chat

fix Evict all user's sessions, clear cached data, send notifications …

Panicf: proxy received evict message with empty uid

// proxyCtrlBroadcast broadcasts a ctrl command to certain sessions attached to this proxy topic.
func (t *Topic) proxyCtrlBroadcast(msg *ServerComMessage) {
	if msg.Ctrl.Code == http.StatusResetContent && msg.Ctrl.Text == "evicted" {
		// We received a ctrl command for evicting a user.
		if msg.uid.IsZero() {
			log.Panicf("topic[%s]: proxy received evict message with empty uid", t.name)
		}
		for sess := range t.sessions {
			// Proxy topic may only have ordinary sessions. No multiplexing or proxy sessions here.
			if _, removed := t.remSession(sess, msg.uid); removed {
				sess.detachSession(t.name)
				if sess.sid != msg.SkipSid {
					sess.queueOut(msg)
				}
			}
		}
	}
}

// TopicProxy is a gRPC endpoint at topic proxy which receives topic master responses.
func (Cluster) TopicProxy(msg *ClusterResp, unused *bool) error {
	// This cluster member received a response from the topic master to be forwarded to the topic.
	// Find appropriate topic, send the message to it.
	if t := globals.hub.topicGet(msg.RcptTo); t != nil {
		msg.SrvMsg.uid = types.ParseUserId(msg.SrvMsg.AsUser)
		t.proxy <- msg
	} else {
		log.Println("cluster: master response for unknown topic", msg.RcptTo)
	}

	return nil
}
+1 -0

0 comment

1 changed file

pr created time in a month

push eventhhy5861/chat

Mike

commit sha bd74fcbceb0d59c374edad6f687df804915a864b

fix Evict all user's sessions, clear cached data, send notifications msg.AsUser empty uid

view details

push time in a month

fork hhy5861/chat

Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots

fork in a month

issue openedtinode/chat

panic: runtime error: index out of range [12] with length 12

version: v0.16.8-rc1

` panic: runtime error: index out of range [12] with length 12

goroutine 11923 [running]: main.(*Topic).remProxiedSession(0xc000766d00, 0xc00088c780, 0xc00088c780) /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:3231 +0x30f main.(*Topic).remSession(0xc000766d00, 0xc00088c780, 0x0, 0xc000201bc0, 0xc000661a78) /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:3293 +0x3c6 main.(*Topic).handleLeaveRequest(0xc000766d00, 0xc0000a5f40, 0xc000354cd0) /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:572 +0xd9 main.(*Topic).runLocal(0xc000766d00, 0xc0000a5f40) /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:325 +0x14ce main.(*Topic).run(0xc000766d00, 0xc0000a5f40) /Users/mike/golang/poizon/duinfr/dewu-chat/server/topic.go:205 +0x58 created by main.topicInit /Users/mike/golang/poizon/duinfr/dewu-chat/server/init_topic.go:129 +0xdb4 `

created time in a month

issue openedtinode/chat

Cluster communication is upgraded to grpc

Have you considered upgrading cluster communication to grpc, so that network overhead and performance should be improved a lot.

created time in a month

startedosixia/docker-openldap

started time in a month

startedTencent/Hippy

started time in a month

startedgomods/athens

started time in a month

startedspotahome/redis-operator

started time in a month

startedVictoriaMetrics/VictoriaMetrics

started time in a month

issue commenttinode/chat

Message loss problem

When the message arrives on the route, no case will be hit.

That is incorrect. The first case h.topicGet is usually satisfied. You can see it by sending messages in web.tinode.co and sandbox.tinode.co. These are cluster setups. Messaging works.

If you believe that there is a problem, please help us identify it by providing steps to reproduce. Thank you.

From the code analysis and printed logs, some {data} did not hit h.getTopic when the route was reached.

This is difficult to submit a case to. My current solution is to add a channel to handle this {data}message as an offline message.

hhy5861

comment created time in a month

issue commenttinode/chat

Message loss problem

Could you please show steps to reproduce. Thanks.

image

If isProxy is true, then the message is proxied to topicMaster. So the message is sent to hub.route. When the message arrives on the route, no case will be hit. Then just discard it.

image

hhy5861

comment created time in a month

startedcli/cli

started time in a month

issue openedtinode/chat

Message loss problem

https://github.com/tinode/chat/blob/7bed3c0bd89469cc3cdafd39775697a79383f936/server/hub.go#L231-L237

  1. I found a problem with discarding {data} data points. When the data is proxyed to another node, the message will be discarded if the session is offline or cannot get the subscription.

  2. I think that when this happens, the message should be stored persistently and handled as an offline message.

created time in a month

startedeyasliu/blog

started time in a month

issue commenttinode/chat

A very strange deadlock

Yes, I think I know what the problem is. I've posted #542. Will likely need an even better hash function in the future.

I updated what you modified to solve the problem, and now the load is fairly balanced.

hhy5861

comment created time in 2 months

startedgrepplabs/kafka-proxy

started time in 2 months

startedprometheus/jmx_exporter

started time in 2 months

startedkubernetes-sigs/external-dns

started time in 2 months

startedstrimzi/strimzi-kafka-operator

started time in 2 months

startedspaolacci/murmur3

started time in 2 months

issue commenttinode/chat

A very strange deadlock

Assuming your heap profile is for node 0, your average topic appears to account for: 10584MB / 24119 = 0.439M Can you check the size distribution of avatars in your database please? And your average session is: 4496.37MB / 21181 = 0.212M Can you post the profile picture with the full allocation call stack please?

I removed the avatar of the user information

image What is the cause of unbalanced load?

hhy5861

comment created time in 2 months

startedShopify/toxiproxy

started time in 2 months

issue commenttinode/chat

A very strange deadlock

Assuming your heap profile is for node 0, your average topic appears to account for: 10584MB / 24119 = 0.439M

Can you check the size distribution of avatars in your database please?

And your average session is: 4496.37MB / 21181 = 0.212M

Can you post the profile picture with the full allocation call stack please?

I removed the avatar of the user information

hhy5861

comment created time in 2 months

issue commenttinode/chat

A very strange deadlock

image

@aforge @or-else

Found that the memory usage is still high.

hhy5861

comment created time in 2 months

startedYolean/kubernetes-kafka

started time in 2 months

fork hhy5861/kubernetes-kafka

Kafka cluster as Kubernetes StatefulSet, plain manifests and config

fork in 2 months

startedShopify/sarama

started time in 2 months

issue commenttinode/chat

A very strange deadlock

@aforge This problem is very serious and will cause all nodes in the cluster to be paralyzed.

hhy5861

comment created time in 2 months

issue commenttinode/chat

A very strange deadlock

8f7a1e2 should help this. I subscribed 5000 sessions to a user's me within a short period of time - and saw no stalling.

@hhy5861 Can you try it and see if it reproduces the problem for you still?

My preliminary judgment should be that this case is blocked, which will cause all cases to fail. https://github.com/tinode/chat/blob/7bed3c0bd89469cc3cdafd39775697a79383f936/server/hub.go#L210

hhy5861

comment created time in 2 months

issue openedtinode/chat

A very strange deadlock

tinode version: 0.16.6

  1. A large number of users subscribe to me will cause a deadlock. There is no response to the ctrl event.
  2. The case has not been found yet.
  3. This phenomenon can be reproduced by pressure test. Stand-alone also appeared.

created time in 2 months

issue commenttinode/chat

too many goruntines cause very high memory usage.

I can't reproduce it locally on a 3-node setup. So with ~10000 active local topics, ~2600 websocket sessions, you've got ~278000 proxy connections - which is a little high for a 20-node setup, let alone incoming proxy connections (~626000).

@hhy5861 did this profile result from a natural load over a period of time or some kind of synthetic test? If latter, what did it look like? this is the real data of the production environment. This is a sample taken at the time point when a node load is the highest. But the load of other nodes is similar.

Also, would it be possible for you to grab these goroutine profiles from the remaining nodes please? I checked the distribution of other nodes are almost the same. The memory usage is still high, mainly because the session recovery mechanism is still problematic. It should be caused by topic proxy.

I'd like to look at the number of topics, number of sessions and and local and remote proxy session distribution.

12 34

# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 549
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP spider_cluster_leader If this cluster node is the cluster leader.
# TYPE spider_cluster_leader gauge
spider_cluster_leader 0
# HELP spider_cluster_nodes_live Number of cluster nodes believed to be live by the current node.
# TYPE spider_cluster_nodes_live gauge
spider_cluster_nodes_live 19
# HELP spider_cluster_size Configured number of cluster nodes.
# TYPE spider_cluster_size gauge
spider_cluster_size 20
# HELP spider_malloced_bytes Number of bytes of memory allocated and in use.
# TYPE spider_malloced_bytes gauge
spider_malloced_bytes 1.0655576848e+10
# HELP spider_sessions_live_count Number of currently active sessions.
# TYPE spider_sessions_live_count gauge
spider_sessions_live_count 780081
# HELP spider_sessions_total Total number of sessions since instance start.
# TYPE spider_sessions_total counter
spider_sessions_total 1.307638e+06
# HELP spider_topics_live_count Number of currently active topics.
# TYPE spider_topics_live_count gauge
spider_topics_live_count 498192
# HELP spider_topics_total Total number of topics used during instance lifetime.
# TYPE spider_topics_total counter
spider_topics_total 1.40286e+06
# HELP spider_up If tinode instance is reachable.
# TYPE spider_up gauge
spider_up 1
# HELP spider_version The version of this tinode instance.
# TYPE spider_version gauge
spider_version 1600
hhy5861

comment created time in 2 months

issue commenttinode/chat

How about the performance of tinode? Will we do some benchmarking

@aforge Because instant message system technology selection

I found that the cluster mode uses very high memory. The Session recycling mechanism also has some problems. 12

warfrogsdf

comment created time in 2 months

issue openedtinode/chat

too many goruntines cause very high memory usage.

goroutine profile: total 920756
626254 @ 0x437070 0x447363 0xb3570f 0x4671c1
#	0xb3570e	main.(*Session).clusterWriteLoop+0x14e	/Users/mike/golang/poizon/duinfr/spider/server/cluster.go:1013

278952 @ 0x437070 0x447363 0xb77212 0xb65cdb 0x4671c1
#	0xb77211	main.(*Topic).runProxy+0x251	/Users/mike/golang/poizon/duinfr/spider/server/topic_proxy.go:22
#	0xb65cda	main.(*Topic).run+0x3a		/Users/mike/golang/poizon/duinfr/spider/server/topic.go:169

9478 @ 0x437070 0x447363 0xb664cd 0xb65cf8 0x4671c1
#	0xb664cc	main.(*Topic).runLocal+0x2bc	/Users/mike/golang/poizon/duinfr/spider/server/topic.go:219
#	0xb65cf7	main.(*Topic).run+0x57		/Users/mike/golang/poizon/duinfr/spider/server/topic.go:167

3289 @ 0x437070 0x447363 0xb3b952 0x4671c1
#	0xb3b951	main.(*Session).writeLoop+0x1c1	/Users/mike/golang/poizon/duinfr/spider/server/hdl_websock.go:76

2678 @ 0x437070 0x43002a 0x42f595 0x4dcfe5 0x4dde7b 0x4dde5d 0x56d7bf 0x57ff3e 0x6500d0 0x47ee31 0x65031c 0x64e8d5 0x6529fb 0x652a06 0x4fe443 0x4fe5af 0x8d2d30 0x8d528c 0x8d631f 0x8d6a9f 0xb3b5ec 0x4671c1
#	0x42f594	internal/poll.runtime_pollWait+0x54			/Users/mike/.gvm/gos/go1.14.6/src/runtime/netpoll.go:203
#	0x4dcfe4	internal/poll.(*pollDesc).wait+0x44			/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:87
#	0x4dde7a	internal/poll.(*pollDesc).waitRead+0x19a		/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:92
#	0x4dde5c	internal/poll.(*FD).Read+0x17c				/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_unix.go:169
#	0x56d7be	net.(*netFD).Read+0x4e					/Users/mike/.gvm/gos/go1.14.6/src/net/fd_unix.go:202
#	0x57ff3d	net.(*conn).Read+0x8d					/Users/mike/.gvm/gos/go1.14.6/src/net/net.go:184
#	0x6500cf	crypto/tls.(*atLeastReader).Read+0x5f			/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:760
#	0x47ee30	bytes.(*Buffer).ReadFrom+0xb0				/Users/mike/.gvm/gos/go1.14.6/src/bytes/buffer.go:204
#	0x65031b	crypto/tls.(*Conn).readFromUntil+0xeb			/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:782
#	0x64e8d4	crypto/tls.(*Conn).readRecordOrCCS+0x114		/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:589
#	0x6529fa	crypto/tls.(*Conn).readRecord+0x15a			/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:557
#	0x652a05	crypto/tls.(*Conn).Read+0x165				/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:1233
#	0x4fe442	bufio.(*Reader).fill+0x102				/Users/mike/.gvm/gos/go1.14.6/src/bufio/bufio.go:100
#	0x4fe5ae	bufio.(*Reader).Peek+0x4e				/Users/mike/.gvm/gos/go1.14.6/src/bufio/bufio.go:138
#	0x8d2d2f	github.com/gorilla/websocket.(*Conn).read+0x3f		/Users/mike/.gvm/pkgsets/go1.14.6/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:370
#	0x8d528b	github.com/gorilla/websocket.(*Conn).advanceFrame+0x5b	/Users/mike/.gvm/pkgsets/go1.14.6/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:798
#	0x8d631e	github.com/gorilla/websocket.(*Conn).NextReader+0x8e	/Users/mike/.gvm/pkgsets/go1.14.6/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:980
#	0x8d6a9e	github.com/gorilla/websocket.(*Conn).ReadMessage+0x2e	/Users/mike/.gvm/pkgsets/go1.14.6/global/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:1064
#	0xb3b5eb	main.(*Session).readLoop+0x14b				/Users/mike/golang/poizon/duinfr/spider/server/hdl_websock.go:54

19 @ 0x437070 0x43002a 0x42f595 0x4dcfe5 0x4dde7b 0x4dde5d 0x56d7bf 0x57ff3e 0x4feaff 0x4754d7 0x506b8f 0x506b42 0x5111e7 0x51173c 0x511a4b 0x5118cd 0x781495 0x780765 0x4671c1
#	0x42f594	internal/poll.runtime_pollWait+0x54			/Users/mike/.gvm/gos/go1.14.6/src/runtime/netpoll.go:203
#	0x4dcfe4	internal/poll.(*pollDesc).wait+0x44			/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:87
#	0x4dde7a	internal/poll.(*pollDesc).waitRead+0x19a		/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:92
#	0x4dde5c	internal/poll.(*FD).Read+0x17c				/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_unix.go:169
#	0x56d7be	net.(*netFD).Read+0x4e					/Users/mike/.gvm/gos/go1.14.6/src/net/fd_unix.go:202
#	0x57ff3d	net.(*conn).Read+0x8d					/Users/mike/.gvm/gos/go1.14.6/src/net/net.go:184
#	0x4feafe	bufio.(*Reader).Read+0x24e				/Users/mike/.gvm/gos/go1.14.6/src/bufio/bufio.go:226
#	0x4754d6	io.ReadAtLeast+0x86					/Users/mike/.gvm/gos/go1.14.6/src/io/io.go:310
#	0x506b8e	io.ReadFull+0x6e					/Users/mike/.gvm/gos/go1.14.6/src/io/io.go:329
#	0x506b41	encoding/gob.decodeUintReader+0x21			/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decode.go:120
#	0x5111e6	encoding/gob.(*Decoder).recvMessage+0x56		/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:81
#	0x51173b	encoding/gob.(*Decoder).decodeTypeSequence+0x10b	/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:143
#	0x511a4a	encoding/gob.(*Decoder).DecodeValue+0xda		/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:211
#	0x5118cc	encoding/gob.(*Decoder).Decode+0x16c			/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:188
#	0x781494	net/rpc.(*gobClientCodec).ReadResponseHeader+0x44	/Users/mike/.gvm/gos/go1.14.6/src/net/rpc/client.go:228
#	0x780764	net/rpc.(*Client).input+0xa4				/Users/mike/.gvm/gos/go1.14.6/src/net/rpc/client.go:109

19 @ 0x437070 0x43002a 0x42f595 0x4dcfe5 0x4dde7b 0x4dde5d 0x56d7bf 0x57ff3e 0x4feaff 0x4754d7 0x506b8f 0x506b42 0x5111e7 0x51173c 0x511a4b 0x5118cd 0x783aa5 0x7852c7 0x784bca 0x784167 0x783fb9 0x4671c1
#	0x42f594	internal/poll.runtime_pollWait+0x54			/Users/mike/.gvm/gos/go1.14.6/src/runtime/netpoll.go:203
#	0x4dcfe4	internal/poll.(*pollDesc).wait+0x44			/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:87
#	0x4dde7a	internal/poll.(*pollDesc).waitRead+0x19a		/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:92
#	0x4dde5c	internal/poll.(*FD).Read+0x17c				/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_unix.go:169
#	0x56d7be	net.(*netFD).Read+0x4e					/Users/mike/.gvm/gos/go1.14.6/src/net/fd_unix.go:202
#	0x57ff3d	net.(*conn).Read+0x8d					/Users/mike/.gvm/gos/go1.14.6/src/net/net.go:184
#	0x4feafe	bufio.(*Reader).Read+0x24e				/Users/mike/.gvm/gos/go1.14.6/src/bufio/bufio.go:226
#	0x4754d6	io.ReadAtLeast+0x86					/Users/mike/.gvm/gos/go1.14.6/src/io/io.go:310
#	0x506b8e	io.ReadFull+0x6e					/Users/mike/.gvm/gos/go1.14.6/src/io/io.go:329
#	0x506b41	encoding/gob.decodeUintReader+0x21			/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decode.go:120
#	0x5111e6	encoding/gob.(*Decoder).recvMessage+0x56		/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:81
#	0x51173b	encoding/gob.(*Decoder).decodeTypeSequence+0x10b	/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:143
#	0x511a4a	encoding/gob.(*Decoder).DecodeValue+0xda		/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:211
#	0x5118cc	encoding/gob.(*Decoder).Decode+0x16c			/Users/mike/.gvm/gos/go1.14.6/src/encoding/gob/decoder.go:188
#	0x783aa4	net/rpc.(*gobServerCodec).ReadRequestHeader+0x44	/Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:397
#	0x7852c6	net/rpc.(*Server).readRequestHeader+0x66		/Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:583
#	0x784bc9	net/rpc.(*Server).readRequest+0x59			/Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:543
#	0x784166	net/rpc.(*Server).ServeCodec+0x96			/Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:458
#	0x783fb8	net/rpc.(*Server).ServeConn+0x158			/Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:449

15 @ 0x437070 0x43002a 0x42f595 0x4dcfe5 0x4dde7b 0x4dde5d 0x56d7bf 0x57ff3e 0x6500d0 0x47ee31 0x65031c 0x64e8d5 0x6511fd 0x651208 0x66a4da 0x6684af 0x668061 0x653151 0x70ec5d 0x4671c1
#	0x42f594	internal/poll.runtime_pollWait+0x54				/Users/mike/.gvm/gos/go1.14.6/src/runtime/netpoll.go:203
#	0x4dcfe4	internal/poll.(*pollDesc).wait+0x44				/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:87
#	0x4dde7a	internal/poll.(*pollDesc).waitRead+0x19a			/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_poll_runtime.go:92
#	0x4dde5c	internal/poll.(*FD).Read+0x17c					/Users/mike/.gvm/gos/go1.14.6/src/internal/poll/fd_unix.go:169
#	0x56d7be	net.(*netFD).Read+0x4e						/Users/mike/.gvm/gos/go1.14.6/src/net/fd_unix.go:202
#	0x57ff3d	net.(*conn).Read+0x8d						/Users/mike/.gvm/gos/go1.14.6/src/net/net.go:184
#	0x6500cf	crypto/tls.(*atLeastReader).Read+0x5f				/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:760
#	0x47ee30	bytes.(*Buffer).ReadFrom+0xb0					/Users/mike/.gvm/gos/go1.14.6/src/bytes/buffer.go:204
#	0x65031b	crypto/tls.(*Conn).readFromUntil+0xeb				/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:782
#	0x64e8d4	crypto/tls.(*Conn).readRecordOrCCS+0x114			/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:589
#	0x6511fc	crypto/tls.(*Conn).readRecord+0x6c				/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:557
#	0x651207	crypto/tls.(*Conn).readHandshake+0x77				/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:973
#	0x66a4d9	crypto/tls.(*serverHandshakeState).doFullHandshake+0x4e9	/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/handshake_server.go:524
#	0x6684ae	crypto/tls.(*serverHandshakeState).handshake+0x31e		/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/handshake_server.go:105
#	0x668060	crypto/tls.(*Conn).serverHandshake+0x100			/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/handshake_server.go:60
#	0x653150	crypto/tls.(*Conn).Handshake+0x1e0				/Users/mike/.gvm/gos/go1.14.6/src/crypto/tls/conn.go:1342
#	0x70ec5c	net/http.(*conn).serve+0x19c					/Users/mike/.gvm/gos/go1.14.6/src/net/http/server.go:1816

tinode version 0.16.6

created time in 2 months

issue closedtinode/chat

cluster thread safety

2020/08/25 22:39:59 in: '{"hi":{"id":"110283","lang":"zh_CN","ua":"dewuApp\4.52.0 (Android 4.14.116; zh_CN_#Hans); dewuApp/4.52.0","ver":"0.16"}}' sid='aF1xOAB_tlw' uid='' fatal error: concurrent map writes2020/08/25 22:39:59 cluster: multiplexing session started usrdjMgjtkej-s-node-02 1274

goroutine 7886 [running]: runtime.throw(0xd6bae1, 0x15) /Users/mike/.gvm/gos/go1.14.6/src/runtime/panic.go:1116 +0x72 fp=0xc003490a10 sp=0xc0034909e0 pc=0x434642 runtime.mapassign_faststr(0xc32940, 0xc00037c3c0, 0xc002541470, 0x21, 0x21) /Users/mike/.gvm/gos/go1.14.6/src/runtime/map_faststr.go:211 +0x3f7 fp=0xc003490a78 sp=0xc003490a10 pc=0x413497 main.(*Cluster).TopicMaster(0xc00019e1e0, 0xc0027017a0, 0xc0006edc88, 0x0, 0x0) /Users/mike/golang/poizon/duinfr/spider/server/cluster.go:399 +0xccb fp=0xc003490bc0 sp=0xc003490a78 pc=0xaf9cbb runtime.call64(0xc0002fc840, 0xc00009c250, 0xc000a0fd40, 0x1800000028) /Users/mike/.gvm/gos/go1.14.6/src/runtime/asm_amd64.s:540 +0x3b fp=0xc003490c10 sp=0xc003490bc0 pc=0x46566b reflect.Value.call(0xc000180900, 0xc00009c250, 0x13, 0xd5cfe8, 0x4, 0xc003490f08, 0x3, 0x3, 0x0, 0x0, ...) /Users/mike/.gvm/gos/go1.14.6/src/reflect/value.go:460 +0x8ab fp=0xc003490e28 sp=0xc003490c10 pc=0x49b8ab reflect.Value.Call(0xc000180900, 0xc00009c250, 0x13, 0xc0026be708, 0x3, 0x3, 0xc0026be768, 0x405f8b, 0xc001843320) /Users/mike/.gvm/gos/go1.14.6/src/reflect/value.go:321 +0xb4 fp=0xc003490ea8 sp=0xc003490e28 pc=0x49adb4 net/rpc.(*service).call(0xc00018ca40, 0xc0000a8000, 0xc0000977a8, 0xc0000977e0, 0xc0000a4400, 0xc000189980, 0xbb70e0, 0xc0027017a0, 0x16, 0xba94a0, ...) /Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:377 +0x17f fp=0xc003490f70 sp=0xc003490ea8 pc=0x781faf runtime.goexit() /Users/mike/.gvm/gos/go1.14.6/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc003490f78 sp=0xc003490f70 pc=0x4671c1 created by net/rpc.(*Server).ServeCodec /Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:474 +0x42b

goroutine 1 [select]: main.listenAndServe(0xc00057b968, 0x4, 0xc0004d2ec0, 0xc000083c80, 0x101, 0x93a80, 0xc0000968d0, 0x10, 0xc0000968e0, 0x10, ...) /Users/mike/golang/poizon/duinfr/spider/server/http.go:79 +0x1c2 main.main() /Users/mike/golang/poizon/duinfr/spider/server/main.go:571 +0x18f6

:

closed time in 2 months

hhy5861

issue openedtinode/chat

cluster thread safety

2020/08/25 22:39:59 in: '{"hi":{"id":"101750","lang":"zh_CN","ua":"dewuApp\4.52.0 (Android 4.14.116; zh_CN_#Hans); dewuApp/4.52.0","ver":"0.16"}}' sid='22SAjSPCXs4' uid='' 2020/08/25 22:39:59 cluster: multiplexing session started usrNc4V9SnPa1o-node-10 1272 2020/08/25 22:39:59 ws: session started -4D8AZpqUVk 175.5.131.183:8576 1273 2020/08/25 22:39:59 in: '{"sub":{"get":{"what":"desc sub"},"id":"66880","topic":"me"}}' sid='EnJpg0SxaJE' uid='9d4d11bLEas' 2020/08/25 22:39:59 in: '{"hi":{"id":"110283","lang":"zh_CN","ua":"dewuApp\4.52.0 (Android 4.14.116; zh_CN_#Hans); dewuApp/4.52.0","ver":"0.16"}}' sid='aF1xOAB_tlw' uid='' fatal error: concurrent map writes2020/08/25 22:39:59 cluster: multiplexing session started usrdjMgjtkej-s-node-02 1274

goroutine 7886 [running]: runtime.throw(0xd6bae1, 0x15) /Users/mike/.gvm/gos/go1.14.6/src/runtime/panic.go:1116 +0x72 fp=0xc003490a10 sp=0xc0034909e0 pc=0x434642 runtime.mapassign_faststr(0xc32940, 0xc00037c3c0, 0xc002541470, 0x21, 0x21) /Users/mike/.gvm/gos/go1.14.6/src/runtime/map_faststr.go:211 +0x3f7 fp=0xc003490a78 sp=0xc003490a10 pc=0x413497 main.(*Cluster).TopicMaster(0xc00019e1e0, 0xc0027017a0, 0xc0006edc88, 0x0, 0x0) /Users/mike/golang/poizon/duinfr/spider/server/cluster.go:399 +0xccb fp=0xc003490bc0 sp=0xc003490a78 pc=0xaf9cbb runtime.call64(0xc0002fc840, 0xc00009c250, 0xc000a0fd40, 0x1800000028) /Users/mike/.gvm/gos/go1.14.6/src/runtime/asm_amd64.s:540 +0x3b fp=0xc003490c10 sp=0xc003490bc0 pc=0x46566b reflect.Value.call(0xc000180900, 0xc00009c250, 0x13, 0xd5cfe8, 0x4, 0xc003490f08, 0x3, 0x3, 0x0, 0x0, ...) /Users/mike/.gvm/gos/go1.14.6/src/reflect/value.go:460 +0x8ab fp=0xc003490e28 sp=0xc003490c10 pc=0x49b8ab reflect.Value.Call(0xc000180900, 0xc00009c250, 0x13, 0xc0026be708, 0x3, 0x3, 0xc0026be768, 0x405f8b, 0xc001843320) /Users/mike/.gvm/gos/go1.14.6/src/reflect/value.go:321 +0xb4 fp=0xc003490ea8 sp=0xc003490e28 pc=0x49adb4 net/rpc.(*service).call(0xc00018ca40, 0xc0000a8000, 0xc0000977a8, 0xc0000977e0, 0xc0000a4400, 0xc000189980, 0xbb70e0, 0xc0027017a0, 0x16, 0xba94a0, ...) /Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:377 +0x17f fp=0xc003490f70 sp=0xc003490ea8 pc=0x781faf runtime.goexit() /Users/mike/.gvm/gos/go1.14.6/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc003490f78 sp=0xc003490f70 pc=0x4671c1 created by net/rpc.(*Server).ServeCodec /Users/mike/.gvm/gos/go1.14.6/src/net/rpc/server.go:474 +0x42b

goroutine 1 [select]: main.listenAndServe(0xc00057b968, 0x4, 0xc0004d2ec0, 0xc000083c80, 0x101, 0x93a80, 0xc0000968d0, 0x10, 0xc0000968e0, 0x10, ...) /Users/mike/golang/poizon/duinfr/spider/server/http.go:79 +0x1c2 main.main() /Users/mike/golang/poizon/duinfr/spider/server/main.go:571 +0x18f6

:

tinode version: 0.16.5

created time in 2 months

issue openedtinode/chat

cluster thread safety

If you are not reporting a bug, please post to https://groups.google.com/d/forum/tinode instead.

Subject of the issue

Describe your issue here.

Your environment

Server-side

  • [ ] web.tinode.co, api.tinode.co
  • [ ] sandbox.tinode.co
  • [ ] Your own setup:
    • platform (Windows, Mac, Linux etc)
    • version of Tinode server, e.g. 0.15.2-rc3
    • database backend
    • cluster or standalone

Client-side

  • [ ] TinodeWeb/tinodejs: javascript client
    • Browser make and version.
  • [ ] Tindroid: Android app
    • Android API level (e.g. 25).
    • Emulator or hardware, if hardware describe it.
  • [ ] Tinodios: iOS app
    • iOS version
    • Simulator or hardware, if hardware describe it.
  • [ ] tn-cli
    • Python version
  • [ ] Chatbot
    • Python version
  • Version of the client, e.g. 0.15.1
  • [ ] Your own client. Describe it:
    • Transport (gRPC, websocket, long polling)
    • Programming language.
    • gRPC version, if applicable.

Steps to reproduce

Tell us how to reproduce this issue.

Expected behaviour

Tell us what should happen.

Actual behaviour

Tell us what happens instead.

Server-side log

Copy server-side log here. You may also attach it to the issue as a file.

Client-side log

Copy client-side log here (Android logcat, Javascript console, etc). You may also attach it to the issue as a file.

created time in 2 months

startedpingcap/tidb-operator

started time in 2 months

startedmitchellh/gox

started time in 2 months

startedJeffail/gabs

started time in 2 months

issue openedtinode/chat

Push message from client problem

2020/08/10 20:20:26 session SBhwbJVbjsU: 31149 message(s) dropped
2020/08/10 20:20:26 topic[grpJQa2klCLCP8]: connection stuck, detaching
2020/08/10 20:20:26 s.queueOutBytes: send queue is full CwqneYpL5sY
2020/08/10 20:20:26 session CwqneYpL5sY: 22341 message(s) dropped
2020/08/10 20:20:26 topic[grpJQa2klCLCP8]: connection stuck, detaching
2020/08/10 20:20:26 s.queueOutBytes: send queue is full 4I-2QOcFzwI
2020/08/10 20:20:26 session 4I-2QOcFzwI: 34729 message(s) dropped
2020/08/10 20:20:26 topic[grpJQa2klCLCP8]: connection stuck, detaching
2020/08/10 20:20:26 s.queueOutBytes: send queue is full VCxD9_pJoRk
2020/08/10 20:20:26 session VCxD9_pJoRk: 14233 message(s) dropped
2020/08/10 20:20:26 topic[grpJQa2klCLCP8]: connection stuck, detaching
2020/08/10 20:20:26 s.queueOutBytes: send queue is full 3B16vD3HLhA
2020/08/10 20:20:26 session 3B16vD3HLhA: 22479 message(s) dropped
2020/08/10 20:20:26 topic[grpJQa2klCLCP8]: connection stuck, detaching
2020/08/10 20:20:26 s.queueOutBytes: send queue is full XLgAb8fcR4Y
2020/08/10 20:20:26 session XLgAb8fcR4Y: 30241 message(s) dropped
2020/08/10 20:20:26 topic[grpJQa2klCLCP8]: connection stuck, detaching
2020/08/10 20:20:26 s.queueOutBytes: send queue is full bF_W21FUBBE
2020/08/10 20:20:26 session bF_W21FUBBE: 24342 message(s) dropped
2020/08/10 20:20:26 topic[grpJQa2klCLCP8]: connection stuck, detaching

version : v0.16.5

created time in 3 months

startedNVIDIA/k8s-device-plugin

started time in 3 months

startedNVIDIA/gpu-monitoring-tools

started time in 3 months

startedkairen/gpu-prometheus-exporter

started time in 3 months

more