profile
viewpoint

Ask questionsfatal error: unexpected signal during runtime execution

Description

In a clean installation of Ubuntu Bionic (server edition for example) and last version of balena-engine, when I execute pull command I have:

avp@servertest:~$ sudo balena-engine pull resin/rpi-raspbian:jessie
error during connect: Post http://%2Fvar%2Frun%2Fbalena-engine.sock/v1.35/images/create?fromImage=resin%2Frpi-raspbian&tag=jessie: EOF

but I have a server running. Anyway I get fatal error: unexpected signal during runtime execution on server log output.

Steps to reproduce the issue:

  1. Ubuntu 18.04 bionic (Ubuntu 18.04.2 LTS) in real machine
  2. curl -sfL https://balena.io/engine/install.sh | sh
  3. one console: sudo balena-engine-daemon
  4. second console: sudo balena-engine pull resin/rpi-raspbian:jessie

Server output log received:

avp@servertest:~$ sudo balena-engine-daemon 
[sudo] password for avp: 
INFO[2019-07-01T21:36:34.838656553Z] libcontainerd: started new balena-engine-containerd process  pid=1524
INFO[0000] starting containerd                           module=containerd revision= version=1.0.0+unknown
INFO[0000] setting subreaper...                          module=containerd
INFO[0000] changing OOM score to -500                    module=containerd
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/balena-engine/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/balena-engine/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd/io.containerd.metadata.v1.bolt
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address=/var/run/balena-engine/containerd/balena-engine-containerd-debug.sock module=containerd/debug
INFO[0000] serving...                                    address=/var/run/balena-engine/containerd/balena-engine-containerd.sock module=containerd/grpc
INFO[0000] containerd successfully booted in 0.006043s   module=containerd
INFO[2019-07-01T21:36:34.903734603Z] [graphdriver] using prior storage driver: overlay2 
INFO[2019-07-01T21:36:34.918017315Z] Graph migration to content-addressability took 0.00 seconds 
WARN[2019-07-01T21:36:34.918569898Z] Your kernel does not support swap memory limit 
WARN[2019-07-01T21:36:34.918702692Z] Your kernel does not support cgroup rt period 
WARN[2019-07-01T21:36:34.918747559Z] Your kernel does not support cgroup rt runtime 
INFO[2019-07-01T21:36:34.920846934Z] Loading containers: start.                   
INFO[2019-07-01T21:36:35.489881058Z] Default bridge (balena0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2019-07-01T21:36:35.660693109Z] Loading containers: done.                    
WARN[2019-07-01T21:36:35.730793277Z] failed to retrieve balena-engine-init version: exec: "balena-engine-init": executable file not found in $PATH 
INFO[2019-07-01T21:36:35.730894152Z] Docker daemon                                 commit=3fdfd0d graphdriver(s)=overlay2 version=v17.12.0
INFO[2019-07-01T21:36:35.730924565Z] Daemon has completed initialization          
INFO[2019-07-01T21:36:35.735918262Z] API listen on /var/run/balena-engine.sock    
WARN[2019-07-01T21:37:11.066849492Z] failed to retrieve balena-engine-init version: exec: "balena-engine-init": executable file not found in $PATH 
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7f21b85ea448]

runtime stack:
runtime.throw(0x174a29a, 0x2a)
	/usr/local/go/src/runtime/panic.go:616 +0x81
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:372 +0x28e

goroutine 195 [syscall]:
runtime.cgocall(0x12bb480, 0xc4204a2df8, 0x29)
	/usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc4204a2db8 sp=0xc4204a2d80 pc=0x4035e4
net._C2func_getaddrinfo(0xc4208a4d70, 0x0, 0xc42093fe90, 0xc42023e5c0, 0x0, 0x0, 0x0)
	_cgo_gotypes.go:92 +0x55 fp=0xc4204a2df8 sp=0xc4204a2db8 pc=0x57fd15
net.cgoLookupIPCNAME.func1(0xc4208a4d70, 0x0, 0xc42093fe90, 0xc42023e5c0, 0xf, 0xf, 0xc420398000)
	/usr/local/go/src/net/cgo_unix.go:149 +0x13b fp=0xc4204a2e40 sp=0xc4204a2df8 pc=0x586a7b
net.cgoLookupIPCNAME(0xc4212b0860, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/cgo_unix.go:149 +0x174 fp=0xc4204a2f38 sp=0xc4204a2e40 pc=0x581384
net.cgoIPLookup(0xc4208f8540, 0xc4212b0860, 0xe)
	/usr/local/go/src/net/cgo_unix.go:201 +0x4d fp=0xc4204a2fc8 sp=0xc4204a2f38 pc=0x581a4d
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4204a2fd0 sp=0xc4204a2fc8 pc=0x45cc01
created by net.cgoLookupIP
	/usr/local/go/src/net/cgo_unix.go:211 +0xaf

goroutine 1 [chan receive]:
github.com/docker/docker/cmd/dockerd.(*DaemonCli).start(0xc4201acd50, 0xc420346770, 0x0, 0x0)
	/balena-engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:270 +0x1605
github.com/docker/docker/cmd/dockerd.runDaemon(0xc420346770, 0xc4202521e0, 0x0)
	/balena-engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:78 +0x68
github.com/docker/docker/cmd/dockerd.newDaemonCommand.func1(0xc420414000, 0x232e208, 0x0, 0x0, 0x0, 0x0)
	/balena-engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5b
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc420414000, 0xc42003a1b0, 0x0, 0x0, 0xc420414000, 0xc42003a1b0)
	/balena-engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:700 +0x46d
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420414000, 0xc4202521e0, 0xc420344d80, 0xc420414000)
	/balena-engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:785 +0x2e4
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420414000, 0xc42000e020, 0x15de301)
	/balena-engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:738 +0x2b
github.com/docker/docker/cmd/dockerd.Main()
	/balena-engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:106 +0xa0
main.main()
	/balena-engine/.gopath/src/github.com/docker/docker/cmd/balena-engine/main.go:29 +0xf9

goroutine 21 [syscall]:
os/signal.signal_recv(0x4353c8)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
...

Additional information:

If I adding group balena-engine and right permissions on balena socket, the error is the same. Also with a systemd registration

Output of balena-engine version:

avp@servertest:~$ balena-engine version
Client:
 Version:	v17.12.0
 API version:	1.35
 Go version:	go1.10.4
 Git commit:	3fdfd0d
 Built:	Wed Oct 17 15:42:29 2018
 OS/Arch:	linux/amd64
 Experimental:	false
 Orchestrator:	swarm

Server:
 Engine:
  Version:	v17.12.0
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.10.4
  Git commit:	3fdfd0d
  Built:	Wed Oct 17 15:42:29 2018
  OS/Arch:	linux/amd64
  Experimental:	false

Output of balena-engine info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: v17.12.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null
 Log: journald json-file
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Runtimes: bare runc
Default Runtime: runc
Init Binary: balena-engine-init
containerd version: 
runc version: 13e66eedaddfbfeda2a73d23701000e4e63b5471
init version: N/A (expected: )
Security Options:
 apparmor
Kernel Version: 4.15.0-54-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.55GiB
Name: servertest
ID: 3HHG:R76P:ZSKJ:REX6:BCMN:2I5G:VRYX:C7JN:ZIJY:VAYG:ZQOH:WF43
Docker Root Dir: /var/lib/balena-engine
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details

servertest                  
    description: Computer
    width: 64 bits
    capabilities: smp vsyscall32
  *-core
       description: Motherboard
       physical id: 0
     *-memory
          description: System memory
          physical id: 0
          size: 15GiB
     *-cpu
          product: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
          vendor: Intel Corp.
          physical id: 1
          bus info: cpu@0
          size: 2887MHz
          capacity: 3500MHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d cpufreq
     *-pci
          description: Host bridge
          product: Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
          vendor: Intel Corporation
          physical id: 100
          bus info: pci@0000:00:00.0
          version: 02
          width: 32 bits
          clock: 33MHz
balena-os/balena-engine

Answer questions roman-mazur

@robertgzr I did a quick search with the following reasoning SIGSEGV most likely arises when calls to C are performed. From the stack traces above, net.cgoLookupIP may be a good candidate of who is causing the trouble.

There are similar issues reported for other projects on github. For example, this one: https://github.com/google/cadvisor/issues/1481

So either bumping Go version we use to build the engine or playing with build flags (netgo and netdns=go) may help.

useful!

Related questions

error during connect: Post hot 1
source:https://uonfu.com/
Github User Rank List