x/build: freebsd-arm-paulzhol builder is failing cmd/link TestDWARF

Ask questionsx/build: freebsd-arm-paulzhol builder is failing cmd/link TestDWARF

https://build.golang.org/log/d0025d71cbb3628b7800d2eebe1d8fb58c79969d

ok  	cmd/internal/src	0.034s
ok  	cmd/internal/test2json	1.729s
--- FAIL: TestDWARF (4.86s)
    dwarf_test.go:41: cmd/link is stale - run go install cmd/link
FAIL
FAIL	cmd/link	56.431s
FAIL
go tool dist: Failed: exit status 1

I've captured the stdout/stderr output from the buildlet to look at the env. variables as I understand it was related to clearing GOROOT_FINAL in similar issues.

2020/08/28 11:21:06 [0x314ba580] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/internal/obj/riscv" "go_test:cmd/internal/obj/x86" "go_test:cmd/internal/objabi"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:22:09 [0x314ba580] Run = ok, after 1m3.510284157s

2020/08/28 11:22:09 [0x314bb1e0] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/internal/src" "go_test:cmd/internal/test2json" "go_test:cmd/link"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:23:49 [0x314bb1e0] Run = exit status 1, after 1m39.66510577s

2020/08/28 11:23:49 [0x3148cdc0] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/link/internal/benchmark" "go_test:cmd/link/internal/ld" "go_test:cmd/link/internal/loader"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:23:50 Halting in 1 second.

Maybe this is a dup of #33598 but on FreeBSD-arm.

golang/go

Answer questions paulzhol

@bcmills The first one is wrong (GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4) and it is the default set by the buildlets main():

https://github.com/golang/build/blob/1f82dd332eb91abb9335fc2abc4d8039f7ac7ba5/cmd/buildlet/buildlet.go#L255-L264

var inheritedGorootBootstrap string

func initGorootBootstrap() {
	// Remember any GOROOT_BOOTSTRAP to use as a backup in handleExec
	// if $WORKDIR/go1.4 ends up not existing.
	inheritedGorootBootstrap = os.Getenv("GOROOT_BOOTSTRAP")

	// Default if not otherwise configured in dashboard/builders.go:
	os.Setenv("GOROOT_BOOTSTRAP", filepath.Join(*workDir, "go1.4"))
}

the second GOROOT_BOOTSTRAP is coming from the (coordinator?):

https://github.com/golang/build/blob/1e76199ae864f03a22993c156ef26ecc4679e7f4/dashboard/builders.go#L844-L857

// Env returns the environment variables this builder should run with.
func (c *BuildConfig) Env() []string {
	env := []string{"GO_BUILDER_NAME=" + c.Name}
	if c.FlakyNet {
		env = append(env, "GO_BUILDER_FLAKY_NET=1")
	}
	if c.IsLongTest() {
		// Set a private hook in cmd/dist to run main Go repository tests
		// without the default -short flag. See golang.org/issue/12508.
		env = append(env, "GO_TEST_SHORT=0")
	}
	env = append(env, c.HostConfig().env...)
	return append(env, c.env...)
}

from: https://github.com/golang/build/blob/18a463e644997ff721e2f319f030b5fd63d4739d/dashboard/builders.go#L347

	"host-freebsd-arm-paulzhol": &HostConfig{
		IsReverse:   true,
		ExpectNum:   1,
		Notes:       "Cubiboard2 1Gb RAM dual-core Cortex-A7 (Allwinner A20), FreeBSD 11.1-RELEASE",
		env:         []string{"GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4"},
		OwnerGithub: "paulzhol",
	},

but eventually this is just passed to an os/exec.Cmd struct in the buildlet exec handler, where

// If Env contains duplicate environment keys, only the last // value in the slice for each duplicate key is used. // As a special case on Windows, SYSTEMROOT is always added if // missing and not explicitly set to the empty string. Env []string

I randomly sampled an active build, and only the correct (last) value is set:

1527  -  S     0:56.71 /var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/go_bootstrap install -gcflags=all= -ldflags=all= std cmd

[paulzhol@arm-virt-12 ~]$ procstat -e 1527
  PID COMM             ENVIRONMENT
 1527 go_bootstrap     CGO_ENABLED=1 RC_PID=21 GO_BUILDER_NAME=freebsd-arm-paulzhol GO_TEST_TIMEOUT_SCALE=30 USER=paulzhol GOENV=off PWD=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/src HOME=/home/paulzhol GOROOT=/var/tmp/workdir-host-freebsd-arm-paulzhol/go TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp MAIL=/var/mail/paulzhol SHLVL=1 WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol BLOCKSIZE=K XDG_CACHE_HOME=/tmp/cache PATH=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin GOCACHE=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/obj/go-build GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4 GOARM=7 _=./cmd/dist/dist TERM=dumb GO386=387 GOARCH=arm GOHOSTARCH=arm GOHOSTOS=freebsd GOOS=freebsd GOMIPS=hardfloat GOMIPS64=hardfloat GOPPC64=power8 GOROOT_FINAL=/var/tmp/workdir-host-freebsd-arm-paulzhol/go LANG=C LANGUAGE=en_US.UTF8 GOPATH=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/obj/gopath GOBIN= CC=clang
[paulzhol@arm-virt-12 ~]$ ls -l /var/tmp/workdir-host-freebsd-arm-paulzhol/go

Looking at the PATH above, and https://github.com/golang/go/issues/31567#issuecomment-484925837 I don't have the bootstrap go1.4 set, or is there any other go installation present.

The other difference I can think of, is my builder has the entire filesystem mounted read-only, so https://golang.org/cl/223746 is needed to ignore failing writes. Although GOROOT=/var/tmp/workdir-host-freebsd-arm-paulzhol/go set above is writable anyway.

useful!