@bcmills The first one is wrong (GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4) and it is the default set by the buildlets main():

https://github.com/golang/build/blob/1f82dd332eb91abb9335fc2abc4d8039f7ac7ba5/cmd/buildlet/buildlet.go#L255-L264

var inheritedGorootBootstrap string

func initGorootBootstrap() {
	// Remember any GOROOT_BOOTSTRAP to use as a backup in handleExec
	// if $WORKDIR/go1.4 ends up not existing.
	inheritedGorootBootstrap = os.Getenv("GOROOT_BOOTSTRAP")

	// Default if not otherwise configured in dashboard/builders.go:
	os.Setenv("GOROOT_BOOTSTRAP", filepath.Join(*workDir, "go1.4"))
}

the second GOROOT_BOOTSTRAP is coming from the (coordinator?):

https://github.com/golang/build/blob/1e76199ae864f03a22993c156ef26ecc4679e7f4/dashboard/builders.go#L844-L857

// Env returns the environment variables this builder should run with.
func (c *BuildConfig) Env() []string {
	env := []string{"GO_BUILDER_NAME=" + c.Name}
	if c.FlakyNet {
		env = append(env, "GO_BUILDER_FLAKY_NET=1")
	}
	if c.IsLongTest() {
		// Set a private hook in cmd/dist to run main Go repository tests
		// without the default -short flag. See golang.org/issue/12508.
		env = append(env, "GO_TEST_SHORT=0")
	}
	env = append(env, c.HostConfig().env...)
	return append(env, c.env...)
}

from: https://github.com/golang/build/blob/18a463e644997ff721e2f319f030b5fd63d4739d/dashboard/builders.go#L347

	"host-freebsd-arm-paulzhol": &HostConfig{
		IsReverse:   true,
		ExpectNum:   1,
		Notes:       "Cubiboard2 1Gb RAM dual-core Cortex-A7 (Allwinner A20), FreeBSD 11.1-RELEASE",
		env:         []string{"GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4"},
		OwnerGithub: "paulzhol",
	},

but eventually this is just passed to an os/exec.Cmd struct in the buildlet exec handler, where

// If Env contains duplicate environment keys, only the last // value in the slice for each duplicate key is used. // As a special case on Windows, SYSTEMROOT is always added if // missing and not explicitly set to the empty string. Env []string

I randomly sampled an active build, and only the correct (last) value is set:

1527  -  S     0:56.71 /var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/go_bootstrap install -gcflags=all= -ldflags=all= std cmd

[paulzhol@arm-virt-12 ~]$ procstat -e 1527
  PID COMM             ENVIRONMENT
 1527 go_bootstrap     CGO_ENABLED=1 RC_PID=21 GO_BUILDER_NAME=freebsd-arm-paulzhol GO_TEST_TIMEOUT_SCALE=30 USER=paulzhol GOENV=off PWD=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/src HOME=/home/paulzhol GOROOT=/var/tmp/workdir-host-freebsd-arm-paulzhol/go TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp MAIL=/var/mail/paulzhol SHLVL=1 WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol BLOCKSIZE=K XDG_CACHE_HOME=/tmp/cache PATH=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin GOCACHE=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/obj/go-build GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4 GOARM=7 _=./cmd/dist/dist TERM=dumb GO386=387 GOARCH=arm GOHOSTARCH=arm GOHOSTOS=freebsd GOOS=freebsd GOMIPS=hardfloat GOMIPS64=hardfloat GOPPC64=power8 GOROOT_FINAL=/var/tmp/workdir-host-freebsd-arm-paulzhol/go LANG=C LANGUAGE=en_US.UTF8 GOPATH=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/obj/gopath GOBIN= CC=clang
[paulzhol@arm-virt-12 ~]$ ls -l /var/tmp/workdir-host-freebsd-arm-paulzhol/go

Looking at the PATH above, and https://github.com/golang/go/issues/31567#issuecomment-484925837 I don't have the bootstrap go1.4 set, or is there any other go installation present.

The other difference I can think of, is my builder has the entire filesystem mounted read-only, so https://golang.org/cl/223746 is needed to ignore failing writes. Although GOROOT=/var/tmp/workdir-host-freebsd-arm-paulzhol/go set above is writable anyway.

https://build.golang.org/log/d0025d71cbb3628b7800d2eebe1d8fb58c79969d

ok  	cmd/internal/src	0.034s
ok  	cmd/internal/test2json	1.729s
--- FAIL: TestDWARF (4.86s)
    dwarf_test.go:41: cmd/link is stale - run go install cmd/link
FAIL
FAIL	cmd/link	56.431s
FAIL
go tool dist: Failed: exit status 1

I've captured the stdout/stderr output from the buildlet to look at the env. variables as I understand it was related to clearing GOROOT_FINAL in similar issues.

2020/08/28 11:21:06 [0x314ba580] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/internal/obj/riscv" "go_test:cmd/internal/obj/x86" "go_test:cmd/internal/objabi"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:22:09 [0x314ba580] Run = ok, after 1m3.510284157s

2020/08/28 11:22:09 [0x314bb1e0] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/internal/src" "go_test:cmd/internal/test2json" "go_test:cmd/link"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:23:49 [0x314bb1e0] Run = exit status 1, after 1m39.66510577s

2020/08/28 11:23:49 [0x3148cdc0] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/link/internal/benchmark" "go_test:cmd/link/internal/ld" "go_test:cmd/link/internal/loader"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:23:50 Halting in 1 second.

Maybe this is a dup of #33598 but on FreeBSD-arm.

@dmitshur I've updated env/freebsd-amd64 to allow building a 11.3-RELEASE images in https://golang.org/cl/186038. I should also test and add a 12.1-RELEASE now, with 12.2 due in two months. I can't create the images though, brad used to do it I think.

Per https://farmer.golang.org:

host-freebsd-arm-paulzhol: 0/0 (1 missing)

/cc @andybons @dmitshur @toothrot @paulzhol

I think I am hitting https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224479#c32 (the work dir, and everything writable on the builder is in a tmpfs /tmp mount and I was using an NFS swapfile - tried with mdconfig and without). I switched the builder to use a partition based swap. It is still backed by an NFS file, but the hypervisor is managing that device.

It has been down for several months, I've restored it only this weekend. It keeps dying from out of memory every few builds as far as I can tell.