/cc @randall77 @griesemer

/cc @FiloSottile

/cc @griesemer

A new Linux ARM builder should be created on EC2. The new builder should help resolve: #40872 and #35628

@dmitshur @toothrot

@bsiegert the non-reverse builder is still planned.

MacStadium has turned over the new hardware. We will be working to incorporate it into the build infrastructure in the coming days.

@esnolte Would you be able to provide insight into why the builders are missing? You are listed as the owner at https://farmer.golang.org/builders

From https://farmer.golang.org/#health:

/cc @andybons @golang/osp-team

@ramereth I was able to get it back up and running by changing ExecStart to ExecStart=/usr/local/bin/rundockerbuildlet -basename=ppc64_ -image=golang/builder -n=5 -cpu=2 -memory= --env=host-linux-ppc64-osu. Thanks again for your help.

@ramereth Thanks for your work on this. We are able to log in.

@prattmic was the original reporter of the issue.

The gomote ssh command is not working because the coordinator is attempting to connect to the remote buildlet via HTTP instead of HTTPS. This is only an issue on the aws builders.

/cc @dmitshur @toothrot

/cc @tombergan

/cc @FiloSottile

Approved because this is a serious issue without a workaround.

Approved because this is a serious issue without a workaround.

Approved because this is a serious issue without a workaround.

This has been approved because it's a serious issue without a workaround.

This has been approved because it's a serious issue without a workaround.

See https://golang.org/wiki/GerritAccess

Approver, Trybot access request for [email protected]

https://go-review.googlesource.com/#/admin/groups/1005,members https://go-review.googlesource.com/#/admin/groups/1030,members

Hi Roberto,

Thank you for contributing to the Go project! I've discussed this request with the team (/cc @golang/osp-team), and we agreed to grant you access. I've added you to the may-start-trybots and approvers groups now.

Please be familiar with the rules for voting and submitting changes as described at https://golang.org/wiki/GerritAccess#approver-access-approvers. If you're unsure or have any questions, please don't hesitate to ask us.

Additionally, as described at https://golang.org/wiki/GerritAccess#trybot-access-may-start-trybots, please be sure to at least skim CLs before starting a trybot run.

Thank you again for contributing, your help is greatly appreciated!

/cc @rsc

/cc @dmitshur @andybons

@sc0Vu Do you encounter the same error if you download and use the pkg from the golang site?

/cc @mikioh @bradfitz @ianlancetaylor

The following context timeout deadlocks on play.golang.org but works fine using several recent versions of the Go compiler.

https://play.golang.org/p/7vqNz9WXa6o

ctx, cancel := context.WithTimeout(context.Background(), time.Millisecond)
defer cancel()

<-ctx.Done()	
err := ctx.Err()

What did you expect to see?

net.Error: context.deadlineExceededError

What did you see instead?

timeout running program

Update Looks like #39016 is related. If we remove the dependency on "net" then the deadlock seems to resolve.

https://play.golang.org/p/NM3xC6_R1_Q

@Xeoncross This does seem similar to the bug reported in #39016. I'm going to close this issue since it seems to be a duplicate issue. Would you mind adding the sample case and description to the previous issue?

/cc @ianlancetaylor @bradfitz @tklauser

/cc @andybons @dmitshur @toothrot

Changed this to an x/build issue since I believe this is an issue with the packaging.

/cc @dmitshur @andybons

/cc @julieqiu @jba @jamalc

/cc @julieqiu @jba @jamalc

/cc @julieqiu @jba @jamalc

From https://farmer.golang.org/#health:

# "macs" status: MacStadium Mac VMs
# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host09a missing, not seen for 60h44m42s
Warn: macstadium_host09b missing, not seen for 60h42m31s
Warn: 2 machines missing, 10% of capacity
Warn: makemac daemon: vm.destroy("mac_10_12_host09b") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Warn: makemac daemon: vm.destroy("mac_10_15_host09a") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

The machine hosting macstadium_host09a and macstadium_host09b is being unresponsive.

/cc @toothrot @andybons @dmitshur

This was fixed by restarting the physical node hosting hosting those VMs.

From https://farmer.golang.org/#health:

# "macs" status: MacStadium Mac VMs
# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host09a missing, not seen for 60h44m42s
Warn: macstadium_host09b missing, not seen for 60h42m31s
Warn: 2 machines missing, 10% of capacity
Warn: makemac daemon: vm.destroy("mac_10_12_host09b") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Warn: makemac daemon: vm.destroy("mac_10_15_host09a") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

The machine hosting macstadium_host09a and macstadium_host09b is being unresponsive.

/cc @toothrot @andybons @dmitshur

I understand this is about limiting the rate of requests, not the total number.

This is about limiting the rate at which we make requests. We do not currently throttle requests made against the AWS API. In the link posted above, they have listed explicit request and resource rate limits. When a request fails because of us exceeding one of those limits, we fail the build.

Any throttling that we do on the coordinator side should be paired with some retry logic (with exponential backoffs) when a transient failure is encountered. That work will be included in a follow up issue.

To help me understand this better, what is an example of a known situation when this happens?

We issue a number of requests to a cloud provider in various situations:

• A new builder is brought online and the coordinator will attempt to create builds for each missing cell for that builder.
• Any CL submission that causes the all of the x repos to trigger a build.

Another scenario to consider: The reduction of capacity for a provider which would lead to a reduction in quota for an account. This would lower the number of simultaneous builds the coordinator can execute.

Approved. This test exposes a pre-existing condition which which can block releases.

Approved. This test exposes a pre-existing condition which which can block releases.

Approved as this is a documentation fix.

@esnolte agreed to become the maintainer for the builder in #34973. It's been down for some time now and they have not responded since the creation of this issue. I believe we should consider removing this builder.

@zeebo, what's the status with the ARM5 Spacemonkey builders?

We haven't seen them in quite some time.

/cc @bcmills @dmitshur

I'm going to close this issue since it seems like we found a new maintainer at this point in time.

@paulzhol Thank you!

/cc @cherrymui

/cc @griesemer

/cc @ianlancetaylor @findleyr

/cc @andybons @dmitshur @toothrot

/cc @jayconrod

/cc @andybons @dmitshur @toothrot

# "macs" status: MacStadium Mac VMs
# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host01a missing, not seen for 2h25m12s
Warn: macstadium_host01b missing, not seen for 2h56m50s
Warn: macstadium_host02a missing, not seen for 2h57m1s
Warn: macstadium_host02b missing, not seen for 2h57m2s
Warn: macstadium_host03a missing, not seen for 2h56m53s
Warn: macstadium_host03b missing, not seen for 2h32m5s
Warn: macstadium_host04a missing, not seen for 2h56m52s
Warn: macstadium_host04b missing, not seen for 2h57m2s
Warn: macstadium_host05a missing, not seen for 2h36m27s
Warn: macstadium_host05b missing, not seen for 2h57m1s
Warn: macstadium_host06a missing, not seen for 2h32m2s
Warn: macstadium_host06b missing, not seen for 2h57m1s
Warn: macstadium_host07a missing, not seen for 2h56m56s
Warn: macstadium_host07b missing, not seen for 3h3m55s
Warn: macstadium_host08a missing, not seen for 2h56m51s
Warn: macstadium_host08b missing, not seen for 2h41m2s
Warn: macstadium_host09a missing, not seen for 2h56m49s
Warn: macstadium_host09b missing, not seen for 2h56m48s
Warn: macstadium_host10a missing, not seen for 3h11m1s
Warn: macstadium_host10b missing, not seen for 2h56m54s
Error: 20 machines missing, 100% of capacity
Error: makemac daemon: failed to fetch status: Get http://macstadiumd.golang.org:8713: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

/cc @andybons @dmitshur @toothrot

The builders are back. I will keep this open until the MacStadium status changes.

/cc @julieqiu @jba @jamalc

/cc @mpvl

@prashantv I misunderstood the issue and incorrectly renamed it with the runtime package. It's been corrected.

MacStadium's status page is reporting a CenturyLink/Level 3 outage.

# "macs" status: MacStadium Mac VMs
# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host01a missing, not seen for 2h25m12s
Warn: macstadium_host01b missing, not seen for 2h56m50s
Warn: macstadium_host02a missing, not seen for 2h57m1s
Warn: macstadium_host02b missing, not seen for 2h57m2s
Warn: macstadium_host03a missing, not seen for 2h56m53s
Warn: macstadium_host03b missing, not seen for 2h32m5s
Warn: macstadium_host04a missing, not seen for 2h56m52s
Warn: macstadium_host04b missing, not seen for 2h57m2s
Warn: macstadium_host05a missing, not seen for 2h36m27s
Warn: macstadium_host05b missing, not seen for 2h57m1s
Warn: macstadium_host06a missing, not seen for 2h32m2s
Warn: macstadium_host06b missing, not seen for 2h57m1s
Warn: macstadium_host07a missing, not seen for 2h56m56s
Warn: macstadium_host07b missing, not seen for 3h3m55s
Warn: macstadium_host08a missing, not seen for 2h56m51s
Warn: macstadium_host08b missing, not seen for 2h41m2s
Warn: macstadium_host09a missing, not seen for 2h56m49s
Warn: macstadium_host09b missing, not seen for 2h56m48s
Warn: macstadium_host10a missing, not seen for 3h11m1s
Warn: macstadium_host10b missing, not seen for 2h56m54s
Error: 20 machines missing, 100% of capacity
Error: makemac daemon: failed to fetch status: Get http://macstadiumd.golang.org:8713: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

/cc @andybons @dmitshur @toothrot

This does seem like a misuse of t.Cleanup. Silently failing seems odd though.

/cc @mpvl

/cc @julieqiu @jba @jamalc

/cc @julieqiu @jba @jamalc

Approved as this is a critical issue without any workaround.

Approved as this is a critical issue without any workaround.

Approved because it is a critical issue without a workaround.

Approved because this is a critical problem without a workaround.

Approved because this is a critical problem without a workaround.

Approved because this is a critical problem without a workaround.

Approved because this is a critical problem without a workaround.

/cc @ianlancetaylor @bradfitz @tklauser

/cc @julieqiu @jba @jamalc

/cc @julieqiu @jba @jamalc

What are the contents of the main.go file being compiled?

As mentioned in #40933, at this point this seems like questions. You will get better and faster help using a forum. Please see https://golang.org/wiki/Questions. Please comment on the issue if you feel like it was closed in error.

Building up on #40988 I tried to complile the app at Mac OS, and got the below error:

cmd.Run() failed with exit status 2:

output:

# command-line-arguments
/usr/local/go/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: unknown option: -z
clang: error: linker command failed with exit code 1 (use -v to see invocation)

What version of Go are you using (go version)?

<pre> $ go version go version go1.15 darwin/amd64 </pre>

What operating system and processor architecture are you using (go env)?

<details><summary><code>go env</code> Output</summary><br><pre> $ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/Users/hajsf/Library/Caches/go-build" GOENV="/Users/hajsf/Library/Application Support/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GOINSECURE="" GOMODCACHE="/Users/hajsf/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/hajsf/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/usr/local/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64" GCCGO="gccgo" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_d/dz1y2j413g18v0xssf6_64x40000gn/T/go-build017554273=/tmp/go-build -gno-record-gcc-switches -fno-common" </pre></details>

What version of Go are you using (go version)?

<pre> $ go version go version go1.15 linux/amd64 </pre>

What operating system and processor architecture are you using (go env)?

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build091967167=/tmp/go-build -gno-record-gcc-switches"

$uname -a
Linux d5badc077a48-sv 4.14.188-mainline-rev1 #1 SMP Mon Jul 20 19:43:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

What did you do?

https://gitlab.com/ahmetozer/net-tools-service/-/jobs/697169653

What did you see instead?

 > [linux/arm/v7 stage-0 4/6] RUN go get -d -v ./...:
#25 2.766 fatal error: unexpected signal during runtime execution
#25 2.797 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1c9f0]
#25 2.803
#25 2.803 runtime stack:
#25 2.806 runtime: unexpected return pc for runtime.notetsleep_internal called from 0xc2e88c
#25 2.818 stack: frame={sp:0xfffefd48, fp:0xfffefd68} stack=[0xff7f0da4,0xfffefde0)
#25 2.828 fffefcc8:  0000001f  0007159c <runtime.fatalthrow.func1+44>  0001c9f0 <runtime.notetsleep_internal+508>  00000000
#25 2.852 fffefcd8:  00000001  00547e17  00046324 <runtime.fatalthrow+68>  00852900
#25 2.885 fffefce8:  00046184 <runtime.throw+92>  fffefd18  008648e0  fffefd18
#25 2.898 fffefcf8:  00046184 <runtime.throw+92>  00852900  00046184 <runtime.throw+92>  fffefd08
#25 2.905 fffefd08:  00071570 <runtime.fatalthrow.func1+0>  00852900  00046184 <runtime.throw+92>  fffefd18
#25 2.918 fffefd18:  0005c5ec <runtime.sigpanic+920>  fffefd20  00071504 <runtime.throw.func1+0>  00550247
#25 2.928 fffefd28:  0000002a  0001c9f0 <runtime.notetsleep_internal+508>  00550247  0000002a
#25 2.937 fffefd38:  00000000  ffffff92  00852900  00042cf8 <runtime.futexsleep+220>
#25 2.946 fffefd48: <00c2e88c  00000000  00000001  00018601 <runtime.(*cpuProfile).addNonGo+253>
#25 2.957 fffefd58:  000543d4 <runtime.preemptall+96>  00c2e6c0  0004bd70 <runtime.forEachP+904>  00c22000
#25 2.962 fffefd68: >0004bd8c <runtime.forEachP+932>  00852a00  00000002  00000000
#25 2.965 fffefd78:  00c10008  0006fa64 <runtime.gcMarkDone.func1+64>  00852a94  000186a0 <runtime.(*cpuProfile).addExtra+88>
#25 2.979 fffefd88:  00000000  00000000  00000001  00000001
#25 2.981 fffefd98:  00000002  00000000  00c23500  00853108
#25 2.983 fffefda8:  00852900  00c10008  00c23500  00076648 <runtime.systemstack+132>
#25 2.988 fffefdb8:  0058370c  00000002  00000004  00c00b60
#25 2.991 fffefdc8:  0004b530 <runtime.mstart+0>  000764b8 <runtime.rt0_go+160>  000764b0 <runtime.rt0_go+152>  00000000
#25 2.998 fffefdd8:  ff7f0da4  00852900
#25 3.005 runtime.throw(0x550247, 0x2a)
#25 3.007       /usr/local/go/src/runtime/panic.go:1116 +0x5c
#25 3.011 runtime.sigpanic()
#25 3.011       /usr/local/go/src/runtime/signal_unix.go:704 +0x398
#25 3.013 runtime: unexpected return pc for runtime.notetsleep_internal called from 0xc2e88c
#25 3.014 stack: frame={sp:0xfffefd48, fp:0xfffefd68} stack=[0xff7f0da4,0xfffefde0)
#25 3.018 fffefcc8:  0000001f  0007159c <runtime.fatalthrow.func1+44>  0001c9f0 <runtime.notetsleep_internal+508>  00000000
#25 3.024 fffefcd8:  00000001  00547e17  00046324 <runtime.fatalthrow+68>  00852900
#25 3.028 fffefce8:  00046184 <runtime.throw+92>  fffefd18  008648e0  fffefd18
#25 3.036 fffefcf8:  00046184 <runtime.throw+92>  00852900  00046184 <runtime.throw+92>  fffefd08
#25 3.044 fffefd08:  00071570 <runtime.fatalthrow.func1+0>  00852900  00046184 <runtime.throw+92>  fffefd18
#25 3.057 fffefd18:  0005c5ec <runtime.sigpanic+920>  fffefd20  00071504 <runtime.throw.func1+0>  00550247
#25 3.062 fffefd28:  0000002a  0001c9f0 <runtime.notetsleep_internal+508>  00550247  0000002a
#25 3.068 fffefd38:  00000000  ffffff92  00852900  00042cf8 <runtime.futexsleep+220>
#25 3.081 fffefd48: <00c2e88c  00000000  00000001  00018601 <runtime.(*cpuProfile).addNonGo+253>
#25 3.086 fffefd58:  000543d4 <runtime.preemptall+96>  00c2e6c0  0004bd70 <runtime.forEachP+904>  00c22000
#25 3.093 fffefd68: >0004bd8c <runtime.forEachP+932>  00852a00  00000002  00000000
#25 3.108 fffefd78:  00c10008  0006fa64 <runtime.gcMarkDone.func1+64>  00852a94  000186a0 <runtime.(*cpuProfile).addExtra+88>
#25 3.123 fffefd88:  00000000  00000000  00000001  00000001
#25 3.128 fffefd98:  00000002  00000000  00c23500  00853108
#25 3.134 fffefda8:  00852900  00c10008  00c23500  00076648 <runtime.systemstack+132>
#25 3.142 fffefdb8:  0058370c  00000002  00000004  00c00b60
#25 3.148 fffefdc8:  0004b530 <runtime.mstart+0>  000764b8 <runtime.rt0_go+160>  000764b0 <runtime.rt0_go+152>  00000000
#25 3.159 fffefdd8:  ff7f0da4  00852900
#25 3.163 runtime.notetsleep_internal(0x852a00, 0x2, 0x0, 0xc10008)
#25 3.167       /usr/local/go/src/runtime/lock_futex.go:205 +0x1fc
#25 3.178
#25 3.182 goroutine 4 [GC worker (idle)]:
#25 3.188 runtime.systemstack_switch()
#25 3.189       /usr/local/go/src/runtime/asm_arm.s:294 +0x4 fp=0xc2bf6c sp=0xc2bf68 pc=0x765b8
#25 3.195 runtime.gcMarkDone()
#25 3.197       /usr/local/go/src/runtime/mgc.go:1467 +0xe4 fp=0xc2bf98 sp=0xc2bf6c pc=0x2c830
#25 3.202 runtime.gcBgMarkWorker(0xc23500)
#25 3.204       /usr/local/go/src/runtime/mgc.go:2018 +0x260 fp=0xc2bfe4 sp=0xc2bf98 pc=0x2dd9c
#25 3.211 runtime.goexit()
#25 3.212       /usr/local/go/src/runtime/asm_arm.s:857 +0x4 fp=0xc2bfe4 sp=0xc2bfe4 pc=0x783e4
#25 3.216 created by runtime.gcBgMarkStartWorkers
#25 3.217       /usr/local/go/src/runtime/mgc.go:1839 +0x68
#25 3.235
#25 3.235 goroutine 1 [running]:
#25 3.238       goroutine running on other thread; stack unavailable
#25 3.485 qemu: uncaught target signal 4 (Illegal instruction) - core dumped
#25 3.492 Illegal instruction
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c go get -d -v ./...]: buildkit-runc did not terminate successfully

I do not get a any error on amd64 on golang official container but when i try to build multi platform with docker buildx, go get command has a error on arm and arm64. So i try golang:alpine it gets a same error, I change kernel version and get same result.

Currently to temporarily, I create a alpine container and install go and git with package manager (apk) and replace FROM golang with my container. Now it is work.

/ # go version
go version go1.13.14 linux/amd64

https://gitlab.com/ahmetozer/net-tools-service/-/jobs/698402255

A better place to report this may be the docker buildx issue tracker. Since docker buildx is an experimental feature and it seems to be initiating the go get command, they may have a clear understanding of what is happening. Closing this issue. Please comment if you disagree or if I have misunderstood.

/cc @robpike @mvdan

/cc @ianlancetaylor @griesemer

/cc @golang/tools-team

/cc @FiloSottile

/cc @julieqiu @jba @jamalc

/cc @cherrymui

/cc @julieqiu @jba @jamalc

May I ask why you are using the replace directive in this manner?

/cc @rsc @dsnet @bradfitz @mvdan

@terryjiang2020 We have been unable to reproduce this issue. Do you have another Windows 10 64 bit machine you can test the installer on? Can you try @alexrousg suggestion and download the zip file and expand it?

Thank you for creating an issue @sethkoehler. If you believe that your suggested fix will resolve your issue, please feel free to create a CL.

/cc @robpike @rsc

/cc @rsc @bradfitz

Per https://farmer.golang.org:

host-freebsd-arm-paulzhol: 0/0 (1 missing)

/cc @andybons @dmitshur @toothrot @paulzhol

Per https://farmer.golang.org:

host-linux-arm5spacemonkey: 0/0 (3 missing)

/cc @andybons @dmitshur @toothrot @esnolte

We have engaged MacStadium with an inquiry into upgrading the builders with newer hardware.

@Alovchin91 We are working out the licensing. As soon as it's worked out we can move forward by adding the builders.

Example 1:

// a.go2
package p

type S [type T] struct {}

// errors_test.go2
package p

import "testing"

func TestFoo(t *testing.T) {}

$ go tool go2go test
type checking failed for p
errors_test.go2:1:1: inconsistent use of () or [] for type parameters

Workaround: Rename the test file name from errors_test.go2 to a_test.go2 will get a PASS.

Example 2:

// a.go2
package p

// a_test.go2
package p

import "testing"
type S [type T] struct {}
func TestFoo(t *testing.T) {}

$ go tool go2go test
type checking failed for p
a_test.go2:1:1: inconsistent use of () or [] for type parameters

Workaround: Delete a.go2 will get a PASS.

This seems to have reported in #40806. I'm going to close this issue. Please let me know if you think this was closed in error.

/cc @julieqiu @jba @jamalc

/cc @mikioh

/cc @mikioh

There should exist a way to limit the number of requests made to the AWS API. Not limiting the request rate can lead to errors when we have a burst of request. The API request throttling is documented at: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html

/cc @andybons @dmitshur @toothrot

# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host09a missing, not seen for 13h21m50s
Warn: macstadium_host09b missing, not seen for 13h26m46s
Warn: 2 machines missing, 10% of capacity
Warn: makemac daemon: vm.destroy("mac_10_12_host09b") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Warn: makemac daemon: vm.destroy("mac_10_11_host09a") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Similar recent issues: #39378, #39306

/cc @andybons @dmitshur @toothrot