I told you why. I added debug to see why cancel was sent. I can give you the line numbers. Again, the client is cancelling the context or the response body is closed before we finishing reading the body.

So I won't comment on the setup but its not exactly trivial and the above well, wasn't enough.

What I ended up seeing were 2 types of reasons for the RST_STREAM:

1. context cancelled
2. missing stream end when the response body was closed

The problem is easier to solve than I thought. The race is close to what is described but slightly incorrect. This problem can occur with all request but its easiest to reproduce with a HEAD/no response body request. The readLoop for request 1 will put the connection back into the idle pool. Request 2 will pick it up and cancel it. Request 1's roundTrip will be notified that the connection has closed because it hasn't received the response. Once it puts the connection back, it should no longer care about any connection issue since a response will be coming.

Here is a minimal testcase to reproduce the issue:

package http_test

import (
	"context"
	"net/http"
	"net/http/httptest"
	"sync"
	"testing"
)

func Test41600(t *testing.T) {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		w.Header().Add("Content-Length", "0")
	}))
	defer ts.Close()

	client := ts.Client()
	transport := client.Transport.(*http.Transport)
	transport.MaxIdleConns = 1
	transport.MaxConnsPerHost = 1

	var wg sync.WaitGroup

	ctx, cancel := context.WithCancel(context.Background())

	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for ctx.Err() == nil {
				reqctx, reqcancel := context.WithCancel(context.Background())
				go reqcancel()
				req, _ := http.NewRequestWithContext(reqctx, http.MethodGet, ts.URL, nil)
				rsp, err := client.Do(req)
				if err == nil {
					defer rsp.Body.Close()
				}
			}
		}()
	}

	for {
		req, _ := http.NewRequest(http.MethodGet, ts.URL, nil)
		if rsp, err := client.Do(req); err != nil {
			t.Errorf("unexpected: %p %v", req, err)
			break
		} else {
			rsp.Body.Close()
		}
	}

	cancel()
	wg.Wait()
}

I am going to have to think about this. There doesn't seem to be an easy fix. The goroutine which reads the response, puts the connection back into the pool and then sends the response back via a channel to roundTrip(). However, the roundTrip() is also watching for contexts that are Done() and will cancel the connection. The connection is already in the idleConn pool. Turns out a request will get this connection before the previous one is actually done so the connection is sometimes broken out of the pool and sometimes healthy but then breaks later.

Sorry about the wild goose chase. I have narrowed down the issue. Still trying to understand how its happening but I do understand why you get the error. The connection handed to a request, not just the non-cancelling one is already broken, closed and cancelled. Now the question is why this connection actually exists in the pool.

A bit difficult to figure out what went wrong but given that sending a request and receiving a response took 12s, I would say something was wrong before we hit this timeout.

Take a look at how many connections you have in TIME_WAIT when it fails.

Will give it a go... its random when it happens.

<!--- Please note the following potential times when an issue might be in Terraform core:

• Configuration Language or resource ordering issues
• State and State Backend issues
• Provisioner issues
• Registry issues
• Spans resources across multiple providers

If you are running into one of these scenarios, we recommend opening an issue in the Terraform core repository instead. --->

<!--- Please keep this note for the community --->

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

<!--- Thank you for keeping this note for the community --->

Terraform (and AzureRM Provider) Version

<!--- Please run terraform -v to show the Terraform core version and provider version(s). If you are not running the latest version of Terraform or the provider, please upgrade because your issue may have already been fixed. Terraform documentation on provider versioning. --->

Affected Resource(s)

<!--- Please list the affected resources and data sources. --->

• azurerm_XXXXX

Terraform Configuration Files

<!--- Information about code formatting: https://help.github.com/articles/basic-writing-and-formatting-syntax/#quoting-code --->

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

<!--- Please provide a link to a GitHub Gist containing the complete debug output. Please do NOT paste the debug output in the issue; just paste a link to the Gist.

To obtain the debug output, see the Terraform documentation on debugging. --->

Panic Output

<!--- If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log. --->

Expected Behavior

<!--- What should have happened? --->

Actual Behavior

<!--- What actually happened? --->

Steps to Reproduce

<!--- Please list the steps required to reproduce the issue. --->

1. terraform apply

Important Factoids

<!--- Are there anything atypical about your accounts that we should know? For example: Running in a Azure China/Germany/Government? --->

References

<!--- Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests

Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Such as vendor documentation? --->

• #0000

Its not so simple. Regardless of where we could put StartGracefulShutdown(), the current code lacks most of the guards that are currently provided via http.Server. One such example is preventing new connections from being served once Shutdown is called. Exposing the current shutdown behavior just affects the current connections but does nothing to prevent new ones.

Its broken, but so is h2c. If you attempt to do a POST instead of GET which is more common with data, you get the same error. There are no tests for h2c, so I am more concerned that there is more than just this example that is broken.

How are you even creating a "standalone" http2 server?

Good news, its easy to fix. Just have to create a test that always fails.

Looking at this further, the issue is when a GOAWAY is sent from the server all outstanding streams are sent a response errClientConnGotGoAway. However, cs.writeRequestBody() may still be executing. Unfortunately, this isn't the only situation where this occurs.

dupe of https://github.com/golang/go/issues/31192

There are variations of this issue even when running this test case.

While CONNECT is the mechanism used, this is about the environment variables. all_proxy was meant to be the catch all but that is not implemented.

that doesnt explain what the arg is supposed to represent

access to AWS is required

did you mention kafkacat is required? should probably check for it

not a good variable name

The zap logger is safe for concurrent use, https://godoc.org/go.uber.org/zap#Logger. I use it myself and have had no issues.

@networkimprov the fix has yet to be approved

@bobcatfish I wouldn't say we are hiding a race. Most loggers are thread safe. t.Log however is a special purpose logger that is tied to a different lifecycle. If you really want to use a testing.T underneath the logger, you need to make a stronger gesture to stop all users of the logger before the test completes. This is a pretty tall task but could be valuable.

The original code used the zaptest Logger which wrote to t.Logf and may call t.Fail. t.Fail is a bit of a dangerous thing when running on random goroutines and does not guarantee that the actual test will fail since it could be after the test has already completed. As for the t.Logf, there is this latest issue, https://github.com/golang/go/issues/40343 The documentation for testing.T states that while Log and Error can be used on other goroutines, the other methods may not.

The issue reported was fixed.

I can't see a way of fixing this without changing the http2 side to something we can detect on both sides.

I don't make the calls on back ports but this does qualify since it does not behave as documented and there is no workaround.