profile
viewpoint
Chris Cotter tritone Google New York

googleapis/google-cloud-go 2712

Google Cloud Client Libraries for Go.

tritone/dotnet-docs-samples 0

.NET code samples used on https://cloud.google.com

tritone/go-genproto 0

Generated code for Google Cloud client libraries.

tritone/golang-samples 0

Sample apps and code written for Google Cloud in the Go programming language.

tritone/google-api-go-client 0

Auto-generated Google APIs for Go.

tritone/google-cloud-cpp 0

C++ Client Libraries for Google Cloud Services

tritone/google-cloud-go 0

Google Cloud Client Libraries for Go.

tritone/google-cloud-python 0

Google Cloud Client Library for Python

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-go

feat(storage): add retry config to ACL handle methods

 func TestRetryer(t *testing.T) { 				o = o.Retryer(tc.objectOptions...) 			} -			if diff := cmp.Diff(-				o.retry,-				tc.want,-				cmp.AllowUnexported(retryConfig{}, gax.Backoff{}),-				// ErrorFunc cannot be compared directly, but we check if both are-				// either nil or non-nil.-				cmp.Comparer(func(a, b func(err error) bool) bool {-					return (a == nil && b == nil) || (a != nil && b != nil)-				}),-			); diff != "" {-				s.Fatalf("retry not configured correctly: %v", diff)+			configHandleCases := []struct {+				r    *retryConfig+				name string+				want *retryConfig+			}{+				{+					name: "object.retry",+					r:    o.retry,+					want: tc.want,+				},+				{+					name: "object.ACL()",+					r:    o.ACL().retry,+					want: tc.want,+				},+				{+					name: "bucket.ACL()",+					r:    b.ACL().retry,+					want: b.retry,+				},+				{+					name: "bucket.DefaultObjectACL()",+					r:    b.DefaultObjectACL().retry,+					want: b.retry,+				},+			}+			for _, ac := range configHandleCases {+				s.Run(ac.name, func(ss *testing.T) {

Didn't realize these could be nested... nice!

BrennaEpp

comment created time in 2 days

Pull request review commentgoogleapis/google-cloud-go

feat(storage): add retry config to ACL handle methods

 func (a *ACLHandle) bucketDelete(ctx context.Context, entity ACLEntity) error { func (a *ACLHandle) objectList(ctx context.Context) ([]ACLRule, error) { 	var acls *raw.ObjectAccessControls 	var err error-	err = runWithRetry(ctx, func() error {+	err = run(ctx, func() error { 		req := a.c.raw.ObjectAccessControls.List(a.bucket, a.object) 		a.configureCall(ctx, req) 		acls, err = req.Do() 		return err-	})+	}, a.retry, true)

You are missing a bunch of calls in this file (mostly stuff that is not retried currently)-- could you add run() wrappings for those as well?

BrennaEpp

comment created time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentGoogleCloudPlatform/golang-samples

feat(storage): update SignedURL samples to handle auth for user

 func TestV4SignedURL(t *testing.T) {  	bucketName := tc.ProjectID + "-signed-url-bucket-name" 	objectName := "foo.txt"-	serviceAccount := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS")-	if serviceAccount == "" {++	if os.Getenv("GOOGLE_APPLICATION_CREDENTIALS") == "" {

I think we can assume that users need to have credentials in order to run these tests.

BrennaEpp

comment created time in 2 days

Pull request review commentGoogleCloudPlatform/golang-samples

feat(storage): update SignedURL samples to handle auth for user

 package objects  // [START storage_generate_upload_signed_url_v4] import (+	"context" 	"fmt" 	"io"-	"io/ioutil" 	"time"  	"cloud.google.com/go/storage"-	"golang.org/x/oauth2/google" ) -// generateV4PutObjectSignedURL generates object signed URL with PUT method.-func generateV4PutObjectSignedURL(w io.Writer, bucket, object, serviceAccount string) (string, error) {+// generateV4GetObjectSignedURL generates object signed URL with PUT method.+// Signing a URL requires credentials authorized to sign a URL. You can pass+// these in as a Google service account private key (obtainable from the Google+// Developers Console) or using a SignBytes function implementing custom signing.+// In this example, neither of these options are used, which means the following+// code only works when the credentials are defined via the environment variable+// GOOGLE_APPLICATION_CREDENTIALS.

Same as above

BrennaEpp

comment created time in 2 days

Pull request review commentGoogleCloudPlatform/golang-samples

feat(storage): update SignedURL samples to handle auth for user

 package objects  // [START storage_generate_signed_url_v4] import (+	"context" 	"fmt" 	"io"-	"io/ioutil" 	"time"  	"cloud.google.com/go/storage"-	"golang.org/x/oauth2/google" )  // generateV4GetObjectSignedURL generates object signed URL with GET method.-func generateV4GetObjectSignedURL(w io.Writer, bucket, object, serviceAccount string) (string, error) {+// Signing a URL requires credentials authorized to sign a URL. You can pass+// these in as a Google service account private key (obtainable from the Google+// Developers Console) or using a SignBytes function implementing custom signing.+// In this example, neither of these options are used, which means the following+// code only works when the credentials are defined via the environment variable+// GOOGLE_APPLICATION_CREDENTIALS.

This is not quite accurate, correct? It could also populate signBlob.

I would also maybe move this comment down to where the creds could be passed in via the SignedURLOptions and make it explicit that that's where you would do it.

BrennaEpp

comment created time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

issue commentgoogleapis/google-cloud-go

storage: Provide a way to resume uploads from another instance of the application

Thanks for the feature request!

I think this is a reasonable idea, but unfortunately it would be a heavy lift to accomplish given the library's design. The current design of the Writer is such that the details of the upload are fully encapsulated in the google.golang.org/api library that we depend on. In the storage library we construct an ObjectsInsertCall, then in google.golang.org/api the upload type is chosen (may be multipart or resumable based on the ChunkSize parameter and the input size) and the buffering and individual chunk uploads (if resumable) are managed. Retry logic for failed uploads is also there. See code here and here. So, there are no guarantees that the upload will be resumable. And we don't have a way to start a Writer or ObjectsInsertCall based on an already-started resumable upload session (or based on a signed URL for that matter).

Of course, all this is potentially surmountable -- with code changes in both libraries and breaking the encapsulation of upload logic that we currently have. I'll leave this issue open but it's unlikely to get prioritized unless it becomes something a lot of users really want.

The idea of using signed URLs for this use case sounds like a good approach! We just added new surfaces to this library to make it easier to generate them without having to worry about auth complications as much (see BucketHandle.SignedURL)

williambrode

comment created time in 6 days

Pull request review commentgoogleapis/storage-testbench

feat: introduce support for return-broken-stream-final-chunk-after-YB

 curl -H "x-retry-test-id: 1d05c20627844214a9ff7cbcf696317d" "http://localhost:91 | ----------------------- | --- | return-X                | Testbench will fail with HTTP code provided for `X`, e.g. return-503 returns a 503 | return-X-after-YK       | Testbench will return X after YKiB of uploaded data+| return-200-with-YB      | Testbench will return 200 with a response of Y bytes from object metadata JSON on resumable upload completion.

The current return-broken-stream causes the failure at the beginning of the response while headers are being sent. I don't see why we shouldn't extend pattern for after-YB to cover failures occurring in the request body (whether data or metadata).

frankyn

comment created time in 7 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-go

test(storage): add tests for clone methods

+// Copyright 2021 Google LLC+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+//      http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++package storage++import (+	"reflect"+	"testing"+	"time"++	"github.com/google/go-cmp/cmp"+)++func TestPostPolicyV4Clone(t *testing.T) {+	t.Parallel()++	opts := &PostPolicyV4Options{+		GoogleAccessID: "accessID",+		PrivateKey:     []byte{},+		SignBytes: func(b []byte) ([]byte, error) {+			return b, nil+		},+		SignRawBytes: func(bytes []byte) ([]byte, error) {+			return bytes, nil+		},+		Expires:             time.Now(),+		Style:               VirtualHostedStyle(),+		Insecure:            true,+		Fields:              &PolicyV4Fields{ACL: "test-acl"},+		Conditions:          []PostPolicyV4Condition{},+		shouldHashSignBytes: true,+	}++	// Check that all fields are set to a non-zero value, so we can check that+	// clone accurately clones all fields and catch newly added fields not cloned+	reflectOpts := reflect.ValueOf(*opts)++	for i := 0; i < reflectOpts.NumField(); i++ {+		if reflectOpts.Field(i).IsZero() {

Ooh, apparently this was added in Go 1.13 https://pkg.go.dev/reflect#Value.IsZero

BrennaEpp

comment created time in 7 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-go

test(storage): add tests for clone methods

+// Copyright 2021 Google LLC+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+//      http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++package storage++import (+	"reflect"+	"testing"+	"time"++	"github.com/google/go-cmp/cmp"+)++func TestPostPolicyV4Clone(t *testing.T) {

nit: change the test name or leave a comment to indicate that it's the clone method for the options struct and not for the PostPolicyV4 method itself. Same thing for the other test below

BrennaEpp

comment created time in 7 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogleapis/storage-testbench

feat: introduce support for return-200-with-YB

 curl -H "x-retry-test-id: 1d05c20627844214a9ff7cbcf696317d" "http://localhost:91 | ----------------------- | --- | return-X                | Testbench will fail with HTTP code provided for `X`, e.g. return-503 returns a 503 | return-X-after-YK       | Testbench will return X after YKiB of uploaded data+| return-200-with-YB      | Testbench will return 200 with a response of Y bytes from object metadata JSON on resumable upload completion.

As discussed, I feel like the name is slightly unclear. IIUC this is causing a broken stream after Y bytes-- so let's call it that instead of return-200

frankyn

comment created time in 8 days

PullRequestReviewEvent

create barnchtritone/google-api-go-client

branch : write-retry

created branch time in 8 days

more