profile
viewpoint

googleapis/google-cloud-cpp 291

C++ Client Libraries for Google Cloud Services

GoogleCloudPlatform/pubsub 186

This repository contains open-source projects managed by the owners of Google Cloud Pub/Sub.

dpcollins-google/bahir 0

Mirror of Apache Bahir

dpcollins-google/beam 0

Apache Beam

dpcollins-google/gapic-generator-python 0

Generate Python API client libraries from Protocol Buffers.

dpcollins-google/gax-java 0

Google API Extensions for Java - shared runtime for clients generated by googleapis/gapic-generator

dpcollins-google/google-cloud-4-words 0

The Google Cloud Developer's Cheat Sheet

Pull request review commentgoogleapis/google-cloud-cpp

feat(pubsublite): stream factories for PSL bidi streams

+// Copyright 2021 Google LLC+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+//      https://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++#ifndef GOOGLE_CLOUD_CPP_GOOGLE_CLOUD_PUBSUBLITE_INTERNAL_STREAM_FACTORY_H+#define GOOGLE_CLOUD_CPP_GOOGLE_CLOUD_PUBSUBLITE_INTERNAL_STREAM_FACTORY_H++#include "google/cloud/internal/async_read_write_stream_impl.h"+#include "google/cloud/version.h"+#include <google/cloud/pubsublite/v1/cursor.grpc.pb.h>+#include <google/cloud/pubsublite/v1/publisher.grpc.pb.h>+#include <google/cloud/pubsublite/v1/subscriber.grpc.pb.h>+#include <unordered_map>++namespace google {+namespace cloud {+namespace pubsublite_internal {+GOOGLE_CLOUD_CPP_INLINE_NAMESPACE_BEGIN++template <class Request, class Response>+using BidiStream = internal::AsyncStreamingReadWriteRpc<Request, Response>;++template <class Request, class Response>+using StreamFactory = std::function<std::unique_ptr<BidiStream<Request, Response>>()>;++using ClientMetadata = std::unordered_map<std::string, std::string>;++inline std::unique_ptr<grpc::ClientContext> MakeGrpcClientContext(+    ClientMetadata const& metadata) {+  auto context = absl::make_unique<grpc::ClientContext>();+  for (const auto& kv : metadata) {

Done.

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-cpp

feat(pubsublite): stream factories for PSL bidi streams

 class CompletionQueue;  namespace internal { std::shared_ptr<CompletionQueueImpl> GetCompletionQueueImpl(-    CompletionQueue& cq);+    CompletionQueue const& cq);

As long as there are no larger fixes, I think it would be simpler to keep this together as one.

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-cpp

feat(pubsublite): stream factories for PSL bidi streams

+// Copyright 2021 Google LLC+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+//      https://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++#include "google/cloud/pubsublite/internal/stream_factory.h"+#include <gmock/gmock.h>++namespace google {+namespace cloud {+namespace pubsublite_internal {++TEST(StreamFactoryTest, CreateStreams) {+    CompletionQueue queue;+    ClientMetadata metadata{{"key1", "value1"}, {"key2", "value2"}};+    auto publish_factory = MakeStreamFactory(std::shared_ptr<pubsublite::v1::PublisherService::StubInterface>(nullptr), queue, metadata);

I rebased and ran the formatter now that there are no typos.

dpcollins-google

comment created time in 3 days

push eventdpcollins-google/google-cloud-cpp

Carlos O'Ryan

commit sha fc6a1af367ed93b0e8a5398758d5e1b69901c47e

feat(generator): generate quickstart scaffold (#7676) Modify the generator to also create a scaffold for the quickstart program. Then I used the generator to create the scaffold for the `logging`, `pubsublite`, `secretmanager`, and `tasks` libraries. The generated code requires some editing, but these are fairly small. These quickstarts are compiled in the `cmake-install` build, where we can compile them in parallel. This was very useful, it found several bugs in the generated CMake files.

view details

Carlos O'Ryan

commit sha 31d387e631d0b93a5a1dcc6293393858b1185761

doc: update CHANGELOG for v1.34.0 release (#7675)

view details

Carlos O'Ryan

commit sha 6080a8b44ab0fb2d7c62b40124e7b11c5991779c

chore: update versions post v1.34.0 (#7678)

view details

Bradley White

commit sha 823c1217e0e46f9edc2c3e75fdd10771b97344ac

feat(common): add support for call-tree-specific options (#7669) Provide access to the prevailing `Options` for an operation without having to plumb function parameters through the various and sundry layers, like connection. This also means that helper libraries can determine things like whether a tracing component is enabled, without having to distort their APIs. For example, if all `ServiceClient::Operation()` calls are implemented like ``` class ServiceClient { public ServiceClient(std::shared_ptr<ServiceConnection> connection, Options options = {}) : connection_(std::move(connection)), options_(ServiceDefaultOptions(std::move(options))) {} T Operation(..., Options options = {}) { internal::OptionsSpan span(internal::MergeOptions(options, options_)); ... } private: std::shared_ptr<ServiceConnection> connection_; Options options_; }; ``` then the connection layer, or any other internal library, can retrieve the prevailing options using `internal::CurrentOptions()`. If the operation is asynchronous, the invocation-time `Options` are also installed during its callbacks.

view details

WhiteSource Renovate

commit sha 052432d89cbff142e3868bab77d2c88f3f80b620

chore(deps): update dependency com_github_googleapis_google_cloud_cpp to v1.34.0 (#7679)

view details

Carlos O'Ryan

commit sha 9fbee10622339a3d0994e0bc54dfd1dcd4a64123

cleanup!: remove beta proto libraries (#7680) Remove beta proto libraries. These are unused by any client library, and the GA versions are now available.

view details

Carlos O'Ryan

commit sha a984b4eaa0f340f2428b787bb8e6bb125036991c

cleanup: remove workaround for old protobuf+compiler (#7681) Old versions of protobuf (prior to 3.15, we require 3.15.8) generated code that could not be compiled with older compilers (GCC < 6, we require >= 6.x). In other words, this workaround is not necessary for two reasons, we can safely remove it I think.

view details

Bradley White

commit sha b33672f83d6561ad64bce32c0af99db392d78db2

doc(common): add a note about AsyncGrpcOperation and OptionsSpan (#7682)

view details

Carlos O'Ryan

commit sha 92cf451d6b107d4a78b0683ce3130e38b4d07efc

cleanup: warn about untested compiler versions (#7684) Add warnings when the library is configured with a compiler we do not test with, and fatal messages for versions where we know our dependencies won't work. I also made some cosmetic fixes to the top-level CMakeLists.txt file, just to make it more readable for us.

view details

Carlos O'Ryan

commit sha 1644b06ddae149c9698941381a55e2b15f97f3fd

cleanup(bigtable): remove workaround for old protobuf versions (#7685)

view details

Carlos O'Ryan

commit sha 6f77c3e87a1a8ffeb5502c0a6ac59fba67f43e63

cleanup: only stop build based on our requirements (#7687)

view details

Bradley White

commit sha b44055be357e5582092f5a8f3aa530cdaee08147

feat(generator): add options support to generated clients (#7683) The generated client constructor now takes a defaulted `Options` argument, which is merged with the default options for the service, and then stored as a member. Similarly, the generated client operations also take a defaulted `Options` argument, which is then merged with the client options from above, and installed as the prevailing options for the duration of the operation by instantiating an `OptionsSpan`. Any module within the scope of that operation, or its continuations, can then retrieve those prevailing options. So, for example, a metadata decorator could add a header based on a per-operation option, or a retry policy might modify its behavior based on another option.

view details

dpcollins-google

commit sha edbe6642ed90b67a68dc25c046ed8f1709bec040

chore: bump checkers typos version so it works on mac and fix typos (#7693) <!-- Reviewable:start --> This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/googleapis/google-cloud-cpp/7693) <!-- Reviewable:end -->

view details

Daniel Collins

commit sha 99f3954ba22b3d1cb4c61fa32102037a7dececd5

feat[pubsublite]: Add stream factories for PSL bidi streams This abstracts away from the GRPC classes at the lower levels and provides a simple surface to build bidi streaming retries on. #7670

view details

Daniel Collins

commit sha 9b58ca6aaf29c8b7e9b0933c9464e8e5a033263e

Extra changes

view details

Daniel Collins

commit sha d2a6c8d139b3a7ec41685f8d8c1d8cb8b182eb91

add includes

view details

Daniel Collins

commit sha 2c956a0c2bf97bf0bdd776d9b8718c2f9a69e66a

Change over BUILD file to GLOB and fix build errors

view details

Daniel Collins

commit sha edcb08a3a025a8f398f1227208f493461eaf675a

push

view details

Daniel Collins

commit sha e5a1e992d945f996e2496bcb8206d6c05452c637

Add tests to CMake and remove bazel file creation

view details

Daniel Collins

commit sha 5a01ba368c68e3fc0e71b59b2d27d9c65c2d4e44

fixes

view details

push time in 3 days

push eventdpcollins-google/google-cloud-cpp

Daniel Collins

commit sha 81c53654d5dc015ae40d9b8449ed715d21230b68

newline

view details

push time in 3 days

push eventdpcollins-google/google-cloud-cpp

Daniel Collins

commit sha ef3f28fce8a2abca113f589bc6151a61d4c25751

Fixes

view details

push time in 3 days

push eventdpcollins-google/google-cloud-cpp

Daniel Collins

commit sha 62a239d552d72c193bc211743533e9a51640e762

Add exclusions

view details

push time in 3 days

push eventdpcollins-google/google-cloud-cpp

Carlos O'Ryan

commit sha fc6a1af367ed93b0e8a5398758d5e1b69901c47e

feat(generator): generate quickstart scaffold (#7676) Modify the generator to also create a scaffold for the quickstart program. Then I used the generator to create the scaffold for the `logging`, `pubsublite`, `secretmanager`, and `tasks` libraries. The generated code requires some editing, but these are fairly small. These quickstarts are compiled in the `cmake-install` build, where we can compile them in parallel. This was very useful, it found several bugs in the generated CMake files.

view details

Carlos O'Ryan

commit sha 31d387e631d0b93a5a1dcc6293393858b1185761

doc: update CHANGELOG for v1.34.0 release (#7675)

view details

Carlos O'Ryan

commit sha 6080a8b44ab0fb2d7c62b40124e7b11c5991779c

chore: update versions post v1.34.0 (#7678)

view details

Bradley White

commit sha 823c1217e0e46f9edc2c3e75fdd10771b97344ac

feat(common): add support for call-tree-specific options (#7669) Provide access to the prevailing `Options` for an operation without having to plumb function parameters through the various and sundry layers, like connection. This also means that helper libraries can determine things like whether a tracing component is enabled, without having to distort their APIs. For example, if all `ServiceClient::Operation()` calls are implemented like ``` class ServiceClient { public ServiceClient(std::shared_ptr<ServiceConnection> connection, Options options = {}) : connection_(std::move(connection)), options_(ServiceDefaultOptions(std::move(options))) {} T Operation(..., Options options = {}) { internal::OptionsSpan span(internal::MergeOptions(options, options_)); ... } private: std::shared_ptr<ServiceConnection> connection_; Options options_; }; ``` then the connection layer, or any other internal library, can retrieve the prevailing options using `internal::CurrentOptions()`. If the operation is asynchronous, the invocation-time `Options` are also installed during its callbacks.

view details

WhiteSource Renovate

commit sha 052432d89cbff142e3868bab77d2c88f3f80b620

chore(deps): update dependency com_github_googleapis_google_cloud_cpp to v1.34.0 (#7679)

view details

Carlos O'Ryan

commit sha 9fbee10622339a3d0994e0bc54dfd1dcd4a64123

cleanup!: remove beta proto libraries (#7680) Remove beta proto libraries. These are unused by any client library, and the GA versions are now available.

view details

Carlos O'Ryan

commit sha a984b4eaa0f340f2428b787bb8e6bb125036991c

cleanup: remove workaround for old protobuf+compiler (#7681) Old versions of protobuf (prior to 3.15, we require 3.15.8) generated code that could not be compiled with older compilers (GCC < 6, we require >= 6.x). In other words, this workaround is not necessary for two reasons, we can safely remove it I think.

view details

Bradley White

commit sha b33672f83d6561ad64bce32c0af99db392d78db2

doc(common): add a note about AsyncGrpcOperation and OptionsSpan (#7682)

view details

Carlos O'Ryan

commit sha 92cf451d6b107d4a78b0683ce3130e38b4d07efc

cleanup: warn about untested compiler versions (#7684) Add warnings when the library is configured with a compiler we do not test with, and fatal messages for versions where we know our dependencies won't work. I also made some cosmetic fixes to the top-level CMakeLists.txt file, just to make it more readable for us.

view details

Carlos O'Ryan

commit sha 1644b06ddae149c9698941381a55e2b15f97f3fd

cleanup(bigtable): remove workaround for old protobuf versions (#7685)

view details

Carlos O'Ryan

commit sha 6f77c3e87a1a8ffeb5502c0a6ac59fba67f43e63

cleanup: only stop build based on our requirements (#7687)

view details

Bradley White

commit sha b44055be357e5582092f5a8f3aa530cdaee08147

feat(generator): add options support to generated clients (#7683) The generated client constructor now takes a defaulted `Options` argument, which is merged with the default options for the service, and then stored as a member. Similarly, the generated client operations also take a defaulted `Options` argument, which is then merged with the client options from above, and installed as the prevailing options for the duration of the operation by instantiating an `OptionsSpan`. Any module within the scope of that operation, or its continuations, can then retrieve those prevailing options. So, for example, a metadata decorator could add a header based on a per-operation option, or a retry policy might modify its behavior based on another option.

view details

Daniel Collins

commit sha 547de97908d4a6b4f0c3fde8a0d61b72329140c8

chore: bump checkers typos version so it works on mac and fix typos

view details

push time in 3 days

create barnchdpcollins-google/google-cloud-cpp

branch : fix-mac-checkers

created branch time in 3 days

Pull request review commentgoogleapis/google-cloud-cpp

feat(pubsublite): stream factories for PSL bidi streams

+// Copyright 2021 Google LLC+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+//      https://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++#ifndef GOOGLE_CLOUD_CPP_GOOGLE_CLOUD_PUBSUBLITE_INTERNAL_STREAM_FACTORY_H+#define GOOGLE_CLOUD_CPP_GOOGLE_CLOUD_PUBSUBLITE_INTERNAL_STREAM_FACTORY_H++#include <unordered_map>++#include "google/cloud/version.h"+#include "google/cloud/internal/async_read_write_stream_impl.h"+

done.

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent

push eventdpcollins-google/google-cloud-cpp

Daniel Collins

commit sha aa89a0271108296bff0eb145520b4cd1121cb8fd

fixes

view details

Daniel Collins

commit sha 3a7f798771d967f694cb01c88a32821415311aff

fixes

view details

push time in 3 days

Pull request review commentgoogleapis/google-cloud-cpp

fix(pubsub): Change AsyncReadWriteStreamAuth to be usable with unique_ptr

 namespace internal {  template <typename Request, typename Response> class AsyncStreamingReadWriteRpcAuth-    : public AsyncStreamingReadWriteRpc<Request, Response>,-      public std::enable_shared_from_this<-          AsyncStreamingReadWriteRpcAuth<Request, Response>> {+    : public AsyncStreamingReadWriteRpc<Request, Response> {  public:   using StreamFactory = std::function<-      std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(+      std::unique_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(           std::unique_ptr<grpc::ClientContext>)>;    AsyncStreamingReadWriteRpcAuth(       std::unique_ptr<grpc::ClientContext> context,       std::shared_ptr<GrpcAuthenticationStrategy> auth, StreamFactory factory)-      : context_(std::move(context)),-        auth_(std::move(auth)),-        factory_(std::move(factory)) {}+      : auth_(std::move(auth)),+        state_(std::make_shared<SharedState>(std::move(factory), std::move(context))) {}    void Cancel() override {-    if (context_) return context_->TryCancel();-    if (stream_) return stream_->Cancel();+    state_->Cancel();   }    future<bool> Start() override {     using Result = StatusOr<std::unique_ptr<grpc::ClientContext>>; -    auto weak =-        std::weak_ptr<AsyncStreamingReadWriteRpcAuth>(this->shared_from_this());-    return auth_->AsyncConfigureContext(std::move(context_))+    std::unique_ptr<grpc::ClientContext> context;+    {+      std::lock_guard<std::mutex> g{state_->mu};+      context = std::move(state_->initial_context);+    }+    auto weak = std::weak_ptr<SharedState>(state_);+    return auth_->AsyncConfigureContext(std::move(context))         .then([weak](future<Result> f) mutable {-          if (auto self = weak.lock()) return self->OnStart(f.get());+          if (auto state = weak.lock()) return state->OnStart(f.get());           return make_ready_future(false);         });   }    future<absl::optional<Response>> Read() override {-    if (!stream_) return make_ready_future(absl::optional<Response>{});-    return stream_->Read();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Read();   }    future<bool> Write(Request const& request,                      grpc::WriteOptions options) override {-    if (!stream_) return make_ready_future(false);-    return stream_->Write(request, std::move(options));+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Write(request, std::move(options));   }    future<bool> WritesDone() override {-    if (!stream_) return make_ready_future(false);-    return stream_->WritesDone();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->WritesDone();   }    future<Status> Finish() override {-    if (!stream_) {-      return make_ready_future(-          Status(StatusCode::kInvalidArgument,-                 "uninitialized GrpcReadWriteStreamAuth<>"));-    }-    return stream_->Finish();+    return state_->Finish();   }   private:-  future<bool> OnStart(StatusOr<std::unique_ptr<grpc::ClientContext>> context) {-    if (!context) {-      stream_ =-          absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(+  struct SharedState {+    SharedState(StreamFactory factory, std::unique_ptr<grpc::ClientContext> initial_context) : factory(std::move(factory)), initial_context(std::move(initial_context)), stream(absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(Status(StatusCode::kFailedPrecondition, "Stream is not yet started."))) {}

Changed to kInternal, I don't have a strong preference.

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-cpp

fix(pubsub): Change AsyncReadWriteStreamAuth to be usable with unique_ptr

 namespace internal {  template <typename Request, typename Response> class AsyncStreamingReadWriteRpcAuth-    : public AsyncStreamingReadWriteRpc<Request, Response>,-      public std::enable_shared_from_this<-          AsyncStreamingReadWriteRpcAuth<Request, Response>> {+    : public AsyncStreamingReadWriteRpc<Request, Response> {  public:   using StreamFactory = std::function<-      std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(+      std::unique_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(           std::unique_ptr<grpc::ClientContext>)>;    AsyncStreamingReadWriteRpcAuth(       std::unique_ptr<grpc::ClientContext> context,       std::shared_ptr<GrpcAuthenticationStrategy> auth, StreamFactory factory)-      : context_(std::move(context)),-        auth_(std::move(auth)),-        factory_(std::move(factory)) {}+      : auth_(std::move(auth)),+        state_(std::make_shared<SharedState>(std::move(factory), std::move(context))) {}    void Cancel() override {-    if (context_) return context_->TryCancel();-    if (stream_) return stream_->Cancel();+    state_->Cancel();   }    future<bool> Start() override {     using Result = StatusOr<std::unique_ptr<grpc::ClientContext>>; -    auto weak =-        std::weak_ptr<AsyncStreamingReadWriteRpcAuth>(this->shared_from_this());-    return auth_->AsyncConfigureContext(std::move(context_))+    std::unique_ptr<grpc::ClientContext> context;+    {+      std::lock_guard<std::mutex> g{state_->mu};

Personal preference to avoid unlocking methods because scopes more clearly deliniate the locked region. But a release method is cleaner

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-cpp

fix(pubsub): Change AsyncReadWriteStreamAuth to be usable with unique_ptr

 #include "google/cloud/version.h" #include <functional> #include <memory>+#include <thread>

Because I don't know the threading headers by heart yet :) done.

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-cpp

fix(pubsub): Change AsyncReadWriteStreamAuth to be usable with unique_ptr

 namespace internal {  template <typename Request, typename Response> class AsyncStreamingReadWriteRpcAuth-    : public AsyncStreamingReadWriteRpc<Request, Response>,-      public std::enable_shared_from_this<-          AsyncStreamingReadWriteRpcAuth<Request, Response>> {+    : public AsyncStreamingReadWriteRpc<Request, Response> {  public:   using StreamFactory = std::function<-      std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(+      std::unique_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(           std::unique_ptr<grpc::ClientContext>)>;    AsyncStreamingReadWriteRpcAuth(       std::unique_ptr<grpc::ClientContext> context,       std::shared_ptr<GrpcAuthenticationStrategy> auth, StreamFactory factory)-      : context_(std::move(context)),-        auth_(std::move(auth)),-        factory_(std::move(factory)) {}+      : auth_(std::move(auth)),+        state_(std::make_shared<SharedState>(std::move(factory), std::move(context))) {}    void Cancel() override {-    if (context_) return context_->TryCancel();-    if (stream_) return stream_->Cancel();+    state_->Cancel();   }    future<bool> Start() override {     using Result = StatusOr<std::unique_ptr<grpc::ClientContext>>; -    auto weak =-        std::weak_ptr<AsyncStreamingReadWriteRpcAuth>(this->shared_from_this());-    return auth_->AsyncConfigureContext(std::move(context_))+    std::unique_ptr<grpc::ClientContext> context;+    {+      std::lock_guard<std::mutex> g{state_->mu};+      context = std::move(state_->initial_context);+    }+    auto weak = std::weak_ptr<SharedState>(state_);+    return auth_->AsyncConfigureContext(std::move(context))         .then([weak](future<Result> f) mutable {-          if (auto self = weak.lock()) return self->OnStart(f.get());+          if (auto state = weak.lock()) return state->OnStart(f.get());           return make_ready_future(false);         });   }    future<absl::optional<Response>> Read() override {-    if (!stream_) return make_ready_future(absl::optional<Response>{});-    return stream_->Read();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Read();   }    future<bool> Write(Request const& request,                      grpc::WriteOptions options) override {-    if (!stream_) return make_ready_future(false);-    return stream_->Write(request, std::move(options));+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Write(request, std::move(options));   }    future<bool> WritesDone() override {-    if (!stream_) return make_ready_future(false);-    return stream_->WritesDone();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->WritesDone();   }    future<Status> Finish() override {-    if (!stream_) {-      return make_ready_future(-          Status(StatusCode::kInvalidArgument,-                 "uninitialized GrpcReadWriteStreamAuth<>"));-    }-    return stream_->Finish();+    return state_->Finish();   }   private:-  future<bool> OnStart(StatusOr<std::unique_ptr<grpc::ClientContext>> context) {-    if (!context) {-      stream_ =-          absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(+  struct SharedState {+    SharedState(StreamFactory factory, std::unique_ptr<grpc::ClientContext> initial_context) : factory(std::move(factory)), initial_context(std::move(initial_context)), stream(absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(Status(StatusCode::kFailedPrecondition, "Stream is not yet started."))) {}++    future<bool> OnStart(StatusOr<std::unique_ptr<grpc::ClientContext>> context) {+      std::lock_guard<std::mutex> g{mu};+      if (cancelled) return make_ready_future(false);+      if (context) {+        stream = factory(*std::move(context));+      } else {+        stream = absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(               std::move(context).status());-      return make_ready_future(false);+      }+      return stream->Start();     }-    stream_ = factory_(*std::move(context));-    return stream_->Start();-  } -  std::unique_ptr<grpc::ClientContext> context_;-  std::shared_ptr<GrpcAuthenticationStrategy> auth_;-  StreamFactory factory_;-  std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>> stream_;+    future<Status> Finish() {+      std::lock_guard<std::mutex> g{mu};+      cancelled = true;  // ensure stream is not recreated after Finish+      return stream->Finish();+    }++    void Cancel() {+      std::lock_guard<std::mutex> g{mu};+      if (cancelled) return;+      cancelled = true;+      if (initial_context) initial_context->TryCancel();+      stream->Cancel();+    }++    const StreamFactory factory;+    std::mutex mu;+    std::unique_ptr<grpc::ClientContext> initial_context;  // ABSL_GUARDED_BY(mu)+    std::unique_ptr<AsyncStreamingReadWriteRpc<Request, Response>> stream;  // ABSL_GUARDED_BY(mu)+    bool cancelled = false;  // ABSL_GUARDED_BY(mu)+  };++  const std::shared_ptr<GrpcAuthenticationStrategy> auth_;

Done.

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-cpp

fix(pubsub): Change AsyncReadWriteStreamAuth to be usable with unique_ptr

 namespace internal {  template <typename Request, typename Response> class AsyncStreamingReadWriteRpcAuth-    : public AsyncStreamingReadWriteRpc<Request, Response>,-      public std::enable_shared_from_this<-          AsyncStreamingReadWriteRpcAuth<Request, Response>> {+    : public AsyncStreamingReadWriteRpc<Request, Response> {  public:   using StreamFactory = std::function<-      std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(+      std::unique_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(           std::unique_ptr<grpc::ClientContext>)>;    AsyncStreamingReadWriteRpcAuth(       std::unique_ptr<grpc::ClientContext> context,       std::shared_ptr<GrpcAuthenticationStrategy> auth, StreamFactory factory)-      : context_(std::move(context)),-        auth_(std::move(auth)),-        factory_(std::move(factory)) {}+      : auth_(std::move(auth)),+        state_(std::make_shared<SharedState>(std::move(factory), std::move(context))) {}    void Cancel() override {-    if (context_) return context_->TryCancel();-    if (stream_) return stream_->Cancel();+    state_->Cancel();   }    future<bool> Start() override {     using Result = StatusOr<std::unique_ptr<grpc::ClientContext>>; -    auto weak =-        std::weak_ptr<AsyncStreamingReadWriteRpcAuth>(this->shared_from_this());-    return auth_->AsyncConfigureContext(std::move(context_))+    std::unique_ptr<grpc::ClientContext> context;+    {+      std::lock_guard<std::mutex> g{state_->mu};+      context = std::move(state_->initial_context);+    }+    auto weak = std::weak_ptr<SharedState>(state_);+    return auth_->AsyncConfigureContext(std::move(context))         .then([weak](future<Result> f) mutable {-          if (auto self = weak.lock()) return self->OnStart(f.get());+          if (auto state = weak.lock()) return state->OnStart(f.get());           return make_ready_future(false);         });   }    future<absl::optional<Response>> Read() override {-    if (!stream_) return make_ready_future(absl::optional<Response>{});-    return stream_->Read();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Read();   }    future<bool> Write(Request const& request,                      grpc::WriteOptions options) override {-    if (!stream_) return make_ready_future(false);-    return stream_->Write(request, std::move(options));+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Write(request, std::move(options));   }    future<bool> WritesDone() override {-    if (!stream_) return make_ready_future(false);-    return stream_->WritesDone();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->WritesDone();   }    future<Status> Finish() override {-    if (!stream_) {-      return make_ready_future(-          Status(StatusCode::kInvalidArgument,-                 "uninitialized GrpcReadWriteStreamAuth<>"));-    }-    return stream_->Finish();+    return state_->Finish();   }   private:-  future<bool> OnStart(StatusOr<std::unique_ptr<grpc::ClientContext>> context) {-    if (!context) {-      stream_ =-          absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(+  struct SharedState {+    SharedState(StreamFactory factory, std::unique_ptr<grpc::ClientContext> initial_context) : factory(std::move(factory)), initial_context(std::move(initial_context)), stream(absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(Status(StatusCode::kFailedPrecondition, "Stream is not yet started."))) {}++    future<bool> OnStart(StatusOr<std::unique_ptr<grpc::ClientContext>> context) {+      std::lock_guard<std::mutex> g{mu};+      if (cancelled) return make_ready_future(false);+      if (context) {+        stream = factory(*std::move(context));+      } else {+        stream = absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(               std::move(context).status());-      return make_ready_future(false);+      }+      return stream->Start();     }-    stream_ = factory_(*std::move(context));-    return stream_->Start();-  } -  std::unique_ptr<grpc::ClientContext> context_;-  std::shared_ptr<GrpcAuthenticationStrategy> auth_;-  StreamFactory factory_;-  std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>> stream_;+    future<Status> Finish() {+      std::lock_guard<std::mutex> g{mu};+      cancelled = true;  // ensure stream is not recreated after Finish+      return stream->Finish();+    }++    void Cancel() {+      std::lock_guard<std::mutex> g{mu};+      if (cancelled) return;+      cancelled = true;+      if (initial_context) initial_context->TryCancel();+      stream->Cancel();+    }++    const StreamFactory factory;

Done. It does, because it explains why it is not mutex guarded: its value never changes.

dpcollins-google

comment created time in 3 days

PullRequestReviewEvent

push eventdpcollins-google/google-cloud-cpp

Daniel Collins

commit sha 80edd6467b9381d6fa643630897a0731b927affa

fixes to stream auth

view details

push time in 3 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-cpp

fix(pubsub): Change AsyncReadWriteStreamAuth to be usable with unique_ptr

 namespace internal {  template <typename Request, typename Response> class AsyncStreamingReadWriteRpcAuth-    : public AsyncStreamingReadWriteRpc<Request, Response>,-      public std::enable_shared_from_this<-          AsyncStreamingReadWriteRpcAuth<Request, Response>> {+    : public AsyncStreamingReadWriteRpc<Request, Response> {  public:   using StreamFactory = std::function<-      std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(+      std::unique_ptr<AsyncStreamingReadWriteRpc<Request, Response>>(           std::unique_ptr<grpc::ClientContext>)>;    AsyncStreamingReadWriteRpcAuth(       std::unique_ptr<grpc::ClientContext> context,       std::shared_ptr<GrpcAuthenticationStrategy> auth, StreamFactory factory)-      : context_(std::move(context)),-        auth_(std::move(auth)),-        factory_(std::move(factory)) {}+      : auth_(std::move(auth)),+        state_(std::make_shared<SharedState>(std::move(factory), std::move(context))) {}    void Cancel() override {-    if (context_) return context_->TryCancel();-    if (stream_) return stream_->Cancel();+    state_->Cancel();   }    future<bool> Start() override {     using Result = StatusOr<std::unique_ptr<grpc::ClientContext>>; -    auto weak =-        std::weak_ptr<AsyncStreamingReadWriteRpcAuth>(this->shared_from_this());-    return auth_->AsyncConfigureContext(std::move(context_))+    std::unique_ptr<grpc::ClientContext> context;+    {+      std::lock_guard<std::mutex> g{state_->mu};+      context = std::move(state_->initial_context);+    }+    auto weak = std::weak_ptr<SharedState>(state_);+    return auth_->AsyncConfigureContext(std::move(context))         .then([weak](future<Result> f) mutable {-          if (auto self = weak.lock()) return self->OnStart(f.get());+          if (auto state = weak.lock()) return state->OnStart(f.get());           return make_ready_future(false);         });   }    future<absl::optional<Response>> Read() override {-    if (!stream_) return make_ready_future(absl::optional<Response>{});-    return stream_->Read();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Read();   }    future<bool> Write(Request const& request,                      grpc::WriteOptions options) override {-    if (!stream_) return make_ready_future(false);-    return stream_->Write(request, std::move(options));+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->Write(request, std::move(options));   }    future<bool> WritesDone() override {-    if (!stream_) return make_ready_future(false);-    return stream_->WritesDone();+    std::lock_guard<std::mutex> g{state_->mu};+    return state_->stream->WritesDone();   }    future<Status> Finish() override {-    if (!stream_) {-      return make_ready_future(-          Status(StatusCode::kInvalidArgument,-                 "uninitialized GrpcReadWriteStreamAuth<>"));-    }-    return stream_->Finish();+    return state_->Finish();   }   private:-  future<bool> OnStart(StatusOr<std::unique_ptr<grpc::ClientContext>> context) {-    if (!context) {-      stream_ =-          absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(+  struct SharedState {+    SharedState(StreamFactory factory, std::unique_ptr<grpc::ClientContext> initial_context) : factory(std::move(factory)), initial_context(std::move(initial_context)), stream(absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(Status(StatusCode::kFailedPrecondition, "Stream is not yet started."))) {}++    future<bool> OnStart(StatusOr<std::unique_ptr<grpc::ClientContext>> context) {+      std::lock_guard<std::mutex> g{mu};+      if (cancelled) return make_ready_future(false);+      if (context) {+        stream = factory(*std::move(context));+      } else {+        stream = absl::make_unique<AsyncStreamingReadWriteRpcError<Request, Response>>(               std::move(context).status());-      return make_ready_future(false);+      }+      return stream->Start();     }-    stream_ = factory_(*std::move(context));-    return stream_->Start();-  } -  std::unique_ptr<grpc::ClientContext> context_;-  std::shared_ptr<GrpcAuthenticationStrategy> auth_;-  StreamFactory factory_;-  std::shared_ptr<AsyncStreamingReadWriteRpc<Request, Response>> stream_;+    future<Status> Finish() {+      std::lock_guard<std::mutex> g{mu};+      cancelled = true;  // ensure stream is not recreated after Finish+      return stream->Finish();+    }++    void Cancel() {+      std::lock_guard<std::mutex> g{mu};+      if (cancelled) return;+      cancelled = true;+      if (initial_context) initial_context->TryCancel();+      stream->Cancel();+    }++    const StreamFactory factory;+    std::mutex mu;+    std::unique_ptr<grpc::ClientContext> initial_context;  // ABSL_GUARDED_BY(mu)

They could be unguarded. I'm hoping I can find a way to get ABSL_GUARDED_BY to do the thing its supposed to with lock guards, then eventually add the annotation proper back. But thats a task for another day.

dpcollins-google

comment created time in 3 days

push eventdpcollins-google/beam

Daniel Collins

commit sha 70eb2dccdfa1e3637298bd2b6ea6f858f48f2719

Increment pubsub python version and fix breakages. This is needed to unblock adding a dependency on google-cloud-pubsublite

view details

push time in 3 days

more