profile
viewpoint
Vadym Matsishevskyi vam-google Google Seattle :ukraine:

googleapis/googleapis 4325

Public interface definitions of Google APIs.

googleapis/google-cloud-java 1609

Google Cloud Client Library for Java

googleapis/repo-automation-bots 276

A collection of bots, based on probot, for performing common maintenance tasks across the open-source repos managed by Google on GitHub.

googleapis/gapic-generator-python 73

Generate Python API client libraries from Protocol Buffers.

googleapis/gapic-generator-java 12

Generates GAPIC Java client libraries from protobufs.

push eventvam-google/gapic-generator-java

Vadym Matsishevskyi

commit sha 3b2dec6adf46823eeb6859e93362e63f00d84ffc

feat: add server streaming support for REST transport (#902) This PR depends on https://github.com/googleapis/gax-java/pull/1599

view details

push time in 10 hours

push eventvam-google/gax-java

Vadym Matsishevskyi

commit sha 3c97529b8bd0e8141c5d722f887cb7ae1ed30b69

feat: introduce HttpJsonClientCall, Listeners infrastructure and ServerStreaming support in REST transport (#1599) This includes the following changes for `HTTP1.1/REST` transport: 1) `HttpJsonClientCall` class (with `HttpJsonClientCall.Listener`) mimicking [io.grpc.ClientCall](https://github.com/grpc/grpc-java/blob/master/api/src/main/java/io/grpc/ClientCall.java#L102) functionality. Most of the complexity of this PR is concentrated in `HttpJsonClientCallImpl` class. 2) The unary callables are rewritten to be based on `HttpJsonClientCall` flow (similarly to how it is already done in gRPC unary calls). 3) Server streaming support for REST transport. The implementation is based on `HttpJsonClientCall` and `HttpJsonClientCall.Listener` (introduced in this PR), similarly to how gRPC streaming is based on `io.grpc.ClientCall` and `io.grpc.ClientCall.Listener` (implemented in [grpc-java](https://github.com/grpc/grpc-java/) library) respectively. The extreme similarity between `HttpJsonClientCall` call and `io.grpc.ClientCall` is intentional and crucial for consistency of the two transports and also intends simplifying creation and maintenance of multi-transport manual wrappers (like [google-ads-java](https://github.com/googleads/google-ads-java)). The server streaming abstractions in gax java are all based on the flow control managed by a ClientCall, so having similar set of abstractions in REST transport is necessary to reuse transport-independent portions of streaming logic in gax and maintain identical user-facing streaming surface. This PR also builds a foundation for the soon-coming [ClientInterceptor](https://github.com/grpc/grpc-java/blob/master/api/src/main/java/io/grpc/ClientInterceptor.java#L42)-like infrastructure in REST transport. This is specifically required to support REST transport in [google-ads-java](https://github.com/googleads/google-ads-java/blob/main/google-ads/src/main/java/com/google/ads/googleads/lib/logging/LoggingInterceptor.java#L42). REST-based client-side streaming and bidirectional streaming is not implemented by this PR and most likely will never be due to limitations of the `HTTP1.1/REST` protocol compared to `HTTP2/gRPC`. Most of the java docs in `HttpJsonClientCall` class is a modified version of the java docs from `io.grpc.ClientCall`, which is intentional, because `HttpJsonClientCall` is designed to be as similar to `io.grpc.ClientCall` in both surface and behavior as possible (while the two classes cannot be a part of the same class hierarchy, because they belong to two independent transport layers). **What server-streaming means in case of REST transport** In REST transport server-streaming methods return a JSON array of response messages (i.e. the array element type is the same one used as a returned type in the corresponding method definition in protobuf). The response is provided as as [Chunck-encoded](https://en.wikipedia.org/wiki/Chunked_transfer_encoding) input stream, containing one big JSON array. To parse the json array we rely on [JsonReader](https://github.com/google/gson/blob/master/gson/src/main/java/com/google/gson/stream/JsonReader.java#L191) from gson library, which gax-httpjson already depended on even prior this PR (check `ProtoMessageJsonStreamIterator` class implementation in this PR for details). Note, we must process elements of the array one-by-one because the size of the full array may be in realm of gigabytes. _**Note**, ideally I need to split this PR at least in two separate ones: 1) HttpJsonClientCall stuff and unary calls based on it in one PR and then 2) server streaming feature in a second PR. Unfortunately the most reasonable way to test `HttpJsonClientCall` infrastructure is by doing it from server streaming logic beause most of the complexity introduced in HttpJsonClient call is induced by necessity to support streaming workflow in the first place (and to support call interceptors (not part of this PR) as a secondary goal)._ _**Note**, there are a few minor breaking changes in gax-httpjson module (and only there) inroduced in this PR. This should be ok, because unlike gax and gax-grpc, gax-httpjson is not GA yet. The breaking changes are very minor (in the space of `HttpJsonCallContext` and `ManagedHttpJsonChannel`) and are backward-compatible with `java-compute` (the main and only officially supported user of gax-httpjson as of now)._

view details

Chanseok Oh

commit sha de285de1d6b280a07939e0431fcc7f648eb82bfa

chore: refactor code (#1606)

view details

release-please[bot]

commit sha 8b33f10dbfafc187130d87dc7fe453621c8cf120

chore(main): release 2.10.0 (#1605) :robot: I have created a release *beep* *boop* --- ## [2.10.0](https://github.com/googleapis/gax-java/compare/v2.9.0...v2.10.0) (2022-01-21) ### Features * add api key support ([#1436](https://github.com/googleapis/gax-java/issues/1436)) ([5081ec6](https://github.com/googleapis/gax-java/commit/5081ec6541da8ca3f5a4c0d20aa75bd20010a642)) * introduce HttpJsonClientCall, Listeners infrastructure and ServerStreaming support in REST transport ([#1599](https://github.com/googleapis/gax-java/issues/1599)) ([3c97529](https://github.com/googleapis/gax-java/commit/3c97529b8bd0e8141c5d722f887cb7ae1ed30b69)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

view details

push time in 10 hours

push eventgoogleapis/gapic-generator-java

Vadym Matsishevskyi

commit sha 3b2dec6adf46823eeb6859e93362e63f00d84ffc

feat: add server streaming support for REST transport (#902) This PR depends on https://github.com/googleapis/gax-java/pull/1599

view details

push time in 11 hours

PR merged googleapis/gapic-generator-java

feat: add server streaming support for REST transport

This PR depends on https://github.com/googleapis/gax-java/pull/1599

+111 -23

2 comments

11 changed files

vam-google

pr closed time in 11 hours

pull request commentgoogleapis/gapic-generator-java

feat: add server streaming support for REST transport

@chanseokoh pushing in now. If you have any comments, please leave them here and I'll address them in a followup PR.

vam-google

comment created time in 11 hours

PullRequestReviewEvent

push eventvam-google/gapic-generator-java

Chanseok Oh

commit sha 128039b37f3bc1f3db62df768b8fead010815d60

chore: make integration test infrastructure sane (#904) The you-can-do-anything-in-bazel mentality can easily lure one to a state where you are programming a build with a build tool instead of configuring a build. I just extracted the shell script portion out of a `.bzl` file into a file. This should also help switching to any build system from Bazel.

view details

vam-google

commit sha 51460c699b5c8935f983d6b02df1d616c0375479

Merge remote-tracking branch 'upstream/main'

view details

vam-google

commit sha b096b17aeb8c6035f119ef61ec899fb8577a206a

depend onlatest gax

view details

push time in 11 hours

PullRequestReviewEvent

pull request commentgoogleapis/gapic-generator-java

chore: make integration test infrastructure sane

@chanseokoh Please consider reverting this PR as a whole or heavily refactoring it to eliminate the issues.

The issues are:

  1. Before the generated code had also been compiled by bazel before going to golden files diff check, now compilation is not happening anymore. This is a big difference. This is so, because the integration_test rules were consuming gapic target directly (which produces a compiled java binary), and was getting the sources from it, which were guaranteed to be already compiled and formatted properly. Now the build depends on srcjar_raw intermediate artifact, which gets created before compilation, and since the subsequent compilation is not needed for the final test target to execute, bazel will truncate in from build execution tree as redundant.

  2. The build file now directly references the implementation-specific aspects of the rules (the names of intermediate build artifacts, which srcjar_raw is): https://github.com/googleapis/gapic-generator-java/pull/904/files#diff-d0ca181e8f4c33142243bd935d28e96c7fb6d1c3f99dbedb386f209c0f3f52a5R53. Like an oversimplified analogy to this would be accessing private variable form another class directly instead of having a getter method. Basically ere bazel gets used as a script-calling tool (a severely complicated one), instead of using it as a rigorous powerful build system.

chanseokoh

comment created time in 12 hours

push eventvam-google/gapic-generator-java

Vadym Matsishevskyi

commit sha bfb35cd31c05b5fbd2ea8073bcdcfdd3496bca12

feat: make generated test values comply with url path template (#903) This includes nested messages creation when there are url paths with subfields mentioned (like `/v1/{field.name=projects/*/databases/*/collectionGroups/*/fields/*}`) This is needed for rest transports tests because, unlike grpc, request fields must match path templates for rest logic pass the tests. Main changes are in `DefaultValueComposer` and `HttpRuleParser` classes. The generated pattern-matching samples are in the following format: given the pattern pattern: `/v1/{field.name=projects/*/databases/*/collectionGroups/*/fields/*}` the value will be: `field.name=projects/project-1234/databases/database-1234/collectionGroups/collectionGroup-1234/fields/field-1234`

view details

vam-google

commit sha 024ebb931a20ed39578c89b7e70443b4f8b09685

Merge remote-tracking branch 'upstream/main'

view details

push time in 12 hours

push eventgoogleapis/gax-java

Chanseok Oh

commit sha de285de1d6b280a07939e0431fcc7f648eb82bfa

chore: refactor code (#1606)

view details

Vadym Matsishevskyi

commit sha 869b91a7d90093a2ebeb345c74814d7b9e59ca22

Merge branch 'main' into release-please--branches--main

view details

push time in 14 hours

PullRequestReviewEvent

issue openedgoogleapis/gapic-generator-python

Showcase CI tests are always skipped

An example of a such showcase skipped run: https://github.com/googleapis/gapic-generator-python/runs/4887995943?check_suite_focus=true.

Run nox -s showcase
nox > Running session showcase
nox > Session showcase skipped: Python interpreter 3.9 not found.

As a result showcase tests are always reported as green even though they never actually ran.

created time in 15 hours

PullRequestReviewEvent

push eventgoogleapis/gapic-generator-java

Vadym Matsishevskyi

commit sha bfb35cd31c05b5fbd2ea8073bcdcfdd3496bca12

feat: make generated test values comply with url path template (#903) This includes nested messages creation when there are url paths with subfields mentioned (like `/v1/{field.name=projects/*/databases/*/collectionGroups/*/fields/*}`) This is needed for rest transports tests because, unlike grpc, request fields must match path templates for rest logic pass the tests. Main changes are in `DefaultValueComposer` and `HttpRuleParser` classes. The generated pattern-matching samples are in the following format: given the pattern pattern: `/v1/{field.name=projects/*/databases/*/collectionGroups/*/fields/*}` the value will be: `field.name=projects/project-1234/databases/database-1234/collectionGroups/collectionGroup-1234/fields/field-1234`

view details

push time in 15 hours

PR merged googleapis/gapic-generator-java

feat: make generated test values comply with url path template

This includes nested messages creation when there are url paths with subfields mentioned (like /v1/{field.name=projects/*/databases/*/collectionGroups/*/fields/*})

This is needed for rest transports tests because, unlike grpc, request fields must match path templates for rest logic pass the tests.

Main changes are in DefaultValueComposer and HttpRuleParser classes.

The generated pattern-matching samples are in the following format: given the pattern pattern: /v1/{field.name=projects/*/databases/*/collectionGroups/*/fields/*} the value will be: field.name=projects/project-1234/databases/database-1234/collectionGroups/collectionGroup-1234/fields/field-1234

+403 -153

3 comments

17 changed files

vam-google

pr closed time in 15 hours

pull request commentgoogleapis/gapic-generator-java

feat: make generated test values comply with url path template

@chanseokoh pushed, PTAL

vam-google

comment created time in 15 hours

push eventvam-google/gapic-generator-java

vam-google

commit sha 3f692cb850f73ab85d24aff6c348837cb1bcc1ec

address PR feedback

view details

push time in 15 hours

Pull request review commentgoogleapis/gapic-generator-java

feat: make generated test values comply with url path template

 public class ComplianceClientTest {     RepeatRequest request =         RepeatRequest.newBuilder()             .setName("name3373707")-            .setInfo(ComplianceData.newBuilder().build())

This is the gist of this change: the fields of the nested object (ComplianceData) is mentioned in url path template (https://github.com/googleapis/gapic-generator-java/blob/main/src/test/java/com/google/api/generator/gapic/testdata/compliance.proto#L66), thus it has to be properly constructed in the tests for rest logic to pass (it needs to construct url to make a call, and to construct it we need the values from input message to do the substitution).

vam-google

comment created time in 16 hours

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogleapis/gapic-generator-java

feat: make generated test values comply with url path template

 private static void checkHttpFieldIsValid(String binding, Message inputMessage,         fieldCondition,         String.format(messageFormat, field.name(), inputMessage.name(), field.type()));   }++  private static Map<String, String> constructPathValuePatterns(String pattern) {+    Map<String, String> varPattern = new HashMap<>();

Order does not matter (see other response for details)

vam-google

comment created time in 16 hours

Pull request review commentgoogleapis/gapic-generator-java

feat: make generated test values comply with url path template

 static AnonymousClassExpr createAnonymousResourceNameClass(String fieldOrMessage         .setMethods(Arrays.asList(getFieldValuesMapMethod, getFieldValueMethod))         .build();   }++  private static String constructValueMatchingPattern(String fieldName, String pattern) {+    if (pattern == null || pattern.isEmpty()) {+      return fieldName + fieldName.hashCode();+    }++    final String suffix = "-" + (Math.abs((fieldName + pattern).hashCode()) % 10000);++    String value = pattern;+    value = value.replace("**", "*");

changed

vam-google

comment created time in 16 hours

PullRequestReviewEvent

Pull request review commentgoogleapis/gapic-generator-java

feat: make generated test values comply with url path template

 public String patternLowerCamel() {     return lowerCamelPattern;   } +  public Map<String, String> getPathParametersValuePatterns() {+    Map<String, String> valuePatterns = new HashMap<>();

So the order here does not matter, as this map is to map field to their patterns and it is never used (and can't reasonably be sued) to determine order of things. This map is being iterated once in DefaultValue composer to create a nested fields map (i.e. filter out all fields, which are outside of the nested field scope) but that is once again - to get another non-ordered map, which eventually will be used for only one thing - get the field pattern by field name, and order does not matter in that case.

vam-google

comment created time in 16 hours

PullRequestReviewEvent

Pull request review commentgoogleapis/gapic-generator-java

feat: make generated test values comply with url path template

 static AnonymousClassExpr createAnonymousResourceNameClass(String fieldOrMessage         .setMethods(Arrays.asList(getFieldValuesMapMethod, getFieldValueMethod))         .build();   }++  private static String constructValueMatchingPattern(String fieldName, String pattern) {+    if (pattern == null || pattern.isEmpty()) {+      return fieldName + fieldName.hashCode();+    }++    final String suffix = "-" + (Math.abs((fieldName + pattern).hashCode()) % 10000);++    String value = pattern;+    value = value.replace("**", "*");++    String prevTempl = null;+    while (!value.equals(prevTempl)) {+      prevTempl = value;+      value = REPLACER_PATTERN.matcher(value).replaceFirst("$1$2$1" + suffix);+    }++    value = value.replace("*", fieldName + suffix);+    return value;

Thanks, I'm surprised that Intellij does not highlight stuff like this =)

vam-google

comment created time in 16 hours

PullRequestReviewEvent

Pull request review commentgoogleapis/gapic-generator-python

feat: add interceptor-like functionality to REST transport

 from google.api_core import grpc_helpers_async from google.api_core import path_template {% if service.has_lro %} from google.api_core import future+from google.api_core import operation

Probably a stupid question, but whasn't this required before? I.e. it is logical to assume that {% if service.has_lro %} that operation should be imported.

software-dov

comment created time in 17 hours

Pull request review commentgoogleapis/gapic-generator-python

feat: add interceptor-like functionality to REST transport

 def test_{{ method_name }}_rest_unset_required_fields():     {% endif %}{# required_fields #}  +{% if not (method.server_streaming or method.client_streaming) %}

just in case, do bidirectional streaming methods have their own flag or are signified by both server_streaming and client_streaming being true?

software-dov

comment created time in 17 hours

more