profile
viewpoint

Ask questionsFirestore: should library handle/retry UNAVAILABLE errors?

Hey all, we see a very rare error when writing data from a app engine application to firestore ('com.google.cloud:google-cloud-firestore:0.71.0-beta'). It seems to be a too old connection to firestore and therefore the connection is being reset: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR Received Goaway max_age Should we expect such errors when using the google-cloud-java lib or this something the library itself should handle?

Environment details

  • OS: Google app engine standard - java 8
  • Java version: Google app engine standard - java 8
  • google-cloud-java version(s): 0.71.0-beta

Steps to reproduce

  1. Have app engine instance running and write stuff to Firestore.
  2. Wait for error.

Stacktrace

org.eclipse.jetty.servlet.ServletHandler doHandle:  (ServletHandler.java:624)
javax.servlet.ServletException: Exception while trying to write to firestore.
	at com.processing.Process.doPost(Process.java:75)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
	at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:119)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)
	at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:171)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:296)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:202)
	at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81)
	at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:699)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:661)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:631)
	at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run(JavaRuntime.java:825)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:273)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway
max_age
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:423)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68)
	at com.FirestoreConnector.addOrUpdate(FirestoreConnector.java:94)
	at com.FirestoreConnector.addOrUpdate(FirestoreConnector.java:61)
	at com.processing.Process.doPost(Process.java:60)
	... 32 more
Caused by: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway
max_age
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:69)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1030)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:507)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:482)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:295)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway
max_age
	at io.grpc.Status.asRuntimeException(Status.java:526)
	... 23 more

Code snippet

private final Firestore db = FirestoreOptions
            .newBuilder()
            .setTimestampsInSnapshotsEnabled(true)
            .build()
            .getService();

public <T> String addOrUpdate(T input) throws Exception {
            return db.collection(collection)
                    .add(data)
                    .get(60, TimeUnit.SECONDS)
                    .getId();
}

External references such as API reference guides used

  • ?

Any additional information below

Thanks!

googleapis/google-cloud-java

Answer questions ajaaym

@eqinox76 as per comment above client library can not safely retry commit operation. Hence please add the retry logic in application code if its safe to retry. Closing this for now, please feel free to reopen if this doesn't address your need.

useful!

Related questions

Dialogflow V2, Springboot Dialogflow Library calls are failing ::: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative_x86_64, netty_tcnative] hot 2
PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first hot 2
Pubsub Subscriber initialization crashes JVM on Alpine Linux with libc6-compat hot 1
Google-cloud-language: No such Method Error on gax grpc hot 1
Duplicate classes error when adding dialogflow dependency 0.98.0-alpha along with firebase hot 1
The newest version of google-cloud-speech jar bug report hot 1
Firestore: Quickstart throws IllegalStateException: Could not find policy 'pick_first' hot 1
PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first hot 1
java.lang.NoSuchMethodError on Google PubSub Publisher.newBuilder().build() hot 1
Error reporting from GCE fails due to auth issues hot 1
Dependency conflict with KMS when running Spark code in Dataproc hot 1
Memory leak from netty/Deadline exceeded after 9999961477ns hot 1
BigQuery: Create new Table using schema from json file hot 1
Unable to make GRPC call in 32bit Windows Machine hot 1
PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first hot 1
Github User Rank List