profile
viewpoint

Ask questionsError reporting from GCE fails due to auth issues

I'm investigating why all our Java error reporting in GKE fails at the moment, whereas the Go based code bases seem to report their errors happily. My impression is that the default authentication on GCE is not working for this library.

I wrote this test class:

import com.google.cloud.errorreporting.v1beta1.ReportErrorsServiceClient;
import com.google.devtools.clouderrorreporting.v1beta1.ProjectName;
import com.google.devtools.clouderrorreporting.v1beta1.ReportErrorEventResponse;
import com.google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent;

import java.io.IOException;

public class ErrorReportTest {
    public static void main(String[] args) throws IOException {
        try (ReportErrorsServiceClient reportErrorsServiceClient = ReportErrorsServiceClient.create()) {
            ProjectName projectName = ProjectName.of("travello-api");
            ReportedErrorEvent event = ReportedErrorEvent.newBuilder()
                    .setMessage(
                            "Exception like thing that Peter is testing error reporting with.\n" +
                            "        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49)\n" +
                            "        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)\n" +
                            "        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)\n" +
                            "        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)\n" +
                            "        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n" +
                            "        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:982)\n" +
                            "        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n" +
                            "        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)\n" +
                            "        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:957)\n" +
                            "        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)\n" +
                            "        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515)\n" +
                            "        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490)\n")
                    .build();
            ReportErrorEventResponse response = reportErrorsServiceClient.reportErrorEvent(projectName, event);
            System.out.println(response.toString());
        }
    }
}

I wrapped this into a shaded JAR using this Maven POM:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.travelloapp</groupId>
    <artifactId>error-reporting-test</artifactId>
    <version>dev-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-errorreporting</artifactId>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>google-cloud-bom</artifactId>
                <version>0.103.0-alpha</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>ErrorReportTest</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

If I run the result on my local machine (where I'm logged into GCP), it submits the error just fine.

If I SCP it onto a GCE VM and run it there I get this:

peter@gce-host:~$ java -jar error-reporting-test-dev-SNAPSHOT.jar 
Exception in thread "main" com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Request had insufficient authentication scopes.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:982)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:957)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:500)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:65)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:592)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:508)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:632)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
                at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
                at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
                at com.google.cloud.errorreporting.v1beta1.ReportErrorsServiceClient.reportErrorEvent(ReportErrorsServiceClient.java:255)
                at com.google.cloud.errorreporting.v1beta1.ReportErrorsServiceClient.reportErrorEvent(ReportErrorsServiceClient.java:188)
                at ErrorReportTest.main(ErrorReportTest.java:28)
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Request had insufficient authentication scopes.
        at io.grpc.Status.asRuntimeException(Status.java:533)
        ... 23 more

I noticed GCE_METADATA_HOST was not set, so I tried this, but the result after is the same:

export GCE_METADATA_HOST=169.254.169.254

This was inspired by https://godoc.org/cloud.google.com/go/compute/metadata#Client.Get

It seems to work if I deploy service keys and use GOOGLE_APPLICATION_CREDENTIALS, but based on the documentation I assumed that would not be needed, and Go seems to be OK without.

Am I doing something wrong? Wrong assumptions?

googleapis/google-cloud-java

Answer questions peterbecker

I tried using GOOGLE_APPLICATION_CREDENTIALS pointing to a JSON key as well:

root@livn-service-56c7954dc7-jrl6j:/# cat $GOOGLE_APPLICATION_CREDENTIALS 
{
  "type": "service_account",
  "project_id": "[...omitted...]",
  "private_key_id": "[...omitted...]",
  "private_key": "-----BEGIN PRIVATE KEY-----\n[...omitted...]\n-----END PRIVATE KEY-----\n",
  "client_email": "[...omitted...]",
  "client_id": "[...omitted...]",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/[...omitted...]"
}
root@livn-service-56c7954dc7-jrl6j:/# java -jar error-reporting-test-dev-SNAPSHOT.jar 
Exception in thread "main" com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
[,...abridged...]

This is now running on a GKE pod (which is the final goal).

useful!

Related questions

Dialogflow V2, Springboot Dialogflow Library calls are failing ::: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative_x86_64, netty_tcnative] hot 2
PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first hot 2
Pubsub Subscriber initialization crashes JVM on Alpine Linux with libc6-compat hot 1
Google-cloud-language: No such Method Error on gax grpc hot 1
Duplicate classes error when adding dialogflow dependency 0.98.0-alpha along with firebase hot 1
The newest version of google-cloud-speech jar bug report hot 1
Firestore: Quickstart throws IllegalStateException: Could not find policy 'pick_first' hot 1
Firestore: should library handle/retry UNAVAILABLE errors? hot 1
PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first hot 1
java.lang.NoSuchMethodError on Google PubSub Publisher.newBuilder().build() hot 1
Dependency conflict with KMS when running Spark code in Dataproc hot 1
Memory leak from netty/Deadline exceeded after 9999961477ns hot 1
BigQuery: Create new Table using schema from json file hot 1
Unable to make GRPC call in 32bit Windows Machine hot 1
PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first hot 1
Github User Rank List