profile
viewpoint
Joe Tsai dsnet @google Mountain View http://digital-static.net Free food critic.

google/go-cmp 2108

Package for comparing Go values in tests

google/safebrowsing 345

Safe Browsing API Go Client

dsnet/compress 319

Collection of compression related Go packages.

dsnet/udptunnel 99

Daemon for creating a simple VPN over UDP.

dsnet/motd-generator 41

Custom message-of-the-day (MOTD) generator intended to be informative about the system you are logging in to.

dsnet/sshtunnel 38

SSH daemon for creating forward and reverse tunnels.

dsnet/golib 16

Collection of mostly unrelated helper Go packages.

dsnet/playground 14

Locally hosted Go playground for more advanced functionality.

dsnet/termijack 14

TermiJack surreptitiously hijacks standard streams (stdin, stdout, and/or stderr) from an already running process.

dsnet/gotab 11

Simple bash tab completion for the go command.

PR opened gogo/protobuf

add config to let user choose whether to render number64 type such as int64, uint64 as integers or string

In some scenarios, we need to render number64 type such as int64, uint64 as integers instead of string values, but found we cannot. So I add an option in config to fix this.

+4 -1

0 comment

1 changed file

pr created time in 20 hours

startedyuppity/unifi-video-api

started time in 21 hours

issue closedgolang/protobuf

How to get a map's value's golang package and golang name when writing a plugin

test/api/test.proto

syntax="proto3";
package test;
option go_package="test/api;api";
import "test/first/first.proto";
service test{
	rpc hello(hreq)returns(hresp);
}
message hreq{
	map<int32,first.a.abc> test=1;
}
message hresp{
	string w=1;
}

test/first/first.proto

syntax="proto3";
package first;
option go_package="test/first;first";
message a{
	enum abc{
		A = 0;
		B = 1;
		C = 2;
	}
}

when generate the test/api/test.proto how to get the first.a.abc go pkg and go name the first.a.abccan change to a message type,how to get the go package and go name

closed time in a day

chenjie199234

issue openedgolang/protobuf

How to get a map's value's golang package and golang name when writing a plugin

test/api/test.proto

syntax="proto3";
package test;
option go_package="test/api;api";
import "test/first/first.proto";
service test{
	rpc hello(hreq)returns(hresp);
}
message hreq{
	map<int32,first.a.abc> test=1;
}
message hresp{
	string w=1;
}

test/first/first.proto

syntax="proto3";
package first;
option go_package="test/first;first";
message a{
	enum abc{
		A = 0;
		B = 1;
		C = 2;
	}
}

when generate the test/api/test.proto how to get the first.a.abc go pkg and go name

created time in a day

startednorvig/pytudes

started time in 2 days

issue commentgolang/protobuf

String and protojson produces non-deterministic whitespaces.

Thanks a lot, sorry for duplicate!

bwplotka

comment created time in 3 days

issue openedgolang/protobuf

String and protojson produces nondeterministic whitespaces.

What version of protobuf and what language are you using?

My go.mod:

	github.com/golang/protobuf v1.4.3
	github.com/opentracing/opentracing-go v1.1.0
	github.com/pkg/errors v0.9.1
	github.com/stretchr/testify v1.6.1
	golang.org/x/net v0.0.0-20190620200207-3b0461eec859
	golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be
	google.golang.org/grpc v1.30.1
	google.golang.org/grpc/examples v0.0.0-20200723182653-9106c3fff523
	google.golang.org/protobuf v1.25.0

protoc-gen-go: require google.golang.org/protobuf v1.25.0 // cmd/protoc-gen-go (We use bingo for tool pinning)

Go 1.15

What did you do?

Okay, it's extremely weird so stay with me. (:

The repro test is here https://github.com/grpc-ecosystem/go-grpc-middleware/blob/repro/grpctesting/testpb/test_repro_test.go you can find proto as well.

Now it's tricky to reproduce, because it depends on... code line you invoke String or protojson? Like some compiler optimization is changing things? I know it's super weird, but it looks like this:

Steps to repro (I am using Goland test invocation):

  1. Run test.
package testpb

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"google.golang.org/protobuf/encoding/protojson"
)

func TestNonDeterministicMarshal(t *testing.T) {
	p := &PingRequest{Value: "something", ErrorCodeReturned: 4}

	// Remove this line to see the test failing (:

	// Sometimes
	// * "value:\"something\" error_code_returned:4"
	// * "value:\"something\"  error_code_returned:4"
	assert.Equal(t, "value:\"something\" error_code_returned:4", p.String())
	b, err := protojson.Marshal(p)
	require.NoError(t, err)

	// Sometimes
	// * "{\"value\":\"something\",\"errorCodeReturned\":4}
	// * "{\"value\":\"something\", \"errorCodeReturned\":4}
	assert.Equal(t, "{\"value\":\"something\",\"errorCodeReturned\":4}", string(b))
}

This passes for me 100%.

/home/bwplotka/.gvm/gos/go1.15/bin/go tool test2json -t /tmp/___TestNonDeterministicMarshal_in_github_com_grpc_ecosystem_go_grpc_middleware_v2_grpctesting_testpb -test.v -test.run ^\QTestNonDeterministicMarshal\E$
=== RUN   TestNonDeterministicMarshal
--- PASS: TestNonDeterministicMarshal (0.00s)
PASS

Process finished with exit code 0
  1. Remove comment // Remove this line to see the test failing (: and newline below:
package testpb

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"google.golang.org/protobuf/encoding/protojson"
)

func TestNonDeterministicMarshal(t *testing.T) {
	p := &PingRequest{Value: "something", ErrorCodeReturned: 4}

	// Sometimes
	// * "value:\"something\" error_code_returned:4"
	// * "value:\"something\"  error_code_returned:4"
	assert.Equal(t, "value:\"something\" error_code_returned:4", p.String())
	b, err := protojson.Marshal(p)
	require.NoError(t, err)

	// Sometimes
	// * "{\"value\":\"something\",\"errorCodeReturned\":4}
	// * "{\"value\":\"something\", \"errorCodeReturned\":4}
	assert.Equal(t, "{\"value\":\"something\",\"errorCodeReturned\":4}", string(b))
}

No this fails for me with:

//home/bwplotka/.gvm/gos/go1.15/bin/go tool test2json -t /tmp/___TestNonDeterministicMarshal_in_github_com_grpc_ecosystem_go_grpc_middleware_v2_grpctesting_testpb -test.v -test.run ^\QTestNonDeterministicMarshal\E$
=== RUN   TestNonDeterministicMarshal
    test_repro_test.go:17: 
        	Error Trace:	test_repro_test.go:17
        	Error:      	Not equal: 
        	            	expected: "value:\"something\" error_code_returned:4"
        	            	actual  : "value:\"something\"  error_code_returned:4"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-value:"something" error_code_returned:4
        	            	+value:"something"  error_code_returned:4
        	Test:       	TestNonDeterministicMarshal
    test_repro_test.go:24: 
        	Error Trace:	test_repro_test.go:24
        	Error:      	Not equal: 
        	            	expected: "{\"value\":\"something\",\"errorCodeReturned\":4}"
        	            	actual  : "{\"value\":\"something\", \"errorCodeReturned\":4}"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-{"value":"something","errorCodeReturned":4}
        	            	+{"value":"something", "errorCodeReturned":4}
        	Test:       	TestNonDeterministicMarshal
--- FAIL: TestNonDeterministicMarshal (0.00s)
  1. Put any comment back in that spot and all works again.

NOTE: This works somehow underministically, so CI, running it with other tests vs single changes behavior (essentially on what line to put comment to affect output)

What did you expect to see?

No matter what and where I put comments in my code I expect the same String and protojson Output =D

What did you see instead?

Putting comment in right spot adds or removes whitespace between poto fields.

Anything else we should know about your project / environment?

Pretty standard, you can see all here: https://github.com/grpc-ecosystem/go-grpc-middleware/tree/repro

We will not touch repro branch so you can take a look.

I know it might be annoying little thing, but deterministic output would be supper nice, especially for reliable unit tests (that's how I noticed this)

created time in 3 days

issue commentgolang/protobuf

protoc-gen-go-grpc: program not found or is not executable when using APIv2

SOLVED: updated path to:

export GOPATH=$HOME/Documents/goworkspace
export PATH="$PATH:$GOPATH/bin"

thank you

ctfrancia

comment created time in 3 days

issue openedgogo/protobuf

EnumsAsInts does not work in jsonpb config

The code is so simple:

package main

import (
	"os"

	"github.com/gogo/protobuf/jsonpb"
	"github.com/gogo/protobuf/proto"
)

func main() {
	marshaller := jsonpb.Marshaler{
		EnumsAsInts:  true,
		EmitDefaults: false,
		OrigName:     true,
	}

	marshaller.Marshal(os.Stdout, &TestInt{
		Ori: 16,
	})
}

type TestInt struct {
	Ori uint64 `protobuf:"varint,1,opt,name=ori,proto3" json:"ori,omitempty"`
}

func (m *TestInt) Reset()         { *m = TestInt{} }
func (m *TestInt) String() string { return proto.CompactTextString(m) }
func (*TestInt) ProtoMessage()    {}

No matter I set EnumsAsInts: true or not, all that I got is {"ori":"16"}. But what I want is {"ori":16}

created time in 4 days

PR closed gogo/protobuf

Go 1.15: Convert int to string using rune()

See https://github.com/golang/go/issues/32479

Signed-off-by: Robert-André Mauchin zebob.m@gmail.com

+1 -1

0 comment

1 changed file

eclipseo

pr closed time in 4 days

issue commentgolang/protobuf

proto: repeated imported enum cause panic

If you can give us actual code with a reproduction we can look into it, but without a way to reproduce the problem I'm afraid there's not much we can do.

purefire

comment created time in 4 days

issue commentgolang/protobuf

proto: repeated imported enum cause panic

I’ve setup a gRPC service with the given proto files, and… it doesn’t crash. It just works, as expected.

purefire

comment created time in 4 days

startedwajahatkarim3/hugo-on-fire

started time in 4 days

issue commentgolang/protobuf

proto: repeated imported enum cause panic

https://github.com/mesg-foundation/engine/issues/1805 also reports the same bug after upgrading to v1.4+

purefire

comment created time in 4 days

issue commentgolang/protobuf

proto: repeated imported enum cause panic

@dsnet We also just ran into this, so I think the issue should be reopened. I'll try to summarise the problem below.


If you have two proto files:

// services.proto

import "common.proto";

message ListServicesRequest {
    repeated Status filter_statuses = 1;
}
// common.proto

enum Status {
    NOT_SET = 0;
    DISABLED = 1;
    ENABLED = 2;
}

whenever you call a RPC method that uses ListServicesRequest, the following error is returned:

2021/01/15 12:19:32 http: panic serving 10.125.39.64:32176: runtime error: index out of range [0] with length 0
goroutine 159 [running]:
net/http.(*conn).serve.func1(0xc00056e640)
        /usr/local/go/src/net/http/server.go:1801 +0x147
panic(0xfdeac0, 0xc0004b6a60)
        /usr/local/go/src/runtime/panic.go:975 +0x3e9
google.golang.org/protobuf/internal/filedesc.(*EnumValues).Get(0xc0000eaa80, 0x0, 0xc0000eaa80, 0xc0003e19a0)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/filedesc/desc_list_gen.go:66 +0x4c
google.golang.org/protobuf/internal/impl.newEnumConverter(0x11d5240, 0xef5060, 0x11d5120, 0xc0002be380, 0xc00000e758, 0xd68861a761eb97d8)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/convert.go:379 +0x8b
google.golang.org/protobuf/internal/impl.newSingularConverter(0x11d5240, 0xef5060, 0x11d5120, 0xc0002be380, 0xc0003e19e8, 0x8de86b)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/convert.go:138 +0xdfb
google.golang.org/protobuf/internal/impl.newListConverter(0x11d5240, 0xc00043a680, 0x11d5120, 0xc0002be380, 0xe6b320, 0x0)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/convert_list.go:17 +0x1f4
google.golang.org/protobuf/internal/impl.NewConverter(0x11d5240, 0xc00043a680, 0x11d5120, 0xc0002be380, 0x11ccc00, 0xc0004076b0)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/convert.go:56 +0x126
google.golang.org/protobuf/internal/impl.fieldInfoForList(0x11d5120, 0xc0002be380, 0xfa7953, 0xe, 0x0, 0x0, 0x11d5240, 0xe74d80, 0xfa7963, 0xa0, ...)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/message_reflect_field.go:172 +0xe5
google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc(0xc0005f4140, 0x58, 0xffffffffffffffff, 0x40, 0xffffffffffffffff, 0xc0004072f0, 0xc000407320, 0xc000407350, 0xc000407380)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/message_reflect.go:61 +0x69d
google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs(0xc0005f4140, 0x11d5240, 0x1004600, 0x58, 0xffffffffffffffff, 0x40, 0xffffffffffffffff, 0xc0004072f0, 0xc000407320, 0xc000407350, ...)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/message_reflect.go:36 +0x65
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc0005f4140)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/message.go:90 +0x192
google.golang.org/protobuf/internal/impl.(*MessageInfo).init(...)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/message.go:72
google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).ProtoMethods(0xc0004b4a10, 0xc0003e2158)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal/impl/message_reflect_gen.go:150 +0x53
google.golang.org/protobuf/proto.protoMethods(...)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/proto/proto_methods.go:18
google.golang.org/protobuf/proto.MarshalOptions.marshal(0xc000000001, 0x0, 0x0, 0x0, 0x11d2de0, 0xc0004b4a10, 0xc0004b4a10, 0x11d2de0, 0xc0004b4a10, 0x1027a20, ...)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/proto/encode.go:143 +0x49
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend(0x1000001, 0x0, 0x0, 0x0, 0x11afde0, 0xc0004b4a10, 0x203000, 0x7f0c1b3effff, 0xc00, 0x8, ...)
        /go/pkg/mod/google.golang.org/protobuf@v1.25.0/proto/encode.go:125 +0x98
github.com/golang/protobuf/proto.marshalAppend(0x0, 0x0, 0x0, 0x11c38e0, 0xc00011a5a0, 0xc00043a500, 0x40, 0x40, 0xfd1700, 0x418745, ...)
        /go/pkg/mod/github.com/golang/protobuf@v1.4.2/proto/wire.go:40 +0xc7
github.com/golang/protobuf/proto.Marshal(...)
        /go/pkg/mod/github.com/golang/protobuf@v1.4.2/proto/wire.go:23

This only occurs if the RPC method uses a request struct, which contains a repeated field that references an Enum in a different file. So if your proto file imports that enum, it will cause an error.

Downgrading to v1.3.5 fixed this, so I think it's a regression from v1.4 onwards.

purefire

comment created time in 4 days

issue closedgolang/protobuf

How to stop protoc-gen-go from downgrading to the previous version (1.23.0)

I can't work out why all of a sudden when I run protoc -I . <service>.proto it generates a pb.go file with an older version of protoc-gen-go (see image below)

Screenshot 2021-01-11 at 12 31 44

Environment

  • I have protobuf version 3.14.0 installed globally via homebrew (brew info protobuf or protoc --version)
  • in the project, I have google.golang.org/protobuf v1.25.0 and github.com/golang/protobuf v1.4.3 installed via go.mod – but this happens for me across all the services which suggests the problem is to do with the global module
  • My go version is: go1.15.6 darwin/amd64 (the latest)

I appreciate any suggestions. Thanks

closed time in 4 days

Neats29

issue commentgolang/protobuf

How to stop protoc-gen-go from downgrading to the previous version (1.23.0)

Thank you. I resolved my issue by using a different version of protobuf.

Neats29

comment created time in 4 days

fork xiaoqidun/compress

Collection of compression related Go packages.

fork in 5 days

issue commentgolang/protobuf

protoc-gen-go: packages named 'type' induce a malformed warnings

I later realized that, although this is annoying, it's not a blocking. I can still import the generated package via path and just ignore the package name.

anzboi

comment created time in 6 days

startedmathworks/thingspeak-arduino

started time in 6 days

issue commentgolang/protobuf

reflect/protoregistry: conflicts with same filename

Has the protobuf team made any further decisions on this issue?

Much like @rcgoodfellow, I have a Go program that depends on two third-party libraries, each of which defines a telemetry.proto file at the top level of their project. With current google.golang.org/protobuf, the program panics unless I set the GOLANG_PROTOBUF_REGISTRATION_CONFLICT environment variable.

This can't be that uncommon a situation. At this point, I'm not really clear how the protobuf team expect the file-by-path registry to work outside of a monorepo setting.

guyguy333

comment created time in 6 days

startedgookit/color

started time in 6 days

PublicEvent

created repositorytailscale/tailetc

created time in 6 days

startedjba/templatecheck

started time in 7 days

issue commentgogo/protobuf

Starting on apiv2 - custom types

I think the basic idea would be to generate code for 'custom types' that implements protoreflect.Message (actually protoreflect.ProtoMessage but this just provides a layer of indirection between the custom type and the protobuf reflective type): https://godoc.org/google.golang.org/protobuf/reflect/protoreflect#Message.

This would be analogous the the various marshalling methods requried by gogoproto.customtype: https://github.com/gogo/protobuf/blob/master/custom_types.md#custom-type-method-signatures.

Types that implement ProtoMessage should use the Message interface to reflect on the type. At that point it is possible to handle the mapping between the protobuf type system and an arbitrary Go type.

I have been exploring the API myself to see whether there are some types that are open for extension but with which I could share most of the implementation. Most of the Message implementing types are internal, here is an example: https://github.com/protocolbuffers/protobuf-go/blob/master/testing/protocmp/reflect.go#L39

turbodonkey

comment created time in 7 days

fork aclements/klog

Leveled execution logs for Go (fork of https://github.com/golang/glog)

fork in 8 days

issue commentgolang/protobuf

How to stop protoc-gen-go from downgrading to the previous version (1.23.0)

I have google.golang.org/protobuf v1.25.0 installed via go.mod …

The protoc-gen-go binary has to be v1.25.0, and it will not automatically use the value in the go.mod, you have to manage the binary yourself.

Neats29

comment created time in 8 days

issue openedgolang/protobuf

How to stop `protoc-gen-go` downgrading to the previous version (1.23.0)

I can't work out why all of a sudden when I run protoc -I . <service>.proto it generates a pb.go file with an older version of protoc-gen-go (see image below)

Screenshot 2021-01-11 at 12 31 44

Environment

  • I have protobuf version 3.14.0 installed globally via homebrew (brew info protobuf or protoc --version)
  • in the project, I have google.golang.org/protobuf v1.25.0 and github.com/golang/protobuf v1.4.3 installed via go.mod
  • My go version is: go1.15.6 darwin/amd64 (the latest)

I appreciate any suggestions. Thanks

created time in 8 days

more