profile
viewpoint
Joe Tsai dsnet @google Mountain View http://digital-static.net Free food critic.

google/go-cmp 1917

Package for comparing Go values in tests

google/safebrowsing 334

Safe Browsing API Go Client

dsnet/compress 311

Collection of compression related Go packages.

dsnet/udptunnel 94

Daemon for creating a simple VPN over UDP.

dsnet/motd-generator 38

Custom message-of-the-day (MOTD) generator intended to be informative about the system you are logging in to.

dsnet/sshtunnel 34

SSH daemon for creating forward and reverse tunnels.

dsnet/golib 14

Collection of mostly unrelated helper Go packages.

dsnet/termijack 14

TermiJack surreptitiously hijacks standard streams (stdin, stdout, and/or stderr) from an already running process.

dsnet/playground 13

Locally hosted Go playground for more advanced functionality.

dsnet/gotab 9

Simple bash tab completion for the go command.

issue commentgolang/protobuf

growslice issue

This identifies gogo/protobuf as the code being use. I think you are filing this bug in the wrong module.

linchuan4028

comment created time in 4 hours

issue commentgolang/protobuf

growslice issue

Without a reproduction, there is not much we can do with this report. Furthermore, it's unclear what "the growslie performance issue" is.

linchuan4028

comment created time in 17 hours

issue commentgolang/protobuf

How to get a field valued zero through reflection in proto3?

The protoreflect.Message.Range method is documented as:

Range iterates over every populated field in an undefined order

Under proto3 semantics, the zero value is not a populated field.

If you want to iterate over every message field regardless of whether it is populated, you can iterate over all the declared fields:

message := m.ProtoReflect()
fields := message.Descriptor().Fields() // all the declared fields in this message type
for i := 0; i < fields.Len(); i++ {
    v := message.Get(fields.Get(i))
    ... // do something with v
}
alpaca2333

comment created time in 2 days

issue commentgolang/protobuf

encoding/protojson: Allow for optional marshaling/unmarshaling of booleans as numeric 1/0

This behavior would have to be first proposed at github.com/protocolbuffers/protobuf for all languages with a JSON implementation to provide consistent behavior. Without uniform agreement that this is to be handled, we can't do anything here.

bignacio

comment created time in 2 days

push eventgolang/protobuf

Joe Tsai

commit sha 91c84e0db17890c2bb64280a6d660e73e4237fd1

travis.yml: update tested versions of Go (#1211) While this module is deprecated, it is still important to ensure that it continues to work with higher versions of Go. Update travis.yml to test up to Go1.15.

view details

push time in 4 days

delete branch golang/protobuf

delete branch : update-travis

delete time in 4 days

PR merged golang/protobuf

travis.yml: update tested versions of Go

While this module is deprecated, it is still important to ensure that it continues to work with higher versions of Go. Update travis.yml to test up to Go1.15.

+6 -0

0 comment

1 changed file

dsnet

pr closed time in 4 days

PR opened golang/protobuf

travis.yml: update tested versions of Go

While this module is deprecated, it is still important to ensure that it continues to work with higher versions of Go. Update travis.yml to test up to Go1.15.

+6 -0

0 comment

1 changed file

pr created time in 4 days

create barnchgolang/protobuf

branch : update-travis

created branch time in 4 days

issue commentgolang/go

encoding/json: use different error type for unknown field if they are disallowed

Given the existence of custom UnmarshalJSON functions that already don't guarantee any semantics regarding the errors they returns, I don't think we should expose semantically identically errors for unknown fields since it would only be a partial guarantee.

Furthermore, the original poster should adequately explain what they want to use this for. It's rare for any proposal to go through without some justification for why it's useful.

Segflow

comment created time in 4 days

push eventgolang/protobuf

Christian Persson

commit sha 3860b2764ff25e103fbe1db40f22248fe7a6dc20

proto: convert integer to rune before converting to string (#1210) Go 1.15 introduced a new `go vet` warning (https://golang.org/doc/go1.15#vet) for conversions of the form `string(x)` where `x` is an integer type other than `rune` or `byte`. This warning is enabled by default when running `go test`. As a consequence, running `go test github.com/golang/protobuf/proto` results in a build failure prior to this commit.

view details

push time in 4 days

PR merged golang/protobuf

proto: convert integer to rune before converting to string

Go 1.15 introduced a new go vet warning (https://golang.org/doc/go1.15#vet) for conversions of the form string(x) where x is an integer type other than rune or byte. This warning is enabled by default when running go test. As a consequence, running go test github.com/golang/protobuf/proto results in a build failure prior to this commit.

Fixes #1209.

+1 -1

5 comments

1 changed file

Saser

pr closed time in 4 days

issue closedgolang/protobuf

proto: tests fail to build on Go 1.15 due to new vet check

What version of protobuf and what language are you using? Version: v1.4.2.

$ go version
go version go1.15.2 linux/amd64

What did you do? Go 1.15 contains a new vet check that warns for conversion of the form string(x) where x is an integer type that is not rune or byte. I initially noticed the build failure when I ran go test all in a project that uses Go 1.15 and depends on this module.

To reproduce:

$ cd $(mktemp -d)

$ go mod init test
go: creating new go.mod: module test

$ go get github.com/golang/protobuf@v1.4.2
go: downloading github.com/golang/protobuf v1.4.2

$ go test github.com/golang/protobuf/proto
go: downloading google.golang.org/protobuf v1.23.0
go: downloading github.com/google/go-cmp v0.4.0
# github.com/golang/protobuf/proto
[redacted $GOPATH]/pkg/mod/github.com/golang/protobuf@v1.4.2/proto/text_decode.go:768:10: conversion from uint64 to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
FAIL    github.com/golang/protobuf/proto [build failed]
FAIL

What did you expect to see? Passing tests.

What did you see instead? A build failure (see details above).

Anything else we should know about your project / environment? N/A

closed time in 4 days

Saser

pull request commentgolang/protobuf

proto: convert integer to rune before converting to string

We consider development of the github.com/golang/protobuf module frozen at this point. However, it is an important goal that the module continues to work with higher versions of Go. Therefore, this change is justified.

The PR looks good to me, but will require the CLA to be signed in order to be merged.

Saser

comment created time in 4 days

delete branch google/go-cmp

delete branch : license

delete time in 5 days

push eventgoogle/go-cmp

Joe Tsai

commit sha d713870ac17fdb9ee5e2ee48ff6562dfb1c0157b

Fix license headers (#236) There is no LICENSE.md file, but there is a LICENSE file.

view details

push time in 5 days

PR merged google/go-cmp

Fix license headers

There is no LICENSE.md file, but there is a LICENSE file.

+47 -47

0 comment

47 changed files

dsnet

pr closed time in 5 days

PR opened google/go-cmp

Fix license headers

There is no LICENSE.md file, but there is a LICENSE file.

+47 -47

0 comment

47 changed files

pr created time in 5 days

create barnchgoogle/go-cmp

branch : license

created branch time in 5 days

issue commentgolang/go

proposal: errors: add ErrUnsupported

my main use case for an error of this form is to skip tests of functionality that is not present on the current platform.

Thanks for clarifying. That's an reasonable use-case, but I doubt that's what most people are thinking of when they see this.

The difference between that and "unimplemented" is that no semantically-equivalent fallback is possible

I'm going to make the case that most people will not understand this, leading to this error being widely misused and therefore the meaning of it being increasingly muddled to the point where the documentation mismatches what is done in practice.

(If a fallback implementation is possible, than the function being called should just do that in the first place.)

I'm not a fan of this recommendation since there is often information that the caller has where they can do a better job at doing a fallback than what the function can do locally.

ianlancetaylor

comment created time in 5 days

issue commentgolang/go

proposal: errors: add ErrUnsupported

Much of the discussion have been regarding the difference between "not supported" and "not implemented" and I'm still not entirely clear what the difference exactly is. If we add this sentinel error, I'm fairly sure a large number of Go users will not understand the difference as well.

When I first saw this proposal, my understanding was closer to "not implemented", but upon reading the discussion, it seems that this is something more like "not supported". If this were to be added and I had not read this discussion, I'm fairly sure I would have used this error as if it were "not implemented".

Code like:

func (w) WriteString(s string) (int, error) {
    if u, ok := w.u.(io.StringWriter); ok {
        return u.WriteString(s)
    }
    return 0, ErrTBD
}

is exactly like something I would have written using ErrUnsupported in place of ErrTBD.

To be honest, I'm not sure what the utility of this sentinel error is. The proposed documentation on this says:

// Functions and methods should not return this error but should instead // return an error including appropriate context that satisfies // errors.Is(err, errors.ErrUnsupported)

Support I do errors.Is(err, errors.ErrUnsupported), what are the things that we're expecting users to do with this? If they use that to switch to some fallback implementation, how is that functionally different than "not implemented"?

ianlancetaylor

comment created time in 5 days

issue openedgolang/go

cmd/vet: flag likely incorrect %T usages on reflect.Type

In code that uses Go reflection, it is common that the user prints the type. However, it easy for people (out of habit) to use %T with a reflect.Type when they should actually be using %v instead.

For example:

t := reflect.TypeOf(...)
if checkValid(t) {
    panic(fmt.Sprintf("invalid type: %T: t)) // incorrect: should be using %v
}

It is unlikely that the wrong verb is detected during code review.

In the instances similar to this, I don't think I saw a single case where a person actually wanted to print the type of reflect.Type itself (which by the way is *reflect.rtype and not particularly interesting for users).

\cc @dominikh in case this is a check better suited for staticcheck than vet.

created time in 5 days

issue commentgolang/protobuf

compiler/protogen: Go name conflicts can occur for non-style compliant protobuf names

That snippet tells us that the code doesn't compile. Can you explain how that's better UX than the Go toolchain not compiling the code? To my knowledge, the go/types package doesn't return a distinguishable error for conflicting identifiers.

ofpiyush

comment created time in 6 days

issue commentgolang/protobuf

compiler/protogen: Go name conflicts can occur for non-style compliant protobuf names

We already do something similar for formatting go code, checking for package errors will not be a lot of additional effort for better UX.

That's 100LOC to maintain for slightly better UX; I'm not convinced this is worth it. If there's anything to do here, it should be in protoc to at least print a warning that non-style compliant names are being used. The problem we're facing here is certainly not unique to Go.

proto4, whenever that comes. Is there any official place to request features/fixes for the next major version or to file things for consideration when that conversation begins?

In my opinion, the introduction of proto3 itself, give the pre-existing proto2 has been technically expensive to maintain and rollout. I'm doubtful there will ever be a proto4. Regardless, any discussion about proto4 should be at github.com/protocolbuffers/protobuf

ofpiyush

comment created time in 6 days

issue commentgolang/protobuf

encoding/protojson: DiscardUnknown doesn't work for unknown enum strings

"ignores" unknown string value by dropping it and hence changing the size of the repeated field.

Wow. That is surprising behavior, but okay.

jjhuff

comment created time in 6 days

issue openedgolang/go

reflect: add StructField.IsExported

The StructField.PkgPath field is populated if and only if the field is unexported. However, there is a hit to readability:

for i := 0; i < t.NumField(); i++ {
    f := t.Field(i)
    if f.PkgPath != "" {
        continue
    }
    ...
}

It's not obvious to most readers why the f.PkgPath != "" check exist. For this reason, a vast majority of cases have a comment that says:

    // Skip unexported fields.
    if f.PkgPath != "" {
        continue
    }

This documentation seems unnecessary if we had just added a helper method on StructField like:

// IsExported reports whether the struct field is exported.
func (f StructField) IsExported() bool {
    return f.PkgPath == ""
}

Thus, use of it becomes obvious from a readability perspective:

for i := 0; i < t.NumField(); i++ {
    f := t.Field(i)
    if f.IsExported() {
        continue
    }
    ...
}

Bikeshedding: I don't have a preference whether this should be called IsExported or simply Exported. The former matches IsValid, IsNil, or IsZero, while the latter matches StructField.Anonymous. It seems that the reflect package is already internally inconsistent about predicate functionality.

created time in 6 days

issue commentgolang/protobuf

encoding/protojson: DiscardUnknown doesn't work for unknown enum strings

I didn't check C++, but I'm going to guess that it doesn't handle this case since there's not an obvious behavior to perform in the case of repeated fields.

jjhuff

comment created time in 6 days

issue commentgolang/protobuf

protojson DiscardUnknown doesn't work for unknown enum strings

\cc @cybrcodr, do you happen to remember what the behavior for equivalent functionality in C++ is?

jjhuff

comment created time in 6 days

issue commentgolang/protobuf

compiler/protogen: Go name conflicts can occur for non-style compliant protobuf names

We can fail explicit when we know generated code will not compile.

What's the point of building conflict detection code into protoc-gen-go if the code already doesn't compile? We'd just be re-implementing the exact same logic as the Go compiler itself. Therefore, It seems entirely appropriate to let the Go compiler be the explicit failure for this class of failures.

this might be a good candidate issue for discussion in v4.

What is v4?

ofpiyush

comment created time in 7 days

issue commentgolang/protobuf

compiler/protogen: Go name conflicts can occur for non-style compliant protobuf names

The world already breaks when those failures happen. It just breaks when the program compiles instead of code generation.

The code breaks if you add another non-compliant name that results in a conflict. That's very different from breaking the code the moment if any non-style compliant name is used.

I should also note that style enforcement is not the responsibility of protoc-gen-go. Personally, I support such a notion, but this is something that should be discussed at the main protocol buffer project and be implemented in protoc.

ofpiyush

comment created time in 7 days

issue commentgolang/go

proposal: spec: type inferred composite literals

As a counter argument to above, the snippet below becomes more readable with type-eliding:

compress.NewReader(r, {Level: compress.BestSpeed})
compress.NewReader(r, compress.NewReaderOptions{Level: compress.BestSpeed}

You'll find examples that may get better and examples that may get worse if eliding were always performed. Personally, I prefer to at least have this choice.

neild

comment created time in 7 days

issue commentgolang/protobuf

compiler/protogen: Go name conflicts can occur for non-style compliant protobuf names

Explicit failure is better than a silent one in case there are no plans to support non-style compliant proto files.

There's no way we can enforce this today with breaking the world. The situation is unfortunate, but it is what it is.

If we want to support the behaviour, we can follow the convention proto2 codegen does and add an _ for the 2nd occurence of a conflicting name.

We already do this for some small set of conflicts. We consider it a grave mistake to have ever done this since it means that generated names are dependent on the order that fields are specified in a .proto file.

ofpiyush

comment created time in 7 days

issue closedgoogle/go-cmp

Inconsistent results from Diff() - intermittent \u00a0 in result

I've found that the raw, "Go-syntax" results of a Diff() intermittently contain the \u00a0 unicode character rather than a space.

Here's a sample that reproduces this:

	type A struct {
		B string
	}
	result := cmp.Diff(A{B: "foo1"}, A{B: "foo2"})
        // Print out the Go-syntax output:
	fmt.Println("Result is: " + fmt.Sprintf("%#v", result))

Running the above sometimes produces: Result is: "\u00a0\u00a0reconciler.A{\n-\u00a0\tB: \"foo1\",\n+\u00a0\tB: \"foo2\",\n\u00a0\u00a0}\n"

and sometimes produces: ```Result is: " reconciler.A{\n- \tB: "foo1",\n+ \tB: "foo2",\n }\n"``

Result is: "\u00a0\u00a0reconciler.A{\n-\u00a0\tB: "foo1",\n+\u00a0\tB: "foo2",\n\u00a0\u00a0}\n"

This problem is only apparent if you are looking at the Go-syntax (%#v) output, and is not apparent by just printing the string. However, there are libraries that look at the Go-syntax value of a string - for example, the various string comparison methods of https://github.com/stretchr/testify. Attempting to use the output of Diff() in test cases with https://github.com/stretchr/testify is not usable due to this inconsistency.

closed time in 7 days

dansimone

issue commentgoogle/go-cmp

Inconsistent results from Diff() - intermittent \u00a0 in result

Closing as working as intended.

The first few versions of cmp unfortunately kept breaking a number of tests that assumed that the output would never change every time the reporter output was changed. At some point we made it random precisely so that brittle tasks cannot be written. The fact that the reporter output has been improved significantly in recent versions is thanks to the freedom to make beneficial changes made possible by deliberate randomization.

dansimone

comment created time in 7 days

issue closedgolang/protobuf

Incorrect Import in plugin implementation

What version of protobuf and what language are you using? Version 1.25.0

What did you do? The GeneratedCode creates wrong import name:

g.QualifiedGoIdent(protogen.GoIdent{GoName: "", GoImportPath: "github.com/scylladb/gocqlx/v2"})

What did you expect to see? import ( gocqlx "github.com/scylladb/gocqlx/v2" ) OR import ( "github.com/scylladb/gocqlx/v2" )

What did you see instead? import ( v2 "github.com/scylladb/gocqlx/v2" )

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment?

closed time in 9 days

ehsannm

issue commentgolang/protobuf

Incorrect Import in plugin implementation

Closing as duplicate of #1205

ehsannm

comment created time in 9 days

issue commentgolang/protobuf

Silent failure for message -> struct name conversion.

The derived Go names that protoc-gen-go uses (or more specifically compiler/protogen produces) has long been known to be problematic and possibly produce conflicts. Unfortunately, there is not much we can do about it since changing the behavior will likely break the world. For that reason, the new google.golang.org/protobuf module did not fix this behavior since it's deeply ingrained into the wider use of protobufs in Go and we wanted the new module to be reasonably compatible with the older github.com/golang/protobuf to the best we can.

I should note that naming styles in most programming languages is typically an aesthetic concern. However, for protocol buffers, it's actually a semantic concern since the naming style used makes a difference in whether generated code is even valid. Therefore adherence to the style is of much greater concern beyond just personal preferences. If the protobuf style guide is properly followed, conflicts should not occur. Otherwise, all bets are off, and we can't guarantee conflict-free name generation.

If you want guaranteed conflict-free name derivation, then the following naming styles must be used for the corresponding descriptor types:

Descriptor Type Naming Style
Enum CamelCase
EnumValue UPPER_SNAKE_CASE
Message CamelCase
Field lower_snake_case
Oneof lower_snake_case
Extension lower_snake_case
Service CamelCase
Method CamelCase

In particular:

  • CamelCase must match the regular expression [A-Z][a-zA-Z0-9]*, where any letters that appear after a number must be uppercase (e.g., Ext4Mount is okay, but Ext4mount is not).
  • UPPER_SNAKE_CASE must match the regular expression [A-Z][_A-Z0-9]* (e.g., GEO_LOCATION is okay).
  • lower_snake_case must match the regular expression [a-z][_a-z0-9]*, where letters must not follow any numbers (e.g., int64_value is okay, but int64value is not).
ofpiyush

comment created time in 9 days

issue commentgolang/protobuf

protogen: support naming a GeneratedFile import with an alias

I should note that ImportAs would be best-effort, since we would presumably still prioritize correctness over cosmetics in the event that there is still a conflict over the chosen name.

noahdietz

comment created time in 10 days

issue commentgolang/go

compress/zlib: compression behavior change since go1.15?

Bisect implicates https://go-review.googlesource.com/c/go/+/193605

k-ye

comment created time in 11 days

issue commentgolang/go

compress/zlib: compression behavior change since go1.15?

Thanks @networkimprov. I've been a bit distant from Go toolchain development for a while, so I've gotten behind on the latest in processes.

k-ye

comment created time in 11 days

issue commentgolang/go

compress/zlib: compression behavior change since go1.15?

Adding to Go1.15.3 milestone since this is a regression relative to Go1.14 and a serious one.

k-ye

comment created time in 11 days

issue commentgolang/go

compress/zlib: compression behavior change since go1.15?

@kokes, difference in output in itself is not particularly interesting since the compress/flate library provides no guarantees about output stability. However, producing corrupted issue is something we take seriously.

Thank you @k-ye. I am able to reproduce which does indeed indicate that it's producing corrupted output. I'm bisecting the root cause now (may very well be the CLs related to #34121 that you pointed to earlier).

k-ye

comment created time in 11 days

issue commentgolang/go

compress/zlib: compression behavior change since go1.15?

I need more information to investigate this. The code snippets provided on do basic compression/decompression. I need real test data that exhibits the problem.

k-ye

comment created time in 12 days

issue commentgolang/go

compress/zlib: compression behavior change since go1.15?

If the output is valid (e.g., can be decompressed), then this is working as expected. The compression libraries make no guarantees that the output remains stable for all time. While that guarantee can be useful in some contexts, if unfortunately means that we can never make changes to the compression algorithm either to improve the speed or to improve the compression ratio, both which are properties that are generally considered more important than stability.

k-ye

comment created time in 13 days

issue commentgolang/go

compress/zlib: compression behavior change since go1.15?

It is unclear to me whether this issue is about the compressed output simply being different or that the compressed output is actually invalid (i.e., cannot be decompressed). Can you please clarify?

k-ye

comment created time in 13 days

issue commentgolang/protobuf

all: cyclic imports bloat go.sum

Given that this isn't a pressing issue, we're not going to do anything about this at the moment. It's a significant amount of release tooling work with little gain. Furthermore, it's really something the Go toolchain should do a better job at pruning useless go.sum entries.

2767mr

comment created time in 14 days

issue commentgolang/protobuf

all: cyclic imports bloat go.sum

I'm not sure how your test repos address the root issue. Both https://github.com/2767mr/prototest2/blob/master/go.sum and https://github.com/2767mr/prototest3/blob/master/go.sum and contain cyclic references to the past history of the other's module.

Stepping back a bit, can you explain why this is an issue that needs resolving? Is this purely a bloat issue or is there a technical issue involved?

2767mr

comment created time in 14 days

issue commentgolang/protobuf

encoding/{protojson,prototext}: add MarshalWriter/UnmarshalReader functionality

thinking I think the best choice might to stay away from the pattern after all, so that we neither reinforce unexpected behavior, or the unexpectedness of the behavior.?

It seems that part of this is figuring out what we want to do with encoding/json related to this issue. It's been on my TODO list to think more heavily about golang/go#36225 (and many other encoding/json issues discovered over the years).

One test even required the buggy behavior in order to pass, because of a typoed extra %s at the end of the format string in the test, resulting in something like {"valid":"json"}%!s(MISSING) being the test input.

😲

dsnet

comment created time in 14 days

issue commentgolang/protobuf

encoding/{protojson,prototext}: add MustUnmarshal function

I‘m not sure about reinforcing a habit of unmarshalling const string into a proto message for initialization

True, I'm not fond of having multiple ways of doing the same thing. This issue was filed primarily based on observation of a common code pattern. I don't feel strongly whether to have this or not.

dsnet

comment created time in 14 days

issue commentgolang/protobuf

all: cyclic imports bloat go.sum

It is possible to reference future releases inside a go.mod.

Yes, for the go.mod file, but not for the go.sum file. The go.sum file contains a hash of the dependencies, but there's a problem. The hash of the dependencies covers the forward version reference to some unreleased module for which we do not know (or rather cannot compute) the hash for.

2767mr

comment created time in 14 days

issue commentgolang/protobuf

all: cyclic imports bloat go.sum

This is a known issue. It's unfortunate, but relatively harmless. The cyclic dependency is necessary:

  • github.com/golang/protobuf depends on google.golang.org/protobuf because v1.4+ is implemented in terms of the newer module.
  • google.golang.org/protobuf "depends" on github.com/golang/protobuf only to enforce that v1.4+ be used. Aside from that constraint, it requires nothing from the older module. This is necessary to avoid problems with disjoint sets of type registries.

The cyclic relationship can't be broken.

If there's anything to be done, it probably be with the Go toolchain to prune useless go.sum entries for cyclic dependencies. If so, there's not much this project can do about this issue.

2767mr

comment created time in 15 days

issue commentgolang/protobuf

How understand ExtensionDesc options and this value without convert to string?

You have have an ExtensionDesc and want to retrieve the value for a field that the extension descriptor represents, you can use proto.GetExtension.

(FYI, we try to keep our issue tracker about actionable issues. Questions like this are best asked on Slack or someone other means where you can more readily get immediate help. Consider joining the #protobuf channel or the Gophers slack workspace: https://gophers.slack.com/?redir=%2Fmessages%2Fgeneral%2F)

mrfoe7

comment created time in 18 days

issue openedgolang/protobuf

encoding/{protojson,prototext}: add MarshalWriter/UnmarshalReader functionality

JSON serialization is very commonly used in HTTP servers where use of net/http means that the user typically starts with a io.Reader and io.Writer on hand. As such, it is slightly inconvenient using protojson since it operates primarily on []byte.

Perhaps we should add MarshalWriter that takes in a io.Writer as input and a UnmarshalReader that takes in a io.Reader as input.

Consideration needs to be given to:

  • #609 which considers adding APIs to the proto package to support streaming read/write and/or scatter/gather functionality.
  • https://github.com/golang/go/issues/36225, where the typical pattern of doing json.NewDecoder(r).Decode(...) is subtly buggy since it permits trailing trash data to go undetected.

(We would probably do the equivalent change to prototext since the two packages have nearly identical API surface despite having obviously different semantic behavior).

created time in 18 days

issue openedgolang/protobuf

encoding/{protojson,prototext}: add MustUnmarshal function

I'm noticing that text serialization (and to some degree JSON) is used to quickly initialize a lot of data in tests (and sometimes in not-test code at init) with a pattern like:

const initData = `...` // text data
m := new(foopb.MyMessage)
if err := prototext.Unmarshal([]byte(initData), m); err != nil {
    panic(err)
}
return m

With the upcoming introduction of generics, perhaps we should add a helper for this:

func MustUnmarshal[M proto.Message](s string) M {
    var m M
    m = m.ProtoReflect().New().Interface()
    if err := Unmarshal([]byte(s), m); err != nil {
        panic(err)
    }
    return m
}

Thus, usage of this would be something like:

m := prototext.MustUnmarshal[*foopb.MyMessage](initData)

created time in 18 days

issue commentgolang/protobuf

panic when use github.com/google/go-cmp on generated messages

If you want to ignore fields in a protobuf message, then you can use protocmp.IgnoreFields. If you want to ignore fields in a normal Go struct, then you can use cmpopts.IgnoreFields. You can compose the two options together if needed.

tiagomelo

comment created time in 20 days

issue commentgolang/protobuf

panic when use github.com/google/go-cmp on generated messages

is it or is not possible to really make cmp.Diff() to work in conjunction with google.golang.org/protobuf/testing/protocmp

Is this a typo? protocmp exists to work with cmp.Diff. Did you mean to ask whether it's possible to use cmp.Diff without protocmp? If so, it was never the intention of cmp since it was created to be used to compare protobuf messages. In fact, protocmp was something I wanted to have available when cmp first became available, but we needed to build protobuf reflection first.

tiagomelo

comment created time in 20 days

issue commentgolang/go

encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions

I have to admit I'm still not getting this. Such a map will only tell you if a type T has a match in the set of interfaces, but not which interfaces T matches, right? You'd still have to check them one by one. This is better for the case where T matches none of the interfaces, but still, the feature doesn't seem O(1) to me.

A simple modification of the cache is to use a map[reflect.Type]int where the int is the index interface that we care about.

Side note: if you intend the CL's performance to remain as-is, I wonder if we should document that we don't recommend registering more than a handful of interface types on a single encoder or decoder.

Sounds reasonable.

rsc

comment created time in 20 days

issue commentgolang/go

encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions

I'd imagine this is necessary either way if you're supporting interfaces.

Can you provide an example?

rsc

comment created time in 20 days

issue commentgolang/go

encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions

If both are available at once, it's likely users will jump straight to interface types without thinking whether using concrete types would be enough. Which is also why the performance difference worries me.

Can you explain how this would be the case? If there was a concrete type T that you wanted to provide an override for, then it seems natural that you would register a concrete type override. It seems odd to me to reach for an interface override unnecessarily for that case.

For example, let's say the user wants to provide a type override for time.Time, what interface would they be using to match that type? It seems more natural that they would register a func(t time.Time) ([]byte, error). Also, it would be easier to implement since working with concrete types is easier than interface types (where you're limited to the methods in the interface).

rsc

comment created time in 20 days

issue commentgolang/go

encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions

(apologies for responding out-of-order; for some reason my browser didn't render the earlier comment)

And would using concrete types be a big problem in those two use cases you mention?

Yes. For some cases, it would require registering dozens (if not hundreds) of concrete types. Even worse, it requires that the user of manually keep the code that declares their types constantly in sync with where those types are used with JSON serialization.

Can you elaborate a bit? I'm not quite seeing how that cache would work in constant time. If we're confident that this can be made performant in the future, then I'm not as worried about adding both features at once.

You can imagine a sync.Map that is functionally a map[reflect.Type]bool. It caches the answer to "will type T implement the set of registered interface type overrides?" If T is in the cache, then the query is O(1), if not, then we perform a linear lookup over all registered interface type overrides (which I contend is usually close to 0).

The challenge with the implementation is that the specific cache we want to use is itself dependent on the set of possible interface types that we may want to match against. Since I don't see having many interface type overrides being a common pattern, I didn't want to implement this for now.

rsc

comment created time in 20 days

issue commentgolang/go

encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions

Googlers have told you what they think of both features, but from what you've said, they haven't tried using concrete types alone

Concrete overrides are used to. I didn't mention them since the question I was responding to was about interface overrides.

And there isn't an easy way for developers outside of Google to do the same kind of testing (and very few people will go through the trouble unless we ship it in a release), so I fear that the feedback doesn't represent Go use cases outside of Google.

Is there something specific that you are proposing? More users of the feature would be great. For users that only care about concrete type overrides, the existence of the interface type override doesn't affect them if they don't use it. Am I missing something?

rsc

comment created time in 20 days

issue commentgolang/go

encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions

At Google, we have https://go-review.googlesource.com/c/go/+/212998 imported as a separate fork that adventurous users already using. The ability to specify type-override based on an interface is widely used. For our cases, it is typically used to capture all types that implement a proto.Message interface, but it's also used for other cases where a type doesn't have natural JSON support, and an interface match allows capturing types based on the public API (as specified by an interface) and to provide custom serialization for that type (based on the methods of that interface).

func([]byte, interface{}) fallback to be used if none of the concrete type funcs match. This won't be as powerful, but it will also remove the expensive linear search with go/types.Type.AssignableTo.

There are ways to implement the linear search in O(1) with type caches, I didn't implement here because it seems like pre-mature for the case where there are many type-overrides given that are all interfaces.

Also, a func([]byte, interface{}) fallback is 1) less performant since it now needs to be called for every node in the tree, rather than just those that match the interface, 2) does not compose well with disjoint sets of interface types, 3) needs to also provide an API for saying; "thanks for calling me, but I actually don't know how to serialize this specific type; please try something else".

rsc

comment created time in 20 days

issue commentgolang/go

compress/gzip: Reader unable to parse headers with long comments

Having reviewed my CL, the reason I didn't push it through is because of concerns with this feature being an trivial avenue for denial-of-service attack where a malicious GZIP file could cause a server to allocate arbitrary amounts of memory.

We have several options:

  1. We could immediately permit parsing of small headers (I proposed up to 32KiB to match the window size of the DEFLATE algorithm itself). However, this won't help use-cases that abuse the comment field to store machine-parsable information where it can easily exceed 32KiB.
  2. Add a new Reader.MaxFieldSize method so that the user can opt-in to parsing larger headers. Use of this option would still be a bit odd, since you would have to do something like:
var zr gzip.Reader
zr.MaxFieldSize(math.MaxInt32)
if err := zr.Reset(r); err != nil {
    ... // handle err
}
... // make use of zr.Header.Comment

Since this would probably require an API change, I'm adding the NeedDecision label. \cc @FiloSottile since he's probably interested in any possible DOS-related issues.

dsnet

comment created time in 21 days

issue commentgolang/go

compress/gzip: Reader unable to parse headers with long comments

Running the snippet on Go1.15 clearly shows it's still an issue.

Huh, apparently I had a fix for this that I never submitted: https://golang.org/cl/53637

dsnet

comment created time in 23 days

issue commentgoogle/go-cmp

proposal: use generic less function when cmpopts.SortSlices argument is nil

I was thinking about this more and I wanted to note a complication with generic less functions.

Suppose you had:

type S struct {
    F1, F2 int
} 

var ss1, ss2 []S = ...
cmp.Equal(ss1, ss2,
    cmpopts.SortSlices(GenericLess(S{})),
    cmpopts.IgnoreFields(S{}, "F1"),
)

There's a subtle problem here. The less operation used by cmpopts.SortSlices needs to be aware of any other options passed to cmp that may affect the semantics of equality. In this specific case, there's another option that ignores field "F1" and that needs to be taken into consideration when performing a sort, otherwise this semantically won't work as expected (it might sometimes get it right, and it might not; its flaky).

To make this work, we would probably need

  1. some way to plumb options down to other options, and
  2. generic less as a first-class feature in cmp itself. It's possible to define a generic less (e.g., cmp.Less), but a complication arises from the presence of Equal methods and Comparer options. Such functionality provides no means to derive ordering information. If there was a cmp.Less function, this property should hold: (cmp.Less(x, y, opts...) == 0) == cmp.Equal(x, y, opts...).
rogpeppe

comment created time in 24 days

CommitCommentEvent

issue closedgolang/protobuf

Mention version/commit of protoc at the top [Code generated by protoc-gen-go. DO NOT EDIT.]

Is your feature request related to a problem? Please describe. It is usually not clear which version/commit generated the code here.

Describe the solution you'd like Mention of the version/commit at the top would make things clear and easier to work with.

closed time in 24 days

emailtovamos

issue commentgolang/protobuf

Mention version/commit of protoc at the top [Code generated by protoc-gen-go. DO NOT EDIT.]

Duplicate of #524, which is already done for google.golang.org/protobuf/cmd/protoc-gen-go. There are no intentions to add this feature to github.com/golang/protobuf/protoc-gen-go.

emailtovamos

comment created time in 24 days

issue commentgolang/protobuf

How understand field options type without convert to string?

I apologize, but I don't understand what this is asking. Can you be more specific about what you trying to do?

Also, it seems like you're using gogoproto.nullable. Perhaps you should be asking this over at https://github.com/gogo/protobuf/issues?

mrfoe7

comment created time in a month

issue commentgolang/protobuf

reflect/protodesc: some descriptors cannot be round-tripped to protos and back

Also, as mentioned in the description, this is strangely inconsistent. I can generate such code to Go using protoc-gen-go and load the descriptor just fine.

I agree it's inconsistent, and I'm inclined to make protoc-gen-go not be able to generate MessageSet code in open-source.

jhump

comment created time in a month

issue commentgolang/protobuf

reflect/protodesc: some descriptors cannot be round-tripped to protos and back

Is there any real code outside Google that actually uses MessageSet? It's an artifact of proto1 and it's a source of significant technical debt. In open-source, we ideally should treat it as if it doesn't exist.

jhump

comment created time in a month

issue commentgolang/protobuf

Well-Known Type parser not used if protojson.MarshalOptions.Resolver is set

I think the issue the relationship between a global resolver and a custom resolver to be somewhat of a red herring.

After all this code:

m := &anypb.Any{
	TypeUrl: "type.googleapis.com/google.protobuf.Timestamp",
	Value: protopack.Message{
		protopack.Tag{1, protopack.VarintType}, protopack.Varint(1599000711),
		protopack.Tag{2, protopack.VarintType}, protopack.Varint(166067539),
	}.Marshal(),
}
b, err := protojson.Marshal(m)
fmt.Println(string(b), err)

fails with:

 proto: google.protobuf.Any: unable to resolve "type.googleapis.com/google.protobuf.Timestamp": not found

assuming that timestamppb is not linked in.

The question at hand is whether protojson should have implicit knowledge of the WKTs for dynamic purposes or not. Either:

  1. protojson has a dependency on every WKT as @neild alluded to (and the associated import-cycle problems), or
  2. protojson has a local copy the descriptor of every WKT and uses it with dynamicpb to create a dynamic version of every WKT.

Neither approach seems particularly palatable for this edge-case.

bufdev

comment created time in a month

issue commentgolang/protobuf

API v2: protodesc uses empty non-nil package when creating FileDescriptorProto for a file with no package

Seems reasonable. Would you like to contribute a fix?

jhump

comment created time in a month

issue commentgolang/protobuf

APIv2: why are all Descriptor interfaces marked with doNotImplement?

So is the main reason for this so that you can add new methods to the interface without breaking users?

Essentially this. The protobuf language has new additions over time (e.g., oneofs, maps, proto3 optional to name a few), and we would want the freedom to adjust the descriptor API accordingly to reflect such functionality.

There was some discussion in the past about adding a "FooDescriptorNotImplemented" type for every Descriptor and require application code to embed that or some other technique. At the time that we released the module, we decided to punt on figuring out if we wanted to allow custom descriptor implementations. Naturally, having an unexported method is the more conservative approach and can always be loosened in the future.

\cc @neild

jhump

comment created time in a month

issue commentgolang/go

json.Unmarshaler breaks DisallowUnknownFields

As @mvdan mentioned, I don't know if there is a specific issue about the top-level option passing problem, but it's one that I've commented about in a number of different encoding/json issues. Adding new features carries O(n!) maintenance cost, since you need to consider how the newly added feature interacts with every feature that has already provided. I don't think we should add another UnmarshalerWithOptions interface without carefully thinking about the implications. Just as a high-level overview, such an interface would probably require a dependency on the encoding/json package, which is already a significant divergence from the Unmarshaler interface itself (which has no dependency on anything other than basic types like []byte, int, and error). If we were to force that dependency through the interface, it also brings into question whether the interface should provide more things than just the options. For example, currently calling nested Unmarshalers is O(n^2) since the encoding/json package has to repeatedly parse the entire sub-tree to find the end of the JSON token before it can call UnmarshalJSON. If we were to pass something similar to json.Decoder, it could be much more efficient.

alvaroaleman

comment created time in a month

issue commentgolang/protobuf

Incorrect Import in plugin implementation

I'm okay with adjusting the heuristic for how protogen determines the local package names, but your description still seems to suggest that this is purely for cosmetic reasons since the "the code is full of vX.Identifier". Can you confirm that the code actually functions correctly even if it doesn't look like what is expected?

Essentially, I'd like to make sure there's not some deeper issue I'm not quite understanding. The purpose of protogen is not to generate code that always follows style guidelines. It's nice if possible, but it's not a goal.

ehsannm

comment created time in a month

issue commentgolang/protobuf

protoc-gen-go: generate allocating accessors for message fields

\cc @neild who has been interested in adding a "MutableFoo" accessor method which does exactly this.

chrisguiney

comment created time in a month

issue commentgolang/protobuf

Incorrect Import in plugin implementation

Can you explain why this is "incorrect"? Within a Go source file, the code can rename import paths to any chosen local package names; and "v2" is a valid Go identifier and shouldn't cause any problems so long as it is unique within the context of where it is used.

ehsannm

comment created time in a month

issue commentgolang/protobuf

How to implement a general tool that can deserialize any proto message's serialization?

There is no .proto parser implementation in this module. You can consider using https://pkg.go.dev/github.com/jhump/protoreflect/desc/protoparse?tab=doc

Hank00AAA

comment created time in a month

issue commentgolang/protobuf

How to implement a general tool that can deserialize any proto message's serialization?

I update the issue, Is it clear enougn? 0 0

It's better. From what I can gather, yes, dynamicpb will do what you want. However, it will be the user's responsibility to provide the protobuf type information needed to parse arbitrary messages. The dynamicpb package creates dynamic messages when given a protoreflect.Message. A protoreflect.Message can be dynamically constructed at runtime using the google.golang.org/protobuf/reflect/protodesc package.

Hank00AAA

comment created time in a month

issue commentgolang/protobuf

Could I decode only a message in golang, not the whold proto file?

If you want to the ability to define a subset of the message at runtime, you could also do that with the google.golang.org/protobuf/types/dynamicpb package.

That said, I still don't understand what the original post is actually asking for.

Hank00AAA

comment created time in a month

created taggoogle/go-cmp

tagv0.5.2

Package for comparing Go values in tests

created time in a month

release google/go-cmp

v0.5.2

released time in a month

issue commentprotocolbuffers/protobuf

lack of specification for serializating NaN and infinity for google.protobuf.Value.number_value

The design document for JSON serialization for protobufs specifies that a google.protobuf.Value message without one of the kind fields explicitly populated should be rejected in order to ensure proper round-trip preservation of data. Given that proper preservation of data seems to be the intention, it would follow that NaN and Infinity should also be rejected in order to preserve the same property.

cybrcodr

comment created time in a month

issue commentprotocolbuffers/protobuf

lack of specification for serializating NaN and infinity for google.protobuf.Value.number_value

This issue is not so much about what users can do instead, but what is supposed to be correct serialization behavior.

cybrcodr

comment created time in a month

delete branch google/go-cmp

delete branch : error-help

delete time in a month

push eventgoogle/go-cmp

Joe Tsai

commit sha d2fcc899bdc2d134b7c00e36137260db963e193c

Suggest use of cmpopts.EquateErrors (#234) If cmp panics because it is trying to access an unexported field, specially suggest the use of cmpopts.EquateErrors if the parent type implements the error interface. Fixes #233

view details

push time in a month

PR merged google/go-cmp

Suggest use of cmpopts.EquateErrors

If cmp panics because it is trying to access an unexported field, specially suggest the use of cmpopts.EquateErrors if the parent type implements the error interface.

Fixes #233

+17 -1

0 comment

2 changed files

dsnet

pr closed time in a month

issue closedgoogle/go-cmp

suggest cmpopts.EquateErrors in error comparison panic

This panics:

cmp.Equal(errors.New("x"), errors.New("x"))
panic: cannot handle unexported field at {*errors.errorString}.s:
	"errors".errorString
consider using a custom Comparer; if you control the implementation of type, you can also consider using an Exporter, AllowUnexported, or cmpopts.IgnoreUnexported

The panic could detect when the type being compared implements error and suggest using cmpopts.EquateErrors.

closed time in a month

neild

issue closedgolang/protobuf

Generated code with the new https://github.com/protocolbuffers/protobuf-go depends on the old packages

When I generate code using the new protoc-gen-go I get the following generated code. It seems strange to depend on the old "github.com/golang/protobuf/proto" just for the "ProtoPackageIsVersion4" constant:


// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// 	protoc-gen-go v1.25.0
// 	protoc        v3.12.3
// source: proto/zipkin_proto3/zipkin.proto

import (
	proto "github.com/golang/protobuf/proto"
)

// This is a compile-time assertion that a sufficiently up-to-date version
// of the legacy proto package is being used.
const _ = proto.ProtoPackageIsVersion4

Am I missing something?

closed time in a month

bogdandrutu

PR opened google/go-cmp

Reviewers
Suggest use of cmpopts.EquateErrors

If cmp panics because it is trying to access an unexported field, specially suggest the use of cmpopts.EquateErrors if the parent type implements the error interface.

Fixes #233

+17 -1

0 comment

2 changed files

pr created time in a month

create barnchgoogle/go-cmp

branch : error-help

created branch time in a month

issue closedgolang/protobuf

proto: nil pointer panic with unknown fields

What version of protobuf and what language are you using? protobuf-go v1.23.0

What did you do? https://github.com/protocolbuffers/protobuf-go/blob/v1.23.0/internal/impl/encode.go image p.Apply(mi.unknownOffset).Bytes() might return a nil pointer,append() will appear panic panic: runtime error: invalid memory address or nil pointer dereference image

What did you expect to see? need add nil pointer check

Anything else we should know about your project / environment? go version go1.13.5 darwin/amd64 runtime: Linux ali-jump 2.6.32-696.10.1.el6.x86_64 #1 SMP Tue Aug 22 18:51:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

closed time in a month

yanxyc

issue commentgolang/protobuf

proto: nil pointer panic with unknown fields

Marking as resolved since this is an issue in the application code.

yanxyc

comment created time in a month

issue commentgolang/protobuf

proto: nil pointer panic with unknown fields

goroutine that has been started at the time of the request timeout did not end immediately, and this goroutine was modifying this protobuff object?

If another Go routine in modifying the same message while you are marshaling it, then that is fundamentally a race condition and expected to fail, cause memory corruption, or other strange side effects.

yanxyc

comment created time in a month

issue commentgolang/protobuf

protoc-gen-go: disable recording the versions in generated file

Given that reframing, it's possible that just hiding sub-versions of protoc might resolve the base problem I'm having.

By sub-version, do you mean the "PATCH" version per semver terminology? That's something we could consider, but PATCH versions themselves can theoretically introduce changes to the generated .pb.go output and will require more thought.

In general, I recommend that the project enforce some means of ensuring everyone is using the same version of tools for checked in code. This applies to protoc and protoc-gen-go and even go fmt.

bmhatfield

comment created time in a month

issue commentgolang/protobuf

Disable versions in generated file header

This use-case for wanting to disable the version markers is suspect as it's bad practices to have a number of developers working on the same module, but using different versions.

bmhatfield

comment created time in a month

issue commentgolang/protobuf

when marshal protobuf struct deal XXX_unrecognized field appear nil pointer panic

Do you have a reproduction for this? The line you are referencing is not the problem since it assumes that p itself is already a non-nil pointer. Thus, a nil pointer must have slipped in from elsewhere.

yanxyc

comment created time in a month

issue commentgolang/protobuf

The fieldGoType function is unexported

\cc @neild.

I vaguely recall some discussion in the past about whether to expose this or not. I don't remember the details, but there are some complexities involved in whether fieldGoType actually returns the expected Go type of the field in certain cases (e.g., for extensions and oneofs). Also, fieldGoType is not a pure function since it mutates the state of protogen.GeneratedFile making it a sharp API with subtle behavior. Any public API would need to be safer to use and will probably involve a noticeable increase in the protogen API surface.

mcesar

comment created time in 2 months

more