google/go-cmp 1515
Package for comparing Go values in tests
Safe Browsing API Go Client
dsnet/compress 284
Collection of compression related Go packages.
Daemon for creating a simple VPN over UDP.
Custom message-of-the-day (MOTD) generator intended to be informative about the system you are logging in to.
SSH daemon for creating forward and reverse tunnels.
TermiJack surreptitiously hijacks standard streams (stdin, stdout, and/or stderr) from an already running process.
dsnet/golib 12
Collection of mostly unrelated helper Go packages.
Locally hosted Go playground for more advanced functionality.
Simple bash tab completion for the go command.
push eventgolang/protobuf
commit sha c4051cd4ec0a2615a1d35d72f6191989a53d8de7
types/known: remove packages present in genproto Remove the generated proto packages that already exist in google.golang.org/genproto. We want to eventually move these packages here, but it doesn't need to happen yet. Add a local copy of fieldmaskpb for use in tests. Refactor proto generation to override import paths using the M<source>=<import_path> compiler option instead of by patching the source files. Change-Id: I8d31f67e931d70140182f19f3e0106111f71c4b4 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219598 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 5 hours
issue commentgolang/protobuf
APIv2: handling of nil map values
Are you speaking from experience regarding a real problem or is this hypothetical?
It is not the same data anymore.
The generated Go code is not a 1:1 match to the protobuf data model. There are other ways where you can express things in the Go data structures that cannot be represented in the protobuf data model. One simply cannot expect that a round-trip marshal/unmarshal reproduces the exact same Go data structure. This is a false expectation and never documented as being valid.
comment created time in 8 hours
push eventgolang/protobuf
commit sha 725bfeaf400d629ea98057ca4b6f11587944acfe
internal/impl: support legacy Merger interface Change-Id: I796be0bb1bb40605073228446364f3a2f1073ef1 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219557 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 10 hours
issue commentgolang/protobuf
protoc-gen-go: Plugin output is unparseable: Active code page: 65001\r\nz\2230\n\ntest.pb.goz\2040
Thanks for the report. However, without knowing what version of protoc-gen-go this is and also the .proto file that was used to produce this issue, it is very difficult for us to do anything with this report.
comment created time in a day
issue commentgolang/protobuf
APIv1: proto: direct fast-path for old messages
My vote is to do nothing to address this in v1, but:
- Highlight in the release notes that the v2 runtime should be use with messages generated by the v2 generator for best performance results.
- If this is a sufficiently noticeable problem, we could do the performance hack I suggested in a hypothetical
v1.4.1. - I prefer to not spend time on technical work that doesn't matter in the near future.
comment created time in a day
issue openedgolang/protobuf
APIv1: proto: direct fast-path for old messages
As things currently stand, the v1.4.0 release of github.com/golang/protobuf/proto is going to use google.golang.org/protobuf/proto under the hood for unmarshaling and marshaling. To accomplish this, it uses the proto.MessageV2 function to convert any generated message to one that satisfies the v2 message interface.
If the message is recently generated, then it implements the interface and things are great. However, if it does not, then the MessageV2 wraps the message in some internal wrapper type so that it implements the v2 message interface. The process of wrapping involves allocating a 16B struct, and would occur for every top-level message being passed to unmarshal or marshal. This is a performance regression as it is an allocation that formerly did not exist.
To avoid this regression, we could hack v2 to expose low-level details of internal/impl, so that the v1 proto package can directly call the fast-path without allocating the stub structure. I didn't look too deeply into this, but I suspect that this is a non-trivial amount of work and addition of technical debt.
How much attention should we give to the performance of the new proto runtime working with old generated messages?
- Long term, all messages should eventually be re-generated so that this won't matter. It's technical debt added to just support a temporary use-case.
- Working on this will delay v2 more than it already is.
- It's possible that the performance hit is offset by the v2 implementation actually being faster (I did not benchmark this).
- My fear is that users will blindly benchmark v2 without regenerating their messages and claim that v2 is slower than v1, and write blog posts that no one should use it.
created time in a day
issue openedgolang/protobuf
APIv2: runtime/protoiface: add Reset fast-path method?
Currently the implementation special-cases the Reset method from the v1 Message interface. Should it accomplish this through a protoiface method in the same way as all the other methods like unmarshal and marshal?
created time in a day
issue openedgolang/protobuf
APIv2: address TODO(blocks) comments
This is a meta-issue to track the addressing of all TODO(blocks) comments in our codebase:
- [ ] testing/prototest: TestMessage should not take in MessageOptions
- [ ] internal/impl: extension descriptor should check for exact message descriptor match
- [ ] reflect/protoreflect: add List.AppendMutable method
- [ ] reflect/protoreflect: add Map.Mutable method
- [ ] reflect/protoregistry: add URL for how to address registration conflicts
created time in a day
PR closed golang/protobuf
Open to suggestions on the error name. Fixes #1037
pr closed time in a day
pull request commentgolang/protobuf
Add typed error for non-linked in Any message, fixes #1037
Per https://github.com/golang/protobuf/issues/1037#issuecomment-587096838, we are no longer accept feature changes to v1.
comment created time in a day
issue commentgolang/protobuf
Typed error for non-linked in Any message
This is a reasonable feature request, but we are no longer accepting changes to the v1 API. The v2 protobuf API provides this feature through the protoregistry.NotFound error value. We're focusing all our efforts on getting v2 out the door than trying to maintain v1. A stable v2 release is slated for the next week or two.
comment created time in a day
issue commentgolang/protobuf
APIv2: release coordination with google.golang.org/genproto
To ensure users use a sufficient new version of genproto once fieldmask is moved over we need to do one of the following:
- Have a dependency on
genproto. This is unfortunately, because it means that the transitive tree ofgenprotodependencies become our dependencies. :( - Add a hack to
protoregistryto check for duplicate registrations of fieldmask. If so, we panic at init-time with a warning to users to upgrade to a newer version ofgenproto.
comment created time in 4 days
push eventgolang/protobuf
commit sha 91b2604634a35b1a566b42e432c235d9e83cf3cb
encoding: re-arrange options Move Multiline and Indent to the top so that there is a separation between options with semantic significance and those that are merely for aesthetic purposes. Change-Id: Icd5ee94ec010db8139a5e720f5b9842274fb3755 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219500 Reviewed-by: Damien Neil <dneil@google.com>
push time in 4 days
push eventgolang/protobuf
commit sha 5b335f7203acd0d523c9b130df6bf0b5599d169d
reflect/protoreflect: update documentation There were a few stale references due to renamed identifiers. Change-Id: I0ac7eda93c46143292799c3806214dc89f1e80cd Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219504 Reviewed-by: Damien Neil <dneil@google.com>
push time in 4 days
push eventgolang/protobuf
commit sha e8e8875f940670046700d644ebe7d055724ab761
proto, runtime/protoiface, internal/impl: add fast-path Merge Comparing -tags=protoreflect to fast-path: name old time/op new time/op delta pkg:google.golang.org/protobuf/internal/benchmarks goos:linux goarch:amd64 /Clone/google_message1_proto2-12 1.70µs ± 1% 0.30µs ± 1% -82.64% (p=0.001 n=7+7) /Clone/google_message1_proto3-12 1.01µs ± 1% 0.19µs ± 1% -80.77% (p=0.000 n=7+8) /Clone/google_message2-12 818µs ± 8% 141µs ± 6% -82.78% (p=0.000 n=8+8) pkg:google.golang.org/protobuf/internal/benchmarks/micro goos:linux goarch:amd64 EmptyMessage/Clone-12 51.1ns ± 1% 39.3ns ± 3% -23.03% (p=0.000 n=7+8) RepeatedInt32/Clone-12 24.5µs ± 1% 1.1µs ± 3% -95.64% (p=0.000 n=8+8) Required/Clone-12 978ns ± 1% 132ns ± 2% -86.46% (p=0.000 n=8+8) name old alloc/op new alloc/op delta pkg:google.golang.org/protobuf/internal/benchmarks goos:linux goarch:amd64 /Clone/google_message1_proto2-12 1.08kB ± 0% 0.74kB ± 0% -31.85% (p=0.000 n=8+8) /Clone/google_message1_proto3-12 872B ± 0% 544B ± 0% -37.61% (p=0.000 n=8+8) /Clone/google_message2-12 602kB ± 0% 411kB ± 0% -31.65% (p=0.000 n=8+8) pkg:google.golang.org/protobuf/internal/benchmarks/micro goos:linux goarch:amd64 EmptyMessage/Clone-12 96.0B ± 0% 64.0B ± 0% -33.33% (p=0.000 n=8+8) RepeatedInt32/Clone-12 25.4kB ± 0% 3.2kB ± 0% -87.33% (p=0.000 n=8+8) Required/Clone-12 416B ± 0% 256B ± 0% -38.46% (p=0.000 n=8+8) name old allocs/op new allocs/op delta pkg:google.golang.org/protobuf/internal/benchmarks goos:linux goarch:amd64 /Clone/google_message1_proto2-12 52.0 ± 0% 21.0 ± 0% -59.62% (p=0.000 n=8+8) /Clone/google_message1_proto3-12 33.0 ± 0% 3.0 ± 0% -90.91% (p=0.000 n=8+8) /Clone/google_message2-12 22.3k ± 0% 7.5k ± 0% -66.41% (p=0.000 n=8+8) pkg:google.golang.org/protobuf/internal/benchmarks/micro goos:linux goarch:amd64 EmptyMessage/Clone-12 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=8+8) RepeatedInt32/Clone-12 1.51k ± 0% 0.00k ± 0% -99.80% (p=0.000 n=8+8) Required/Clone-12 51.0 ± 0% 18.0 ± 0% -64.71% (p=0.000 n=8+8) Change-Id: Ife9018097c34cb025dc9c4fdd9a61b2f947853c6 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219147 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 4 days
push eventgolang/protobuf
commit sha 7059483cfdd60e17f54504b0a985b9f7e9b71ad7
all: update go.mod in submodules Change-Id: Ibeabaecd3ba55b03dea554fc9fd28f647e9c66fd Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219398 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 4 days
push eventgolang/protobuf
commit sha 06e51b76b48e871f46db99a22166af58da41222c
internal/weakdeps: put APIv1 weak dependency behind a build constraint By using a build constraint that is never satisfied, we can add a weak dependency on a module. If the module is part of a build, our go.mod enforces a minimum version on it, but we never add it to the build dependencies ourselves. This is the same trick used for tool dependencies: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module Dropped the TODO to remove the APIv1 dependency, since I think this removes any need to do so. Change-Id: I45b1a3f45535bcdc9abf34fb562d2869f1712bb6 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219499 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 4 days
push eventgolang/protobuf
commit sha b08bc6ee84ab8c0d66b033d20e814446278c11e8
proto: minor doc changes Change-Id: I9e74b6b68a0bfe96381f224df7c56e49795d17d6 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219501 Reviewed-by: Damien Neil <dneil@google.com>
commit sha 5ca8f849d36bf85cbf114a0b1aa64c6dbaa467e3
reflect/protoregistry: minor doc change Clarify what the return bool of the Range callback does. Change-Id: Ib584e48df9fa1c8053652cd4105b343316729719 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219503 Reviewed-by: Damien Neil <dneil@google.com>
push time in 4 days
push eventgolang/protobuf
commit sha d7b9f5c09351b4f94d9f43c672716e42d4b27ce3
proto: document the relationship between v1 and v2 messages Change-Id: I1da929ce1d4e44adce9ef406cfd937973d2a8cc6 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219139 Reviewed-by: Damien Neil <dneil@google.com>
commit sha 6320bdfa7e23d2bc50133edda4cabb566b9055e6
types/dynamicpb: minor doc change Change-Id: I13439b810e18142d85f4466be006d3abf3a66c30 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219502 Reviewed-by: Damien Neil <dneil@google.com>
push time in 4 days
issue closedgolang/protobuf
APIv2: conversion functions between v1/v2 Message types
We need a simple way to turn a protoV1.Message into a protoV2.Message, at the least. The impl package has a function which does this already, so it's hopefully just a matter of providing an exported API somewhere.
closed time in 5 days
neild
push eventgolang/protobuf
commit sha aa735f3ccadf6250925abd4c99e7c02b88f8fd41
internal/testprotos: add missing go_package option Change-Id: If28f15266349b2c05c209202c38bc98dc43f7dbd Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219297 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 5 days
push eventgolang/protobuf
commit sha baf64d5536c1219775e356cc42c08346524e8e68
all: depend on github.com/golang/protobuf@1.4.0-rc1 While this module has no actual dependency on APIv1, we want to ensure that programs using both APIv1 and APIv2 use a version of APIv1 that wraps APIv2. Updates golang/protobuf#1032. Change-Id: Ie69ce2930c843a7232c79b21ff418c53b28ed1af Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219382 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 5 days
issue commentgolang/protobuf
APIv2: handling of nil map values
To me, if protocol format description document allows it - it is legal. (If protocol allows dropping of required fields - so be it. I guess that's why there are no required fields anymore, but wire encoding did not change - it always allowed it. Encoders were enforcing it.)
The v2 behavior strictly adheres to the protobuf data model. It was a bug in the v1 implementation that transient information not in the protobuf data model was accidentally leaking to/from the wire format.
One more thing - I think you will have to support "bad" maps anyway because...
The Go protobuf compatibility agreement reserves the right to fix bugs. This change in behavior is to fix buggy behavior in v1 to be compliant with the protobuf data model.
That being said, I understand that this change could potentially be problematic for existing data that is written to disk. If so, we would need to understand whether this is a real problem or just a hypothetical. And if real, how often it happens. Supporting the incorrect behavior is costly as well because it causes incompatibility with the other language implementations.
comment created time in 6 days
issue commentgolang/protobuf
APIv2: release coordination with github.com/protocolbuffers/protobuf
Renaming the issue to reflect the fact that we need to update the go_package option in relevant .proto files in the protobuf toolchain. This issue does not block v2 release since v2 needs to be released first, then the protobuf toolchain be updated shortly afterwards.
After doing so, we can remove the hacks in our integration tests to work around this issue: https://github.com/golang/protobuf/blob/709e7c8474b1872537f6062d469dda2899b8f338/integration_test.go#L320-L377
comment created time in 6 days
issue commentgolang/protobuf
APIv2: proto-gen-go links to ptypes any implementation
The Go import path used for any.proto is determined by any.proto itself. This will require a change to the any.proto file in the https://github.com/protocolbuffers/protobuf project. Since that is a different project from this one, it is fundamentally impossible to atomically release both projects simultaneously.
The initial release of APIv2 will still produce Go import paths to the old "github.com/golang/protobuf/ptypes/any" package, and will be fixed when any.proto is updated and a new release of the protoc toolchain is made.
I should note that the old "github.com/golang/protobuf/ptypes/any" package will forward everything to "google.golang.org/protobuf/types/known/anypb", so it won't really matter which package is being imported.
comment created time in 6 days
push eventgolang/protobuf
commit sha 709e7c8474b1872537f6062d469dda2899b8f338
protoreflect: adjust documentation After a header, there needs to be a paragraph in order for godoc to recognize the header as a header. Change-Id: I6c814b24dbb5eb2d5efcb0040556521c05587393 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219140 Reviewed-by: Damien Neil <dneil@google.com>
push time in 6 days
push eventgolang/protobuf
commit sha c7aa53a3e0e697061ca81cc4289ea0dc85be11f1
proto: refactor merge tests Use protobuild to generate test messages, both to simplify generating proto2/proto3/extension variants of each test where appropriate and to make it easier to test alternative message generators in the future. Add various additional merge tests. Change-Id: I4ba3ce232304e1d8325543680e2b6aae61de7364 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219146 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 6 days
push eventgolang/protobuf
commit sha cf33a9a2745f026b6e1675348c4687eb1671bb4d
proto: remove shallow merge support, and MergeOptions We're still debating what the semantics of shallow merges should be, and what the use case for them (if any) is. There's no need to commit ourselves to this feature at this time; drop it for now. Remove the exported MergeOptions type, since the only option is gone. We can trivially add it back in the future if we need it. Change-Id: I4c8697cae69cc66eed1ea65de84b982f20e1be44 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219145 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 6 days
push eventgolang/protobuf
commit sha 316febd1ab5f96d4223979c1c6de8a9c02435c56
internal/impl: pass *coderFieldInfo into fast-path functions Refactor the fast-path size, marshal, unmarshal, and isinit functions to take the *coderFieldInfo for the field as input. This replaces a number of closures capturing field-specific information with functions taking that information as an explicit parameter. Change-Id: I8cb39701265edb7b673f6f04a0152d5f4dbb4d5d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218937 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 6 days
push eventgolang/protobuf
commit sha f26172248d71952940c147431ff5598dea7c882e
proto: add Merge tests for aberrant inputs The generated Go code does not perfectly represent the protobuf data model. As such, it is possible to construct Go message values that could not otherwise be constructed through Unmarshal or Merge. This CL adds a test to exercise some of the boundary conditions. The tests do not necessarily lock in the current behavior, but provides a way for us to know when they have changed (either intentionally or not). Change-Id: I389ef97c2fcf037595ad553baf5270d7bee9673b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196738 Reviewed-by: Damien Neil <dneil@google.com>
push time in 6 days
push eventgolang/protobuf
commit sha e76af4be5e8ecc89b68ca63c581a0290bf7fed2c
proto: document reset memory aliasing guarantees Change-Id: I2dc79c362278b6bc9ccf531067701cba5c392a50 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219141 Reviewed-by: Damien Neil <dneil@google.com>
push time in 6 days
push eventgolang/protobuf
commit sha 0fd14f96108ad68d2571a78c2d8ef5287ee57c43
proto: add MessageV1, MessageV2, and MessageReflect The MessageV1 and MessageV2 functions convert to/from the v1 and v2 message interfaces. The MessageReflect function provides a reflective view over message. These functions do not have an "Of" suffix to be consistent with the existing MessageName and MessageType functions. Furthermore, we drop the "Of" suffix from functions in the descriptor package to be consistent. This is a safe change since none of those functions have seen a stable release. We move the descriptor.GeneratedXXX types to the proto package for documentation purposes. Fixes #956 Change-Id: I566b74367798e2e3399db9902b58ffeb673199ca Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219137 Reviewed-by: Damien Neil <dneil@google.com>
push time in 6 days
push eventgolang/protobuf
commit sha 25fc6fb4f2169332533e95c002b95560df66dbed
reflect/protodesc: add FileOptions The FileOptions type provides the ability to specify specialized options for how a file descriptor is constructed. It follows the same optional arguments pattern as used in the proto package. The resolver is not an option since it almost always necessary when constructing a file descriptor. Change-Id: Ib98ac6289881ad8402dd615f6c895da5899cb8d9 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218940 Reviewed-by: Damien Neil <dneil@google.com>
push time in 7 days
issue commentgolang/protobuf
APIv2: protoc-gen-go: generate ExtensionDescriptor instead of ExtensionType
We decided that this does not block v2 release. We still want to do this, but not at the present moment.
Our rationale is as follows:
- It is already the case that generated code have a
E_MyExtensionvariable that is aprotoreflect.ExtensionType. It is also already the case that users are expected to interact with extensions using theproto.{Get,Set,Has,Clear}Extensionfunctions using that variable. Therefore, the top-level extension functions taking in aprotoreflect.ExtensionTypeis probably the right API. - In the future, we would still want to generate a different variable (e.g.,
MyExtension_field) that implementsprotoreflect.ExtensionTypeDescriptor, but also has methods on it that allows a user to access the extension field in a type-safe way. If we do that, users won't need the top-levelprotofunctions for dealing with extensions, and it won't matter whether they take in aprotoreflect.ExtensionTypeDescriptoror not. See https://go-review.googlesource.com/c/protobuf/+/183337 for an example of how the generated code can directly make experience with extension fields much better.
comment created time in 7 days
issue closedgolang/protobuf
APIv2: generated messages are incompatible with current v1 package
The APIv2 protoc-gen-go is generating code that doesn't work with the current APIv1 proto package, failing with a panic at runtime when passed to a v1 function:
panic: protobuf tag not enough fields in Message.state:
We will have an updated APIv1 package that's compatible with these messages, but that doesn't help the case when someone is using the older package. It's okay for new generated code to require newer package versions, but compatibility failures need to be detected at compile time (or, worst case, init time), not runtime.
closed time in 7 days
neild
issue commentgolang/protobuf
APIv2: generated messages are incompatible with current v1 package
https://go-review.googlesource.com/c/protobuf/+/218939 adds a compile-time assertion on a future v1.4 of github.com/golang/protobuf.
We'll remove this assertion after some period of time, but we think it's probably necessary for the initial release.
comment created time in 7 days
issue commentgolang/protobuf
APIv2: release coordination with google.golang.org/grpc
This still needs to happen, but probably after the first rc1 release.
comment created time in 7 days
issue commentgolang/protobuf
APIv2: release coordination with google.golang.org/genproto
This still needs to happen, but probably after the first rc1 release.
comment created time in 7 days
issue closedgolang/protobuf
APIv2: add protoreflect.Message.ExtensionFields
Currently, there is no efficient way to lookup whether an extension field is populated in a message given only the name or the field number. The best you can do is to loop through all populated fields and look for that field.
Perhaps we should add the following method:
type Message interface {
...
// ExtensionFields returns a list of field descriptors for all extensions fields
// that are populated in the message.
ExtensionFields() ExtensionTypeDescriptors
}
type ExtensionTypeDescriptors interface {
// Len reports the number of fields.
Len() int
// ByName returns the extension descriptor for a field named s.
// It returns nil if not found.
ByName(s FullName) ExtensionTypeDescriptor
// ByNumber returns the extension descriptor for a field numbered n.
// It returns nil if not found.
ByNumber(n FieldNumber) ExtensionTypeDescriptor
// Range iterates over all extension descriptors.
Range(func(ExtensionTypeDescriptor) bool)
}
That allows you to:
- Quickly check whether any extension fields are populated in a message:
m.ExtensionFields().Len() > 0 - Lookup a extension field by name or field number.
- Quickly range over only the extension fields without ranging over all the other populated fields.
EDIT: Added a Range method.
closed time in 7 days
dsnet
issue commentgolang/protobuf
APIv2: add protoreflect.Message.ExtensionFields
We decided against this. It's already a wart that the Go implementation of extensions makes extension type information at the granularity of each concrete message value. This is different from how extensions work in the other major language implementations. This API would accentuate this inconsistency with respect to other languages.
The use-cases that wanted this behavior were actually better off looking at the global registry.
comment created time in 7 days
push eventgolang/protobuf
commit sha d2ece139c61886cffb2ed8812e44ad0886661ff6
encoding/protojson: refactor to follow prototext pattern All unmarshaling error messages now contain line number and column information, except for the following errors: - `unexpected EOF` - `no support for proto1 MessageSets` - `required fields X not set` Changes to internal/encoding/json: - Moved encoding funcs in string.go and number.go into encode.go. - Separated out encoding kind constants from decoding ones. - Renamed file string.go to decode_string.go. - Renamed file number.go to decode_number.go. - Renamed Type struct to Kind. - Renamed Value struct to Token. - Token accessor methods no longer return error. Name, Bool, ParsedString will panic if called on the wrong kind. Float, Int, Uint has ok bool result to check against. - Changed Peek to return Token and error. Changes to encoding/protojson: - Updated internal/encoding/json API calls. - Added line info on most unmarshaling error messages and kept description simple and consistent. Change-Id: Ie50456694f2214c5c4fafd2c9b9239680da0deec Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218978 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 7 days
push eventgolang/protobuf
commit sha 3b512245dc29bb790d2d75d3199ea653d5ef2c07
cmd/protoc-gen-go: add compile-time assertion for legacy proto package version Change-Id: I2bc71dae34b5af379838239210cc04e3e3547d2b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218939 Reviewed-by: Damien Neil <dneil@google.com>
push time in 8 days
push eventgolang/protobuf
commit sha f5a698d60c203675919c762fb7fdec4e7d5bd138
proto: add ProtoPackageIsVersion4 Change-Id: I73a3da6d46e82c4c996ad9360e5b602efbbbcc83 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218938 Reviewed-by: Damien Neil <dneil@google.com>
push time in 8 days
push eventgolang/protobuf
commit sha 93bccf763ed3bb7b73829a600f447b078d99af1c
all: scrub all TODOs TODOs that we do not intend to address have been deleted. Those that are blocking v2 release are marked with "blocks". Change-Id: I7efa9e546d0637b562101d0edc7009893d762722 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218878 Reviewed-by: Damien Neil <dneil@google.com>
push time in 8 days
issue closedgolang/protobuf
APIv2: consider Message.IsValidValue
Should we have a way to test a Value for compatibility with a message field, along the lines of ExtensionType.IsValidValue?
This could either be a method on MessageType or Message.
closed time in 8 days
neild
issue commentgolang/protobuf
APIv2: consider Message.IsValidValue
Nice to have, but we both agreed that the use cases didn't seem all that compelling.
comment created time in 8 days
issue commentgolang/protobuf
APIv2: conversion functions between v1/v2 Message types
We decided to put MessageV1Of, MessageV2Of, and ReflectMessageOf in the v1 proto package.
comment created time in 8 days
issue commentgolang/protobuf
APIv2: re-implement v1 in terms of the v2 API
This is done.
comment created time in 8 days
issue closedgolang/protobuf
APIv2: re-implement v1 in terms of the v2 API
We need to re-implement the following in v1 to forward to v2:
- Wire marshaling
- Wire unmarshaling
- JSON marshaling
- JSON unmarshaling
- Text marshaling
- Text unmarshaling
- Registration
Some parts of this are already done: https://github.com/golang/protobuf/tree/api-v1/proto
closed time in 8 days
dsnet
push eventgolang/protobuf
commit sha 604cdd2d2a5734ba1778f5977183f946028ab492
proto: add package docs Change-Id: I7c0b1e3a3b6cc1830b8e39e0ba66f1e4c06b7d12 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218621 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 10 days
push eventgolang/protobuf
commit sha ee206b9211ec487ce811c8bf0b65c8908778a88a
proto: add tests for groups in oneofs Fixes golang/protobuf#1000. Change-Id: I9904f6496240c544b0190a8a1bc3e6d067f98f82 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218581 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 11 days
push eventgolang/protobuf
commit sha 01b51b4f96e6f04345af6153c1b69345f8e075b9
proto, internal/errors: add Error sentinel, errors.Wrap Add a sentinel proto.Error error which matches all errors returned by packages in this module. Document that protoregistry.NotFound is an exact sentinel value for performance reasons. Add a Wrap function to the internal/errors package and use it to wrap errors from outside sources (resolvers). Wrapped errors match proto.Error. Fixes golang/protobuf#1021. Change-Id: I45567df3fd6c8dc9a5caafdb55654827f6fb1941 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215338 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 11 days
push eventgolang/protobuf
commit sha f9d4fdf0549357f6108db511aa5ff5b93a3c0b2f
internal/impl: fix validation of required group fields Change-Id: I3c3b5cfbea599dc08096aa5992b7829c2e50f25d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218578 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha 9afe9bb78b4f5e9015dd310c03875443c14125ec
internal/impl: validate messagesets Change-Id: Id90bb386e7481bb9dee5a07889f308f1e1810825 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218438 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 11 days
push eventgolang/protobuf
commit sha 952a08d7c4752cd613ac312b66304dbd6f1af1d5
encoding/prototext: make unexpected EOF error into proto.Error Also fixed/added comments on exported vars/funcs. Change-Id: I6c42b2afb90058e026a5310598bb3ebfcd01b989 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218357 Reviewed-by: Damien Neil <dneil@google.com>
push time in 11 days
issue commentgolang/protobuf
APIv2: define structured errors
I think we're mixing two things. The use of %w requires go1.12 at minimum, but the use of error wrapping is something we can support today since it only requires satisfaction of certain interfaces.
comment created time in 11 days
issue closedgolang/protobuf
jsonpb: allow byte array encoding as hex
Is your feature request related to a problem? Please describe. Internally we use Protobufs as our serialization structure. We would also like to archive this data in a form that is human readable for debugging/data analysis purposes. We wanted to simply marshal our protos into JSON, for the most part this is perfect, however for all of the binary fields in our protos it would be much more usable to us if we could see it as hex instead of base64.
Describe the solution you'd like
Expose an option on the jsonpb.Marshaller struct that allows the user to choose how binary data is encoded.
Describe alternatives you've considered
- Write our own marshaller/unmarshaller into a human readable format. This would be a significant amount of work considering the jsonpb marshaller is almost perfect.
- Fork this repo and add the feature ourselves. This would mean we would have to manage a fork for a fairly small change.
- Hackily unmarshal the returned jsonpb value into a generic map and scrape every field, converting all fields that look like base64 into hex. This can be error prone, there are workarounds to get more accurate but involve significant boilerplate and are still very hacky.
Additional context The data we archive is first dumped to S3 and then moved to a cloud data platform. We could do the data transformation during this export process, however not all data we archive to S3 is exported. It can be bothersome to modify the data that is only in S3 when viewing it for debugging.
closed time in 11 days
RobertMic
issue commentgolang/protobuf
jsonpb: allow byte array encoding as hex
Hi, thanks for the feature request! Handling binary data as hexadecimal rather than base64 is a reasonable expectation, but it's not what the protobuf<->JSON specification says to do as it currently says to use base64.
In the golang/protobuf project, we strictly adhere to implementing protobufs as specified by the main protobuf project and avoid any features that are specific to Go.
I recommend filing this request in https://github.com/protocolbuffers/protobuf for the general JSON specification to support this. Without official blessing of such a feature, we can't add it.
comment created time in 11 days
push eventgolang/protobuf
commit sha 6fb29949b8126d606d76d19ed3e1d9120c2577ec
all: tests, tweaks for lazy extension decoding Add a test to confirm that extensions are lazily decoded when we expect. Drop the UnmarshalDefaultResolver flag. I added it thinking for some reason that internal/impl couldn't depend on protoregistry; since it can (and does), it's simpler to just test if the resolver is the expected value. Use a default set of options when lazily unmarshaling extensions. Change-Id: Ied7666ffdc3bf90630260a80c9568d9a945048bc Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218038 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 12 days
push eventgolang/protobuf
commit sha 4e6b903e61af140281d3f72c13d5710ce980d644
internal/encoding/text: fix eof crash when parsing list of scalars Need to check for EOF and return proper error. Bug caught by fuzz test: https://oss-fuzz.com/testcase-detail/6258064955277312. Change-Id: I63d5c12c301f2ddefc9a0813c13abef40d745e91 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218258 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 12 days
push eventgolang/protobuf
commit sha 4eefd77886311f75601b5f4078580e2bfcb621b0
internal/impl: init map value MessageInfos in validator I'm not sure how to write a good test for this one, since it's so specific to both the code and the ordering of initialization. Just sticking the fuzzer-provided case into our standard test message set doesn't do it, because something else has initialized the MessageInfo by the time the test gets there. Fixes: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=20543 Change-Id: I508222b43e52287f73e2ed32ce9b954a5f81717b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218257 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 12 days
push eventgolang/protobuf
commit sha 2eb18f0e62b138c7471b7a4831fb9e3497e23c90
internal/encoding/text: fix error construction in parseTypeName Fuzz test caught the following issue -- https://oss-fuzz.com/testcase-detail/6288731021770752 Change-Id: Idcbce7953b465d1b83c01b1d123c9d43907d402a Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218037 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 13 days
push eventgolang/protobuf
commit sha 4eeab367abd814d79e0be9fc576b0b7757747557
internal/fuzztest: set cap to len on test byte slice Change-Id: I807995997fe7addf005eb639b0d4843cfb58f52b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218018 Reviewed-by: Herbie Ong <herbie@google.com>
commit sha 9dd7148ccd3fde5cddae6e63af037a4bacc1e5ba
internal/fuzz: add oss-fuzz build script Move the build script from OSS-Fuzz's repo into ours, allowing us to make changes without sending them a PR. Change-Id: I557c3be2b6d9fd221ac7e6b1331bf3d53fd3ca51 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217768 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
commit sha 40cba14b2608da2a4fc8d64c5e8412e9670f38e2
internal/impl: fix for lazy decoding of groups Bit of a weird case in why this wasn't caught by tests: When validating extension groups, we were validating an empty buffer rather than the message content. For groups, this validation always fails due to a lack of a group end tag. We'd then skip lazy decoding of the extension field and proceed with eager decoding, which would behave correctly. Change extension validation to report an error immediately on an invalid result from the validator, which is both safe (assuming we trust the validator) and would have caught this problem (by failing to decode the extension field, rather than silently failing to eager decoding). Change-Id: Id6c2d21fb687062bc74d9eb93760a1c24a6fe883 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217767 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 13 days
Pull request review commentgoogle/cel-spec
Handle nested Any comparisons in simple tests.
func MatchValue(tag string, expected *exprpb.Value, actual *exprpb.Value) error } default: // By default, just compare the protos.- if !proto.Equal(expected, actual) {- return fmt.Errorf("%s: Eval got [%v], want [%v]", tag, actual, expected)+ // Compare the canonical string marshaling which is closer+ // to protobuf equality semantics than proto.Equal:+ // - properly compares Any messages, which might be+ // equivalent even with different byte encodings;+ // - surfaces sign differences for floating-point zero.+ // Text marshaling isn't documented as deterministic,+ // but it appears to be so in practice.
Text serialization is deterministic (i.e. produces the same bytes for the same message within the same program), so this hack works. It is definitely not stable though (i.e., produces the same bytes for the same message for all time).
Longer term, you'll want to use the v2 protocmp library, which is designed for comparing protobuf messages in tests:
cmp.Equal(want, got, protocmp.Transform())
comment created time in 13 days
issue commentgolang/go
proposal: encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions
To clarify, are you suggesting that the type-override registration be a top-level Register function or are you merely suggesting that the Decoder and Encoder methods be named Register out of consistency with gob.Register?
comment created time in 13 days
push eventgolang/protobuf
commit sha 0f783d864bdddb741841818ecda4f126bd76dac1
internal/impl: fix off-by-one in varint validation Fixes: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=20532 Change-Id: I670698a1ef780f341f336929384132febe2b40a1 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217766 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 13 days
push eventgolang/protobuf
commit sha cadb4ab3b1c6d76d0e3c08bd0f0b35440342b367
internal/impl: refactor validation a bit Return the size of the field read from the validator, permitting us to avoid an extra parse when skipping over groups. Return an UnmarshalOutput from the validator, since it already combines two of the validator outputs: bytes read and initialization status. Remove initialization status from the ValidationStatus enum, since it's covered by the UnmarshalOutput. Change-Id: I3e684c45d15aa1992d8dc3bde0f608880d34a94b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217763 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 14 days
push eventgolang/protobuf
commit sha 9b3d97c4736df34651b938ebe7b85ca85b3e4724
encoding/prototext: rewrite of internal/encoding/text * Fixes golang/protobuf#842. Unmarshal can now parse singular or repeated message fields without the field separator. * Fixes golang/protobuf#1011. Handles negative 0 properly. * For unknown fields with fixed 32-bit and 64-bit wire types, output is now in hex format with 0x prefix similar to C++ lib output. Previous Go implementation simply outputs these as decimal numbers %d. * All parsing errors, except for unexpected EOF should now contain line and column number info. * Fixed following conformance-related features: * Parse nan,inf,-inf,infinity,-infinity as case-insensitive. * Interpret float32 overflows as inf or -inf. * Parse large int-like number as proto float. * Discard unknown map field if DiscardUnknown=true. * Allow whitespaces/comments in Any type URL and extension field names per spec. * Improves performance and memory usage. It is now as fast and efficient as protojson, if not better on most benchmarks. name old time/op new time/op delta Text/Unmarshal/google_message1_proto2-4 14.1µs ±43% 8.7µs ±12% -38.27% (p=0.000 n=10+10) Text/Unmarshal/google_message1_proto3-4 11.6µs ±18% 7.7µs ± 9% -33.69% (p=0.000 n=10+10) Text/Unmarshal/google_message2-4 6.20ms ±27% 4.10ms ± 5% -33.95% (p=0.000 n=10+10) Text/Marshal/google_message1_proto2-4 12.8µs ± 6% 10.3µs ±23% -19.54% (p=0.000 n=9+10) Text/Marshal/google_message1_proto3-4 11.9µs ±16% 8.6µs ±10% -27.45% (p=0.000 n=10+10) Text/Marshal/google_message2-4 5.59ms ± 5% 5.30ms ±22% ~ (p=0.356 n=9+10) JSON/Unmarshal/google_message1_proto2-4 12.3µs ±61% 13.9µs ±26% ~ (p=0.190 n=10+10) JSON/Unmarshal/google_message1_proto3-4 7.51µs ± 6% 7.86µs ± 1% +4.66% (p=0.010 n=10+9) JSON/Unmarshal/google_message2-4 3.74ms ± 2% 3.94ms ± 2% +5.32% (p=0.000 n=10+10) JSON/Marshal/google_message1_proto2-4 9.90µs ±12% 9.95µs ± 4% ~ (p=0.315 n=9+10) JSON/Marshal/google_message1_proto3-4 7.55µs ± 4% 7.93µs ± 3% +4.98% (p=0.000 n=10+10) JSON/Marshal/google_message2-4 4.29ms ± 5% 4.49ms ± 2% +4.53% (p=0.001 n=10+10) name old alloc/op new alloc/op delta Text/Unmarshal/google_message1_proto2-4 12.5kB ± 0% 2.0kB ± 0% -83.87% (p=0.000 n=10+10) Text/Unmarshal/google_message1_proto3-4 12.2kB ± 0% 1.8kB ± 0% -85.33% (p=0.000 n=10+10) Text/Unmarshal/google_message2-4 5.35MB ± 0% 0.89MB ± 0% -83.28% (p=0.000 n=10+9) Text/Marshal/google_message1_proto2-4 12.0kB ± 0% 1.4kB ± 0% -88.15% (p=0.000 n=10+10) Text/Marshal/google_message1_proto3-4 12.4kB ± 0% 1.9kB ± 0% -84.91% (p=0.000 n=10+10) Text/Marshal/google_message2-4 5.64MB ± 0% 1.02MB ± 0% -81.85% (p=0.000 n=10+9) JSON/Unmarshal/google_message1_proto2-4 2.29kB ± 0% 2.29kB ± 0% ~ (all equal) JSON/Unmarshal/google_message1_proto3-4 2.08kB ± 0% 2.08kB ± 0% ~ (all equal) JSON/Unmarshal/google_message2-4 899kB ± 0% 899kB ± 0% ~ (p=1.000 n=10+10) JSON/Marshal/google_message1_proto2-4 1.46kB ± 0% 1.46kB ± 0% ~ (all equal) JSON/Marshal/google_message1_proto3-4 1.36kB ± 0% 1.36kB ± 0% ~ (all equal) JSON/Marshal/google_message2-4 1.19MB ± 0% 1.19MB ± 0% ~ (p=0.197 n=10+10) name old allocs/op new allocs/op delta Text/Unmarshal/google_message1_proto2-4 133 ± 0% 89 ± 0% -33.08% (p=0.000 n=10+10) Text/Unmarshal/google_message1_proto3-4 108 ± 0% 67 ± 0% -37.96% (p=0.000 n=10+10) Text/Unmarshal/google_message2-4 60.0k ± 0% 38.7k ± 0% -35.52% (p=0.000 n=10+10) Text/Marshal/google_message1_proto2-4 65.0 ± 0% 25.0 ± 0% -61.54% (p=0.000 n=10+10) Text/Marshal/google_message1_proto3-4 59.0 ± 0% 22.0 ± 0% -62.71% (p=0.000 n=10+10) Text/Marshal/google_message2-4 27.4k ± 0% 7.3k ± 0% -73.39% (p=0.000 n=10+10) JSON/Unmarshal/google_message1_proto2-4 95.0 ± 0% 95.0 ± 0% ~ (all equal) JSON/Unmarshal/google_message1_proto3-4 74.0 ± 0% 74.0 ± 0% ~ (all equal) JSON/Unmarshal/google_message2-4 36.3k ± 0% 36.3k ± 0% ~ (all equal) JSON/Marshal/google_message1_proto2-4 27.0 ± 0% 27.0 ± 0% ~ (all equal) JSON/Marshal/google_message1_proto3-4 30.0 ± 0% 30.0 ± 0% ~ (all equal) JSON/Marshal/google_message2-4 11.3k ± 0% 11.3k ± 0% ~ (p=1.000 n=10+10) Change-Id: I377925facde5535f06333b6f25e9c9b358dc062f Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204602 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 14 days
push eventgolang/protobuf
commit sha 521f2a0475fd742a42cea92f934bf1509d1a14f7
internal/filedesc: remove dependency on proto file name Look up a ProtoFile via a message's ParentFile method. Makes it easier to run the test in environments (blaze, bazel) where the file path may have changed. Change-Id: I824f8412ef7db8299961e8df6edac14a13f3e263 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217761 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha 3c5fb5f87920cfc5d1510867c2be0855a6bbcaa4
all: make .proto file names relative to module root Change the protoc flags such that when one of our test .proto files imports another, the filename is consistently specified relative to the module root. Change-Id: I690282795cef23347c8794c1c6357e4fe9560d8a Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217762 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 14 days
push eventgolang/protobuf
commit sha 1c33e1125a5f1e2500ced9568c31ffad13d752ec
proto: make one test more general Tweak the "nested unknown extension" test case's resolver to not depend on the exact message being tested. Useful for if/when we want to run these tests on other message implementations. Change-Id: Id1722afd8e094ddb59cb3e5440f7994c20cfa681 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217760 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 14 days
issue closedgolang/protobuf
TestGolden fails w/ Go 1.14 beta1
What version of protobuf and what language are you using? 1.3.3, go 1.14 beta 1, Fedora Rawhide
What did you do? Run tests
What did you expect to see? No failure.
What did you see instead?
Testing in: /builddir/build/BUILD/protobuf-1.3.3/_build/src
PATH: /builddir/build/BUILD/protobuf-1.3.3/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
GOPATH: /builddir/build/BUILD/protobuf-1.3.3/_build:/usr/share/gocode
GO111MODULE: off
command: go test -buildmode pie -compiler gc -ldflags "-X github.com/golang/protobuf/version=1.3.3 -extldflags '-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '"
testing: github.com/golang/protobuf
github.com/golang/protobuf/jsonpb
PASS
ok github.com/golang/protobuf/jsonpb 0.016s
github.com/golang/protobuf/proto
--- Dump of b ---
0: t= 1 bytes [7] 08 01 12 .. 52 6f 62
9: t= 1 bytes [7] 08 04 12 .. 49 61 6e
18: t= 1 bytes [8] 08 08 12 .. 61 76 65
2020/02/04 18:33:47 proto: don't know how to compare extension 200 of test_proto.MyMessage
PASS
ok github.com/golang/protobuf/proto 0.054s
github.com/golang/protobuf/protoc-gen-go
--- FAIL: TestGolden (0.39s)
golden_test.go:346: RUNNING: protoc --plugin=protoc-gen-go=/tmp/go-build385470712/b001/protoc-gen-go.test -Itestdata --go_out=plugins=grpc,paths=source_relative:/tmp/proto-test639569157 testdata/my_test/test.proto
golden_test.go:349: my_test/test.proto:40:1: warning: Import multi/multi1.proto is unused.
golden_test.go:112: golden file differs: issue780_oneof_conflict/test.pb.go
--- testdata/issue780_oneof_conflict/test.pb.go 2020-01-28 18:10:22.000000000 +0000
+++ /tmp/proto-test639569157/issue780_oneof_conflict/test.pb.go 2020-02-04 18:33:49.848538575 +0000
@@ -89,7 +89,9 @@
proto.RegisterType((*Foo)(nil), "oneoftest.Foo")
}
-func init() { proto.RegisterFile("issue780_oneof_conflict/test.proto", fileDescriptor_48462cafc802a68e) }
+func init() {
+ proto.RegisterFile("issue780_oneof_conflict/test.proto", fileDescriptor_48462cafc802a68e)
+}
var fileDescriptor_48462cafc802a68e = []byte{
// 107 bytes of a gzipped FileDescriptorProto
FAIL
exit status 1
FAIL github.com/golang/protobuf/protoc-gen-go 0.636s
closed time in 14 days
eclipseo
issue commentgolang/protobuf
TestGolden fails w/ Go 1.14 beta1
Thanks for the report, but these tests will be gone soon. In fact they have already been removed in the api-v1 branch.
comment created time in 14 days
push eventgolang/protobuf
commit sha f68f17085ab19c624b417a0973014d0e0e6f2ce0
internal/testprotos: minor .proto file fixes Add a missing syntax and package statement. Change-Id: I3c95aa055ad84fa0bbb2bbdd341b51df9359f499 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217759 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 14 days
push eventgolang/protobuf
commit sha 886c32637fb557f6901c5c0ba0b59dca46a6dc14
internal/encoding/json: add tests for negative zeros. Updates golang/protobuf#1011. Copied tests from http://golang.org/cl/217501. Change-Id: I58ea1111beccee9691929b062eb87a1a752f81e0 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217578 Reviewed-by: Damien Neil <dneil@google.com>
push time in 15 days
push eventgolang/protobuf
commit sha d025c951108d83677b27bc83f99c8d8c78f5a6d5
proto, internal/protobuild: add test proto template builder The proto package tests often test several variations of messages with a similar shape. For example, most tests are performed with a proto2 message with a regular field, a proto2 message with an extension field, and a proto3 message. Add a protobuild package which can initialize all these variations from a single template. For example, these three messages: &testpb.TestAllTypes{OptionalInt32: proto.Int32(1)} &test3pb.TestAllTypes{OptionalInt32: 1} m := &testpb.TestAllExtensions{} proto.SetExtension(m, &testpb.E_OptionalInt32, 1) can all be constructed from the template: protobuild.Message{"optional_int32": 1} This reduces redundancy in tests and will make it more practical to test alternative code generators. Change-Id: I3245a4bf74ee1bce957bc772fed513d427720677 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217457 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 15 days
push eventgolang/protobuf
commit sha 4d918167a9bc88cc4e0075853bdc776683c0d043
internal/impl: catch varint overflow in validator Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=20477 Change-Id: I6afe82e3818f8b4e9cf5eded2125317eae8be49d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217309 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 15 days
issue commentgolang/protobuf
Support for architecture to cross-compile
Hi, I'm not entirely sure what you're are expecting us to do. Is this about the generated code or the protobuf runtime library? I should note that none of the protobuf code makes assumptions about the specific architecture that its built for (we do use unsafe, but in architecture agnostic ways).
comment created time in 17 days
issue commentgoogle/go-cmp
How to apply cmpopts.EquateEmpty() recursively?
Hi, sorry to hear about your troubles. Unfortunately, I'm not able to replicate the issue: https://play.golang.org/p/Zw00duqujKn. It seems to print true for me.
comment created time in 18 days
issue commentgolang/go
proposal: encoding/json: allow per-Encoder/per-Decoder registration of marshal/unmarshal functions
Why is it RegisterFunc and not Register?
Could be either. I don't feel strong about this name.
comment created time in 18 days
issue commentgolang/protobuf
protoc-gen-go: emit versioning information
This is already resolved in the v2 branch. The v2 release is targeting end of Q1.
comment created time in 18 days
issue closedgolang/protobuf
The v1 API provides the following functions for text (un)marshaling:
func MarshalText(w io.Writer, pb protoiface.MessageV1) error {}
func MarshalTextString(pb protoiface.MessageV1) string {}
func UnmarshalText(s string, pb protoiface.MessageV1) error {}
func (*TextMarshaler) Marshal(w io.Writer, pb protoiface.MessageV1) error {}
func (*TextMarshaler) Text(pb protoiface.MessageV1) string {}
The new API currently contains:
func Marshal(m protoreflect.ProtoMessage) ([]byte, error) {}
func Unmarshal(m protoreflect.ProtoMessage, b []byte) error {}
func (MarshalOptions) Marshal(m protoreflect.ProtoMessage) ([]byte, error) {}
func (UnmarshalOptions) Unmarshal(m protoreflect.ProtoMessage, b []byte) error {}
I think there are some places where the ergonomics of the new API could be improved:
-
There should be an easy way to convert a message into a string. Compare:
fmt.Println(proto.MarshalTextString(m))b, _ := textpb.Marshal(m) // And what do I do if there's an error? fmt.Println(string(b)) -
There should be a default indentation level for multi-line output. Currently, the default is compact output, with multi-line output specified by passing an indentation string in via
MarshalOptions. Since the user is not only allowed but required to choose their own indentation level, this will result in a lot of inconsistent output. I think we should default to multi-line output (which is vastly more readable for messages of any size) with a default indentation level, with compact output opted into via aCompactbool onMarshalOptions. -
Unmarshal's first parameter is the message to marshal into. While this does follow the usual Go parameter ordering of (destination, source), it is inconsistent with other every other function namedUnmarshalI can find:
closed time in 18 days
neild
issue closedgolang/protobuf
Regression in handling of 0 values for google.protobuf.Int64Value types (and maybe others)
What version of protobuf and what language are you using? Version: Ruby 3.11.2
What did you do? Simplified/Minimal Example.
message Wrapper {
google.protobuf.Int64Value id = 1;
}
Wrapper.decode(Wrapper.new( id: Google::Protobuf::Int64Value.new( value: 0) ).to_proto)
OUTOUT <Wrapper: id: nil>
What did you expect to see?
<Wrapper: id: <Google::Protobuf::Int64Value: value: 0>>
What did you see instead?
<Wrapper: id: nil>
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
Anything else we should know about your project / environment?
This works on version 3.7.1 of the gem as expected. This is a very dangerous bug in my opinion, as it can break applications...
closed time in 19 days
shidel-dev
issue commentgolang/protobuf
Regression in handling of 0 values for google.protobuf.Int64Value types (and maybe others)
This is the repository for the Go implementation of protobufs, I believe you want to file this at https://github.com/protocolbuffers/protobuf instead. We don't maintain the Ruby implementation.
comment created time in 19 days
push eventgolang/protobuf
commit sha 74b1460c5b521ae9e54fe8e558a34017bd66d584
encoding: add Format helper function and method The Format function and MarshalOptions.Format method are helper functions for directly obtaining the formatted string for a message without having to deal with errors or convert a []byte to string. It is only intended for human consumption (e.g., debugging or logging). We also add a MarshalOptions.Multiline option to specify that the output should use some default indentation in a multiline output. This assists in the v1 to v2 migration where: protoV1.CompactTextString(m) => prototext.MarshalOptions{}.Format(m) protoV1.MarshalTextString(m) => prototext.Format(m) At Google, there are approximately 10x more usages of MarshalTextString than CompactTextString, so it makes sense that the top-level Format function does multiline expansion by default. Fixes #850 Change-Id: I149c9e190a6d99b985d3884df675499a3313e9b3 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213460 Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Herbie Ong <herbie@google.com>
push time in 20 days
push eventgolang/protobuf
commit sha 1887ff702c132e0d4e4ebfb5c1bb093d0f1d8809
internal/impl: better fast-path init checks for extensions Unknown extensions are initialized. Valid extensions with no isInit func are initialized. Change-Id: I2975c7ef85d2b777eca467d3b1861d20de8e24fc Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216960 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 20 days
push eventgolang/protobuf
commit sha 6f2977906dc22db9d90e4ff5dd008b32ecec7cf6
internal/impl: fix validator bytes field length decoding Missing a bounds check on the first byte. Change-Id: I089fa8dcc1a14d11faca1acba758b6b811b16ac4 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216957 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 20 days
push eventgolang/protobuf
commit sha ca6f40c8805ac1ec4c94afccfc99a3f539dbd7f4
proto: make use of fast-path initialization checks Lost the check for the fast-path's Initialized output somewhere. Put it back in. name old time/op new time/op delta Required/Wire/Unmarshal-12 37.8ns ± 1% 30.5ns ± 1% -19.36% (p=0.000 n=8+8) Change-Id: Ica733366d00efba41023339a2bbd68167ab0df53 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216897 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
commit sha c70f5d59d1d7dd75d0850f4ac329c5d77a5db39f
internal/impl: avoid redundant lazy extension inits After taking the lock on a lazy extension's state, check to see if it was initialized while we were waiting for the lock. Change-Id: I1cbd52e9d655eec6c9142c97689ae36f219a28f2 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216898 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 20 days
push eventgolang/protobuf
commit sha 0ae1c9789a684aa8955cb295a56c556e7a1c4525
internal/impl: lazy extension decoding Historically, extensions have been placed in the unknown fields section of the unmarshaled message and decoded lazily on demand. The current unmarshal implementation decodes extensions eagerly at unmarshal time, permitting errors to be immediately reported and correctly detecting unset required fields in extension values. Add support for validated lazy extension decoding, where the extension value is fully validated at initial unmarshal time but the fully unmarshaled message is only created lazily. Make this behavior conditional on the protolegacy flag for now. Change-Id: I9d742496a4bd4dafea83fca8619cd6e8d7e65bc3 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216764 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 20 days
push eventgolang/protobuf
commit sha a522d5fa0c2268a489b0c8118de34e92830bd602
internal/impl: fix tag decoding when field num doesn't fit in int32 Discoverd by OSS-Fuzz. Change-Id: Ie2feefacee4ae632802fa920ac9694b525690eb2 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216619 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 20 days
issue commentgoogle/go-cmp
In the past, the choice to use hexadecimal was to ensure []byte (which is really an alias for []uint8) printed in hexadecimal form, but I agree that it causes standalone unsigned integers to print in sub-par fashion. Perhaps we should always print unsigned integers in decimal form and special case []uint8 instead.
comment created time in 21 days
issue closedgoogle/go-cmp
If a type defines an Equal method that in turn uses cmp.Equal, then it will enter an infinite loop. It will be quite helpful if the type's Equal method could be implemented using cmp.Equal with complex options. Any idea of how to handle this situation?
closed time in 21 days
guiguan
issue commentgoogle/go-cmp
You can try the following:
func (x T) Equal(y T) bool {
type T2 T
return cmp.Equal(T2(x), T2(y))
}
The type declaration of T2 has the exact same underlying memory layout as T, but lacks any of the methods. Per the Go specification:
It does not inherit any methods bound to the given type
This ensures that it does not inherit the Equal method.
You can now cast to T2 and call cmp.Equal on the values.
The same thing works for pointers as well:
func (x *T) Equal(y *T) bool {
type T2 T
return cmp.Equal((*T2)(x), (*T2)(y))
}
comment created time in 21 days
push eventgolang/protobuf
commit sha 524c60670a939e1e814ff232af2d0a4d6b2365ff
runtime/protoiface: use more efficient options representation Change the representation of option flags in protoiface from bools to a bitfield. This brings the representation of options in protoiface in sync with that in internal/impl. This change has several benefits: 1. We will probably find that we need to add more option flags over time. Converting to the more efficient representation of these flags as high in the call stack as possible minimizes the performance implication of the struct growing. 2. On a similar note, this avoids the need to convert from the compact representation to the larger one when passing from internal/impl to proto, since the {Marshal,Unmarshal}State methods take the compact form. 3. This removes unused options from protoiface. Instead of documenting that AllowPartial is always set, we can just not include an AllowPartial flag in the protoiface options. 4. Conversely, this provides a way to add option flags to protoiface that we don't want to expose in the proto package. name old time/op new time/op delta EmptyMessage/Wire/Marshal-12 11.1ns ± 7% 10.1ns ± 1% -9.35% (p=0.000 n=8+8) EmptyMessage/Wire/Unmarshal-12 7.07ns ± 0% 6.74ns ± 1% -4.58% (p=0.000 n=8+8) EmptyMessage/Wire/Validate-12 4.30ns ± 1% 3.80ns ± 8% -11.45% (p=0.000 n=7+8) RepeatedInt32/Wire/Marshal-12 1.17µs ± 1% 1.21µs ± 7% +4.09% (p=0.000 n=8+8) RepeatedInt32/Wire/Unmarshal-12 938ns ± 0% 942ns ± 3% ~ (p=0.178 n=7+8) RepeatedInt32/Wire/Validate-12 521ns ± 4% 543ns ± 7% ~ (p=0.157 n=7+8) Required/Wire/Marshal-12 97.2ns ± 1% 95.3ns ± 1% -1.98% (p=0.001 n=7+7) Required/Wire/Unmarshal-12 41.0ns ± 9% 38.6ns ± 3% -5.73% (p=0.048 n=8+8) Required/Wire/Validate-12 25.4ns ±11% 21.4ns ± 3% -15.62% (p=0.000 n=8+7) Change-Id: I3ac1b00ab36cfdf61316ec087a5dd20d9248e4f6 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216760 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 21 days
push eventgolang/protobuf
commit sha a60e709ac81a100c95a2a09da877d2acd4f36b97
proto: fix DiscardUnknown UnmarshalOptions.DiscardUnknown was simply not working. Oops. Fix it. Add a test. Change-Id: I76888eae1221d99a007f0e9cdb711d292e6856b1 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216762 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha 212b05b8089ca06b9230f97b5d5c117bdfb97868
internal/testprotos: make TestAllExtensions recursive Tweak the test message to allow creating messages with extensions that contain extensions that contain extensions, etc. Change-Id: I41844ae699c88ab96bf0d30db3a3fbaf09616161 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216761 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 21 days
push eventgolang/protobuf
commit sha f25a6ca994fa9b2cc1a51b81929f9f0cc6488542
internal/benchmarks/micro: add validator microbenchmarks Change-Id: Ice768d90e650bf51d619b8cd5f5d51e6b00c53b6 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216623 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha 5d82883c5a9dc1e3f8e72cba2233aa5e68dcfc1d
internal/impl: inline small tag decoding in the validator name old time/op new time/op delta EmptyMessage/Wire/Validate-12 4.59ns ± 0% 4.51ns ± 1% -1.74% (p=0.001 n=8+8) RepeatedInt32/Wire/Validate-12 1.28µs ± 0% 0.91µs ± 0% -28.71% (p=0.000 n=7+8) Required/Wire/Validate-12 48.3ns ± 2% 34.5ns ± 0% -28.69% (p=0.001 n=7+7) Change-Id: If7c431ee23d930d44af0fc26b7bd2149d3aded64 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216624 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha 8fa11b11227be39ae4b9afb16cadc0c329e030d2
internal/impl: inline most field decoding in the validator name old time/op new time/op delta EmptyMessage/Wire/Validate-12 4.51ns ± 1% 4.57ns ± 0% +1.19% (p=0.045 n=8+8) RepeatedInt32/Wire/Validate-12 910ns ± 0% 726ns ± 3% -20.13% (p=0.000 n=8+8) Required/Wire/Validate-12 34.5ns ± 0% 29.6ns ± 5% -13.99% (p=0.000 n=7+8) Change-Id: I8ac90ed3fc79dfef7f2500f13b33fd2593fc0fc1 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216625 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha cb0bfd0f407cb799143a95fe378f07164f4d0a65
internal/impl: reduce redundant MessageInfo initializations in validator name old time/op new time/op delta EmptyMessage/Wire/Validate-12 4.58ns ± 0% 4.29ns ± 1% -6.22% (p=0.000 n=7+8) RepeatedInt32/Wire/Validate-12 702ns ± 1% 518ns ± 0% -26.12% (p=0.001 n=7+7) Required/Wire/Validate-12 30.6ns ± 6% 22.1ns ± 0% -27.81% (p=0.000 n=8+7) Change-Id: I0d1db8583aa0bf4468bc385c213eb6adff001297 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216627 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 21 days
push eventgolang/protobuf
commit sha 5e73c4cec291ce9a2236b182e17eba758d5bbcfe
grpc: accept interface in NewClient functions Change-Id: I62123fccd689bdda9612942cc79b0a91527158cd Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216399 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 21 days
push eventgolang/protobuf
commit sha d23c5127dc24889085f8ccea5c9d560a57a879d8
grpc: accept interface in NewClient functions (#1025) * grpc: accept interface in NewClient functions * fix deprecated.pb.go and grpc_empty.pb.go * add go.mod and go.sum to testdata/grpc * remove comment
push time in 21 days
PR merged golang/protobuf
Addresses https://github.com/grpc/grpc-go/issues/1287
I will send a v2 version of this change shortly.
pr closed time in 21 days
pull request commentgolang/protobuf
grpc: accept interface in NewClient functions
I'm curious, do you have the ability to squash and merge, or should I do that for you?
comment created time in 22 days
push eventgolang/protobuf
commit sha adbbc8ec4728b46698c20e557b1ec673984d2631
internal/impl: inline some small varint decoding Inline decoding of 1- and 2-byte varints in generated unmarshal functions. name old time/op new time/op delta EmptyMessage/Wire/Unmarshal 40.2ns ± 2% 40.1ns ± 1% ~ (p=0.355 n=37+37) EmptyMessage/Wire/Unmarshal-12 7.12ns ± 1% 6.87ns ± 1% -3.49% (p=0.000 n=37+39) RepeatedInt32/Wire/Unmarshal 6.46µs ± 1% 5.78µs ± 1% -10.65% (p=0.000 n=35+33) RepeatedInt32/Wire/Unmarshal-12 1.05µs ± 2% 0.98µs ± 2% -6.79% (p=0.000 n=33+40) Required/Wire/Unmarshal 251ns ± 1% 216ns ± 1% -13.69% (p=0.000 n=38+36) Required/Wire/Unmarshal-12 42.4ns ± 1% 37.7ns ± 2% -11.02% (p=0.000 n=37+39) Change-Id: Iecfc38fcae00979b89a093368821cca7f2357578 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216421 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
push time in 22 days
push eventgolang/protobuf
commit sha 8729675da7e470dd015dd541f25ee77a9884f210
internal/benchmarks/micro: add a place for microbenchmarks Add a place to put microbenchmarks used to justify performance-related changes. Change-Id: I6e90a3500594b3f6297cee0b8e321a50d0a556ca Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216480 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha ce8f7f63536c227ee1f914c3d51a49d26f096985
internal/impl: inline small tag decoding Inline varint decoding of small (1- and 2-byte) field tags in the fast-path unmarshaler. name old time/op new time/op delta EmptyMessage/Wire/Unmarshal 40.6ns ± 1% 40.2ns ± 1% -1.02% (p=0.000 n=37+35) EmptyMessage/Wire/Unmarshal-12 6.77ns ± 2% 7.13ns ± 5% +5.32% (p=0.000 n=37+37) RepeatedInt32/Wire/Unmarshal 9.46µs ± 1% 6.57µs ± 1% -30.56% (p=0.000 n=38+39) RepeatedInt32/Wire/Unmarshal-12 1.50µs ± 2% 1.05µs ± 2% -30.00% (p=0.000 n=39+37) Required/Wire/Unmarshal 371ns ± 1% 258ns ± 1% -30.44% (p=0.000 n=38+32) Required/Wire/Unmarshal-12 60.3ns ± 1% 44.3ns ± 2% -26.45% (p=0.000 n=38+36) Change-Id: Ie80415dea8cb6b840eafa52f0572046a1910a9b1 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216419 Reviewed-by: Joe Tsai <joetsai@google.com>
commit sha 170b2bfca6e6b56c3de20f939798b56635e600bf
internal/impl: precompute required bit in validator Required field validation populates a bitmask of observed required fields. Store a uint64 containing the bit to set in the validationInfo rather than the index of the bit. Provides a noticeable speed increase in validation. name old time/op new time/op delta EmptyMessage/Wire/Unmarshal 40.2ns ± 1% 40.2ns ± 2% ~ (p=0.860 n=35+37) EmptyMessage/Wire/Unmarshal-12 7.13ns ± 5% 7.12ns ± 1% ~ (p=0.112 n=37+37) RepeatedInt32/Wire/Unmarshal 6.57µs ± 1% 6.46µs ± 1% -1.56% (p=0.000 n=39+35) RepeatedInt32/Wire/Unmarshal-12 1.05µs ± 2% 1.05µs ± 2% ~ (p=0.659 n=37+33) Required/Wire/Unmarshal 258ns ± 1% 251ns ± 1% -2.87% (p=0.000 n=32+38) Required/Wire/Unmarshal-12 44.3ns ± 2% 42.4ns ± 1% -4.36% (p=0.000 n=36+37) Change-Id: Ib1cb74d3e348355a6a2f66aecf8fdc4b58cd84d4 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216420 Reviewed-by: Joe Tsai <joetsai@google.com>
push time in 23 days