This identifies gogo/protobuf as the code being use. I think you are filing this bug in the wrong module.

Without a reproduction, there is not much we can do with this report. Furthermore, it's unclear what "the growslie performance issue" is.

The protoreflect.Message.Range method is documented as:

Range iterates over every populated field in an undefined order

Under proto3 semantics, the zero value is not a populated field.

If you want to iterate over every message field regardless of whether it is populated, you can iterate over all the declared fields:

message := m.ProtoReflect()
fields := message.Descriptor().Fields() // all the declared fields in this message type
for i := 0; i < fields.Len(); i++ {
    v := message.Get(fields.Get(i))
    ... // do something with v
}

This behavior would have to be first proposed at github.com/protocolbuffers/protobuf for all languages with a JSON implementation to provide consistent behavior. Without uniform agreement that this is to be handled, we can't do anything here.

While this module is deprecated, it is still important to ensure that it continues to work with higher versions of Go. Update travis.yml to test up to Go1.15.

While this module is deprecated, it is still important to ensure that it continues to work with higher versions of Go. Update travis.yml to test up to Go1.15.

Given the existence of custom UnmarshalJSON functions that already don't guarantee any semantics regarding the errors they returns, I don't think we should expose semantically identically errors for unknown fields since it would only be a partial guarantee.

Furthermore, the original poster should adequately explain what they want to use this for. It's rare for any proposal to go through without some justification for why it's useful.

Go 1.15 introduced a new go vet warning (https://golang.org/doc/go1.15#vet) for conversions of the form string(x) where x is an integer type other than rune or byte. This warning is enabled by default when running go test. As a consequence, running go test github.com/golang/protobuf/proto results in a build failure prior to this commit.

Fixes #1209.

What version of protobuf and what language are you using? Version: v1.4.2.

$ go version
go version go1.15.2 linux/amd64

What did you do? Go 1.15 contains a new vet check that warns for conversion of the form string(x) where x is an integer type that is not rune or byte. I initially noticed the build failure when I ran go test all in a project that uses Go 1.15 and depends on this module.

To reproduce:

$ cd $(mktemp -d)

$ go mod init test
go: creating new go.mod: module test

$ go get github.com/golang/protobuf@v1.4.2
go: downloading github.com/golang/protobuf v1.4.2

$ go test github.com/golang/protobuf/proto
go: downloading google.golang.org/protobuf v1.23.0
go: downloading github.com/google/go-cmp v0.4.0
# github.com/golang/protobuf/proto
[redacted $GOPATH]/pkg/mod/github.com/golang/protobuf@v1.4.2/proto/text_decode.go:768:10: conversion from uint64 to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
FAIL    github.com/golang/protobuf/proto [build failed]
FAIL

What did you expect to see? Passing tests.

What did you see instead? A build failure (see details above).

Anything else we should know about your project / environment? N/A

We consider development of the github.com/golang/protobuf module frozen at this point. However, it is an important goal that the module continues to work with higher versions of Go. Therefore, this change is justified.

The PR looks good to me, but will require the CLA to be signed in order to be merged.

There is no LICENSE.md file, but there is a LICENSE file.

There is no LICENSE.md file, but there is a LICENSE file.

my main use case for an error of this form is to skip tests of functionality that is not present on the current platform.

Thanks for clarifying. That's an reasonable use-case, but I doubt that's what most people are thinking of when they see this.

The difference between that and "unimplemented" is that no semantically-equivalent fallback is possible

I'm going to make the case that most people will not understand this, leading to this error being widely misused and therefore the meaning of it being increasingly muddled to the point where the documentation mismatches what is done in practice.

(If a fallback implementation is possible, than the function being called should just do that in the first place.)

I'm not a fan of this recommendation since there is often information that the caller has where they can do a better job at doing a fallback than what the function can do locally.

Much of the discussion have been regarding the difference between "not supported" and "not implemented" and I'm still not entirely clear what the difference exactly is. If we add this sentinel error, I'm fairly sure a large number of Go users will not understand the difference as well.

When I first saw this proposal, my understanding was closer to "not implemented", but upon reading the discussion, it seems that this is something more like "not supported". If this were to be added and I had not read this discussion, I'm fairly sure I would have used this error as if it were "not implemented".

Code like:

func (w) WriteString(s string) (int, error) {
    if u, ok := w.u.(io.StringWriter); ok {
        return u.WriteString(s)
    }
    return 0, ErrTBD
}

is exactly like something I would have written using ErrUnsupported in place of ErrTBD.

To be honest, I'm not sure what the utility of this sentinel error is. The proposed documentation on this says:

// Functions and methods should not return this error but should instead // return an error including appropriate context that satisfies // errors.Is(err, errors.ErrUnsupported)

Support I do errors.Is(err, errors.ErrUnsupported), what are the things that we're expecting users to do with this? If they use that to switch to some fallback implementation, how is that functionally different than "not implemented"?

In code that uses Go reflection, it is common that the user prints the type. However, it easy for people (out of habit) to use %T with a reflect.Type when they should actually be using %v instead.

For example:

t := reflect.TypeOf(...)
if checkValid(t) {
    panic(fmt.Sprintf("invalid type: %T: t)) // incorrect: should be using %v
}

It is unlikely that the wrong verb is detected during code review.

In the instances similar to this, I don't think I saw a single case where a person actually wanted to print the type of reflect.Type itself (which by the way is *reflect.rtype and not particularly interesting for users).

\cc @dominikh in case this is a check better suited for staticcheck than vet.

That snippet tells us that the code doesn't compile. Can you explain how that's better UX than the Go toolchain not compiling the code? To my knowledge, the go/types package doesn't return a distinguishable error for conflicting identifiers.

We already do something similar for formatting go code, checking for package errors will not be a lot of additional effort for better UX.

That's 100LOC to maintain for slightly better UX; I'm not convinced this is worth it. If there's anything to do here, it should be in protoc to at least print a warning that non-style compliant names are being used. The problem we're facing here is certainly not unique to Go.

proto4, whenever that comes. Is there any official place to request features/fixes for the next major version or to file things for consideration when that conversation begins?

In my opinion, the introduction of proto3 itself, give the pre-existing proto2 has been technically expensive to maintain and rollout. I'm doubtful there will ever be a proto4. Regardless, any discussion about proto4 should be at github.com/protocolbuffers/protobuf

"ignores" unknown string value by dropping it and hence changing the size of the repeated field.

Wow. That is surprising behavior, but okay.

The StructField.PkgPath field is populated if and only if the field is unexported. However, there is a hit to readability:

for i := 0; i < t.NumField(); i++ {
    f := t.Field(i)
    if f.PkgPath != "" {
        continue
    }
    ...
}

It's not obvious to most readers why the f.PkgPath != "" check exist. For this reason, a vast majority of cases have a comment that says:

    // Skip unexported fields.
    if f.PkgPath != "" {
        continue
    }

This documentation seems unnecessary if we had just added a helper method on StructField like:

// IsExported reports whether the struct field is exported.
func (f StructField) IsExported() bool {
    return f.PkgPath == ""
}

Thus, use of it becomes obvious from a readability perspective:

for i := 0; i < t.NumField(); i++ {
    f := t.Field(i)
    if f.IsExported() {
        continue
    }
    ...
}

Bikeshedding: I don't have a preference whether this should be called IsExported or simply Exported. The former matches IsValid, IsNil, or IsZero, while the latter matches StructField.Anonymous. It seems that the reflect package is already internally inconsistent about predicate functionality.

I didn't check C++, but I'm going to guess that it doesn't handle this case since there's not an obvious behavior to perform in the case of repeated fields.

\cc @cybrcodr, do you happen to remember what the behavior for equivalent functionality in C++ is?

We can fail explicit when we know generated code will not compile.

What's the point of building conflict detection code into protoc-gen-go if the code already doesn't compile? We'd just be re-implementing the exact same logic as the Go compiler itself. Therefore, It seems entirely appropriate to let the Go compiler be the explicit failure for this class of failures.

this might be a good candidate issue for discussion in v4.

What is v4?

The world already breaks when those failures happen. It just breaks when the program compiles instead of code generation.

The code breaks if you add another non-compliant name that results in a conflict. That's very different from breaking the code the moment if any non-style compliant name is used.

I should also note that style enforcement is not the responsibility of protoc-gen-go. Personally, I support such a notion, but this is something that should be discussed at the main protocol buffer project and be implemented in protoc.

As a counter argument to above, the snippet below becomes more readable with type-eliding:

compress.NewReader(r, {Level: compress.BestSpeed})
compress.NewReader(r, compress.NewReaderOptions{Level: compress.BestSpeed}

You'll find examples that may get better and examples that may get worse if eliding were always performed. Personally, I prefer to at least have this choice.

Explicit failure is better than a silent one in case there are no plans to support non-style compliant proto files.

There's no way we can enforce this today with breaking the world. The situation is unfortunate, but it is what it is.

If we want to support the behaviour, we can follow the convention proto2 codegen does and add an _ for the 2nd occurence of a conflicting name.

We already do this for some small set of conflicts. We consider it a grave mistake to have ever done this since it means that generated names are dependent on the order that fields are specified in a .proto file.

I've found that the raw, "Go-syntax" results of a Diff() intermittently contain the \u00a0 unicode character rather than a space.

Here's a sample that reproduces this:

	type A struct {
		B string
	}
	result := cmp.Diff(A{B: "foo1"}, A{B: "foo2"})
        // Print out the Go-syntax output:
	fmt.Println("Result is: " + fmt.Sprintf("%#v", result))

Running the above sometimes produces: Result is: "\u00a0\u00a0reconciler.A{\n-\u00a0\tB: \"foo1\",\n+\u00a0\tB: \"foo2\",\n\u00a0\u00a0}\n"

and sometimes produces: ```Result is: " reconciler.A{\n- \tB: "foo1",\n+ \tB: "foo2",\n }\n"``

Result is: "\u00a0\u00a0reconciler.A{\n-\u00a0\tB: "foo1",\n+\u00a0\tB: "foo2",\n\u00a0\u00a0}\n"

This problem is only apparent if you are looking at the Go-syntax (%#v) output, and is not apparent by just printing the string. However, there are libraries that look at the Go-syntax value of a string - for example, the various string comparison methods of https://github.com/stretchr/testify. Attempting to use the output of Diff() in test cases with https://github.com/stretchr/testify is not usable due to this inconsistency.

Closing as working as intended.

The first few versions of cmp unfortunately kept breaking a number of tests that assumed that the output would never change every time the reporter output was changed. At some point we made it random precisely so that brittle tasks cannot be written. The fact that the reporter output has been improved significantly in recent versions is thanks to the freedom to make beneficial changes made possible by deliberate randomization.

What version of protobuf and what language are you using? Version 1.25.0

What did you do? The GeneratedCode creates wrong import name:

g.QualifiedGoIdent(protogen.GoIdent{GoName: "", GoImportPath: "github.com/scylladb/gocqlx/v2"})

What did you expect to see? import ( gocqlx "github.com/scylladb/gocqlx/v2" ) OR import ( "github.com/scylladb/gocqlx/v2" )

What did you see instead? import ( v2 "github.com/scylladb/gocqlx/v2" )

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment?

Closing as duplicate of #1205

The derived Go names that protoc-gen-go uses (or more specifically compiler/protogen produces) has long been known to be problematic and possibly produce conflicts. Unfortunately, there is not much we can do about it since changing the behavior will likely break the world. For that reason, the new google.golang.org/protobuf module did not fix this behavior since it's deeply ingrained into the wider use of protobufs in Go and we wanted the new module to be reasonably compatible with the older github.com/golang/protobuf to the best we can.

I should note that naming styles in most programming languages is typically an aesthetic concern. However, for protocol buffers, it's actually a semantic concern since the naming style used makes a difference in whether generated code is even valid. Therefore adherence to the style is of much greater concern beyond just personal preferences. If the protobuf style guide is properly followed, conflicts should not occur. Otherwise, all bets are off, and we can't guarantee conflict-free name generation.

If you want guaranteed conflict-free name derivation, then the following naming styles must be used for the corresponding descriptor types:

In particular:

• CamelCase must match the regular expression [A-Z][a-zA-Z0-9]*, where any letters that appear after a number must be uppercase (e.g., Ext4Mount is okay, but Ext4mount is not).
• UPPER_SNAKE_CASE must match the regular expression [A-Z][_A-Z0-9]* (e.g., GEO_LOCATION is okay).
• lower_snake_case must match the regular expression [a-z][_a-z0-9]*, where letters must not follow any numbers (e.g., int64_value is okay, but int64value is not).

I should note that ImportAs would be best-effort, since we would presumably still prioritize correctness over cosmetics in the event that there is still a conflict over the chosen name.

Bisect implicates https://go-review.googlesource.com/c/go/+/193605

Thanks @networkimprov. I've been a bit distant from Go toolchain development for a while, so I've gotten behind on the latest in processes.

Adding to Go1.15.3 milestone since this is a regression relative to Go1.14 and a serious one.

@kokes, difference in output in itself is not particularly interesting since the compress/flate library provides no guarantees about output stability. However, producing corrupted issue is something we take seriously.

Thank you @k-ye. I am able to reproduce which does indeed indicate that it's producing corrupted output. I'm bisecting the root cause now (may very well be the CLs related to #34121 that you pointed to earlier).

I need more information to investigate this. The code snippets provided on do basic compression/decompression. I need real test data that exhibits the problem.

If the output is valid (e.g., can be decompressed), then this is working as expected. The compression libraries make no guarantees that the output remains stable for all time. While that guarantee can be useful in some contexts, if unfortunately means that we can never make changes to the compression algorithm either to improve the speed or to improve the compression ratio, both which are properties that are generally considered more important than stability.

It is unclear to me whether this issue is about the compressed output simply being different or that the compressed output is actually invalid (i.e., cannot be decompressed). Can you please clarify?

Given that this isn't a pressing issue, we're not going to do anything about this at the moment. It's a significant amount of release tooling work with little gain. Furthermore, it's really something the Go toolchain should do a better job at pruning useless go.sum entries.

I'm not sure how your test repos address the root issue. Both https://github.com/2767mr/prototest2/blob/master/go.sum and https://github.com/2767mr/prototest3/blob/master/go.sum and contain cyclic references to the past history of the other's module.

Stepping back a bit, can you explain why this is an issue that needs resolving? Is this purely a bloat issue or is there a technical issue involved?

thinking I think the best choice might to stay away from the pattern after all, so that we neither reinforce unexpected behavior, or the unexpectedness of the behavior.?

It seems that part of this is figuring out what we want to do with encoding/json related to this issue. It's been on my TODO list to think more heavily about golang/go#36225 (and many other encoding/json issues discovered over the years).

One test even required the buggy behavior in order to pass, because of a typoed extra %s at the end of the format string in the test, resulting in something like {"valid":"json"}%!s(MISSING) being the test input.

😲

I‘m not sure about reinforcing a habit of unmarshalling const string into a proto message for initialization

True, I'm not fond of having multiple ways of doing the same thing. This issue was filed primarily based on observation of a common code pattern. I don't feel strongly whether to have this or not.

It is possible to reference future releases inside a go.mod.

Yes, for the go.mod file, but not for the go.sum file. The go.sum file contains a hash of the dependencies, but there's a problem. The hash of the dependencies covers the forward version reference to some unreleased module for which we do not know (or rather cannot compute) the hash for.

This is a known issue. It's unfortunate, but relatively harmless. The cyclic dependency is necessary:

• github.com/golang/protobuf depends on google.golang.org/protobuf because v1.4+ is implemented in terms of the newer module.
• google.golang.org/protobuf "depends" on github.com/golang/protobuf only to enforce that v1.4+ be used. Aside from that constraint, it requires nothing from the older module. This is necessary to avoid problems with disjoint sets of type registries.

The cyclic relationship can't be broken.

If there's anything to be done, it probably be with the Go toolchain to prune useless go.sum entries for cyclic dependencies. If so, there's not much this project can do about this issue.

You have have an ExtensionDesc and want to retrieve the value for a field that the extension descriptor represents, you can use proto.GetExtension.

(FYI, we try to keep our issue tracker about actionable issues. Questions like this are best asked on Slack or someone other means where you can more readily get immediate help. Consider joining the #protobuf channel or the Gophers slack workspace: https://gophers.slack.com/?redir=%2Fmessages%2Fgeneral%2F)

JSON serialization is very commonly used in HTTP servers where use of net/http means that the user typically starts with a io.Reader and io.Writer on hand. As such, it is slightly inconvenient using protojson since it operates primarily on []byte.

Perhaps we should add MarshalWriter that takes in a io.Writer as input and a UnmarshalReader that takes in a io.Reader as input.

Consideration needs to be given to:

• #609 which considers adding APIs to the proto package to support streaming read/write and/or scatter/gather functionality.
• https://github.com/golang/go/issues/36225, where the typical pattern of doing json.NewDecoder(r).Decode(...) is subtly buggy since it permits trailing trash data to go undetected.

(We would probably do the equivalent change to prototext since the two packages have nearly identical API surface despite having obviously different semantic behavior).

I'm noticing that text serialization (and to some degree JSON) is used to quickly initialize a lot of data in tests (and sometimes in not-test code at init) with a pattern like:

const initData = `...` // text data
m := new(foopb.MyMessage)
if err := prototext.Unmarshal([]byte(initData), m); err != nil {
    panic(err)
}
return m

With the upcoming introduction of generics, perhaps we should add a helper for this:

func MustUnmarshal[M proto.Message](s string) M {
    var m M
    m = m.ProtoReflect().New().Interface()
    if err := Unmarshal([]byte(s), m); err != nil {
        panic(err)
    }
    return m
}

Thus, usage of this would be something like:

m := prototext.MustUnmarshal[*foopb.MyMessage](initData)

If you want to ignore fields in a protobuf message, then you can use protocmp.IgnoreFields. If you want to ignore fields in a normal Go struct, then you can use cmpopts.IgnoreFields. You can compose the two options together if needed.

is it or is not possible to really make cmp.Diff() to work in conjunction with google.golang.org/protobuf/testing/protocmp

Is this a typo? protocmp exists to work with cmp.Diff. Did you mean to ask whether it's possible to use cmp.Diff without protocmp? If so, it was never the intention of cmp since it was created to be used to compare protobuf messages. In fact, protocmp was something I wanted to have available when cmp first became available, but we needed to build protobuf reflection first.

I have to admit I'm still not getting this. Such a map will only tell you if a type T has a match in the set of interfaces, but not which interfaces T matches, right? You'd still have to check them one by one. This is better for the case where T matches none of the interfaces, but still, the feature doesn't seem O(1) to me.

A simple modification of the cache is to use a map[reflect.Type]int where the int is the index interface that we care about.

Side note: if you intend the CL's performance to remain as-is, I wonder if we should document that we don't recommend registering more than a handful of interface types on a single encoder or decoder.

Sounds reasonable.

I'd imagine this is necessary either way if you're supporting interfaces.

Can you provide an example?

If both are available at once, it's likely users will jump straight to interface types without thinking whether using concrete types would be enough. Which is also why the performance difference worries me.

Can you explain how this would be the case? If there was a concrete type T that you wanted to provide an override for, then it seems natural that you would register a concrete type override. It seems odd to me to reach for an interface override unnecessarily for that case.

For example, let's say the user wants to provide a type override for time.Time, what interface would they be using to match that type? It seems more natural that they would register a func(t time.Time) ([]byte, error). Also, it would be easier to implement since working with concrete types is easier than interface types (where you're limited to the methods in the interface).

(apologies for responding out-of-order; for some reason my browser didn't render the earlier comment)

And would using concrete types be a big problem in those two use cases you mention?

Yes. For some cases, it would require registering dozens (if not hundreds) of concrete types. Even worse, it requires that the user of manually keep the code that declares their types constantly in sync with where those types are used with JSON serialization.

Can you elaborate a bit? I'm not quite seeing how that cache would work in constant time. If we're confident that this can be made performant in the future, then I'm not as worried about adding both features at once.

You can imagine a sync.Map that is functionally a map[reflect.Type]bool. It caches the answer to "will type T implement the set of registered interface type overrides?" If T is in the cache, then the query is O(1), if not, then we perform a linear lookup over all registered interface type overrides (which I contend is usually close to 0).

The challenge with the implementation is that the specific cache we want to use is itself dependent on the set of possible interface types that we may want to match against. Since I don't see having many interface type overrides being a common pattern, I didn't want to implement this for now.

Googlers have told you what they think of both features, but from what you've said, they haven't tried using concrete types alone

Concrete overrides are used to. I didn't mention them since the question I was responding to was about interface overrides.

And there isn't an easy way for developers outside of Google to do the same kind of testing (and very few people will go through the trouble unless we ship it in a release), so I fear that the feedback doesn't represent Go use cases outside of Google.

Is there something specific that you are proposing? More users of the feature would be great. For users that only care about concrete type overrides, the existence of the interface type override doesn't affect them if they don't use it. Am I missing something?

At Google, we have https://go-review.googlesource.com/c/go/+/212998 imported as a separate fork that adventurous users already using. The ability to specify type-override based on an interface is widely used. For our cases, it is typically used to capture all types that implement a proto.Message interface, but it's also used for other cases where a type doesn't have natural JSON support, and an interface match allows capturing types based on the public API (as specified by an interface) and to provide custom serialization for that type (based on the methods of that interface).

func([]byte, interface{}) fallback to be used if none of the concrete type funcs match. This won't be as powerful, but it will also remove the expensive linear search with go/types.Type.AssignableTo.

There are ways to implement the linear search in O(1) with type caches, I didn't implement here because it seems like pre-mature for the case where there are many type-overrides given that are all interfaces.

Also, a func([]byte, interface{}) fallback is 1) less performant since it now needs to be called for every node in the tree, rather than just those that match the interface, 2) does not compose well with disjoint sets of interface types, 3) needs to also provide an API for saying; "thanks for calling me, but I actually don't know how to serialize this specific type; please try something else".

Having reviewed my CL, the reason I didn't push it through is because of concerns with this feature being an trivial avenue for denial-of-service attack where a malicious GZIP file could cause a server to allocate arbitrary amounts of memory.

We have several options:

1. We could immediately permit parsing of small headers (I proposed up to 32KiB to match the window size of the DEFLATE algorithm itself). However, this won't help use-cases that abuse the comment field to store machine-parsable information where it can easily exceed 32KiB.
2. Add a new Reader.MaxFieldSize method so that the user can opt-in to parsing larger headers. Use of this option would still be a bit odd, since you would have to do something like:

var zr gzip.Reader
zr.MaxFieldSize(math.MaxInt32)
if err := zr.Reset(r); err != nil {
    ... // handle err
}
... // make use of zr.Header.Comment

Since this would probably require an API change, I'm adding the NeedDecision label. \cc @FiloSottile since he's probably interested in any possible DOS-related issues.

Running the snippet on Go1.15 clearly shows it's still an issue.

Huh, apparently I had a fix for this that I never submitted: https://golang.org/cl/53637

I was thinking about this more and I wanted to note a complication with generic less functions.

Suppose you had:

type S struct {
    F1, F2 int
} 

var ss1, ss2 []S = ...
cmp.Equal(ss1, ss2,
    cmpopts.SortSlices(GenericLess(S{})),
    cmpopts.IgnoreFields(S{}, "F1"),
)

There's a subtle problem here. The less operation used by cmpopts.SortSlices needs to be aware of any other options passed to cmp that may affect the semantics of equality. In this specific case, there's another option that ignores field "F1" and that needs to be taken into consideration when performing a sort, otherwise this semantically won't work as expected (it might sometimes get it right, and it might not; its flaky).

To make this work, we would probably need

1. some way to plumb options down to other options, and
2. generic less as a first-class feature in cmp itself. It's possible to define a generic less (e.g., cmp.Less), but a complication arises from the presence of Equal methods and Comparer options. Such functionality provides no means to derive ordering information. If there was a cmp.Less function, this property should hold: (cmp.Less(x, y, opts...) == 0) == cmp.Equal(x, y, opts...).

Is your feature request related to a problem? Please describe. It is usually not clear which version/commit generated the code here.

Describe the solution you'd like Mention of the version/commit at the top would make things clear and easier to work with.

Duplicate of #524, which is already done for google.golang.org/protobuf/cmd/protoc-gen-go. There are no intentions to add this feature to github.com/golang/protobuf/protoc-gen-go.

I apologize, but I don't understand what this is asking. Can you be more specific about what you trying to do?

Also, it seems like you're using gogoproto.nullable. Perhaps you should be asking this over at https://github.com/gogo/protobuf/issues?

Also, as mentioned in the description, this is strangely inconsistent. I can generate such code to Go using protoc-gen-go and load the descriptor just fine.

I agree it's inconsistent, and I'm inclined to make protoc-gen-go not be able to generate MessageSet code in open-source.

Is there any real code outside Google that actually uses MessageSet? It's an artifact of proto1 and it's a source of significant technical debt. In open-source, we ideally should treat it as if it doesn't exist.

I think the issue the relationship between a global resolver and a custom resolver to be somewhat of a red herring.

After all this code:

m := &anypb.Any{
	TypeUrl: "type.googleapis.com/google.protobuf.Timestamp",
	Value: protopack.Message{
		protopack.Tag{1, protopack.VarintType}, protopack.Varint(1599000711),
		protopack.Tag{2, protopack.VarintType}, protopack.Varint(166067539),
	}.Marshal(),
}
b, err := protojson.Marshal(m)
fmt.Println(string(b), err)

fails with:

 proto: google.protobuf.Any: unable to resolve "type.googleapis.com/google.protobuf.Timestamp": not found

assuming that timestamppb is not linked in.

The question at hand is whether protojson should have implicit knowledge of the WKTs for dynamic purposes or not. Either:

1. protojson has a dependency on every WKT as @neild alluded to (and the associated import-cycle problems), or
2. protojson has a local copy the descriptor of every WKT and uses it with dynamicpb to create a dynamic version of every WKT.

Neither approach seems particularly palatable for this edge-case.

Seems reasonable. Would you like to contribute a fix?

So is the main reason for this so that you can add new methods to the interface without breaking users?

Essentially this. The protobuf language has new additions over time (e.g., oneofs, maps, proto3 optional to name a few), and we would want the freedom to adjust the descriptor API accordingly to reflect such functionality.

There was some discussion in the past about adding a "FooDescriptorNotImplemented" type for every Descriptor and require application code to embed that or some other technique. At the time that we released the module, we decided to punt on figuring out if we wanted to allow custom descriptor implementations. Naturally, having an unexported method is the more conservative approach and can always be loosened in the future.

\cc @neild

As @mvdan mentioned, I don't know if there is a specific issue about the top-level option passing problem, but it's one that I've commented about in a number of different encoding/json issues. Adding new features carries O(n!) maintenance cost, since you need to consider how the newly added feature interacts with every feature that has already provided. I don't think we should add another UnmarshalerWithOptions interface without carefully thinking about the implications. Just as a high-level overview, such an interface would probably require a dependency on the encoding/json package, which is already a significant divergence from the Unmarshaler interface itself (which has no dependency on anything other than basic types like []byte, int, and error). If we were to force that dependency through the interface, it also brings into question whether the interface should provide more things than just the options. For example, currently calling nested Unmarshalers is O(n^2) since the encoding/json package has to repeatedly parse the entire sub-tree to find the end of the JSON token before it can call UnmarshalJSON. If we were to pass something similar to json.Decoder, it could be much more efficient.

I'm okay with adjusting the heuristic for how protogen determines the local package names, but your description still seems to suggest that this is purely for cosmetic reasons since the "the code is full of vX.Identifier". Can you confirm that the code actually functions correctly even if it doesn't look like what is expected?

Essentially, I'd like to make sure there's not some deeper issue I'm not quite understanding. The purpose of protogen is not to generate code that always follows style guidelines. It's nice if possible, but it's not a goal.

\cc @neild who has been interested in adding a "MutableFoo" accessor method which does exactly this.

Can you explain why this is "incorrect"? Within a Go source file, the code can rename import paths to any chosen local package names; and "v2" is a valid Go identifier and shouldn't cause any problems so long as it is unique within the context of where it is used.

There is no .proto parser implementation in this module. You can consider using https://pkg.go.dev/github.com/jhump/protoreflect/desc/protoparse?tab=doc

I update the issue, Is it clear enougn? 0 0

It's better. From what I can gather, yes, dynamicpb will do what you want. However, it will be the user's responsibility to provide the protobuf type information needed to parse arbitrary messages. The dynamicpb package creates dynamic messages when given a protoreflect.Message. A protoreflect.Message can be dynamically constructed at runtime using the google.golang.org/protobuf/reflect/protodesc package.

If you want to the ability to define a subset of the message at runtime, you could also do that with the google.golang.org/protobuf/types/dynamicpb package.

That said, I still don't understand what the original post is actually asking for.

The design document for JSON serialization for protobufs specifies that a google.protobuf.Value message without one of the kind fields explicitly populated should be rejected in order to ensure proper round-trip preservation of data. Given that proper preservation of data seems to be the intention, it would follow that NaN and Infinity should also be rejected in order to preserve the same property.

This issue is not so much about what users can do instead, but what is supposed to be correct serialization behavior.

If cmp panics because it is trying to access an unexported field, specially suggest the use of cmpopts.EquateErrors if the parent type implements the error interface.

Fixes #233

This panics:

cmp.Equal(errors.New("x"), errors.New("x"))

panic: cannot handle unexported field at {*errors.errorString}.s:
	"errors".errorString
consider using a custom Comparer; if you control the implementation of type, you can also consider using an Exporter, AllowUnexported, or cmpopts.IgnoreUnexported

The panic could detect when the type being compared implements error and suggest using cmpopts.EquateErrors.

When I generate code using the new protoc-gen-go I get the following generated code. It seems strange to depend on the old "github.com/golang/protobuf/proto" just for the "ProtoPackageIsVersion4" constant:

// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// 	protoc-gen-go v1.25.0
// 	protoc        v3.12.3
// source: proto/zipkin_proto3/zipkin.proto

import (
	proto "github.com/golang/protobuf/proto"
)

// This is a compile-time assertion that a sufficiently up-to-date version
// of the legacy proto package is being used.
const _ = proto.ProtoPackageIsVersion4

Am I missing something?

Duplicate of #1077

If cmp panics because it is trying to access an unexported field, specially suggest the use of cmpopts.EquateErrors if the parent type implements the error interface.

Fixes #233

What version of protobuf and what language are you using? protobuf-go v1.23.0

What did you do? https://github.com/protocolbuffers/protobuf-go/blob/v1.23.0/internal/impl/encode.go p.Apply(mi.unknownOffset).Bytes() might return a nil pointer，append() will appear panic panic: runtime error: invalid memory address or nil pointer dereference

What did you expect to see? need add nil pointer check

Anything else we should know about your project / environment? go version go1.13.5 darwin/amd64 runtime: Linux ali-jump 2.6.32-696.10.1.el6.x86_64 #1 SMP Tue Aug 22 18:51:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Marking as resolved since this is an issue in the application code.

goroutine that has been started at the time of the request timeout did not end immediately, and this goroutine was modifying this protobuff object?

If another Go routine in modifying the same message while you are marshaling it, then that is fundamentally a race condition and expected to fail, cause memory corruption, or other strange side effects.

Given that reframing, it's possible that just hiding sub-versions of protoc might resolve the base problem I'm having.

By sub-version, do you mean the "PATCH" version per semver terminology? That's something we could consider, but PATCH versions themselves can theoretically introduce changes to the generated .pb.go output and will require more thought.

In general, I recommend that the project enforce some means of ensuring everyone is using the same version of tools for checked in code. This applies to protoc and protoc-gen-go and even go fmt.

This use-case for wanting to disable the version markers is suspect as it's bad practices to have a number of developers working on the same module, but using different versions.

Do you have a reproduction for this? The line you are referencing is not the problem since it assumes that p itself is already a non-nil pointer. Thus, a nil pointer must have slipped in from elsewhere.

\cc @neild.

I vaguely recall some discussion in the past about whether to expose this or not. I don't remember the details, but there are some complexities involved in whether fieldGoType actually returns the expected Go type of the field in certain cases (e.g., for extensions and oneofs). Also, fieldGoType is not a pure function since it mutates the state of protogen.GeneratedFile making it a sharp API with subtle behavior. Any public API would need to be safer to use and will probably involve a noticeable increase in the protogen API surface.