For protoc-gen-swift, we added the ability for it to read the serialized CodeGeneratorRequest from a file. You can see that in main.swift. Then you need to capture a CodeGeneratorRequest into a file, and you can then debug by running protoc-gen-swift saved_request to run it with the captured request.

The easiest way I've found to capture the CodeGeneratorRequest from protoc is to create a shell script like the following:

#/usr/bin/sh
cat >saved_request

Save this script as protoc-gen-capture and then you can run protoc --capture_out to get protoc

Hello, just to follow up on another issue but got closed. I wonder how do we typically debug during development (custom generator)? For example, we make some changes and then swift build to get the new protoc-gen-swift version and protoc against the new build? It seems many steps for testing a change. Wonder if there is a way to debug in xcode, i.e. run in xcode? Thank you!

Please be sure to include:

what OS you are developing on (Linux or macOS, including the version) macOS

for macOS, what version of Xcode you are using (xcodebuild -version), Xcode 12.3 Build version 12C33

for Linux, what version of Swift (swift --version)

what version of Swift is your code set to compile with (i.e. from project settings, etc.) Apple Swift version 5.3.2

what branch/tag of SwiftProtobuf you are using (1.0.0, etc.) the latest release 1.14.0

if you are getting compile errors, please be sure include all errors/warnings, sometimes the error before the one you are stuck on is important. unknownField

lastly, if it all possible, provide a snippet of .proto and/or source that shows the problem.

syntax = "proto2";

message SearchRequest { required string query = 1; required int32 page_number = 2; required int32 result_per_page = 3; }

Thank you!

The main goal of issues is bugs/help/problems with this library. This is more a question about writing your own protoc plugin (which happens to be in Swift); that's not really something we're setting out to support. It is more of a general development question for writing a Swift executable that just happens to have to interface with the protoc plugin interface.

Since you are writing your own plugin, that's entirely up to you.

What you likely ran into when the raw proto files were being parse by the plugin was how we've choice to do this. You'll notice it has a back door to help with this, it uses PROTOC_GEN_SWIFT_LOG_REQUEST in the env to write out the request it got from protoc. Then when looking at its command line arguments, if there are files, it assumes they were from one of those dumps and reads them, if there aren't any files, it is the normal protoc invocation and it reads from stdin.

Hello, just to follow up on another issue but got closed. I wonder how do we typically debug during development (custom generator)? For example, we make some changes and then swift build to get the new protoc-gen-swift version and protoc against the new build? It seems many steps for testing a change. Wonder if there is a way to debug in xcode, i.e. run in xcode? Thank you!

Please be sure to include:

• what OS you are developing on (Linux or macOS, including the version)

• for macOS, what version of Xcode you are using (xcodebuild -version), for Linux, what version of Swift (swift --version)

• what version of Swift is your code set to compile with (i.e. from project settings, etc.)

• what branch/tag of SwiftProtobuf you are using (1.0.0, etc.)

• if you are getting compile errors, please be sure include all errors/warnings, sometimes the error before the one you are stuck on is important.

• lastly, if it all possible, provide a snippet of .proto and/or source that shows the problem.

Thank you!

Hello @thomasvl, I wonder how do we typically debug during development? For example, we make some changes and then swift build and protoc against the new build? It seems too many steps for testing a change. Wonder if there is a way to debug in xcode, i.e. run in xcode? Thank you!

Hello, I am trying to create a custom generator. So, I am trying to use Google_Protobuf_Compiler_CodeGeneratorRequest(serializedData:data) to parse and get the Message object based on a proto file. the data object is read from a simple proto file with content of

syntax = "proto2";

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2;
  required int32 result_per_page = 3;
}

However, it keeps failing with error: unknownField data is read like this let data = FileHandle.init(forReadingAtPath: proto_path)?.readDataToEndOfFile() I debugged a bit, and seems it is trying to parse the first letter ("s", in the first line) as a fieldNumber but failed. I wonder if I am missing anything? or how can I correctly create a Message or CodeGeneratorRequest object for a proto file. Thank you!

Please be sure to include:

• what OS you are developing on (Linux or macOS, including the version)
  macOS

• for macOS, what version of Xcode you are using (xcodebuild -version), Xcode 12.3 Build version 12C33

  for Linux, what version of Swift (swift --version)

• what version of Swift is your code set to compile with (i.e. from project settings, etc.) Apple Swift version 5.3.2

• what branch/tag of SwiftProtobuf you are using (1.0.0, etc.) the latest release 1.14.0

• if you are getting compile errors, please be sure include all errors/warnings, sometimes the error before the one you are stuck on is important. unknownField

• lastly, if it all possible, provide a snippet of .proto and/or source that shows the problem.

syntax = "proto2";

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2;
  required int32 result_per_page = 3;
}

Thank you!

protoc parses the .proto files, and builds up the binary protobuf message. It then swaps you plugin via the mentiond --foo_out directive, and passes the binary serialization of the message. The code in main then reads it and parses it to then generate from it.

Thank you @tbkka for your explanation. That makes more sense now. I wonder how I can get hold of Google_Protobuf_Compiler_CodeGeneratorRequest? I see in the main.swift, looks like it's initialized by requestData request = try Google_Protobuf_Compiler_CodeGeneratorRequest(serializedData: requestData) I wonder how does protoc inject the CodeGeneratorRequest? Thank yoU!

@thomasvl thank you for your reply. I see. The use case we have now is that we want to customize the generated code. We would like to add some features/generated-code that are particular to our use case. Do you know where/how I can get parsed Google_Protobuf_CodeGeneratorRequest object? and be able to customize the code-gen? or whether that's feasible at all. Thank you!

As Thomas mentioned, you cannot easily run a protobuf code generator directly. Instead, you run it via protoc. If your generator is called protoc-gen-foo, then you run it with the --foo_out argument to protoc. The protoc program will parse the .proto file and then run your protoc-gen-foo program giving it a properly-constructed CodeGeneratorRequest object.

I debugged a bit, and seems it is trying to parse the first letter ("s", in the first line) as a fieldNumber but failed.

Can you explain how you are trying to use your plugin? If it is getting the s in _ syntax = "proto2";_ then it sounds like you are trying to directly pass a proto plugin a raw .proto file. That won't work. Plugins are used via protoc. protoc does all the parsing of the .proto files and invokes plugins based on the arguments given to protoc.

Hello, I am trying to create a custom generator. So, I am trying to use Google_Protobuf_Compiler_CodeGeneratorRequest(serializedData:data) to parse and get the Message object based on a proto file. the data object is read from a simple proto file with content of

syntax = "proto2";

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
}

However, it keeps failing with error: unknownField I debugged a bit, and seems it is trying to parse the first letter ("s", in the first line) as a fieldNumber but failed. I wonder if I am missing anything? or how can I correctly create a Message or CodeGeneratorRequest object for a proto file. Thank you!

Please be sure to include:

• what OS you are developing on (Linux or macOS, including the version)
  macOS

• for macOS, what version of Xcode you are using (xcodebuild -version), Xcode 12.3 Build version 12C33

  for Linux, what version of Swift (swift --version)

• what version of Swift is your code set to compile with (i.e. from project settings, etc.) Apple Swift version 5.3.2

• what branch/tag of SwiftProtobuf you are using (1.0.0, etc.) the latest release 1.14.0

• if you are getting compile errors, please be sure include all errors/warnings, sometimes the error before the one you are stuck on is important. unknownField

• lastly, if it all possible, provide a snippet of .proto and/or source that shows the problem.

syntax = "proto2";

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
}

Thank you!

Updated to match syntax in README.md.

Maybe these were intentionally left because the package has a minimum tools version of 4.2?

Thanks!

@tbkka looks like the checks failed to run again for some reason, something get disabled again?

Updated to match syntax in README.md.

Maybe these were intentionally left because the package has a minimum tools version of 4.2?

I'd love to hear more details about the "weird Objective-C runtime data corruption" you saw.

There might be, but those distributions would need to hide the presence of SwiftProtobuf. Swift Protobuf does not support itself being built in library evolution mode, so it cannot be present in the public ABI of any framework that is built in library evolution mode. That means it can only ever be imported @_implementationOnly, and its types can never be public.

I'm trying to ship one binary Swift framework that uses SwiftProtobuf. I think this should be a pretty common use case. It's hard for me to believe that there's isn't a single SDK out there that uses SwiftProtobuf and is built for distribution as a XCFramework?

You cannot easily do this, and this limitation is ultimately derived from the fact that Swift Protobuf doesn’t really support being used in this way. In principle with sufficient design work it would be possible for Protobuf’s code generation to create types that would support this mode of operation, but it would require a separate mode for invoking the protoc plugin as it’s fundamentally incompatible with the current mode of operation.

What are you trying to achieve by using library evolution mode?

Even if that would work, how can someone generate protos that contain a @_implementationOnly import SwiftProtobuf instead of a import SwiftProtobuf?

@Lukasa The generated code with protoc-gen-swift contains a import SwiftProtobuf, so effectively this is included in the only module that I built. I tried to hack the generated code to change this to an @_implementationOnly import SwiftProtobuf, but since the public API of my library also uses the generated type, it's not allowed.

With the library you’re building for library distribution, are you using @_implementationOnly imports of Swift Protobuf? If you aren’t, the bundled version of Protobuf may be exposed in your module’s API and so you have a multiple dependency issue.

At first glance, this doesn't really seem specific to Swift Protobuf and seems more like it is an issue mixing compile modes, so you might need to reach out on the Swift forums or some place else where folks might have more knowledge how how the different flags you might be using change how the compiler behaves.

Hello! I'm seeing a pretty weird linker error related to SwiftProtobuf symbols. I'm going to describe the situation as much as possible.

1. We build a static library that contains some SwiftProtobuf generated code. This library is built for library distribution, since we want to support multiple Xcode and Swift versions.
2. When we integrate this library in a consumer, that builda SwiftProtobuf from source (we had problems with distributing SwiftProtobuf as a binary built with library distribution in the past since it would cause some weird Objective-C runtime data corruption), the compilation works fine.
3. When the app gets to linking stage, it throws the following errors because some symbols included in the prebuilt static library aren't found in SwiftProtobuf. We are using the same version everywhere, so I'm pretty sure it's not a version mismatch kind of error.

Undefined symbols for architecture x86_64:
  "_$s13SwiftProtobuf19_ProtoNameProvidingP17_protobuf_nameMapAA01_dH0VvgZTq", referenced from:
  "_$s13SwiftProtobuf26_MessageImplementationBaseP29_protobuf_generated_isEqualTo5otherSbx_tFTq", referenced from:
  "_$s13SwiftProtobuf7DecoderP14decodeMapField9fieldType5valueyAA01_bE0Vyqd__qd_0_Gm_SDy04BaseH0Qyd__AJQyd_0_GztKAA0e3KeyH0Rd__AA0e5ValueH0Rd_0_r0_lFTj", 
  "_$s13SwiftProtobuf7DecoderP15nextFieldNumberSiSgyKFTj", referenced from:
  "_$s13SwiftProtobuf7DecoderP24decodeSingularInt32Field5valueys0F0VSgz_tKFTj", referenced from:
  "_$s13SwiftProtobuf7DecoderP24decodeSingularInt64Field5valueys0F0VSgz_tKFTj", referenced from:
  "_$s13SwiftProtobuf7DecoderP25decodeSingularStringField5valueySSSgz_tKFTj", referenced from:
  "_$s13SwiftProtobuf7MessageP05protoC4NameSSvgZTq", referenced from:
  "_$s13SwiftProtobuf7MessageP06decodeC07decoderyqd__z_tKAA7DecoderRd__lFTq", referenced from:
  "_$s13SwiftProtobuf7MessageP13isInitializedSbvgTq", referenced from:
  "_$s13SwiftProtobuf7MessageP13unknownFieldsAA14UnknownStorageVvMTq", referenced from:
  "_$s13SwiftProtobuf7MessageP13unknownFieldsAA14UnknownStorageVvgTq", referenced from:
  "_$s13SwiftProtobuf7MessageP13unknownFieldsAA14UnknownStorageVvsTq", referenced from:
  "_$s13SwiftProtobuf7MessageP4hash4intoys6HasherVz_tFTq", referenced from:
  "_$s13SwiftProtobuf7MessageP8traverse7visitoryqd__z_tKAA7VisitorRd__lFTq", referenced from:
  "_$s13SwiftProtobuf7MessageP9isEqualTo7messageSbAaB_p_tFTq", referenced from:
  "_$s13SwiftProtobuf7MessagePxycfCTq", referenced from:
  "_$s13SwiftProtobuf7VisitorP13visitMapField9fieldType5value0G6NumberyAA01_bE0Vyqd__qd_0_Gm_SDy04BaseH0Qyd__AKQyd_0_GSitKAA0e3KeyH0Rd__AA0e5ValueH0Rd_0_r0_lFTj", "_$s13SwiftProtobuf7VisitorP23visitSingularInt32Field5value11fieldNumberys0F0V_SitKFTj", referenced from:
  "_$s13SwiftProtobuf7VisitorP23visitSingularInt64Field5value11fieldNumberys0F0V_SitKFTj", referenced from:
  "_$s13SwiftProtobuf7VisitorP24visitSingularStringField5value11fieldNumberySS_SitKFTj", referenced from:
  "_$s13SwiftProtobuf8_NameMapV0C11DescriptionO4sameyAEs12StaticStringV_tcAEmFWC", referenced from:
  "_$s13SwiftProtobuf8_NameMapV0C11DescriptionO8standardyAEs12StaticStringV_tcAEmFWC", referenced from:
ld: symbol(s) not found for architecture x86_64

(I redacted the from references since they contain internal code, but these symbols are referenced from the static library built with library distribution turned on)

4. SwiftProtobuf is built as a static library in our project, and I noticed that building it with SwiftProtobuf with library distribution in our project would fix the issue, but given the other issue raised above, we can't do that.

Do you have any idea how what the problem could be here? I'm pretty sure someone has used SwiftProtobuf in a Swift SDK before, so this problem should have surfaced. Why would building some Swift proto generated code for library distribution include some symbols that are not present when linking it to a SwiftProtobuf which is not built for library distribution.

• what OS you are developing on (Linux or macOS, including the version) macOS 10.15.
• for macOS, what v7ersion of Xcode you are using (xcodebuild -version), for Linux, what version of Swift (swift --version) Xcode 12.0 but reproducible on 12.2.
• what version of Swift is your code set to compile with (i.e. from project settings, etc.) Swift 5.2
• what branch/tag of SwiftProtobuf you are using (1.0.0, etc.) 1.12.0 but can reproduce with 1.14.0 as well.

Thanks for the help!

Thank you @tbkka. The discussion is helpful. I think keep as-is make sense.

If you'd like to continue pursuing this, please rebase against the new main branch and re-submit. Thank you!

If you'd like to continue pursuing this, please rebase against the new main branch and re-submit. Thank you!

If you'd like to continue pursuing this, please rebase against the new main branch and re-submit. Thank you!