The problem is definitely on the encoding side, where we'll not even emit the bytes for myType if its value is 0. @stephenh, what's the reason for that behavior? I definitely think we should change it for oneof=unions, but I'd argue it should probably be changed for oneof=properties as well...

Actually, this somewhat consistent with the default (oneof=properties) behavior.

const eg3 = Example.encode({
  name: "David",
  myType: TypeA.x,
  enabled: undefined,
}).finish();
console.log(Example.decode(eg3));

also yields

{ name: 'David' }

Nice catch. Yes, I think that's a bug. Will provide a fix.

I don't find ts-proto's generated types particularly helpful because it forces me to always provide all fields (even if I set it to undefined), which is especially annoying when you have huge nested oneofs.

As an example, consider the following proto:

message Person {
  string name = 1;
  uint32 age = 2;
  oneof legal_document {
    Passport passport = 3;
    DriversLicense dl = 4;
    GovtIssuedId govt_id  = 5;
  }
}

ts-proto translates this to

export interface Person {
  name: string;
  age: number;
  passport: Passport | undefined;
  dl: DriversLicense | undefined;
  GovtIssuedId: govtId | undefined;
}

If I wanted to encode one of these messages, I'd have specify all fields:

Person.encode({
  name: "Bob",
  age: 37,
  passport: undefined,
  dl: undefined,
  govId: undefined,
})

That is tedious and you can imagine with large messages, especially nested oneofs, it becomes entirely impractical. I'm also debating whether we should need to specify age and name at this stage. proto3 says that all fields are essentially optional, though google.protobuf.*Value is the de-facto way to explicitly declare an optional value.

(Also, I personally would also consider code that explicitly sets a value to undefined to be bad form. IMHO, undefined is a fallback value that the JS/TS language provides when the thing you're trying to access cannot be found or was not defined. Code that explicitly wants to provide a null value should use null. That's my interpretation of JS's built-in nullish types.)

I think pbts actually gets this right. It would translate the above to

                interface IPerson {
                    name?: (string|null);
                    age?: (number|null);
                    passport?: (Passport|null);
                    dl?: (DriversLicense|null);
                    govtId?: (GovtIssuedId|null);
                }

which is the interface that consumed by Person.encode(). This means I can write

Person.encode({name: "Bob"})

and it's perfectly valid. We can debate whether name and age should be optional and nullable (probably not), but all the fields in the oneof definitely should be! And for clarity, I would argue that any value that's not a basic type should be optional too, to save tedious typing, having to set them to null (or, worse, undefined, which as said above, is not a value I'd ever expect application code to have to set, it's something to check for).

I'm happy to help with the implementation, and I'm also happy to hide any changes behind a compiler flag if that's preferable.

It is, thanks a bunch!

Woot! Thank you!

Awesome, thanks

Ping?

@stephenh any chance you could look at this over the next few days?

I think I understand the fromPartial concern now. @cliedeman, your example illustrates it perfectly and I think your proposed solution

T extends { $case: string }
  ? { [K in keyof Omit<T, '$case'>]?: DeepPartial<T[K]> } & { $case: T['$case'] }

works great. I think it's fine to require that if choice is filled out, you have to provide $case, but you don't have to provide anything else. I'll update the PR accordingly.

I'll also remove the optionality of message payloads inside the oneof as discussed.

(Btw, in your example you can do switch (data.choice?.$case) and avoid the outermost if that way.)

the From Partial allows the user to omit the case.

Yes, I think that's the whole point of fromPartial yes? :)

I think that the generated code should actually be ` | { $case: 'aMessage'; aMessage: PleaseChoose_Submessage /* undefined removed */ }

Hmm, I need to think about this some more, but I think you might be right. Either you chose the aMessage case and then you need to provide the value, or you didn't want to provide a value at all, but then there's no point to provide a $case. Yeah, I think that makes sense, but before I go back into the code, I might want @stephenh and @timostamm to weigh in.

I can't see what the failure is, circleci wants me to log in. :(

@cliedeman yeah, sorry, too many distractions, but PR is up now.

Addresses https://github.com/stephenh/ts-proto/issues/74

• replace for...of loops + manual index count with forEach loops which provide us with item + index out of the box
• replace boolean parameters with an object for better readability, as discussed in previous PR

All that's required is a bash version that's slightly more modern than the iPhone... The scripts work on Mac if you brew install bash. (Note how the shebang refers to /usr/bin/env bash and doesn't hardcode /bin/bash.)

Sure, the object type sgtm.

Ok I'll send a separate PR for that some time since this PR is ready for review.

I'm not sure I like the two optional bools at the end of the messageToTypeName parameter list:

  keepValueType: boolean = false,
  repeated: boolean = false

IMHO bool params are an anti-pattern anyway because a call like messageToTypeName(typeMap, field.typeName, options, true, false) gives you little insight into what true and false are doing.

What do you think about taking an object there instead? Something like this:

export function messageToTypeName(
  typeMap: TypeMap,
  protoType: string,
  options: Options,
  typeOptions: {keepValueType?: boolean, repeated?: boolean}
): TypeName {

This would translate the above call to messageToTypeName(typeMap, field.typeName, options, {keepValueType: true, repeated: false}) (or you could even leave out repeated).

Rebased on top of master, fixed linter errors, and added another commit that makes options a required param in messageToTypeName.

We can. The only callsites of messageToTypeName that don't currently have access to options are requestType and responseType. We could pass options into those, though, I think. I was just shying away from a bigger change.

Also includes a change not to generate default entries in base objects when the value would be undefined anyway as a separate commit. Probably best to review each commit individually.

Thanks everyone for your comments!

Let's split the oneofs/ADTs out from the optional option? I.e. a useOptionals that defaults to false and on true sets all non-primitive fields to firstName?: string instead of firstName: string | undefined seems good as a dedicated PR.

Agreed. I already have that pulled out and will send a separate PR.

@timostamm wrote:

The idea was that oneofCase is much more unlikely to clash with a field name.

@stephenh wrote:

Fwiw I've seen kind generally/conventionally used for these sorts of type unions.

@cliedeman wrote:

You can easily sidestep collisions by using a special character like $case or something.

All good points! 'case' might clash with a field name just as much as 'none' might. I do like the idea of $case, that's a brilliant way of using an identifier that's allowed in JS/TS but not in proto! I'd be open to $kind too but part of the reason I like the word "case" is because of its close association with the switch statement. I don't feel strongly about it, though.

Possible solution (3): case: undefined

That's basically the same as an optional field, since demo.result?.$kind would resolve to undefined if demo.result was not provided.

and it seems nice to have all N clauses of "could be kind A or kind B or kind C or kind none" all handled at the "same level"

The optional field would still allow you to do that:

interface Demo1 {
   result?: { $case: 'error', error: string }
          | { $case: 'value', value: number },
}

function process1(demo: Demo1) {
    switch (demo.result?.$case) {
        case 'error':
            break;
        case 'value':
            break;
        case undefined:
            break;
        default:
            throw new Error('Should not get here!');
            break;
    }
}

I will work on a PR formulating the above approach.

@timostamm your proposal was this:

interface Demo {
   result: { oneofCase: 'error', error: string }
         | { oneofCase: 'value', value: number }
         | { oneofCase: 'none' },
}

I find oneofCase a bit wordy, any reason we can't just call it case? It reads quite nicely IMHO:

interface Demo {
   result: { case: 'error', error: string }
         | { case: 'value', value: number }
         | { case: 'none' },
}

But more importantly, I'm not sure I like the magic string 'none'. It might very well be a valid field name within a oneof clause! We'd have a clash.

I see two possible solutions: (1) The oneofCase values come from an enum we define, similar to how google-protobuf does it:

interface Demo {
  enum ResultCase {
    NOT_SET = 0,
    ERROR = 1, // this would correspond to the protobuf field index,
    VALUE = 2, // so it might be anything except 0
  }
  result: { case: ResultCaseERROR, error: string }
        | { case: ResultCase.VALUE, value: number },
        | { case: ResultCase.NOT_SET },
}

(2) Instead of another type union, the result property is made optional:

interface Demo {
   result?: { case: 'error', error: string }
          | { case: 'value', value: number },
}

I personally much prefer the latter due because of its brevity. Yes, consumers will have to check that demo.result isn't null-ish before switch (demo.result.case), but that seems acceptable to me. Your thoughts?

nit: else after return is unnecessary noise

Addresses https://github.com/stephenh/ts-proto/issues/74

Closing this in favor of a future PR that will have different flags for non-scalar optionals and better oneof typing.

Woo thanks! I'll continue to work on #74. Let me know if you have any thoughts there, otherwise you may get a PR in the next few days or so...

addresses https://github.com/stephenh/ts-proto/issues/80

sure thing

As discussed in https://github.com/stephenh/ts-proto/issues/74, repeated fields can never have nullish values. This includes non-scalar fields, which are currently translated to Array<Field | undefined>, but should be Field[].

I like the way you're going with that, @timostamm. I was going to open another ticket + PR for adding a member to decoded objects that would tell you which oneof case is populated. I had originally thought of something like this:

interface Demo {
  result: 'error' | 'value' | null,
  error?: string,
  value?: number,
}

but your proposed solution is superior because it links which result is populated to its value type.

I propose two separate compiler flags then:

• useOptionals turns all non-scalar fields into optional properties rather than typing them as xyz | undefined (doesn't touch oneof fields)
• oneofUnions emits oneofs as you proposed

Members of repeated fields and maps cannot be null, regardless whether the element type is a message or scalar.

Yeah, which I think means types like https://github.com/stephenh/ts-proto/blob/master/integration/simple/simple.ts#L70-L71 are not correct and should probably be fixed. Should we file a separate ticket for that?

Addresses https://github.com/stephenh/ts-proto/issues/74

Closing in favor of https://github.com/stephenh/ts-proto/pull/76

Addresses https://github.com/stephenh/ts-proto/issues/74

I also care about type correctness, including strictNullChecks. I'm less concerned about the practical implications regarding checking null and undefined but I hear your point. I'd be happy with your proposal. I'll send a PR.

You mention giving non-scalars (and I'm assuming also fields within oneof clauses, which is one my motivators) this treatment. Does that mean you'd also advocate that it should apply to repeated fields? I've been going back and forth on that personally, but left them to be always non-optional and not-nullable for now (as I mentioned above, ts-proto in default mode actually has a bit of an inconsistency regarding the array value type IMHO.)

Thanks @timostamm. Yes, if all fields were optional, you'd have to do a lot of x.name ?? 'fallback type checks and I agree that would get very verbose.

As mentioned above, I think my initial post wasn't as clear as it could have been. I propose the nullableOptionals compiler flag (as implemented in the PR, please have a look) makes optional and nullable

• message fields
• fields within oneof

With that, a variation of my original example

message Person {
  string name = 1;
  google.protobuf.Uint32Value age = 2;
  oneof legal_document {
    Passport passport = 3;
    DriversLicense dl = 4;
    GovtIssuedId govt_id  = 5;
  }
}

would translate to

export interface Person {
  name: string;
  age?: number | null;
  passport?: Passport | null;
  dl?: DriversLicense | null;
  GovtIssuedId?: govtId | null;
}

So I can do Person.encode({name: 'Bob', passport: {...}}) but I can't do Person.encode({passport: {...}}) because name is a required field. Hope that makes sense.

Hmm, should

  coins: number[];
  snacks: string[];

really be

  coins: Array<number | null>;
  snacks: Array<string | null>;

perhaps? Not that it's actually possible to encode a "hole" in a repeated field currently. If I do this using ts-proto's default options:

    const s = SimpleWithWrappers.encode({
      name: undefined,
      age: undefined,
      enabled: undefined,
      coins: [undefined, 3],
      snacks: [],
    }).finish()
    const d = SimpleWithWrappers.decode(s)

I get d = { coins: [ 0, 3 ], snacks: [] } as a result, so Array<number | undefined> may as well have been Array<number>?

Addresses https://github.com/stephenh/ts-proto/issues/74

The drawback to making all fields optional in typescript is that it breaks strictNullChecks.

I don't quite follow. Can you elaborate with an example?

I think you guys have clarified an important distinction though. fromPartial would allow me to do the equivalent of Person.encode({name: 'Bob'}), as @timostamm explains. It's a pity that fromPartial is only generated when JSON methods are generated -- that's part of the reason why I missed that potential solution. Perhaps that's something we could change with a compiler flag? I might want fromPartial but none of the toJSON and fromJSON stuff.

But fromPartial is still different from what I was actually proposing but did a poor job defining well: making optional and nullable (a) message fields, (b) fields within aoneofclause, if thenullableOptionals=true` compiler flag were set. I have this working locally and this proto message

message SimpleWithWrappers {
  google.protobuf.StringValue name = 1;
  google.protobuf.Int32Value age = 2;
  google.protobuf.BoolValue enabled = 3;
  repeated google.protobuf.Int32Value coins = 6;
  repeated google.protobuf.StringValue snacks = 7;
}

translates to this

export interface SimpleWithWrappers {
  name?: string | null;
  age?: number | null;
  enabled?: boolean | null;
  coins: number[];
  snacks: string[];
}

const baseSimpleWithWrappers: object = {
};

// and the usual encode/decode functions

I'd be very happy with that as I'd still get type checking on all the properties that are supposed to be there, but I can either not specify at all or pass null for those that are clearly optional.

I don't find ts-proto's generated types particularly helpful because it forces me to always provide all fields (even if I set it to undefined), which is especially annoying when you have huge nested oneofs.

As an example, consider the following proto:

message Person {
  string name = 1;
  uint32 age = 2;
  oneof legal_document {
    Passport passport = 3;
    DriversLicense dl = 4;
    GovtIssuedId govt_id  = 5;
  }
}

ts-proto translates this to

export interface Person {
  name: string;
  age: number;
  passport: Passport | undefined;
  dl: DriversLicense | undefined;
  GovtIssuedId: govtId | undefined;
}

If I wanted to encode one of these messages, I'd have specify all fields:

Person.encode({
  name: "Bob",
  age: 37,
  passport: undefined,
  dl: undefined,
  govId: undefined,
})

That is tedious and you can imagine with large messages, especially nested oneofs, it becomes entirely impractical. I'm also debating whether we should need to specify age and name at this stage. proto3 says that all fields are essentially optional, though google.protobuf.*Value is the de-facto way to explicitly declare an optional value.

(Also, I personally would also consider code that explicitly sets a value to undefined to be bad form. IMHO, undefined is a fallback value that the JS/TS language provides when the thing you're trying to access cannot be found or was not defined. Code that explicitly wants to provide a null value should use null. That's my interpretation of JS's built-in nullish types.)

I think pbts actually gets this right. It would translate the above to

                interface IPerson {
                    name?: (string|null);
                    age?: (number|null);
                    passport?: (Passport|null);
                    dl?: (DriversLicense|null);
                    govtId?: (GovtIssuedId|null);
                }

which is the interface that consumed by Person.encode(). This means I can write

Person.encode({name: "Bob"})

and it's perfectly valid. We can debate whether name and age should be optional and nullable (probably not), but all the fields in the oneof definitely should be! And for clarify, I would argue that any value that's not a basic type should be optional too, to save tedious typing, having to set them to null (or, worse, undefined, which as said above, is not a value I'd ever expect application code to have to set, it's something to check for).

I'm happy to help with the implementation, and I'm also happy to hide any changes behind a compiler flag if that's preferable.