JSON Serialization with default value

[seanliu1]

Developing using iOS12, Swift 5, proto3 . I am about to add an extension which can support to output fields with their default values. I just want to check whether it is already implemented.

Based on proto doc, it looks like

JSON options
A proto3 JSON implementation may provide the following options:

Emit fields with default values: Fields with default values are omitted by default in proto3 JSON output. An implementation may provide an option to override this behavior and output fields with their default values.

I wonder does swift version has option to output fileds with their default values. I found python version has it MessageToJson(message, including_default_value_fields=False)

https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.json_format-module

[thomasvl]

The supported JSON encoding options: https://github.com/apple/swift-protobuf/blob/master/Sources/SwiftProtobuf/JSONEncodingOptions.swift

[thomasvl]

p.s. - adding this as an option would be tricky since JSON encoding is built on the visitor pattern which is where it is skipping default values.

[seanliu1]

@thomasvl thanks for your quick reply. Yes, I realized that default value is skipped in the generated code.

  func traverse<V: SwiftProtobuf.Visitor>(visitor: inout V) throws {
    if !self.cards.isEmpty {
      try visitor.visitRepeatedMessageField(value: self.homeCards, fieldNumber: 1)
    }
    if self.test != false {
      try visitor.visitSingularBoolField(value: self.test, fieldNumber: 2)
    }
    try unknownFields.traverse(visitor: &visitor)
  }

[tbkka]

Is there a way to do this without breaking the API between generated code and the library?

I see how old generated code could continue to work with an updated library, but I don't see how new generated code (with modified default handling) could work with an old library.

If we can't preserve this compatibility, we would probably want to increment the major version. It's been a while since 1.0, so this is probably okay. We should certainly consider whether there are other issues that might require API breakage that should be adopted along with this.

[seanliu1]

What I found is that current we do the check in codegen xx.pb.swift.

  func traverse<V: SwiftProtobuf.Visitor>(visitor: inout V) throws {
    if self.showID != 0 {
      try visitor.visitSingularUInt32Field(value: self.showID, fieldNumber: 1)
    }
    try unknownFields.traverse(visitor: &visitor)
  }

What if we add an option to JSONEncodingOptions, then

we change

  mutating func visitSingularUInt32Field(value: UInt32, fieldNumber: Int) throws {

     if value != 0 || options.includeDefaultParameters {
            try startField(for: fieldNumber)
            encoder.putUInt32(value: value)
        }
}

Then API will remain same, and then we do check inside actual visitSingularUInt32Field or visitSingularUInt64Field or visitSingularStringField

[tbkka]

This is a good idea. I'd need to think about how this works for proto2, though.

[thomasvl]

The visitor api shouldn't know about the JSONEncodingOptions, we'd need to make a some other (internal) "options" to expose what ever subset makes sense for all possible visitors.

[seanliu1]

I think JSONEncodingVisitor currently owns JSONEncodingOptions, and we are using JSONEncodingOptions.alwaysPrintEnumsAsInts in the JSONEncodingVisitor Implementation. If this is something we want to support across all visitors, you are right, it probably should some other internal options.

 mutating func visitSingularEnumField<E: Enum>(value: E, fieldNumber: Int) throws {
    try startField(for: fieldNumber)
    if !options.alwaysPrintEnumsAsInts, let n = value.name {
      encoder.appendQuoted(name: n)
    } else {
      encoder.putEnumInt(value: value.rawValue)
    }
  }

[thomasvl]

The call to JSONEncodingVisitor is happening on the generic visitor interface and that call is blocked on the value not being the default, so all the JSON specific types are removed from where the change is needed.

[seanliu1]

The call to JSONEncodingVisitor is happening on the generic visitor interface and that call is blocked on the value not being the default, so all the JSON specific types are removed from where the change is needed.

Oh, I see. Yes, the original approach I proposed required all visitor to include default value check, which is not really flexible. Another way in my mind.

We add a function into Visitor protocol

protocol Visitor {
    func shouldInlcludeDefault() ->Bool
    ....
}

We provide default implementations as other methods in Visitor protocol

extension Visitor {
    func shouldInlcludeDefault() ->Bool {
        return false
    }
    ...
}

In JSONEncodingVisitor, we can implement it by checking shouldIncludeDefaultValue from options.

The last change will be codegen.

All traverse function will require to call

  func traverse<V: SwiftProtobuf.Visitor>(visitor: inout V) throws {
    if self.id != 0 || visitor.shouldIncludeDefaultValue() {
      try visitor.visitSingularUInt32Field(value: self.showID, fieldNumber: 1)
    }
}

I guess this can be also used by formatTextString.

@tbkka @thomasvl let me know what do you think.

[tbkka]

@seanliu1's idea would have:

• Old generated code with new library: default values would always be omitted
• New generated code with old library: default values would always be included

I'll have to think about whether this is enough compatibility to avoid the major version bump.

[seanliu1]

@tbkka see this comments. #861 (comment)

People wont notice any change, unless shouldIncludeDefaultValue is provided.
So

Old generated code with new library: default values would always be omitted
New generated code with old library: default values would always be omitted