[libSyntax] Make the ByteTree format forwards-compatible

[ahoppen]

Adjust the ByteTree format so that the server can add new versions to the tree it serialises while the client is still able to deserialise it.

To test the forwards compatibility, I added a flag that can be passed to the serialisation of the syntax tree as a ByteTree that will add additional garbage fields that the client should ignore. While this is not exactly beautiful, it is way easier to maintain than generating an entire class structure, constructing an object tree and adding the same class layout on the Swift side just to diff the output.

[ahoppen]

@swift-ci Please smoke test

[rintaro]

Incompatibility happens if we:

• Change structure of byte stream
• Increase number of fields
• Decrease number of fields
• Change order of fields
• Add new kind of node
• Drop kind of node
• Change meaning of values

IIUC, This PR only improves "Increase number of fields" case. I'm not sure it's worth to do it.
Could you give me a big picture?

[ahoppen]

Yes, the only change that is made compatible by this patch is to add new fields, but I think it comes at a pretty low cost and we should be able to get pretty much the same compatibility that we used to have from JSON from it:

• Change structure of byte stream
  • Yes, changing that structure is not compatible but I don't see a way to make it compatible either
• Increase number of fields
  • This is made compatible by this PR
• Decrease number of fields
  • If we want to make a compatible change, we can just leave the old fields there while we add the new data to new fields at the end of an object. Same as in JSON where we can't simply drop a key.
• Change order of fields
  • Since we use the order of the fields as the keys for the fields, this is like wanting to change the keys in a JSON object. We were able to do without changing the key names so far, so I don't think it's a huge deal that we can't reorder fields.
• Add new kind of node
  • Coming to think of it, we might want to default to an unknown node for this. I'll do it in a follow-up PR.
• Drop kind of node
  • This has always been compatible. We only continue to reuse a subset of the nodes that we used to and so the client is still able to deserialise all of them
• Change meaning of values
  • This again, is like changing the values that we used to use in JSON which we weren't able to do previously as well.

The general idea is that I just want us to be able to do minor modifications without breaking serialisation compatibility and not regretting later that we didn't implement this earlier. If we need to make an incompatible change, we can still increase the version number.

A few examples that come to my mind right now:

• If we decide that coupling a node's text length is too much of a performance problem, we might want to pass it over from the server as an additional field. Old versions of the deserialiser continue to compute the text length themselves.
• Deserialising the text contents of tokens and trivia is currently a performance bottleneck on the client side. To solve this in a compatible way, we could add two new fields offset a length to those nodes that point to an internal buffer that is already stored on the client. It would simply ignore the old string that is still being passed and slice it's own internal buffer for the text. Old versions of the deserialiser continue to decode the string that is still being passed.

[ahoppen]

@swift-ci Please smoke test

[ahoppen]

@swift-ci Please smoke test

[rintaro]

Thanks for the example use case in your mind!
The patch itself looks good.

[rintaro]

It would be nice if we could implement independent unit tests to test serialization/deserialization.

[ahoppen]

@swift-ci Please smoke test and merge