Meta: adding new keys to the schema

[SleeplessByte]

I've changed the title to be a meta discussion, so that we don't have the same discussion in multiple issues:

• optional flags Proposal: Add flag to mark test cases as optional to canonical schema #1492
• immutability flags High-Scores: add immutability test #1493
• other types of tests Schema optimises for testing f(inputs) = output. What of other types of tests? #1225
• multiline "type"
• identity field Proposal: Add canonical data "identifier" field to separate concern from test "description" #1473 (to avoid compromising descriptions Robot simulator descriptions too long #1525 to fit this purpose)

It seems there really is a lot of ask for new fields and there also seems to be a notion that we don't want to break the current generators.

Personally, I am currently parsing the comments for special behaviour and this means that my generators will all break when someone decides the comments should be a special format. I can derive no special meaning from the english in these comments. I don't think this is the right place to hold special cases, but it is for things not covered in the current scheme.

I know that you (@pgaspar @ErikSchierboom @coriolinus @sshine @SaschaMann and @NobbZ) are all involved with writing and maintaining the generators, or at least the discussions on GitHub:

• I think it would be wise to discuss first if we'll add new keys, if at all, because so far each issue has already raised the "yeah okay but we don't really need it and it costs me work"-type comment
• ...and then which ones are candidates, so we can merge it at once and the burden on maintainers is much lower, as the raised point just mentioned is valid.

---

Original issue about multiline.

There is canonical data such as food-chain and twelve-days that correctly comment:

JSON doesn't allow for multi-line strings, so all verses are presented
here as arrays of strings. It's up to the test generator to join the
lines together with line breaks.

Okay, that's great! But in this case I think it would be great if we'd add a meta property (and keep a list of those, as there are more like these (such as a property preferably being a constant or at least immutable) so that we can automate generation of these exercises.

[SaschaMann]

I think it would be wise to discuss first if we'll add new keys, if at all, because so far each issue has already raised the "yeah okay but we don't really need it and it costs me work"-type comment

Perhaps to address the first point it would be useful to know which tracks have generators that would stop working when solely adding keys (without changing the existing ones).
Usually adding non-mandatory fields to JSON is not considered a breaking change in semver, so I'd expect many/most parsers to handle this just fine (though that doesn't mean generators follow that versioning and can handle it!). Doing this could help estimating how much work it would cause.

---

Wouldn't break:

• C#
• Dart (according to @Stargator)
• Erlang (according to @NobbZ on Slack)
• F#
• JavaScript
• Julia (though the generator is in very early development, so it should probably not be counted here, just adding it for completeness)
• Go
• Pharo
• Ruby
• TypeScript

Would break:

Unsure (from #1411):

• Bash
• ColdFusion
• Common Lisp
• Factor
• OCaml
• Perl 6
• Rust
• Scala
• Vimscript

(feel free to edit my comment to move tracks around in the list)

[SleeplessByte]

@SaschaMann Great suggestion!

[macta]

Pharo (Smalltalk) has a reasonably comprehensive (although slightly hacked) generator that wouldn’t break by addition.

I’ve been trying to normalise inconsistent specs where I’ve encountered my generated exercises weren’t great (and am curious how the current generators handled them - allergies and etl were recent ones).

I’d love some extra flags to use as hints.

I currently treat exceptions specially - I have to look for an expected map with a single key of ‘error’ and then generate an exception check instead of an equality one.

I didn’t manage to handle clock very well with its instance equality comparison (I badly generated an isEqual(dict1,dict2) type call against the model instead of: assertEquals(clock1, clock2).

I also name test cases by concatenating subsequent levels of case descriptions - which works ok, but some cases are very long and often repetitive. Fortunately Pharo doesn’t mind 100+ char method names ;)

Some of the input variable names are not ideal , and some conventions around what would be a Boolean sounding property are not great.

I’m 25 exercises in - spot checked ~20 more but the others might well reveal more useful flags.

_{Sent with GitHawk}

[SleeplessByte]

@sshine You've responded in #1225 that you prefer comments because of what @coriolinus , which seems to be "because it's ambitious and might break current generators". Does this mean you have a generator that breaks?

[sshine]

@SleeplessByte: I should have been more clear:

Because property based test frameworks [...] have a programmatic interface.

[SleeplessByte]

👍

@sshine So is that instead of adding new flags, or on top of new flags.

[ErikSchierboom]

I'm a little confused as to the status of this issue. @sshine is it correct to say that you are not in favor of adding new flags/keys to the schema?

[kytrinyx]

I, personally, would be in favor of adding new flags/keys to the schema, as it seems the only way to support the kind of flexibility that we require at the track level (individual tracks want to include/exclude selectively).

While this would potentially break some generators, I am of the firm opinion that generators should be able to handle new keys by ignoring them, and if we're breaking because of that, it's worth fixing.