Add debug descriptions for Lexeme and LexemeSequence

[ahoppen]

The debug description of Lexeme outputs the current token’s text and the debug description of LexemeSequence outputs the source code that’s remaining in the LexemeSequence. In practice this means that po currentToken output’s the current token’s text and po lexemes shows the remaining source code.

[ahoppen]

@swift-ci Please test

[CodaFi]

shows the remaining source code.

That's... a lot to print. Perhaps we can display the code up to the next newline or some fixed set of bytes to lookahead like 100.

[ahoppen]

That's... a lot to print. Perhaps we can display the code up to the next newline or some fixed set of bytes to lookahead like 100.

Just printing to the end of the next newline is sometimes now sufficient to disambiguate where you are in the source file if you’ve got code snippets that are repeated.

I also thought about capping at a character limit but decided against capping it because

• I couldn’t decide on a limit
• I tried printing the remaining source code in one of the self-parser tests and it was fine
• I expect this to mostly be used in test cases that have already been reduced, so there’s no need to cap it
• I have a vague memory that the remaining source code is capped at some point by the debugger and that annoyed me once

I don’t have super strong feelings about it though.

[CodaFi]

This is backwards - it’ll print the next then the current source text behind it. If there were delimiters here that call that out I’d be happier.

[ahoppen]

My ide was that po self.lexemes should be analogous to po *Ptr (or something like this) in the C++ parser, i.e. print the remaining source text, except for the current token. And AFAICT that’s what it does. self.nextToken prints the next token and SyntaxText(baseAddress: self.cursor.input.baseAddress, count: self.cursor.input.count) prints the remaining source text thereafter.

In my experience printing self.lexemes is useful to disambiguate where in the source file the current token is positioned, e.g. if there are multiple function declarations in the file and we’re currently at a func keyword.

If you want to inspect the next token in particular, you can always po self.peek(), which will print the next token similar to self.currentToken.

[CodaFi]

Like *Ptr

I think that line of reasoning works for Lexer.Cursor, but for the lexeme sequence I would very much expect to see the current token, the lookahead token, and then the rest of the buffer.

[ahoppen]

How exactly would you expect those to be printed?

[CodaFi]

Since this has sat long enough, I'll just ask for a delimiter between these two pieces to separate the head and tail of the sequence. I still think we ought to limit the amount of text returned in that tail (even lldb gives up after a while and gives you ...), but it's not as important.

[CodaFi]

It’s a good start. Maybe we can render the flagset here. And even dump the trivia.

[ahoppen]

The flags are still being rendered if you po currentToken in the parser. This is what the debug output looks like with this change.

(lldb) po self.currentToken
▿ func 
  - tokenKind : SwiftSyntax.RawTokenKind.funcKeyword
  ▿ flags : Flags
    - rawValue : 0
  ▿ start : 0x0000000104014000
    - pointerValue : 4362158080
  - leadingTriviaByteLength : 0
  - textByteLength : 4
  - trailingTriviaByteLength : 1

[ahoppen]

@swift-ci Please test