Skip to content

Add debug descriptions for Lexeme and LexemeSequence #758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 22, 2022
Merged

Add debug descriptions for Lexeme and LexemeSequence #758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b604a48
Add debug description to lexeme printing the lexeme’s content
ahoppen Sep 20, 2022
f9d8e9f
Make LexemeSequence CustomDebugStringConvertible
ahoppen Sep 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions Sources/SwiftParser/Lexer.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ public struct Lexer {
/// A lexeme is the fundamental output unit of lexical analysis. Each lexeme
/// represents a fully identified, meaningful part of the input text that
/// will can be consumed by a ``Parser``.
public struct Lexeme {
public struct Lexeme: CustomDebugStringConvertible {
public struct Flags: OptionSet {
public var rawValue: UInt8

Expand Down Expand Up @@ -101,13 +101,17 @@ public struct Lexer {
SyntaxText(baseAddress: start.advanced(by: leadingTriviaByteLength+textByteLength),
count: trailingTriviaByteLength)
}

public var debugDescription: String {
return String(syntaxText: SyntaxText(baseAddress: start, count: byteLength))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s a good start. Maybe we can render the flagset here. And even dump the trivia.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flags are still being rendered if you po currentToken in the parser. This is what the debug output looks like with this change.

(lldb) po self.currentToken
▿ func 
  - tokenKind : SwiftSyntax.RawTokenKind.funcKeyword
  ▿ flags : Flags
    - rawValue : 0
  ▿ start : 0x0000000104014000
    - pointerValue : 4362158080
  - leadingTriviaByteLength : 0
  - textByteLength : 4
  - trailingTriviaByteLength : 1

}
}
}

extension Lexer {
/// A sequence of ``Lexer/Lexeme`` tokens starting from a ``Lexer/Cursor``
/// that points into an input buffer.
public struct LexemeSequence: IteratorProtocol, Sequence {
public struct LexemeSequence: IteratorProtocol, Sequence, CustomDebugStringConvertible {
fileprivate let start: Lexer.Cursor
fileprivate var cursor: Lexer.Cursor
fileprivate var nextToken: Lexer.Lexeme
Expand Down Expand Up @@ -154,6 +158,15 @@ extension Lexer {
func peek() -> Lexer.Lexeme {
return self.nextToken
}

public var debugDescription: String {
let remainingText = self.nextToken.debugDescription + String(syntaxText: SyntaxText(baseAddress: self.cursor.input.baseAddress, count: self.cursor.input.count))
if remainingText.count > 100 {
return remainingText.prefix(100) + "..."
} else {
return remainingText
}
}
}

@_spi(RawSyntax)
Expand Down
2 changes: 1 addition & 1 deletion Sources/SwiftParser/SwiftParser.docc/FixingBugs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Guide to write test cases in the parser’s test suite and how to debug failures

The general approach to fixing bugs in the parser is to first write an automated test case that reproduces the test case in isolation. This allows you to invoke the parser with minimal dependencies and allows you to set breakpoints inside of it.

Once you’ve written a test case (see below), set a breakpoint in `Parser.parseSourceFile` and navigate the debugger to the place where the parser behaves unexpectedly. While the debugger is stopped at an instance function in <doc:SwiftParser/Parser>, `po self.currentToken` can show you the next token that will be parsed.
Once you’ve written a test case (see below), set a breakpoint in `Parser.parseSourceFile` and navigate the debugger to the place where the parser behaves unexpectedly. While the debugger is stopped at an instance function in <doc:SwiftParser/Parser>, `po self.currentToken` can show you the next token that will be parsed. `po self.lexemes` can show the next tokens that will be parsed.

## Round-Trip Failure or Parser Crash

Expand Down