Implement semantic highlighting for Swift

[fwcd]

This is a reboot of @prostakm's implementation of semantic highlighting in #279, using the LSP structures from #388 and the suggestions from #279 (comment).

Both lexical and semantic tokens are now stored in Documents and updated whenever a document is updated or changed. Lexical tokens are updated incrementally in openDocument and changeDocument and semantic tokens are updated all at once in handleDocumentUpdate.

Implementation Progress

• Parse syntax map and annotations from SourceKit
• Update lexical tokens on document open and change events from SourceKit
  • Remove and shift existing lexical tokens correctly on edit events in DocumentManager
  • Handle edge cases where lexical tokens aren't removed correctly, e.g. when typing let document = ... the do stays highlighted as a keyword and isn't removed, since the edit range never overlaps with it (this is solved by checking whether an edit that is directly adjacent to a token uses a whitespace character at the adjacent position)
• Update semantic tokens on document update events from SourceKit
• Implement (full) semantic token requests
• Implement ranged semantic token requests
• Implement semantic token modifiers
• Provide semantic tokens for declarations
• Add test case for lexical tokens
• Add test case for semantic tokens

Future Work

• Add LSP config options so users can selectively enable/disable the different kinds of tokens (semantic, lexical)
  • This could also be useful for testing
• Implement semantic token delta responses
• Parse syntactic tokens from the edit response's keys.substructure (i.e. revert 1e65730) and address the performance issues in big files (see this thread)
  • A possible approach would be to parse the substructure asynchronously into syntactic tokens, this should be possible since they aren't updated incrementally like the lexical tokens

[ahoppen]

OK, I’ve left a few™ comments inline. Don’t be afraid by the amount. Overall, I think the PR is really good and I don’t think it’s anything too major.

[ahoppen]

I’ve added some more comments inline.

[ahoppen]

Thanks a lot for continuing to push this through. 😍

I’ve got a few more comments inline, but I think we’re nearly there.

[fwcd]

I have removed the syntactic tokens for now, which should resolve the performance issues in large files.

Also the language server now parses lexical syntaxtype_identifier tokens into a new semantic token type identifier. The reason for not using one of the standard LSP token types is that none of them provides a clear mapping for 'plain' identifiers that could be either a variable, function, module, parameter or something else. By using a custom token type, clients will ignore these by default and can choose a more specific highlighting from their own grammar instead (e.g. as in the case of VSCode), while others may choose to find these tokens useful.

[benlangmuir]

I was trying this out in vscode, and had a couple of questions:

• Is defaultLibrary not working, or is vscode just not using that from its builtin themes? I don't see any differentiation.
• It seems like in "public func" the theme wants to emphasize the "public" (modifier) instead of the "func" (keyword). I would have expected the opposite, or that they would look the same. Do you know what other language servers are doing here?

This doesn't need to block the PR though.

---

Other than my comment about the timeout, I think this is probably ready to go, although @ahoppen should probably sign off on it as well. Thanks for your hard work @fwcd !

[benlangmuir]

@swift-ci please test

[DavidGoldman]

Is defaultLibrary not working, or is vscode just not using that from its builtin themes? I don't see any differentiation.

The default theme doesn't use it, but you can confirm it's there by using the Developer: Inspect Editor Tokens and Scopes command and clicking on one of the library symbols

[fwcd]

I've bisected the PR and found that 4bbff3d broke (at least) the document symbol test, since it changes the semantics of the SKDResponseDictionary accessors slightly. Since returning nil in the absence of a value seems to be the 'right thing' to do, I will update the tests accordingly.

[benlangmuir]

@swift-ci please test

[ahoppen]

I just went through this again and have a couple more inline comments. Once they’re taken care of, I’m go for merge 🚀.

[ahoppen]

I also filed SR-15058 with some areas for improvement that are out of scope for this PR, to make sure we don’t loose track of them.

[benlangmuir]

@swift-ci please etst

[DavidGoldman]

@swift-ci please test

[ahoppen]

I’ve got two minor follow-up comments to my last review.

Could you also squash your commits?

[fwcd]

Sure, should I squash the entire PR into a single commit or do you think that a more granular splitting would be better, given that the PR has become pretty large already?

[ahoppen]

I can’t really think of any way the changes can be split up into commits that are at least somewhat independent, so I think one big commit is fine.

[fwcd]

Okay, I'd vote to keep 4bbff3d separate though, since it is a (slight) change to the accessors' semantics that isn't directly related to semantic highlighting per se.

[ahoppen]

Sounds good. 👍 If you have more of these commits, I think keeping them separate would be good. All I tried to say is that I don’t think it’s worth the effort to artificially split up the PR into smaller commits.

[ahoppen]

@swift-ci Please test

[ahoppen]

LanguageServerProtocolTests.CodingTests testValueCoding failed

Test Case '-[LanguageServerProtocolTests.CodingTests testValueCoding]' started.
/Users/buildnode/jenkins/workspace/swift-sourcekit-lsp-PR-macOS/branch-main/sourcekit-lsp/Tests/LanguageServerProtocolTests/CodingTests.swift:333: error: -[LanguageServerProtocolTests.CodingTests testValueCoding] : XCTAssertEqual failed: ("[
  {
    "kind" : 12,
    "name" : "mySymbol",
    "range" : {
      "end" : {
        "character" : 0,
        "line" : 6
      },
      "start" : {
        "character" : 23,
        "line" : 5
      }
    },
    "selectionRange" : {
      "end" : {
        "character" : 0,
        "line" : 6
      },
      "start" : {
        "character" : 23,
        "line" : 5
      }
    }
  }
]") is not equal to ("[
  {
    "children" : [

    ],
    "kind" : 12,
    "name" : "mySymbol",
    "range" : {
      "end" : {
        "character" : 0,
        "line" : 6
      },
      "start" : {
        "character" : 23,
        "line" : 5
      }
    },
    "selectionRange" : {
      "end" : {
        "character" : 0,
        "line" : 6
      },
      "start" : {
        "character" : 23,
        "line" : 5
      }
    }
  }
]")
Test Case '-[LanguageServerProtocolTests.CodingTests testValueCoding]' failed (1.950 seconds).

[fwcd]

Good catch, should be fixed now.

[DavidGoldman]

@swift-ci Please test