Implementation of semantic highlighting

[prostakm]

I took a liberty to implement basic textDocument/semanticTokens request handling based on proposal here : https://github.com/microsoft/vscode-languageserver-node/blob/master/protocol/src/protocol.semanticTokens.proposed.ts.
Adjusted VSCode plugin accordingly

Semantic token disabled :	enabled :

Notes:

• I'm using editor.open sourcekit query to get as much data as possible to construct sematic tokens, it wasn't enough, so I needed to access indexed symbols (for type references detection) thus IndexStoreDB dependency added for server implementations.
• token modifier are not covered
• token legend is hardcoded - eventually it should be negotiated between server and client
• Requires Visual Studio Code Insiders build (nightly builds) as it uses proposed API
• Could use some more tests

About editor.open request and indexed symbols - do you see any other way how to gather token data? I think index and doc_info requests from SourceKit would provide more than enough additional information to properly parse semantic tokens, but I guess since workspace management and index is handled externally, those requests can't be used directly (I was getting 'missing primary file' errors when experimenting with it, altough I didn't spent much time trying to make it work)? Any suggestions here?

Cheers!

[benlangmuir]

Wow, very cool!

You should be able to get everything you need directly from sourcekitd. Using the index is straightforward for a document that hasn't been modified, but it will quickly get difficult to keep up with changes.

Luckily, sourcekitd provides both syntactic tokens (what you are using now) as well as semantic tokens, which should allow you to colour type references, etc. The trick is that the semantic tokens are not returned directly by opening the document - instead they get calculated after we build an AST, at the same time we produce semantic diagnostics. Here is how it could work:

• Instead of opening a new document, capture the syntax map from the existing one -- when we do textDocument/didOpen we currently ignore the syntax map, but you can capture it there.
• We can get semantic tokens after an AST is built, which will send a notification_documentupdate. To capture the tokens, you will want to modify handleDocumentUpdate. The same sourcekit response that we are using for diagnostics there will also contain the semantic tokens in the same format as the syntax map. https://github.com/apple/sourcekit-lsp/blob/5cc541312d6221f58dbf38e2c3376eb98721ce61/Sources/SourceKit/sourcekitd/SwiftLanguageServer.swift#L132
• Update the information after edits. During textDocument/didChange, we perform a "replacetext" request to sourcekitd, which returns the set of modified tokens.

I don't know if the exact details of which tokens will be returned after an edit are documented, but feel free to ask any questions you have on the forums and I or someone can help out. @nathawes was the last person to make major changes to the highlighting API in sourcekitd, so maybe he knows if there was any documentation.

[shahmishal]

The Swift project moved the default branch to main and deleted master branch, so GitHub automatically closed the PR. Please re-create the pull request with main branch.

More detail about the branch update - https://forums.swift.org/t/updating-branch-names/40412

[mackoj]

Is this completely dead ? Since semantic token have moved out of proposed state.

[prostakm]

Sorry, I was quite busy with other things past few months - I have plan to pick it up starting today and push some changes in next few weeks

[DavidGoldman]

I have #388 to add the Language Server Protocol types for semantic highlighting as well as clangd support, once that is submitted we'll just need to rework this PR in order to add Swift semantic highlighting.