Support HEEx/Surface markup

[jonatanklosko]

We need to handle these syntaxes. These should be standalone grammar handling .heex/.surface files, but we will also use tree-sitter injection to parse the relevant sigil nodes.

@connorlay already worked on both grammars, so we can just integrate them.

[jonatanklosko]

Regarding code organisation, on one hand it seems like a tree-sitter convention to keep all grammars as separate repositories, but separating heex/surface feels quite scattered. I saw that in tree-sitter-typescript they keep typescript and tsx grammars in separate directories, but they are bundled together (as the node/rust package), however that's a rather specific case because these are two variants of the same language and the grammars share most of the code. Since we are dealing with separate grammars, I feel that we should have separate tree-sitter projects, but we could place them in a single repo, each in its own directory. I'm just not sure if this will work well when installing the package directly from GitHub (which seems GitHub does). That said, I would be totally ok with separate repositories, just wouldn't want to spam elixir-lang with these :)

[josevalim]

My main concern about being a single repo or multiple repos is about further functionality. If we implement "Go to definition", what are the odds we want to share functionality between them? Especially when HEEx and Surface templates do allow calling local functions?

There is also EEx, which is definitely part of Elixir. If we want to support it, would we make it a separate language package? That sounds a bit too much to me.

If we are not sure, we can also wait until we start "Go to definition" before making a decision.

[jonatanklosko]

To support code navigation for <Mod.fun> I think we would parse Mod.fun into a single node and then use injections to parse it as Elixir code, which would be a regular function call. Not sure about <.fun> though, if we skip the dot and parse fun then it's gonna be an identifier rather than call 🤔 Either way, we certainly want separate heex/surface grammars, hence injections seems like the only reasonable way to integrate go-to-definition there.

As for sharing functionality, as far as I understand we need to add Elixir support in github/semantic, so go-to-definition code will live outside tree-sitter-elixir anyway.

There is also EEx, which is definitely part of Elixir. If we want to support it, would we make it a separate language package? That sounds a bit too much to me.

There is already an official tree-sitter-embedded-template grammar for ERB and EJS, we could probably reuse that given how similar EEx is?

[josevalim]

Alright, so it is either three directories in this repo or three separate repositories. Both are fine by me. It depends on how many colocated changes we will have to perform.

There is already an official tree-sitter-embedded-template grammar for ERB and EJS, we could probably reuse that given how similar EEx is?

Sounds good to me!

[jonatanklosko]

Alright, so it is either three directories in this repo or three separate repositories. Both are fine by me. It depends on how many colocated changes we will have to perform.

I don't expect any collocated changes, as the only integration point should be the injection queries.

So I'd lean towards separate repositories to adhere to the tree-sitter convention.

[josevalim]

So perhaps it is best to start a discussion about moving all of those to the tree-sitter organization. Or alternatively we move all of them to elixir-editors (or we start an elixir-tree org).

[jonatanklosko]

Ideally we would move to the tree-sitter org yeah, otherwise the elixir-editors sounds good to me (just so we don't have elixir-tree-sitter/tree-sitter-elixir ^^).

[josevalim]

Alright, so let's give tree-sitter-elixir a try on neovim, and, if solid, we should request GitHub to move to tree-sitter-elixir/surface/heex for syntax highlighting.

[connorlay]

Hey all, I've been busy the past few days and now am catching up on this thread :)

To support code navigation for <Mod.fun> I think we would parse Mod.fun into a single node and then use injections to parse it as Elixir code, which would be a regular function call. Not sure about <.fun> though, if we skip the dot and parse fun then it's gonna be an identifier rather than call thinking Either way, we certainly want separate heex/surface grammars, hence injections seems like the only reasonable way to integrate go-to-definition there.

This is the approach I took when integrating Surface with Neovim. I suggest we try to keep the HEEx/Surface/EEx grammars as similar to each other as possible to keep the maintenance overhead low when wiring up these injections.

As for sharing functionality, as far as I understand we need to add Elixir support in github/semantic, so go-to-definition code will live outside tree-sitter-elixir anyway.

I found this merged PR that added Rust support that illustrates what might be involved integrating Elixir into semantic. I think tag definitions are what drive the code navigation capability?

Alright, so let's give tree-sitter-elixir a try on neovim, and, if solid, we should request GitHub to move to tree-sitter-elixir/surface/heex for syntax highlighting.

Can GitHub move to tree-sitter-elixir for syntax highlighting before semantic support is added? Or is semantic integration blocking that as well? Either way, I'll get started on a branch migrating Neovim to the new grammar and see how that turns out.

[josevalim]

Can GitHub move to tree-sitter-elixir for syntax highlighting before semantic support is added?

From my understanding, yes. :)

Either way, I'll get started on a branch migrating Neovim to the new grammar and see how that turns out.

Yes, beautiful.

[jonatanklosko]

I think tag definitions are what drive the code navigation capability?

I think so, although code navigation is not supported for Rust at this point. I think the Rust tags are missing the normalization parts, see Ruby/Tags.hs. Generally Ruby seems like a minimal example, it has just three files, and AST.hs is automatically generated, so it's pretty much just Tags.hs as far as I can see.

[connorlay]

@josevalim My PR migrating Neovim to the new parser is open for full review now nvim-treesitter/nvim-treesitter#1904 🚀

[josevalim]

@connorlay beautiful! 😍

[josevalim]

Hi @connorlay, quick question: do you think we are ready to ask the GitHub team to move Elixir and Surface (and HEEx?) upstream?

[connorlay]

@josevalim For Elixir, I think we are ready to ask GitHub to switch over. @jonatanklosko I'm curious to hear what you think.

For Surface and HEEx, I have a couple questions:

1. How often does GitHub update their parser versions? If new syntax is introduced to the parsers, what does the upgrade path look like? Surface and HEEx are pre-1.0, so I expect more changes will occur to those compared to Elixir.
2. I wrote both Surface and HEEx parsers initially for neovim, which utilizes additional query capture definitions than what tree-sitter provides alone. Will we need to port the highlight queries from nvim-treesitter to tree-sitter-surface and tree-sitter-heex before GitHub can utilize them for syntax highlighting? @the-mikedavis has a PR integrating HEEx with the Helix editor that might solve this for us.
3. If GitHub is going to use the Surface and HEEx parsers, would it make sense to transfer the those repositories out of my personal namespace and to the elixir-lang organization? I am happy to do so as long as I retain my status as a maintainer 🙂