Inaccurate GATv2Conv Documentation

[Animiral]

The documentation on GATv2Conv says:

the attention coefficients $\alpha_{ij}$ are given by

$$\alpha_{ij} = \frac{1}{z_i} \exp(\mathbf{a}^T LeakyReLU([W_2 \mathbf{x}_i; W_1 \mathbf{x}_j]))$$

This is not exactly the same as the Equation (7) from “How Attentive are Graph Attention Networks?”:

$$e(\mathbf{h}_i, \mathbf{h}_j) = \mathbf{a}^T LeakyReLU(\mathbf{W} \cdot [\mathbf{h}_i || \mathbf{h}_j])$$

I also checked the relevant piece of code (conv.jl):

    function message(Wix, Wjx, e)
        Wx = Wix + Wjx
        # [...]
        eij = sum(l.a .* leakyrelu.(Wx, l.negative_slope), dims=1)   # 1 × heads × nedges
        α = exp.(eij)
        return (α = α, β = α .* Wjx)
    end

Thanks to my project supervisor, I now realize that this is all correct and equivalent to the paper definition.
So it appears that the issue lies with the documentation.
Instead of a vector concatenation, it should state addition, or stick with the original order of operations.

As a small side note, the markup on GATConv is a bit broken, and it says “Dafault” instead of “Default”.

[CarloLucibello]

Thanks to my project supervisor, I now realize that this is all correct and equivalent to the paper definition.

thanks for checking this out, the equivalence is not immediately obvious, maybe we should add a comment about that in the code

So it appears that the issue lies with the documentation. Instead of a vector concatenation, it should state addition, or stick with the original order of operations.

I agree, it should state addition. I'll leave you the honor of filing a PR if you have time, otherwise I'll see to it soon.