Skip to content

dataloader support for vector of graphs #143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 5, 2022
Merged

Conversation

CarloLucibello
Copy link
Member

@CarloLucibello CarloLucibello commented Mar 5, 2022

Batching graphs is faster than selecting graphs from a batched graph, so we should reconsider how we feed data to the DataLoader.

using Flux, GraphNeuralNetworks, BenchmarkTools

function test(g)
    loader = Flux.DataLoader(g, batchsize=100)
    a = 0 
    for d in loader
        a += 1
    end
    return a
end

n = 5000
s = 10
data = [rand_graph(s, s, ndata = rand(1, s)) for i in 1:n] 
x1 = Flux.batch(data) 
x2 = Flux.batch([rand(s + s + s + s) for i in 1:n]) #source+target+data+extra

@btime test($x1);   #  1.295 s (2502 allocations: 6.17 MiB)
@btime test($x2);   #  357.595 μs (152 allocations: 1.61 MiB)
@btime test($data); #  65.288 ms (227002 allocations: 27.00 MiB) # this PR

Close #141.

@codecov
Copy link

codecov bot commented Mar 5, 2022

Codecov Report

Merging #143 (9b59000) into master (f935e8d) will increase coverage by 0.16%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #143      +/-   ##
==========================================
+ Coverage   85.38%   85.55%   +0.16%     
==========================================
  Files          15       15              
  Lines        1259     1253       -6     
==========================================
- Hits         1075     1072       -3     
+ Misses        184      181       -3     
Impacted Files Coverage Δ
src/GNNGraphs/gnngraph.jl 80.00% <100.00%> (+5.71%) ⬆️
src/GNNGraphs/utils.jl 74.00% <0.00%> (-1.25%) ⬇️
src/GNNGraphs/convert.jl 90.08% <0.00%> (-0.32%) ⬇️
src/GNNGraphs/query.jl 92.98% <0.00%> (-0.05%) ⬇️
src/GNNGraphs/transform.jl 96.71% <0.00%> (-0.04%) ⬇️
src/GNNGraphs/sampling.jl 100.00% <0.00%> (ø)
src/layers/conv.jl 79.52% <0.00%> (+0.17%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f935e8d...9b59000. Read the comment docs.

@CarloLucibello CarloLucibello merged commit 05daca6 into master Mar 5, 2022
@CarloLucibello CarloLucibello deleted the cl/graphdata branch March 23, 2022 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Slow interaction with DataLoader
1 participant