Skip to content

Don't use @assert, and print some debugging info if C is not approximately equal to reference #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 1, 2021
Merged

Don't use @assert, and print some debugging info if C is not approximately equal to reference #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
df4cd5c
Don't use `@assert`, and print some debugging info if `C` is not appr…
DilumAluthge Feb 1, 2021
406c154
Update runbenchmark.jl
DilumAluthge Feb 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "BLASBenchmarksCPU"
uuid = "5fdc822c-4560-4d20-af7e-e5ee461714d5"
authors = ["Chris Elrod <[email protected]> and contributors"]
version = "0.2.1"
version = "0.3.0"

[deps]
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
Expand Down
28 changes: 25 additions & 3 deletions src/runbenchmark.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,23 @@ function maybe_sleep(x)
end

function benchmark_fun!(
f!::F, summarystat, C, A, B, sleep_time, force_belapsed = false, reference = nothing
f!::F,
summarystat,
C,
A,
B,
sleep_time,
force_belapsed,
reference,
comment::String, # `comment` is a place to put the library name, the dimensions of the matrices, etc.
) where {F}
maybe_sleep(sleep_time)
t0 = @elapsed f!(C, A, B)
isnothing(reference) || @assert C ≈ reference
if (reference !== nothing) && (!(C ≈ reference))
msg = "C is not approximately equal to reference"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reference is the first of the libraries passed to runbenchmark, so perhaps the order does matter -- we should trust that the reference is correct.
When we get this error, we know either the ref or lib is wrong.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite sure what you mean. Can you elaborate on this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see what you mean.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we instead just always use LinearAlgebra.mul! to compute the reference C0?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That could take potentially many extra seconds at larger sizes, and do we trust it more than BinaryBuilder OpenBLAS or MKL (or BLIS)?
The time issue would be major when it's integers, and mul! calls the generic matmul.
I guess it'd be nice to always know what the reference is, though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. Hmmm.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep the current behavior for now.

@error(msg, comment)
throw(ErrorException(msg))
end
if force_belapsed || 2t0 < BenchmarkTools.DEFAULT_PARAMETERS.seconds
maybe_sleep(sleep_time)
br = @benchmark $f!($C, $A, $B)
Expand Down Expand Up @@ -287,8 +299,18 @@ function runbench(
last_perfs[1] = (:Size, (M,K,N) .% Int)
for i ∈ eachindex(funcs)
C, ref = i == 1 ? (C0, nothing) : (fill!(C1,junk(T)), C0)
lib = library[i]
comment = "lib=$(lib), M=$(M), K=$(K), N=$(N)"
t = benchmark_fun!(
funcs[i], summarystat, C, A, B, sleep_time, force_belapsed, ref
funcs[i],
summarystat,
C,
A,
B,
sleep_time,
force_belapsed,
ref,
comment,
)
gffactor = 2e-9M*K*N
@inbounds for k ∈ 1:4
Expand Down