Add progress info for OptimizationBBO #336

SebastianM-C · 2022-08-10T21:40:03Z

It would be useful to be able to use OptimizationBBO without having all the output printed to stdout. In order to do this, we can use the TraceMode keyword argument, but that's specific to OptimizationBBO and doesn't translate to all the optimizers.
I propose in this PR a new common keyword argument, verbose that could be translated to backend specific keywords, like TraceMode in BBO. This is the first commit of the PR.

A more general approach would be to avoid anything from getting to stdout and use the logging system for any messages. This would allow us to translate the stdout in progress information and more optimization backends could show progress bars.
I have WIP implementation in the second commit which shows how this could work for OptimizationBBO. The stdout forwarding is based on JuliaLang/julia#12711 (comment)

For the moment you need to set the verbose keyword to :compact and progress to true to get progress logging, but maybe we could auto-set verbose to that if the user uses progress=true.

@Vaibhavdixit02 @ChrisRackauckas Let me know what you think about this. Should I try to go more on the logging route or should we just do the verbose keyword for now?

As an example, this is how VS Code would show the progress with the current implementation.

WIP implementation

codecov · 2022-08-10T21:50:40Z

Codecov Report

Merging #336 (0f31a23) into master (fe73975) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #336   +/-   ##
=======================================
  Coverage   79.34%   79.34%           
=======================================
  Files           9        9           
  Lines         276      276           
=======================================
  Hits          219      219           
  Misses         57       57

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ChrisRackauckas · 2022-08-11T00:15:36Z

lib/OptimizationBBO/src/OptimizationBBO.jl

 function SciMLBase.__solve(prob::SciMLBase.OptimizationProblem, opt::BBO, data=Optimization.DEFAULT_DATA;
    callback=(args...) -> (false),
    maxiters::Union{Number,Nothing}=nothing,
    maxtime::Union{Number,Nothing}=nothing,
    abstol::Union{Number,Nothing}=nothing,
    reltol::Union{Number,Nothing}=nothing,
-    progress=false, kwargs...)
+    verbose::Symbol=:compact,


we only allow true/false

Do we separate between progress and verbose?

I'm not sure how to handle

progress=false, verbose=true. Do we show all logs with @info here?

progress=true, verbose=false. Do we just show the progress here?

progress=true, verbose=true. And here we also add other info like in the screenshot?

Progress and verbose are different. Trace logs should all be handled by progress. Verbose should be left to things like throwing warnings.

Ok, so in that case all the functionality of the PR should be captured by progress and verbose will be reserved for warnings?
For the progress, should we display anything besides number of steps/maxiters?

@ChrisRackauckas Should I remove verbose from this PR and just always redirect the stdout and use it for progress logging when progress=true?

We shouldn't just redirect to stdout. We should always control it with progress and verbose ourselves.

Thanks for the comment. I reimplemented the PR using the callback and querying the BlackBoxOptim.OptRunController for info. I've also implemented time based or maxiters based progress.

Vaibhavdixit02 · 2022-08-11T05:49:42Z

lib/OptimizationBBO/src/OptimizationBBO.jl

+        if !isnothing(m)
+            s = tryparse(Int, m[1])
+            isnothing(s) && continue
+            isnothing(maxiters) && continue


We can assume the default for MaxSteps from BBO in this case as well https://github.com/robertfeldt/BlackBoxOptim.jl/blob/83c811671c00fffdcaadf042ad07ee2e2194ecbf/src/default_parameters.jl#L17

Is maxiters silently using that value? Should we have the maxiters set to that value if the user doesn't provide anything? I think we shouldn't assume a maxiters in the progress logging and have something else passed to BBO.

From what I understood from @ChrisRackauckas we want maxiters to be always explicitely provided by the users.

No. If the solver can do exiting properly, let it exit when tolerances are hit. A lot of solvers just don't do this. When there is no other exit criteria, maxiters is required.

lib/OptimizationBBO/src/OptimizationBBO.jl

SebastianM-C · 2022-08-12T12:57:05Z

lib/OptimizationBBO/src/OptimizationBBO.jl

+        wait(task)
+    end
+end
+
 function SciMLBase.__solve(prob::SciMLBase.OptimizationProblem, opt::BBO, data=Optimization.DEFAULT_DATA;


What is data and why do we set maxiters to length(data) if this is provided.

It's used for passing a dataset if you want to write a data based loss, for example if you have observations from a system and then write a dynamic model for it and estimate the parameters. Unfortunately it looks like we don't use it much in the docs, we should probably do that more. Minibatching is a good use case for this https://docs.sciml.ai/dev/modules/Optimization/tutorials/minibatch/.

why do we set maxiters to length(data) if this is provided.

I think this was a consequence of copy-pasting the optimizationflux implementation, it is done for two reasons, that the callback iterates over the data hence it is required that the number of calls of it are <= length of data else there will be a index error and second for minibatching where the length is the number of batches. I agree it should probably not be set to that

Co-authored-by: Vaibhav Kumar Dixit <[email protected]>

Reimplement progress logging using a callback. The required information is now extracted directly from the OptRunController.

SebastianM-C · 2022-08-18T19:45:52Z

I've reimplemented progress logging without doing any stdout redirection. The implementation is now simpler and it should be more robust. I've also added progress for time based limits. The only remaining part would be tests (and formatting).

Vaibhavdixit02 · 2022-08-19T08:18:55Z

I think it looks good now, I will go over it once more later today and merge, the test failures are unrelated

lib/OptimizationBBO/Project.toml

lib/OptimizationBBO/src/OptimizationBBO.jl

ChrisRackauckas · 2022-08-19T21:41:33Z

This looks good to go, just needs to run the formatter.

SebastianM-C · 2022-08-20T00:15:05Z

lib/OptimizationBBO/src/OptimizationBBO.jl

+
+    if progress
+        # Set progressbar to 1 to finish it
+        Base.@logmsg(Base.LogLevel(-1), msg, progress=1, _id=:OptimizationBBO)


msg is not defined here. Should it be ""?
Also, shouldn't this be after the bboptimize call?

I've added a commit to do that

SebastianM-C added 2 commits August 11, 2022 00:27

Add verbose kwarg

ced514e

Redirect stdout and extract progress

d2043c7

WIP implementation

add missing kwarg

5bfae34

ChrisRackauckas reviewed Aug 11, 2022

View reviewed changes

Vaibhavdixit02 reviewed Aug 11, 2022

View reviewed changes

SebastianM-C commented Aug 12, 2022

View reviewed changes

SebastianM-C and others added 4 commits August 15, 2022 22:25

Remove ProgressLogging dep

d69cd92

Update lib/OptimizationBBO/src/OptimizationBBO.jl

5b55512

Co-authored-by: Vaibhav Kumar Dixit <[email protected]>

Use callbacks

5ed421d

Reimplement progress logging using a callback. The required information is now extracted directly from the OptRunController.

Merge branch 'master' into redirect_stdout

119f9ce

SebastianM-C changed the title ~~[RFC] Redirect stdout and extract progress info~~ Add progress info for OptimizationBBO Aug 18, 2022

SebastianM-C added 4 commits August 18, 2022 22:46

Format log message

ded00d7

Format code

4cfaf78

customize log message

2fd5793

Add tests

186c4b9

SebastianM-C marked this pull request as ready for review August 18, 2022 20:18

Vaibhavdixit02 reviewed Aug 19, 2022

View reviewed changes

lib/OptimizationBBO/Project.toml Outdated Show resolved Hide resolved

remove ProgressLogging

a63752c

ChrisRackauckas reviewed Aug 19, 2022

View reviewed changes

lib/OptimizationBBO/src/OptimizationBBO.jl Show resolved Hide resolved

ChrisRackauckas added 3 commits August 19, 2022 16:45

Update lib/OptimizationBBO/src/OptimizationBBO.jl

943b612

Update OptimizationBBO.jl

45194fa

Update OptimizationBBO.jl

76db227

move progressbar finish after optimization

0f31a23

SebastianM-C commented Aug 20, 2022

View reviewed changes

ChrisRackauckas merged commit 5fc872e into SciML:master Aug 20, 2022

SebastianM-C deleted the redirect_stdout branch August 20, 2022 08:01

Uh oh!

Add progress info for OptimizationBBO #336

Add progress info for OptimizationBBO #336

Uh oh!

Conversation

SebastianM-C commented Aug 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Aug 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChrisRackauckas Aug 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SebastianM-C commented Aug 18, 2022

Uh oh!

Vaibhavdixit02 commented Aug 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChrisRackauckas commented Aug 19, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SebastianM-C commented Aug 10, 2022 •

edited

Loading

codecov bot commented Aug 10, 2022 •

edited

Loading

ChrisRackauckas Aug 18, 2022 •

edited

Loading

Vaibhavdixit02 commented Aug 19, 2022 •

edited

Loading