Hyperparameter optimization for multi-input and multi-output datasets

[madhavkrishnan94]

Hello all.

I am trying to carry out the hyperparameter optimization for my problem that is multi-input and multi-output in nature. I tried following this tutorial.

To start off, I ran the tutorial as is and it ran without errors. Then, I tried to modify train_X as a three-dimensional one by defining it as

uniform_tensor = torch.linspace(0,1,15)
train_X=torch.stack((uniform_tensor,uniform_tensor,uniform_tensor), -1).unsqueeze(1)

When I run the notebook after this change, I get a

'RuntimeError: You must train on the training inputs!'

error at the following line:

output = model(train_X)

Any leads on correct/better ways of running the hyperparameter optimization for these kinds of multi-dimensional datasets would be greatly appreciated.

Thanks in advance!

[saitcakmak]

Hi @madhavkrishnan94. I think the issue might be the .unsqueeze(1) part in train_X=torch.stack((uniform_tensor,uniform_tensor,uniform_tensor), -1).unsqueeze(1). I believe the tutorial expects train_X to be n x d, while the combination of torch.stack and unsqueeze gets you to n x 1 x d. If you remove the unsqueeze, you should end up with n x d.

In general, that tutorial is not a great resource for your purpose. For a tutorial on multi-objective optimization in BoTorch, I'd recommend this one. If your goal is to just do hyperparameter optimization on your problem without developing a custom BO algorithm, I'd actually recommend skipping BoTorch and using Ax instead. You can find a tutorial on multi-objective optimization with Ax here. Ax uses BoTorch under the hood.

[madhavkrishnan94]

Hello @saitcakmak ,

I have been unable to get rid of this error still.

'RuntimeError: You must train on the training inputs!'

I am posting a screenshot of the error message

In this snippet, train_x_normalized is the name of the tensor that I used as train_X in defining the SingleTaskGP model.

I wanted to check, so I tried running the example notebook from the tutorial. It runs properly because it is a single-input, single-output toy example. But my problem is multi-input, multi-output by design.

Can you let me know how else I can carry out the optimization of my model hyperparameters in botorch? You told me that this tutorial isn't a great resource for my purpose. In that case, can you recommend some others?

[saitcakmak]

Hi @madhavkrishnan94. Writing custom training loops requires working with data as it is represented in GPyTorch. For multi-output single task GP models, GPyTorch uses a batched data representation, while BoTorch keeps the output dimension at the end. Working with models directly (e.g. calling model or mll directly) requires advanced knowledge of GPyTorch, and it is not something we want to support on the BoTorch side. I'd instead recommend you to use the model fitting utilities defined here https://github.com/pytorch/botorch/blob/main/botorch/optim/fit.py or here https://github.com/pytorch/botorch/blob/main/botorch/fit.py For most use cases, we would just call fit_gpytorch_mll(mll) to train the model.

The multi-objective BO tutorial demonstrates how to use a multi-output single task GP for multi-objective optimization in BoTorch: https://botorch.org/tutorials/multi_objective_bo

[madhavkrishnan94]

Thanks for the clarification, @saitcakmak

I have been using fit_gpytorch_model(mll) to train the model so far. By printing some of the hyperparameters before and after running fit_gpytorch_model(mll), I can see that the effect of the fitting on the hyperparameter values. But I wonder if its default settings are going to give me the best set of optimized hyperparameters of the model (crudely speaking, go as close to the global optimum point as possible). In order to check the same out, I wanted to run the custom training loop as presented in the tutorial, which is largely borrowed from GPyTorch.

While the multi-objective BO tutorial illustrates the BO loop for a multi-output single task GP, it doesn't tell much about getting the best set of optimized hyperparameters.

But I think I may have a few more options in playing around with the optimizer, as suggested in your reply.

[madhavkrishnan94]

Hello @saitcakmak ,

On that note, I have a few more questions.

1. I see that the fit_gpytorch_mll(mll) uses the L-BFGS-B optimizer under the hood. Is there a way for me to check the (default) hyperparameters passed to the optimizer?

2. Also, when I try to run my own custom optimization of the GP model parameters using

fit_gpytorch_mll_torch(mll, optimizer = Adam([{"params": model.parameters()}], lr=0.1))

the optimization gets truncated and this is the message that gets printed.

OptimizationResult(step=53, fval=4.572842653307676, status=<OptimizationStatus.STOPPED: 4>, runtime=0.7145364573225379, message='torch_minimize stopped due to stopping_criterion.')

Am I missing something here wrt the other optimizer arguments like the fval, steps/iterations, runtime? How do I ensure that the optimization runs fully (successfully)?

3. Can you help me better understand what a closure means? I see that closure is one of the oft mentioned hyperparameters in the docs. Do we need to define it manually? This link discusses some wrappers for building closures- how do we decide which one to use?

Thanks in advance!

[saitcakmak]

I see that the fit_gpytorch_mll(mll) uses the L-BFGS-B optimizer under the hood. Is there a way for me to check the (default) hyperparameters passed to the optimizer?

If you're referring to the hyper-parameters of the model, the hyper-parameters as they are before calling the optimizer will be passed in as the starting conditions for optimization. You can check model.state_dict() to see what these values are. If the model fitting fails for some reason, it will re-try model fitting up to 5 times by default, with the initial conditions generated by sampling values from the priors: https://github.com/pytorch/botorch/blob/main/botorch/optim/utils/model_utils.py#L187

If you're curious about the arguments passed to L-BFGS-B, we use the defaults set in Scipy. You can find the documentation for that here: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html

Torch optimizer stopping early:

By default, fit_gpytorch_mll_torch uses an exponential moving average stopping criterion. If you'd like to customize it, you can include a stopping criterion in the optimizer options: https://github.com/pytorch/botorch/blob/main/botorch/optim/fit.py#L178-L179

Can you help me better understand what a closure means? I see that closure is one of the oft mentioned hyperparameters in the docs. Do we need to define it manually? This link discusses some wrappers for building closures- how do we decide which one to use?

Closure is just a helper that evaluates the model with the given parameters and returns the evaluation result (e.g. MLL) and any gradients (if applicable). For most use cases, you should not worry about it. I don't understand that part of the code too well either, though I never had any reason for customizing it.