Add NeuralNetOneKernel

[sharanry]

Issue #44

Neural network kernel function with a single parameter for the distance measure. The kernel function is parameterized as:

$$k(x,z) = sf2 * asin(x'*P*z / sqrt[(1+x'*P*x)*(1+z'*P*z)])$$

References

• https://github.com/alshedivat/gpml/blob/master/cov/covNNone.m
• http://www.gaussianprocess.org/gpml/code/matlab/doc/manual.pdf pg 44

[sharanry]

I am not sure about the name. The "one" should imply that it is a single parameter kernel function.

[theogf]

I think the name is extremly confusing... It sounds like it's a deep kernel (neural net transform on the inputs) but it's not. Is it really a necessary kernel?

[sharanry]

@theogf http://www.gaussianprocess.org/gpml/chapters/RW4.pdf pg 90 talks about how it came about.

[sharanry]

@theogf I found the significance of this kernel in Andrew Gordon's thesis and the GPML book itself.

Accordingly, Neal (1996) pursued the limits of large models, and showed that a Bayesian neural network becomes a Gaussian process with a neural network kernel as the number of units approaches infinity. This observation inspired Williams and Rasmussen (1996) to further explore Gaussian process models.

We get such a kernel by assuming a single hidden layer neural network with infinite units.

[sharanry]

@theogf @devmotion Do you suggest any other changes? Or, can this be merged?

[devmotion]

IMO we should wait for (at least some of) the kernelmatrix changes. This PR seems to add many methods related to kernelmatrix, so I think by postponing it slightly we might be able to make it compatible with the kernelmatrix and kappa refactoring.

[sharanry]

@devmotion @theogf @willtebbutt Can this be merged?

[willtebbutt]

This broadly looks good, and is fine to be merged, but it's won't currently be efficient for kernelmatrix for multi-dimensional inputs. @sharanry if you could open an issue about this for multi-dimensional inputs, that would be very helpful, just so that we don't forget about it.

edit: looks like the tests for the Weiner kernel have appeared in this PR somehow. If you could remove before merging, that would be very helpful.