Add Gabor Kernel

[sharanry]

Issue #44

Gabor kernel with length scale ell and period p. Given by

    κ(x,y) =  h(x-z), h(t) = exp(-sum(t.^2./(2*ell.^2)))*cos(2*pi*sum(t./p))

I don't think we can get rid of period and length scale using ScaledKernel like we did in #45. Please let me know if I am wrong or if there any other way.

References:

• http://www.gaussianprocess.org/gpml/code/matlab/doc/manual.pdf pg44
• https://github.com/alshedivat/gpml/blob/master/cov/covGabor.m

[devmotion]

Actually, I think we do not need the length scale and could only keep the period, since if d is rescaled to d/ell one could just use p/ell to obtain the same results, it seems. That would be nice in a way since it would allow to reuse the existing input transformations in KernelFunctions.

[theogf]

Can't we just define it as a product of a Squared exponential and cosine actually?

[sharanry]

@theogf I am not exactly sure to how to accommodate the the different scaling for inputs of SqExponentialKernel and CosineKernel if we intend to implement using KernelProduct. Did you have something in mind?

[theogf]

I think this would look like transform(SqExponential(), ell)*transform(CosineKernel(), p)

[sharanry]

@theogf Shouldn't it be this?
transform(SqExponentialKernel(), sqrt(2)*ell)*transform(CosineKernel(), p)

[theogf]

transform(SqExponentialKernel(), 1/(sqrt(2)*ell))*transform(CosineKernel(), 1/p). The scale transform multiply by the inputs

[devmotion]

I guess the sqrt(2) is not necessarily needed here. The division by 2 is also used in the squared exponential kernel in GPML whereas in KernelFunctions the squared exponential kernel is defined without scaling. So without sqrt(2) it would be consistent with our definition of SqExponentialKernel whereas with sqrt(2) it would be according to the definition in GPML.

[devmotion]

I'm wondering, if we actually need a separate type for the Gabor kernel since it can be composed of SqExponentialKernel and CosineKernel. We could even try to remove the use of unneeded transforms with something like

function GaborKernel(; ell = nothing, p = nothing)
    if ell === nothing
        if p === nothing
            return SqExponentialKernel() * CosineKernel()
        else
            return SqExponentialKernel() * transform(CosineKernel(), 1 ./ p)
        end
    elseif p === nothing
        return transform(SqExponentialKernel(), 1 ./ ell) * CosineKernel()
    else
        return transform(SqExponentialKernel(), 1 ./ ell) * transform(CosineKernel(), 1 ./ p)
    end
end

[sharanry]

IMO it would be good to be able to use the GaborKernel out of box.
Would removing the transforms make much of a difference in terms of performance?

[devmotion]

IMO it would be good to be able to use the GaborKernel out of box.

Isn't that "out of the box"? You just call GaborKernel() (possibly with length scale and period) and you get a Gabor kernel.

transform(SqExponentialKernel(), 1 ./ ell) * transform(CosineKernel(), 1 ./ p)

I haven't benchmarked it but I would assume that most of the time (in particular for higher dimensional inputs) is spent in evaluating the kernels and not in computing the transforms. It just felt a bit unnecessary to perform these additional computations if it's so easy to avoid them.

[sharanry]

@devmotion Sorry, I think I misinterpreted what you meant.

If you mean to say that we construct just a function called GaborKernel which gives a KernelProduct object, I think I agree with you. This should be sufficient. However, we need to keep in mind that we can't access the parameters, ell and p easily at a later point if required for whatever reason as we don't create GaborKernel object at any point which can hold these values.

[theogf]

I see @sharanry point, we could eventually write the GaborKernel as a wrapper around what you described :

struct GaborKernel{K<:Kernel} <: Kernel
 k::K
end

And create the constructor as @devmotion described it. One can then create get functions...

[sharanry]

@theogf But would this also create problems if we intend to make the parameters learnable?

[theogf]

No it would be fine as it would just be a wrapper, one would need to create the trainable function on GaborKernel and the training parameters would be propagated through

[sharanry]

@theogf @devmotion Can we merge this?