Add datatype for multi-output GP input

[sharanry]

Related Issue: AbstractGPs#9
Related old PR: AbstractGPs#28

This PR proposes a data type for inputs for a multi-output GP.
A class of multi-output kernels which we hope to add to this repository will handle this datatype.

[willtebbutt]

It would probably be a good idea to also add a single multi-output GP kernel to this PR. Would be good to have this functionality used for something before we merge it.

[sharanry]

It would probably be a good idea to also add a single multi-output GP kernel to this PR. Would be good to have this functionality used for something before we merge it.

Any preference on which kernel to start-off with?

[willtebbutt]

It would probably make sense to do something really trivial, like a kernel that assumes each output is independent. So maybe you just have a kernel that wraps a single output kernel and a number that represents the number of outputs? I can't imagine why you would want this in practice, but it would probably give us what we need for this PR and would be really easy to test against single-output GPs.

[sharanry]

@willtebbutt Could you take look at IndependentKernel which I just added and let me know what you think?

For each pair of MOInput, the kernel returns a matrix of dimension (out_dim x out_dim).

I haven't added kernelmatrix support yet.

Edit:
I realize that the kernel is not agnostic to the the datatype MOInput. But, without the kernels acknowledging such a datatype I am not sure if we can make the computation more efficient when there is a possibility like in the case of IndependentKernel.

[theogf]

That would be great to always have an indicator that one is using a multi-output kernel. For example having all multi-output kernels written as prefix+MOKernel, because IndependentKernel is a very confusing name.
Also independently from that, should we actually create a new package for multi output kernels? I feel multi-output kernels are a very specific task

[willtebbutt]

Also independently from that, should we actually create a new package for multi output kernels? I feel multi-output kernels are a very specific task

I wouldn't personally be in favour of that -- certainly they have their own peculiarities, but they're fundamentally kernels like any other. Indeed, part of my reason for wanting them to conform to the same API as regular kernels is to make them feel less like something complicated and separate from other kernels.

I agree that all of the multi-output stuff should probably be kept in its own folder within this package though.

[sharanry]

Indeed, part of my reason for wanting them to conform to the same API as regular kernels is to make them feel less like something complicated and separate from other kernels.

@willtebbutt Could you clarify what you mean by this? Wouldn't multi-output kernels accept a different type(vector of tuples) of inputs from regular kernels(vector of reals)?

[willtebbutt]

Looks like good progress. Just a few questions.

[willtebbutt]

@willtebbutt Could you clarify what you mean by this? Wouldn't multi-output kernels accept a different type(vector of tuples) of inputs from regular kernels(vector of reals)?

Yes, that's correct. My point was more on the output side that it's not the case that our multi-output kernels will produce kernel matrix whose side-length is larger than the length of the input vector -- which is not what you get if you go down the matrix-valued kernel route. My point is that they only differ in the domain of the input -- the rest of the concepts are the same: you provide an AbstractVector of inputs of length N (the MOInput) and get a kernel matrix of size N x N.

[willtebbutt]

This is looking great. Just a couple of things.

[willtebbutt]

I'm happy with this now. Thanks for the great works as always @sharanry

[sharanry]

Currently we are using dim(x::AbstractVector{Tuple{Any,Int}}) = 1 which doesn't make much sense but helps pass the validate_dims for MOInput and vector of tuples without stringent check. Should I instead override validate_dims using type or is this fine?

[sharanry]

Also, the build seems to fail for some inexact error outside the test.

Got exception outside of a @test

  InexactError: Int64(1.0e-20)

Seems to be a problem with AD tests which I haven't touched in this PR.

[theogf]

Given the error messages I think it's due to JuliaDiff/FiniteDifferences.jl#99
@willtebbutt probably has a better idea of what's going on

[theogf]

Ok main issue is that Delta metric returns an Int and now FiniteDifferences is not happy about it.
The fix would be to replace in distances/delta.jl

@inline function Distances._evaluate(::Delta, a::AbstractVector, b::AbstractVector) where {T}
    @boundscheck if length(a) != length(b)
        throw(DimensionMismatch("first array has length $(length(a)) which does not match the length of the second, $(length(b))."))
    end
    return a == b
end

by

@inline function Distances._evaluate(::Delta, a::AbstractVector{Ta}, b::AbstractVector{Tb}) where {Ta, Tb}
    @boundscheck if length(a) != length(b)
        throw(DimensionMismatch("first array has length $(length(a)) which does not match the length of the second, $(length(b))."))
    end
    T = promote_type(Ta, Tb)
    return T(a == b)
end

[willtebbutt]

Good catch btw @theogf . You're correct that this was introduced in the most recent PR. I think FiniteDifferences shouldn't be erroring with this, so I'll make + tag a fix today so that we can get this merged today.

[willtebbutt]

@sharanry any idea where the drop in coverage is coming from?

[sharanry]

@sharanry any idea where the drop in coverage is coming from?

Few Base functions like iterate and few error throws were not tested. Should be fixed now.