Add Recurrent Layers

[tanmayb123]

#52

[tanmayb123]

Would we prefer to have forgetWeight and forgetBias or forgetW and forgetB?

[dan-zheng]

Would we prefer to have forgetWeight and forgetBias or forgetW and forgetB?

The full names Weight and Bias are preferable.

[tanmayb123]

Done.

[dan-zheng]

Please add a test to Tests/DeepLearningTests/LayerTests.swift!

[eaplatanios]

It would also be nice if we can have an RNN cell protocol, so that all valid cells have to conform to it and can be used in code that expects RNN cells (e.g., simple RNN, bi-directional RNN, etc).

[tanmayb123]

@rxwei What do you think such a protocol would look like? i.e. What kind of functionality would it define?

[rxwei]

Just a heads-up: I probably won't have time to review this or make suggestions until late evening.

[rxwei]

Here's a sketch:

public struct RNNInput<StepInput: Differentiable, State: RNNState>: Differentiable {
    public var stepInput: StepInput
    public var previousState: State
    public init(stepInput: StepInput, previousState: State) {
        self.stepInput = stepInput
        self.previousState = previousState
    }
}

public struct RNNState<CellState: Differentiable, HiddenState: Differentiable>: Differentiable {
    public var cell: CellState
    public var hidden: HiddenState
    public init(cell: CellState, hidden: HiddenState) {
        self.cell = cell
        self.hidden = hidden
    }
}

public protocol RNNCell: Layer
    where Input == RNNInput<StepInput, State>, Output == RNNState<CellState, HiddenState> {
    associatedtype StepInput: Differentiable
    associatedtype CellState: Differentiable
    associatedtype HiddenState: Differentiable
    typealias State = Output

    init(inputSize: Int, hiddenSize: Int)
    var zeroState: State { get }
}

public extension RNNCell {
    @differentiable
    func applied(to stepInput: StepInput, _ previousState: State) -> State {
        return applied(to: Input(stepInput: stepInput, previousState: previousState))
    }
}

[rxwei]

It would be more Swifty to define this as a computed property, since the computational complexity of this is relatively trivial.

[superbobry]

I think you might want to have zeroState (or maybe initialState) accept batchSize as the inputs are likely to be batched.

[rxwei]

Remove redundant parentheses.

[rxwei]

Always prefer literal conversion when a contextual type exists. In this case the contextual type is TensorShape and it conforms to ExpressibleByArrayLiteral.

Suggested change

	self.stateShape = TensorShape([1, concatenatedInputSize])
	self.stateShape = [1, concatenatedInputSize]

[tanmayb123]

Love the sketch, Richard - thanks :)
Quick question: only LSTMs have a "cell state". Others, like GRU and SimpleRNN only have a "hidden state". Therefore, their "state" wouldn't be a struct, rather just a single Tensor. That's why I was a bit confused on how to create that protocol.

[eaplatanios]

Sorry I wasn't able to respond earlier because I've been kept busy. I was thinking of something a bit more abstract where RNNState does not need to consist of two parts but is rather a generic type itself (called just State should be fine), as it can vary between different RNN cells.

[rxwei]

Yeah, the cell state is definitely weird. I think that part may not need a fixed structure, like @eaplatanios said it can just be a generic type.

public struct RNNInput<StepInput: Differentiable, State: Differentiable>: Differentiable {
    public var stepInput: StepInput
    public var previousState: State
    public init(stepInput: StepInput, previousState: State) {
        self.stepInput = stepInput
        self.previousState = previousState
    }
}

public protocol RNNCell: Layer where Input == RNNInput<StepInput, State> {
    associatedtype StepInput: Differentiable
    typealias State = Output

    init(inputSize: Int, hiddenSize: Int)
    var zeroState: State { get }
}

public extension RNNCell {
    @differentiable
    func applied(to stepInput: StepInput, _ previousState: State) -> State {
        return applied(to: Input(stepInput: stepInput, previousState: previousState))
    }
}

[rxwei]

I'm going to check the protocol in. If you can make these new layers work with the new protocol, that'd be really great!

[tanmayb123]

I'll do that shortly - thanks Richard!

[superbobry]

I think you might want to have zeroState (or maybe initialState) accept batchSize as the inputs are likely to be batched.

[superbobry]

A rule of a thumb is to initialize the forget bias with ~1, see http://proceedings.mlr.press/v37/jozefowicz15.pdf

Also, is there a reason for not having a bias for other gates?

[rxwei]

Hi @tanmayb123, we are preparing a Swift for TensorFlow v0.3 release this Wednesday. Would you like to update this PR and check it in so that it can be part of the release?

[tanmayb123]

Sure thing Richard, working on that now. Quick concern that I didn't notice before: RNNCellOutput has both an Output and State, but that's a bit problematic for the cells since they don't have any outputs per time-step - their output is the state. What should I do in that case? Should I pass the new state to Output and State?

[rxwei]

I didn't see James's comment. Definitely address his concerns first :)

[tanmayb123]

Also, related to this PR, I've opened #91 to see how to handle sequential inputs and keeping track of hidden states over time automatically.

[tanmayb123]

The build is failing, but not on any of the Recurrent Cell code.

[rxwei]

I'm going to merge it once tests pass.

[rxwei]

Tests are failing because your branch is still old. Could you pull and merge?

[tanmayb123]

Sure thing.

[rxwei]

Merged. Thanks for iterating on this!

[tanmayb123]

Of course :)