Skip to content
This repository was archived by the owner on Jul 1, 2023. It is now read-only.

use .moments() in LayerNorm and BatchNorm layers #384

Merged
merged 6 commits into from
Jul 24, 2019

Conversation

mikowals
Copy link
Contributor

Mean and variance in the layers are now calculated using Tensor.moments(). I also added tests for both BatchNorm and LayerNorm layers.

The tests turned up a flaw in how the shape of the scale and offset which were always of shape [featureCount] irrespective of the input shape or axis for normalisation. That shape leads to incorrect broadcasting when the axis being normalized along is not the last axis.

I have fixed this by always reshaping scale and offset before they are used. This seems hacky in that I get the shapes from the calculated mean and variance. Without the input shape being known at initialization time though I couldn't see a better way to do this.

I think the axis argument is probably there to be consistent with Keras but most of the Swift api layers assume inputs and activations are NHWC. So requiring NHWC, eliminating the axis argument, and the setting the correct shapes in init() would be another option.

Copy link
Contributor

@rxwei rxwei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@rxwei rxwei requested a review from eaplatanios July 24, 2019 06:04
@rxwei rxwei added the enhancement New feature or request label Jul 24, 2019
@rxwei rxwei merged commit f5222cd into tensorflow:master Jul 24, 2019
@BradLarson
Copy link
Contributor

@mikowals - The inference changes introduced here were triggering shape mismatches within the MiniGo sample project and its tests on tensorflow/swift-models. I've submitted a PR to roll those back here: #426 , but if you have a better way to approach this, I'm all ears.

eaplatanios added a commit to eaplatanios/swift-apis that referenced this pull request Aug 9, 2019
@mikowals mikowals deleted the bn-test branch May 20, 2020 04:32
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants