Skip to content

Implemented Swish Function #7357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from Oct 28, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions maths/sigmoid_linear_unit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
"""
This script demonstrates the implementation of the Sigmoid Linear Unit (SiLU)
or swish function.
* https://en.wikipedia.org/wiki/Rectifier_(neural_networks)
* https://en.wikipedia.org/wiki/Swish_function

The function takes a vector x of K real numbers as input and returns x * sigmoid(x).
Swish is a smooth, non-monotonic function defined as f(x) = x * sigmoid(x).
Extensive experiments shows that Swish consistently matches or outperforms ReLU
on deep networks applied to a variety of challenging domains such as
image classification and machine translation.

This script is inspired by a corresponding research paper.
* https://arxiv.org/abs/1710.05941
"""

import numpy as np


def sigmoid(vector: np.array) -> np.array:
"""
Mathematical function sigmoid takes a vector x of K real numbers as input and
returns 1/ (1 + e^-x).
https://en.wikipedia.org/wiki/Sigmoid_function

>>> sigmoid(np.array([-1.0, 1.0, 2.0]))
array([0.26894142, 0.73105858, 0.88079708])
"""
return 1 / (1 + np.exp(-vector))


def sigmoid_linear_unit(vector: np.array) -> np.array:
"""
Implements the Sigmoid Linear Unit (SiLU) or swish function

Parameters:
vector (np.array): A numpy array consisting of real
values.

Returns:
swish_vec (np.array): The input numpy array, after applying
swish.

Examples:
>>> sigmoid_linear_unit(np.array([-1.0, 1.0, 2.0]))
array([-0.26894142, 0.73105858, 1.76159416])

>>> sigmoid_linear_unit(np.array([-2]))
array([-0.23840584])
"""
return vector * sigmoid(vector)


if __name__ == "__main__":
import doctest

doctest.testmod()