Skip to content
This repository was archived by the owner on Jul 1, 2023. It is now read-only.

Added support for 'softplus' and 'logSigmoid' and their VJPs. #148

Closed
wants to merge 14 commits into from
Closed

Added support for 'softplus' and 'logSigmoid' and their VJPs. #148

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
930cf5f
Enhanced the 'matmul' wrapper so that it matches the behavior of the …
eaplatanios May 30, 2019
1e80a1e
Addressed Richard's feedback.
eaplatanios May 30, 2019
3b60a9e
Addressed Richard's feedback.
eaplatanios May 31, 2019
9ef8db8
Added one more tests helper.
eaplatanios May 31, 2019
561a842
Minor bug fix.
eaplatanios May 31, 2019
399aba6
Merge branch 'matmul' into log-sigmoid
eaplatanios May 31, 2019
a30c098
Added support for 'softplus' and 'logSigmoid'.
eaplatanios May 31, 2019
7b7585e
Minor tweak.
eaplatanios May 31, 2019
795e2cf
Merged upstream changes.
eaplatanios Jun 20, 2019
8869b75
Merged upstream changes.
eaplatanios Jun 20, 2019
2b6a5ba
Enabled the 'logSigmoid' test.
eaplatanios Jun 20, 2019
9463619
Merged upstream changes.
eaplatanios Jun 21, 2019
fdbc8e3
Merged upstream changes.
eaplatanios Jun 21, 2019
465db74
Merged upstream changes.
eaplatanios Jun 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Sources/TensorFlow/Operators/Math.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -923,6 +923,14 @@ internal func _vjpSigmoid<T: TensorFlowFloatingPoint>(
(sigmoid(x), { v in Raw.sigmoidGrad(x, dy: v) })
}

/// Returns the log-sigmoid of the specified tensor element-wise. Specifically,
/// `y = log(1 / (1 + exp(-x)))`. For numerical stability, we use `y = -softplus(-x)`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// `y = log(1 / (1 + exp(-x)))`. For numerical stability, we use `y = -softplus(-x)`.
/// `log(1 / (1 + exp(-x)))`. For numerical stability, we use `-softplus(-x)`.

@inlinable
@differentiable
public func logSigmoid<T: TensorFlowFloatingPoint>(_ x: Tensor<T>) -> Tensor<T> {
-softplus(-x)
}

/// Returns the softplus of the specified tensor element-wise.
/// Specifically, computes `log(exp(features) + 1)`.
@inlinable
Expand Down
7 changes: 7 additions & 0 deletions Tests/TensorFlowTests/OperatorTests/MathTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,12 @@ final class MathOperatorTests: XCTestCase {
XCTAssertEqual(result.scalars, [12.5, 6.5])
}

func testLogSigmoid() {
let x = Tensor<Float>([[1, 2, 3, 4, 5], [1, 2, 3, 4, 5]])
let y = logSigmoid(x)
assertEqual(y, log(sigmoid(x)), accuracy: 0.0001)
}

func testSoftplus() {
let x = Tensor<Float>([1.0, 2.0, 3.0])
let y = softplus(x)
Expand Down Expand Up @@ -319,6 +325,7 @@ final class MathOperatorTests: XCTestCase {
("testCosineSimilarity", testCosineSimilarity),
("testElu",testElu),
("testArgmax", testArgmax),
("testLogSigmoid", testLogSigmoid),
("testSoftplus", testSoftplus),
("testSoftsign", testSoftsign),
("testLeakyRelu", testLeakyRelu),
Expand Down