[AutoDiff upstream] Add the _Differentiation module. #27511

dan-zheng · 2019-10-03T02:17:18Z

The _Differentiation module is the experimental support library for
differentiable programming. It is built when the build-script flag
--enable-experimental-differentiable-programming is enabled.

The Differentiable protocol generalizes all types that work with
differentiation. It is a core piece of the differentiable programming
project. Other parts depending on the Differentiable protocol will
be upstreamed piece by piece.

The Differentiable protocol is compiler-known and will be used during
type-checking, SILGen, and the SIL differentiation transform.

Forum discussion on upstreaming differentiable programming.

stdlib/public/core/Differentiable.swift

Rename the `Differentiable` protocol to `_Differentiable`. This matches upstreaming PR #27511, which adds `protocol _Differentiable` in a new `Differentiable.swift` file. `tensorflow` branch defines `typealias Differentiable = _Differentiable` for usability. There should be no user-facing changes, except that some error messages now show '_Differentiable' instead of 'Differentiable'.

dan-zheng · 2019-10-14T22:14:52Z

Hi @DougGregor: could you please help review this patch (or forward to appropriate reviewers)?

Compilation of the _Differentiable protocol is gated by the --enable-experimental-differentiable-programming build-script flag, which is on by default to ensure testing. The design and naming will go through Swift Evolution.

rxwei · 2019-10-15T18:02:00Z

This patch is a direct dependency of many upcoming patches that can be done in parallel, including TF-828, TF-829, and TF-830. Any review on this patch will be greatly appreciated!

DougGregor · 2019-10-16T20:30:52Z

Why is this going into the standard library vs. an auto-diff support library? Is it truly the only library piece for auto-diff?

rxwei · 2019-10-16T20:42:39Z

@DougGregor It is because we don't yet support retroactive derivative registration, and we will need to define derivatives for Float for testing purposes. Adding a derivative requires the use of @differentiating attribute, whose type checking requires the Differentiable protocol.

That said, we now aim to finish support for retroactive derivative registration this week with SIL differentiability witnesses. So we can check everything into the _Differentiation module and not make modifications to the Swift module today.

rxwei · 2019-10-16T20:45:04Z

@dan-zheng Could you update the PR to add the Differentiable protocol (with no underscores) to a _Differentiation module in stdlib/public?

@DougGregor We'll ping you when it's ready for review again in an hour or so.

dan-zheng · 2019-10-16T21:32:50Z

Why is this going into the standard library vs. an auto-diff support library? Is it truly the only library piece for auto-diff?

@DougGregor: a separate experimental _Differentiable library has been created in 35edb18 and the Differentiable protocol has been moved there. Could you please take a look?

dan-zheng · 2019-10-21T22:31:19Z

Hi @DougGregor (and others): could you please help re-review this patch?

It adds a flag-gated experimental _Differentiation module to stdlib/public - all contents are experimental and subject to change. I hope the patch isn't controversial - it's blocking our other autodiff upstreaming patches, so I hope it's okay if we merge this within 24 hours if no other reviewers respond.

dan-zheng · 2019-10-21T22:31:28Z

@swift-ci Please smoke test

dan-zheng · 2019-10-24T05:32:30Z

@DougGregor: ping in case you have time for review.

include/swift/AST/KnownIdentifiers.def

test/Differentiation/differentiable_protocol.swift

DougGregor

LGTM

dan-zheng · 2019-10-30T16:34:24Z

@swift-ci Please smoke test

dan-zheng · 2019-10-30T19:31:27Z

There appears to be an orthogonal SourceKit-LSP test failure (https://ci.swift.org/job/swift-PR-Linux-smoke-test/18001):

10:56:44 $ "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llvm-linux-x86_64/bin/FileCheck" "--input-file" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/none-swift_package_sandbox_linux-x86_64/tests/test-sourcekit-lsp/Output/test-sourcekit-lsp.py.tmp.run-log" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift-integration-tests/test-sourcekit-lsp/test-sourcekit-lsp.py"
10:56:44 # command stderr:
10:56:44 /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift-integration-tests/test-sourcekit-lsp/test-sourcekit-lsp.py:124:11: error: CHECK: expected string not found in input
10:56:44  # CHECK: "items":[
10:56:44           ^
10:56:44 /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/none-swift_package_sandbox_linux-x86_64/tests/test-sourcekit-lsp/Output/test-sourcekit-lsp.py.tmp.run-log:55:253: note: scanning from here
10:56:44 {"jsonrpc":"2.0","id":4,"result":{"items":[{"kind":2,"label":"foo()","insertTextFormat":1,"detail":"Void","insertText":"foo()","filterText":"foo()"},{"kind":14,"label":"self","insertTextFormat":1,"detail":"Lib","filterText":"self","insertText":"self"}],"isIncomplete":false}}Content-Length: 337
10:56:44                                                                                                                                                                                                                                                             ^
10:56:44 /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/none-swift_package_sandbox_linux-x86_64/tests/test-sourcekit-lsp/Output/test-sourcekit-lsp.py.tmp.run-log:57:78: note: possible intended match here
10:56:44 {"method":"textDocument\/publishDiagnostics","jsonrpc":"2.0","params":{"diagnostics":[],"uri":"file:\/\/\/home\/buildnode\/jenkins\/workspace\/swift-PR-Linux-smoke-test\/branch-master\/buildbot_linux\/none-swift_package_sandbox_linux-x86_64\/tests\/test-sourcekit-lsp\/Output\/test-sourcekit-lsp.py.tmp.dir\/pkg\/Sources\/clib\/clib.c"}}
10:56:44                                                                              ^
10:56:44 
10:56:44 error: command failed with exit status: 1
10:56:44 
10:56:44 --
10:56:44 
10:56:44 ********************
10:56:48 Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
10:56:52 Testing Time: 12.59s
10:56:52 ********************
10:56:52 Failing Tests (1):
10:56:52     swift-package-tests :: test-sourcekit-lsp/test-sourcekit-lsp.py

I'll rebase on top of master and rerun tests.

dan-zheng · 2019-10-30T19:36:01Z

@swift-ci Please smoke test

Move differentiable-programming-related stdlib sources to `stdlib/public/core/Differentiation`. Related master branch PR: swiftlang#27511. master branch will have a `_Differentiation` support library containing the `Differentiable` protocol and related APIs. On tensorflow branch, the Swift source files in `stdlib/public/core/Differentiation` will be directly built as part of swiftCore for simplicity. Swift for TensorFlow users can continue to use the `Differentiable` protocol and related APIs without adding `import _Differentiation`. Rename `protocol _Differentiable` back to `protocol Differentiable`.

The `_Differentiation` module is the experimental support library for differentiable programming. It is built when the build-script flag `--enable-experimental-differentiable-programming` is enabled. The `Differentiable` protocol generalizes all types that work with differentiation. It is a core piece of the differentiable programming project. Other parts depending on the `Differentiable` protocol will be upstreamed piece by piece. The `Differentiable` protocol is compiler-known and will be used during type-checking, SILGen, and the SIL differentiation transform.

dan-zheng · 2019-11-06T01:36:29Z

Note since last review: the test/Differentiation subdirectory is renamed to test/AutoDiff for consistency with test/AutoDiff on tensorflow branch.

dan-zheng · 2019-11-06T01:37:01Z

@swift-ci Please smoke test

…#28089) Move differentiable-programming-related stdlib sources to `stdlib/public/core/Differentiation`. Related master branch PR: #27511. master branch will have a `_Differentiation` support library containing the `Differentiable` protocol and related APIs. On tensorflow branch, the Swift source files in `stdlib/public/core/Differentiation` will be directly built as part of swiftCore for simplicity. Swift for TensorFlow users can continue to use the `Differentiable` protocol and related APIs without adding `import _Differentiation`. - Create empty `_Differentiation` module on tensorflow branch to avoid `#if canImport(_Differentiation)` checks in upstreamed AutoDiff tests. - Rename `protocol _Differentiable` back to `protocol Differentiable`. - Expose SR-11723. Workaround applied to fix validation-test/ParseableInterface/verify_stdlib.swift.

rxwei reviewed Oct 3, 2019

View reviewed changes

stdlib/public/core/Differentiable.swift Outdated Show resolved Hide resolved

dan-zheng force-pushed the upstream-differentiable-protocol branch 4 times, most recently from 3d6bb24 to 154ab48 Compare October 8, 2019 19:29

dan-zheng mentioned this pull request Oct 8, 2019

[AutoDiff] Underscore the Differentiable protocol. #27577

Merged

rxwei approved these changes Oct 8, 2019

View reviewed changes

rxwei requested review from airspeedswift and stephentyrone October 8, 2019 20:45

dan-zheng requested a review from jckarter October 9, 2019 18:00

dan-zheng mentioned this pull request Oct 10, 2019

[Autodiff upstream] Add build-script flag for differentiable programm… #27595

Merged

dan-zheng force-pushed the upstream-differentiable-protocol branch from 154ab48 to 36214fd Compare October 14, 2019 22:09

dan-zheng requested a review from DougGregor October 14, 2019 22:11

dan-zheng force-pushed the upstream-differentiable-protocol branch from 36214fd to 5dc37c3 Compare October 16, 2019 21:31

dan-zheng changed the title ~~[AutoDiff upstream] Add the _Differentiable protocol.~~ [AutoDiff upstream] Add the _Differentiation module. Oct 16, 2019

dan-zheng force-pushed the upstream-differentiable-protocol branch from 5dc37c3 to 35edb18 Compare October 16, 2019 21:44

dan-zheng mentioned this pull request Oct 20, 2019

[AutoDiff] Add build-script flag for differentiable programming. #27596

Merged

DougGregor reviewed Oct 30, 2019

View reviewed changes

include/swift/AST/KnownIdentifiers.def Outdated Show resolved Hide resolved

test/Differentiation/differentiable_protocol.swift Outdated Show resolved Hide resolved

rxwei requested a review from DougGregor October 30, 2019 05:30

DougGregor approved these changes Oct 30, 2019

View reviewed changes

dan-zheng force-pushed the upstream-differentiable-protocol branch from 5aecece to 5191338 Compare October 30, 2019 19:34

dan-zheng mentioned this pull request Nov 5, 2019

[AutoDiff] Move stdlib sources to stdlib/public/core/Differentiation. #28089

Merged

dan-zheng force-pushed the upstream-differentiable-protocol branch 3 times, most recently from c2d8e98 to f02f008 Compare November 5, 2019 23:58

dan-zheng force-pushed the upstream-differentiable-protocol branch from f02f008 to f2b6365 Compare November 6, 2019 01:34

dan-zheng merged commit 53e61a9 into master Nov 6, 2019

dan-zheng deleted the upstream-differentiable-protocol branch November 6, 2019 19:31

[AutoDiff upstream] Add the _Differentiation module. #27511

[AutoDiff upstream] Add the _Differentiation module. #27511

Uh oh!

Conversation

dan-zheng commented Oct 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dan-zheng commented Oct 14, 2019

Uh oh!

rxwei commented Oct 15, 2019

Uh oh!

DougGregor commented Oct 16, 2019

Uh oh!

rxwei commented Oct 16, 2019

Uh oh!

rxwei commented Oct 16, 2019

Uh oh!

dan-zheng commented Oct 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dan-zheng commented Oct 21, 2019

Uh oh!

dan-zheng commented Oct 21, 2019

Uh oh!

dan-zheng commented Oct 24, 2019

Uh oh!

Uh oh!

Uh oh!

DougGregor left a comment

Choose a reason for hiding this comment

Uh oh!

dan-zheng commented Oct 30, 2019

Uh oh!

dan-zheng commented Oct 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dan-zheng commented Oct 30, 2019

Uh oh!

dan-zheng commented Nov 6, 2019

Uh oh!

dan-zheng commented Nov 6, 2019

Uh oh!

Uh oh!

dan-zheng commented Oct 3, 2019 •

edited

Loading

dan-zheng commented Oct 16, 2019 •

edited

Loading

dan-zheng commented Oct 30, 2019 •

edited

Loading