[Autodiff upstream] Add build-script flag for differentiable programm… #27595

dan-zheng · 2019-10-10T02:32:17Z

…ing.

Add --enable-experimental-differentiable-programming build-script flag.

The build-script flag enables/disables standard library additions
related to differentiable programming. This will allow official Swift
releases to disable these additions.

The build-script flag is on by default to ensure testing of
differentiable programming standard library additions. An additional
driver flag must be enabled to use differentiable programming features:
#27446

dan-zheng

Verified the following invocations:

utils/build-script --enable-experimental-differentiable-programming: enabled.
utils/build-script: enabled (by default).
utils/build-script --enable-experimental-differentiable-programming=false: disabled.

dan-zheng · 2019-10-10T02:38:10Z

As discussed here, this PR adds the -enable-experimental-differentiable-programming build-script and CMake flag to gate standard library additions.

@jckarter: could you please help review (or loop in reviewers)? That would be a big help for enabling our progress.

DougGregor · 2019-10-10T22:32:44Z

Looks okay to me. @gottesmm does the build-script fu look good here?

dan-zheng · 2019-10-10T22:38:20Z

stdlib/public/core/CMakeLists.txt

+# Compile differentiable programming sources only if enabled.
+set(SWIFTLIB_DIFFERENTIABLE_PROGRAMMING_SOURCES)
+if(SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING)
+  # TODO: Add `_Differentiable` protocol.


Note: #27511 will add the _Differentiable protocol - separate patch to facilitate review.

dan-zheng · 2019-10-11T01:35:07Z

@swift-ci Please test

swift-ci · 2019-10-11T02:43:47Z

Build failed
Swift Test OS X Platform
Git Sha - 482f3e7514561a8dc370d9ee0930f35961f807f8

swift-ci · 2019-10-11T02:47:12Z

Build failed
Swift Test Linux Platform
Git Sha - 482f3e7514561a8dc370d9ee0930f35961f807f8

dan-zheng · 2019-10-11T03:09:11Z

@swift-ci Please test

swift-ci · 2019-10-11T03:09:45Z

Build failed
Swift Test OS X Platform
Git Sha - 482f3e7514561a8dc370d9ee0930f35961f807f8

swift-ci · 2019-10-11T03:09:51Z

Build failed
Swift Test Linux Platform
Git Sha - 482f3e7514561a8dc370d9ee0930f35961f807f8

dan-zheng · 2019-10-12T22:21:06Z

Pinging @gottesmm for review, as requested by him

gottesmm · 2019-10-13T17:20:00Z

utils/build-script

@@ -601,6 +601,9 @@ class BuildScriptInvocation(object):
                        args.build_jobs)
                ]

+        if args.enable_experimental_differentiable_programming:
+            impl_args += ["--enable-experimental-differentiable-programming"]


I don't think you need to do this in build-script-impl. You should be able to add this as a cmake option like this: https://github.com/apple/swift/blob/master/utils/swift_build_support/swift_build_support/products/swift.py

Thanks! Done in 4bef66b - could you please re-review?

gottesmm

There are tests for this, no?

dan-zheng · 2019-10-14T05:39:24Z

There are tests for this, no?

I updated the utils/build_swift/tests/expected_options.py test and did some manual verification. I feel that's reasonable - please let me know if you'd like further changes.

EDIT: @gottesmm shared that there are extra CMake flag tests. Added in cc2d822, mimicking existing tests:

$ python -m unittest discover -s utils/swift_build_support
Ran 140 tests in 0.236s

OK (skipped=3)

gottesmm

Can you squash/make sure this passes pylint? Then LGTM

utils/swift_build_support/tests/products/test_swift.py

…ing. Add `--enable-experimental-differentiable-programming` build-script flag. The build-script flag enables/disables standard library additions related to differentiable programming. This will allow official Swift releases to disable these additions. The build-script flag is on by default to ensure testing of differentiable programming standard library additions. An additional driver flag must be enabled to use differentiable programming features: swiftlang#27446

dan-zheng · 2019-10-14T18:03:00Z

Can you squash/make sure this passes pylint? Then LGTM

Squashed. (I tend to use "Squash and merge" when merging.)

I ran pylint with the default configuration and newly added Python code passes (or fails with the same errors as surrounding code, e.g. method name too long).

dan-zheng · 2019-10-14T18:03:12Z

@swift-ci Please test

swift-ci · 2019-10-14T18:03:39Z

Build failed
Swift Test OS X Platform
Git Sha - 777ebcd7f2932cae83f8fd60ecd5a600b9c31e49

swift-ci · 2019-10-14T18:05:34Z

Build failed
Swift Test Linux Platform
Git Sha - 777ebcd7f2932cae83f8fd60ecd5a600b9c31e49

dan-zheng · 2019-10-14T19:47:08Z

Linux CI failed due to a seemingly unrelated SourceKitLSP error:

12:35:59 2019-10-14 19:35:59.512 SourceKitLSPPackageTests.xctest[72164:9f7fe700] manifest parse error(s):
12:35:59 
/home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/tmp/TemporaryDirectory.N7tywe/pkg/Package.swift:3:5: error: type annotation missing in pattern
12:35:59 let pack
12:35:59     ^

I'll try triggering it again.

dan-zheng · 2019-10-14T19:47:22Z

@swift-ci Please test linux

dan-zheng · 2019-10-14T21:34:42Z

Thanks to reviewers!

…ing. (swiftlang#27595) Add `--enable-experimental-differentiable-programming` build-script flag. The build-script flag enables/disables standard library additions related to differentiable programming. This will allow official Swift releases to disable these additions. The build-script flag is on by default to ensure testing of differentiable programming standard library additions. An additional driver flag must be enabled to use differentiable programming features: swiftlang#27446

) Add `--enable-experimental-differentiable-programming` build-script flag. The build-script flag enables/disables standard library additions related to differentiable programming. This will allow official Swift releases to disable these additions. The build-script flag is on by default to ensure testing of differentiable programming standard library additions. An additional driver flag must be enabled to use differentiable programming features: #27446 Mirror of #27595, with minor changes specific to tensorflow branch.

dan-zheng commented Oct 10, 2019

View reviewed changes

dan-zheng requested review from rxwei and jckarter October 10, 2019 02:34

dan-zheng mentioned this pull request Oct 10, 2019

[AutoDiff] Add build-script flag for differentiable programming. #27596

Merged

rxwei approved these changes Oct 10, 2019

View reviewed changes

rxwei requested a review from DougGregor October 10, 2019 06:30

dan-zheng commented Oct 10, 2019

View reviewed changes

dan-zheng requested a review from gottesmm October 11, 2019 01:34

gottesmm requested changes Oct 13, 2019

View reviewed changes

dan-zheng requested a review from gottesmm October 14, 2019 03:20

gottesmm reviewed Oct 14, 2019

View reviewed changes

utils/swift_build_support/tests/products/test_swift.py Show resolved Hide resolved

gottesmm approved these changes Oct 14, 2019

View reviewed changes

dan-zheng force-pushed the upstream-differentiable-programming branch from cc2d822 to 7f8d133 Compare October 14, 2019 18:00

dan-zheng merged commit 2bd55f6 into swiftlang:master Oct 14, 2019

dan-zheng deleted the upstream-differentiable-programming branch October 14, 2019 21:34

[Autodiff upstream] Add build-script flag for differentiable programm… #27595

[Autodiff upstream] Add build-script flag for differentiable programm… #27595

Uh oh!

Conversation

dan-zheng commented Oct 10, 2019

Uh oh!

dan-zheng left a comment

Choose a reason for hiding this comment

Uh oh!

dan-zheng commented Oct 10, 2019

Uh oh!

DougGregor commented Oct 10, 2019

Uh oh!

dan-zheng Oct 10, 2019

Choose a reason for hiding this comment

Uh oh!

dan-zheng commented Oct 11, 2019

Uh oh!

swift-ci commented Oct 11, 2019

Uh oh!

swift-ci commented Oct 11, 2019

Uh oh!

dan-zheng commented Oct 11, 2019

Uh oh!

swift-ci commented Oct 11, 2019

Uh oh!

swift-ci commented Oct 11, 2019

Uh oh!

dan-zheng commented Oct 12, 2019

Uh oh!

gottesmm Oct 13, 2019

Choose a reason for hiding this comment

Uh oh!

dan-zheng Oct 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gottesmm left a comment

Choose a reason for hiding this comment

Uh oh!

dan-zheng commented Oct 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gottesmm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dan-zheng commented Oct 14, 2019

Uh oh!

dan-zheng commented Oct 14, 2019

Uh oh!

swift-ci commented Oct 14, 2019

Uh oh!

swift-ci commented Oct 14, 2019

Uh oh!

dan-zheng commented Oct 14, 2019

Uh oh!

dan-zheng commented Oct 14, 2019

Uh oh!

dan-zheng commented Oct 14, 2019

Uh oh!

Uh oh!

dan-zheng Oct 13, 2019 •

edited

Loading

dan-zheng commented Oct 14, 2019 •

edited

Loading