[Python] fix UInt.pythonObject for Python 3.7

[turbolent]

Fix UInt.pythonObject for Python 3.7. The existing conversion worked for Python 2.7, but crashed for 3.

cc @pvieito @dan-zheng

[dan-zheng]

Thank you!

[dan-zheng]

@swift-ci Please test tensorflow

[dan-zheng]

@turbolent
Oops, I forgot to ask: could you please add a unit test to test/Python/python_runtime.swift?

[turbolent]

@dan-zheng The existing unit tests already contain the necessary tests: https://github.com/apple/swift/blob/bc50eb5816162aee0b90e0cafb53266218332f71/test/Python/python_runtime.swift#L229-L233

I tried duplicating this test case and running it using Python 3.7, but it seems like subsequent calls to PythonLibrary.useVersion actually don't load a new version? Is there any way to run multiple versions in a process (not necessarily at the same time, e.g. through unloading one)?

[turbolent]

I added the fix also to PythonKit: pvieito/PythonKit#8, which tests only Python 3.7

[dan-zheng]

@dan-zheng The existing unit tests already contain the necessary tests:

Gotcha, thanks! Though the unit tests explicitly use Python 2.7, as you noted.

I tried duplicating this test case and running it using Python 3.7, but it seems like subsequent calls to PythonLibrary.useVersion actually don't load a new version? Is there any way to run multiple versions in a process (not necessarily at the same time, e.g. through unloading one)?

I believe the intended usage of PythonLibrary.useVersion is at the beginning of a program, before any references to Python. (relevant PR comment)

Dynamically switching between Python versions in a program is not supported. It may be that we do not want to support this due to bad interactions between Python versions: for example, what happens if you perform operations on PythonObjects from different Python versions?

@pvieito: could you please confirm the behavior of PythonLibrary.useVersion? What are your thoughts on dynamically switching Python versions?

[pvieito]

I believe the intended usage of PythonLibrary.useVersion is at the beginning of a program, before any references to Python. (relevant PR comment)

Exactly. The first reference to the Python, calls PyInitialize which loads the Python library dynamically. The version loaded is by default the last one found in the system or the version defined by the PYTHON_VERSION environment variable. Calling PythonLibrary.useVersion simply sets this environment variable to the specified version, so it must be set before any reference to Python.

@pvieito: could you please confirm the behavior of PythonLibrary.useVersion? What are your thoughts on dynamically switching Python versions?

Loading two versions of Python in the same process is not possible, as a lot of common symbols would clash.

Switching Python versions (in the sense of unloading a Python library and loading other version) could be technically possible in some OS (dlclose?), but would make the implementation a lot more complex and unstable (what happens when unloading Python 3 while holding a Python 3 str reference and then passing it to a Python 2 function?).

[turbolent]

@dan-zheng @pvieito Thanks for the explanation, that makes sense!

Is there any way to still have tests for both major versions?

[dan-zheng]

@dan-zheng @pvieito Thanks for the explanation, that makes sense!

Is there any way to still have tests for both major versions?

Duplicating the test file for both major versions would be straightforward, but we shouldn't do it because it's not maintainable.

---

I think a good solution is to define all Python tests under a common PythonUnittest module in stdlib/private, kind of like stdlib/private/TensorFlowUnittest.

Then, we can have two test files (test/Python/python2_runtime.swift and test/Python/python3_runtime.swift) that call Python.useVersion then import and run all tests.

I filed TF-336 to track adding this unit test infra - it's a nice starter bug if you're interested. 🙂

[pvieito]

For reference, I have hooked up a simple Travis CI on PythonKit that it simply launches the swift test with the PYTHON_VERSION environment variable set to 3 and 2 (after removing all references to Python.useVersion):

- PYTHON_VERSION=2 swift test
- PYTHON_VERSION=3 swift test

Thanks to @turbolent for porting the tests to XCTest! 🙂