Add Python 3 support.

[dan-zheng]

Note: This PR should not be merged until Python 3 support is discussed with the community.

---

This approach for adding Python 3 support creates two pairs of CPython and Python modules:

• CPython2 and Python2
• CPython3 and Python3

Python2 and Python3 compile the same source code,
stdilb/public/Python/Python.swift.

This separation of the Python modules is not ideal for other modules that want
to support both Python 2 and 3. Assuming pre-built packages will be released
with support for both Python 2 and 3, modules importing Python could be
compiled with either a -DPYTHON2 or -DPYTHON3 flag.

Example:

// Default to Python 2.
// Specify -DPYTHON3 to use Python 3.
#if canImport(Python2)
  import Python2
#elseif canImport(Python3) && PYTHON3
  import Python3
#else
#error("Toolchain does not support Python2 or Python3.")
#endif

 // Use same `PythonObject` APIs.

Todo:

• Add Python 3 tests. It would be nice to avoid duplicating test files if possible, using lit configuration or double RUN lines.

[dan-zheng]

This change (adding IDEMPOTENT) should be upstreamed to master.

[dan-zheng]

I didn't check if the original linker error still persists. Perhaps we can remove this code now?

[dan-zheng]

Todo: make the #if PYTHON2 ... #elseif PYTHON3 code in Python.swift smarter.

For symbols defined by Python 2 but not Python 3 (like PyString_AsString), we can rebind the equivalent Python 3 symbol to the same name.

Instead of this:

#if PYTHON2
let pyStringCastFunction = PyString_AsString
#elseif PYTHON3
let pyStringCastFunction = PyUnicode_AsUTF8
#endif

guard let cString = pyStringCastFunction(pyObject) else {

Do this:

#if PYTHON3
internal let PyString_AsString = PyUnicode_AsUTF8
#endif

guard let cString = PyString_AsString(pyObject) else {

[pvieito]

As a technical curiosity, to which library path will the CPython3 module be linked against on macOS?

For example, when using the python.org installer the library is installed at /Library/Frameworks/Python.framework/Versions/3.X/Python while when using the Homebrew version it is located at /usr/local/Frameworks/Python.framework/Versions/3.X/Python and the paths explicitly depend on the exact Python version.

Would we have to rely on DYLD @rpath or something like that?

[dan-zheng]

@pvieito The Python 3 library path is determined by the following CMake logic:

set(Python_ADDITIONAL_VERSIONS 3.4 3.5 3.6 3.7)
find_package(PythonLibs "${python_version}" REQUIRED)

On my Mac with Homebrew Python 3.7, the library path successfully resolves to:

-- Found PythonLibs: /usr/local/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib (found suitable version "3.7.0", minimum required is "3")

I believe the CMake logic can find the python.org installer path, too (/Library/Frameworks/...). I will test this before merging.

[pvieito]

@dan-zheng Yes, but my question is not from a compile-time perspective but from a runtime one. When the prebuilt binaries are distributed they would only be linked to the Python version (and path) that was found at build time.

This is not a problem on Python 2, as its path is guaranteed by the OS but it will be problematic for Python 3 as it can be found in various locations and versions.

This is one of the problems that could be solved by dynamically linking the Python library at runtime with dlopen (we would have to search for all known paths in each platform).

[dan-zheng]

#20674 introduces a flexible, runtime approach for Python 3 support. Hurray!