[CORE UPDATE - PART I] Refactor python recipes + openssl + sqlite3 (#1537)

* Refactor hostpython3's recipe into a new base class HostPythonRecipe

To make it use as a base class for hostpython2 and hostpython3 recipes in a near future.

* Refactor python3's recipe into a new base class GuestPythonRecipe

To make it use as a base class for python2 and python3 recipes in a near future.

* Move hostpython2 and python2 recipes to a new name hostpython2legacy and python2legacy.

This recipes will be useless until properly fixed or maybe it will be removed...

* Add the new hostpython2/python2 recipes (v2.7.15)

Those new recipes will use the same build method than python3. Also added a set of patches, some of them are enabled by default because they are necessary to perform a successful build. The others are added because maybe we will need them.

* Adapt sdl2's stuff to the new python2's build system

* Adapt pythonforandroid's core files to the new python2's build system

Cause we share the same build system for python2 and python3 recipes, we can safely remove some specific python2's code in favour of the python3's code, otherwise we will surely find troubles in our builds.

* Adapt python2's test apps to the new python2's build system

The python2's test with openssl and sqlite it will probably success, but without the libs until those are enabled. The pygame's test app will fail until properly fixed.

* Enable sqlite support for python recipes

* Add ability to compile recipes with clang

The android ndk has been migrated his default compiler from gcc to clang and soon will be removed from the ndk. Here we prepare p4a to compile recipes with clang, which will allow us to successfully compile an updated openssl libs. Without this openssl upgrade we will not be able to grant support to python3's _ssl.so module.

Note: The default p4a compiler still is gcc...so...no changes on the compile behaviour unless explicitly requested

References: #1478

* Update OpenSSL to 1.1.1 (uses clang for compilation)

This openssl version is an LTS, supported until 11th September 2023.

Important: There is now a distinction between the version we use for downloading, and the version used for creating the library: OpenSSL 1.1 now have prefix with "1.1.so". This explains why others recipes that depends on it require the "library" version. Those recipes, still not are modified to support the new version.

References: #1478

* Enhance openssl recipe and remove lib_version

Several actions done:
  - The openssl recipe has been enhanced by introducing a couple of methods (include_flags and link_flags) which should help us to link other recipes with the openssl libs.
  - Remove lib_version introduced in the last openssl version update, so we don't have to modify all dependant recipes
  - Add a new variable `url_version`, used to download our recipe in the subclassed method versioned_url (this replaces the removed lib_version)
  - Add D__ANDROID_API__ to the configure command (needed to successfully build the libraries and link them with python)
  - Add documentation

* Enable openssl support for python recipes

This commit grants openssl support for both python versions (2 and 3). The python2 recipe needs some extra work to make it link with python, we need to patch the setup.py file in order to that the python's build system recognises the libraries. This is not the case for the python3 recipe which works like a charm, except for the fact that we have to modify our static class variable ` config_args ` for the python's recipe, i found this to be necessary in my tests, otherwise the openssl libs aren't detected.

* Fix hardcoded python2 entry for Arch

* Fix hardcoded python2 entries for BootstrapNDKRecipes

* Fix duplicated key in options for python2 test

* Fix numpy's recipe and adds python3's compatibility.

Also:
  - add numpy to CI tests for python3: this module is some kind of special and probably a dependency for a lot of python packages so better make sure that we not break anything...as we already do with python2
  - Add numpy to python2 and python3 test apps, cause already have some buttons that allows to test it from the running app.

* Add python3 test for sqlite and openssl and add python3's compatibility to dependant recipes

To be able to run successfully the new test we must add python3 compatibility to some recipes that is why we touch them here.

* Make that travis test python3 with sqlite3 and openssl

* Update android ndk to r17c for docker images

* Add one more dependency to our docker images

Or the ci tests where libffi is involved will fail.

* Update documentation about the new python core

* Prevent crash on create_python_bundle

When building an apk for service_only's bootstrap the build will crash if we don't add some pure python packages, because the directory "python-install/<distribution directory>" will not be ever created, so...this will make p4a crash when we try to enter into this directory.

* Replace exit for BuildInterruptingException

¡¡¡Thanks @AndreMiras!!!

* Remove note of quickstart.rst

¡¡¡Thanks @inclement!!!

* Redaction corrections

¡¡¡Thanks @inclement!!!

* Remove tuple of python versions

Because all 3 are automatically added by PythonRecipe.

¡¡¡Thanks @inclement!!!

* Replace backslashes

To be more readable and pythonic

Thanks @KeyWeeUsr!!!

* Fix hardcoded toolchain version and arch for python recipe

* Fix hardcoded toolchain version and arch for clang

* Fix android host for python recipes

To allow us to build python 2 or 3 for any arch supported for p4a

* Fix hardcoded target for python recipes and clang

* Add missing includes for clang

Or we will get errors complaining about missing includes related with asm

* Fix hardcoded arch for numpy

* Fix openssl build for arch x86_64

* Fix openssl build for arch arm64-v8a

* Drop commented code from python2legacy recipe

* Replace backslashes for openssl recipe

To enhance the readability of the sourcecode

¡¡¡Thanks KeyWeeUsr!!!

* Fix hardcoded arch flags for libffi recipe

To fix libffi build for any arch

* Remove unneeded warning for numpy

* Fix libffi build for all archs

Adds some flags lost (removed when were fixed the hardcoded arch entries)

* Remove lines from openssl recipe that seems not needed anymore

The removed code was intended to check if the library were build with the right symbols, and if not, clean the build and trigger a rebuild with a pause of 3 seconds (to show a message warning about the rebuild). It seems that new version of the openssl libs doesn't need this anymore because the symbols checked with the removed source code seems to be build without problems. So we remove those lines in order the clean up a little the code and to be reintroduced again if needed (but properly documented)

References: #1537

* Replace sum with append for cflags list

* Remove unneeded message for build.py

¡¡¡Thanks @inclement!!!

* Remove commented line

¡¡¡Thanks @inclement!!!

* Remove unneeded files for the project

¡¡¡Thanks @inclement!!!

* Remove some unneeded lines for some of the test apps

¡¡¡Thanks @AndreMiras!!!

* Remove more unneeded lines for test app and shortens line

* Fix initsite for python2's site module

* Enhance python2 patches

  - Move all locale related code from fix-locale.patch to fix-api-minor-than-21.patch, because for api >= 21 localeconv is properly defined in android
  - Remove fix-locale.patch and the remaining code moved into new patches (and enhanced thanks to some cpython's patches I found in bug tracker), to be neatest
  - Enhance definitions of fix-api-minor-than-21.patch to be more suitable for a potential upstream merge into cpython source (¡¡¡Thanks @Jonast!!!)

References: https://bugs.python.org/issue20306 and https://bugs.python.org/issue26863,

* Remove unused patches

* Some fixes/enhancements for python2legacy

In case we decide to make it work and maintain it

¡¡¡Thanks @AndreMiras!!!

Note: At this point the python2legacy recipe will not work, it would require some more work to make it successfully build:
  - Fix conflicts between python recipes
  - Add python2legacy compatibility to some basic recipes (For sdl2, at least we need: android, pyjnius, sdl2, setuptools and six)
  - Add python2legacy compatibility to sdl2 bootstrap (at least)
  - Add python2legacy compatibility to some core files (bootstrap.py, build.py and recipe.py)

* Change ndk downloads page

To be safe for the foreseeable future

¡¡¡Thanks @inclement!!!

* Fix clang path for crystax

* Python2legacy: fix conflicts between python recipes

* Python2legacy: provisional fix to the hardcoded job count

This should be implemented properly as described in #1558, but for now we will avoid issues

* Python2legacy: make work some basic recipes with python2legacy

To make it work the python2legacy recipe with the sdl2's bootstrap

* Python2legacy: make work sdl2 bootstrap with python2legacy

* Python2legacy: fix get_site_packages_dir for python2legacy

* Python2legacy: fix strip_libraries for python2legacy

* Python2legacy: make the necessary changes to recipe.py to make it work the python2legacy recipe

Note: With this commit, we will be able to successfully build using the python2legacy recipe, but be aware that you may need to add python2legacy compatibility to some recipes to make it work with your app, because only some minimal set of recipes has been touched (the ones needed to make work the python2legacy with the sdl2 bootstrap and kivy)

* Python2legacy: make it work the openssl libs with python2legacy and python3crystax

Also this should fix the the openssl build issues reported when using crystax. Notice that this changes will leave us with two versions of openssl libs. We need the most updated one for our python2 and python3 recipes, and the legacy version for python2legacy and python3crystax. The python2legacy version is too old to support the openssl version 1.1+ and python3crystax is not building fine with the new openssl libs so to fix the crystax issues we force to use the legacy version, which previously has been proved to work.

* Python2legacy: add a test app for python2legacy (with openssl and sqlite3)

To make easier for us to test if python2legacy is working

* Python2legacy: update README file

To inform about the python2legacy recipe

* Move libraries from LDFLAGS to LIBS

Because this is how you are supposed to do it, you must use LDFLAGS for linker flags and LDLIBS (or the equivalent LOADLIBES) for the libraries

* Fix linkage problems with python's versioned library

The Java method called to load the libraries only supports a .so suffix, so we must remove the version for our python libraries, or we will get linkage errors

Resolves: #1501

* Revert "Fix linkage problems with python's versioned library"

This reverts commit 99ee28b

Author: opacam

.travis.yml

@@ -18,7 +18,7 @@ env:
     - ANDROID_SDK_HOME=/opt/android/android-sdk
     - ANDROID_NDK_HOME=/opt/android/android-ndk
   matrix:
-    - COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements sdl2,pyjnius,kivy,python3'
+    - COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools'
     # overrides requirements to skip `peewee` pure python module, see:
     # https://github.com/kivy/python-for-android/issues/1263#issuecomment-390421054
     - COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python2_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements sdl2,pyjnius,kivy,python2,openssl,requests,sqlite3,setuptools'

Dockerfile.py2

@@ -26,7 +26,7 @@ RUN apt -y update -qq \
 
 
 ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
-ENV ANDROID_NDK_VERSION="16b"
+ENV ANDROID_NDK_VERSION="17c"
 ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"
 
 # get the latest version from https://developer.android.com/ndk/downloads/index.html
@@ -104,7 +104,7 @@ RUN dpkg --add-architecture i386 \
 # specific recipes dependencies (e.g. libffi requires autoreconf binary)
 RUN apt -y update -qq \
     && apt -y install -qq --no-install-recommends \
-        autoconf automake cmake gettext libltdl-dev libtool pkg-config \
+        libffi-dev autoconf automake cmake gettext libltdl-dev libtool pkg-config \
     && apt -y autoremove \
     && apt -y clean
 

Dockerfile.py3

@@ -26,7 +26,7 @@ RUN apt -y update -qq \
 
 
 ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
-ENV ANDROID_NDK_VERSION="16b"
+ENV ANDROID_NDK_VERSION="17c"
 ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"
 
 # get the latest version from https://developer.android.com/ndk/downloads/index.html
@@ -104,7 +104,7 @@ RUN dpkg --add-architecture i386 \
 # specific recipes dependencies (e.g. libffi requires autoreconf binary)
 RUN apt -y update -qq \
     && apt -y install -qq --no-install-recommends \
-        autoconf automake cmake gettext libltdl-dev libtool pkg-config \
+        libffi-dev autoconf automake cmake gettext libltdl-dev libtool pkg-config \
     && apt -y autoremove \
     && apt -y clean
 

README.rst

@@ -35,6 +35,25 @@ issues and PRs relating to this branch are still accepted. However, the
 new toolchain contains all the same functionality via the built in
 pygame bootstrap.
 
+In the last quarter of 2018 the python recipes has been changed, the new recipe
+for python3 (3.7.1) has a new build system which has been applied to the ancient
+python recipe, allowing us to bump the python2 version number to 2.7.15. This
+change, unifies the build process for both python recipes, and probably solve
+some issues detected over the years, but unfortunately, this change breaks the
+pygame bootstrap (in a near future we will fix it or remove it). Also should be
+mentioned that the old python recipe is still usable, and has been renamed to
+`python2legacy`. This `python2legacy` recipe allow us to build with a minimum
+api lower than 21, which is not the case for the new python recipes which are
+limited to a minimum api of 21. All this work has been done using android ndk
+version r17c, and your build should success with that version...but be aware
+that the project is in constant development so...the ndk version will change
+at some time.
+
+Those mentioned changes has been done this way to make easier the transition
+between python3 and python2. We will slowly phase out python2 support
+towards 2020...so...if you are using python2 in your projects you should
+consider to migrate it into python3.
+
 Documentation
 =============
 

doc/source/quickstart.rst

@@ -93,11 +93,27 @@ named ``tools``, and you will need to run extra commands to install
 the SDK packages needed. 
 
 For Android NDK, note that modern releases will only work on a 64-bit
-operating system. If you are using a 32-bit distribution (or hardware),
-the latest useable NDK version is r10e, which can be downloaded here:
+operating system. The minimal, and recommended, NDK version to use is r17c:
+
+ - `Go to ndk downloads page <https://developer.android.com/ndk/downloads/>`_
+ - Windows users should create a virtual machine with an GNU Linux os
+   installed, and then you can follow the described instructions from within
+   your virtual machine.
+
+If you are using a 32-bit distribution (or hardware),
+the latest usable NDK version is r10e, which can be downloaded here:
 
 - `Legacy 32-bit Linux NDK r10e <http://dl.google.com/android/ndk/android-ndk-r10e-linux-x86.bin>`_
 
+.. warning::
+    **32-bit distributions**
+
+    Since the python2 recipe updated to version 2.7.15, the build system has
+    been changed and you should use an old release of python-for-android, which
+    contains the legacy python recipe (v2.7.2). The last python-for-android
+    release with the legacy version of python is version
+    `0.6.0 <https://github.com/kivy/python-for-android/archive/0.6.0.zip>`_.
+
 First, install a platform to target (you can also replace ``27`` with
 a different platform number, this will be used again later)::
 
@@ -113,8 +129,8 @@ Then, you can edit your ``~/.bashrc`` or other favorite shell to include new env
 
     # Adjust the paths!
     export ANDROIDSDK="$HOME/Documents/android-sdk-27"
-    export ANDROIDNDK="$HOME/Documents/android-ndk-r10e"
-    export ANDROIDAPI="27"  # Target API version of your application
+    export ANDROIDNDK="$HOME/Documents/android-ndk-r17c"
+    export ANDROIDAPI="26"  # Target API version of your application
     export NDKAPI="19"  # Minimum supported API version of your application
     export ANDROIDNDKVER="r10e"  # Version of the NDK you installed
 

pythonforandroid/archs.py

@@ -1,5 +1,6 @@
-from os.path import (exists, join, dirname)
+from os.path import (exists, join, dirname, split)
 from os import environ, uname
+from glob import glob
 import sys
 from distutils.spawn import find_executable
 
@@ -30,13 +31,30 @@ def include_dirs(self):
                 d.format(arch=self))
             for d in self.ctx.include_dirs]
 
-    def get_env(self, with_flags_in_cc=True):
+    @property
+    def target(self):
+        target_data = self.command_prefix.split('-')
+        return '-'.join(
+            [target_data[0], 'none', target_data[1], target_data[2]])
+
+    def get_env(self, with_flags_in_cc=True, clang=False):
         env = {}
 
-        env['CFLAGS'] = ' '.join([
-            '-DANDROID', '-mandroid', '-fomit-frame-pointer'
-            ' -D__ANDROID_API__={}'.format(self.ctx.ndk_api),
-            ])
+        cflags = [
+            '-DANDROID',
+            '-fomit-frame-pointer',
+            '-D__ANDROID_API__={}'.format(self.ctx.ndk_api)]
+        if not clang:
+            cflags.append('-mandroid')
+        else:
+            cflags.append('-target ' + self.target)
+            toolchain = '{android_host}-{toolchain_version}'.format(
+                android_host=self.ctx.toolchain_prefix,
+                toolchain_version=self.ctx.toolchain_version)
+            toolchain = join(self.ctx.ndk_dir, 'toolchains', toolchain, 'prebuilt', 'linux-x86_64')
+            cflags.append('-gcc-toolchain {}'.format(toolchain))
+
+        env['CFLAGS'] = ' '.join(cflags)
         env['LDFLAGS'] = ' '
 
         sysroot = join(self.ctx._ndk_dir, 'sysroot')
@@ -45,6 +63,8 @@ def get_env(self, with_flags_in_cc=True):
             # https://android.googlesource.com/platform/ndk/+/ndk-r15-release/docs/UnifiedHeaders.md
             env['CFLAGS'] += ' -isystem {}/sysroot/usr/include/{}'.format(
                 self.ctx.ndk_dir, self.ctx.toolchain_prefix)
+            env['CFLAGS'] += ' -I{}/sysroot/usr/include/{}'.format(
+                self.ctx.ndk_dir, self.command_prefix)
         else:
             sysroot = self.ctx.ndk_platform
             env['CFLAGS'] += ' -I{}'.format(self.ctx.ndk_platform)
@@ -82,8 +102,20 @@ def get_env(self, with_flags_in_cc=True):
             env['NDK_CCACHE'] = self.ctx.ccache
             env.update({k: v for k, v in environ.items() if k.startswith('CCACHE_')})
 
-        cc = find_executable('{command_prefix}-gcc'.format(
-            command_prefix=command_prefix), path=environ['PATH'])
+        if clang:
+            llvm_dirname = split(
+                glob(join(self.ctx.ndk_dir, 'toolchains', 'llvm*'))[-1])[-1]
+            clang_path = join(self.ctx.ndk_dir, 'toolchains', llvm_dirname,
+                              'prebuilt', 'linux-x86_64', 'bin')
+            environ['PATH'] = '{clang_path}:{path}'.format(
+                clang_path=clang_path, path=environ['PATH'])
+            exe = join(clang_path, 'clang')
+            execxx = join(clang_path, 'clang++')
+        else:
+            exe = '{command_prefix}-gcc'.format(command_prefix=command_prefix)
+            execxx = '{command_prefix}-g++'.format(command_prefix=command_prefix)
+
+        cc = find_executable(exe, path=environ['PATH'])
         if cc is None:
             print('Searching path are: {!r}'.format(environ['PATH']))
             raise BuildInterruptingException(
@@ -93,20 +125,20 @@ def get_env(self, with_flags_in_cc=True):
                 'installed. Exiting.')
 
         if with_flags_in_cc:
-            env['CC'] = '{ccache}{command_prefix}-gcc {cflags}'.format(
-                command_prefix=command_prefix,
+            env['CC'] = '{ccache}{exe} {cflags}'.format(
+                exe=exe,
                 ccache=ccache,
                 cflags=env['CFLAGS'])
-            env['CXX'] = '{ccache}{command_prefix}-g++ {cxxflags}'.format(
-                command_prefix=command_prefix,
+            env['CXX'] = '{ccache}{execxx} {cxxflags}'.format(
+                execxx=execxx,
                 ccache=ccache,
                 cxxflags=env['CXXFLAGS'])
         else:
-            env['CC'] = '{ccache}{command_prefix}-gcc'.format(
-                command_prefix=command_prefix,
+            env['CC'] = '{ccache}{exe}'.format(
+                exe=exe,
                 ccache=ccache)
-            env['CXX'] = '{ccache}{command_prefix}-g++'.format(
-                command_prefix=command_prefix,
+            env['CXX'] = '{ccache}{execxx}'.format(
+                execxx=execxx,
                 ccache=ccache)
 
         env['AR'] = '{}-ar'.format(command_prefix)
@@ -123,12 +155,13 @@ def get_env(self, with_flags_in_cc=True):
         env['READELF'] = '{}-readelf'.format(command_prefix)
         env['NM'] = '{}-nm'.format(command_prefix)
 
-        hostpython_recipe = Recipe.get_recipe('hostpython2', self.ctx)
-
-        # This hardcodes python version 2.7, needs fixing
+        hostpython_recipe = Recipe.get_recipe(
+            'host' + self.ctx.python_recipe.name, self.ctx)
         env['BUILDLIB_PATH'] = join(
             hostpython_recipe.get_build_dir(self.arch),
-            'build', 'lib.linux-{}-2.7'.format(uname()[-1]))
+            'build', 'lib.linux-{}-{}'.format(
+                uname()[-1], self.ctx.python_recipe.major_minor_version_string)
+        )
 
         env['PATH'] = environ['PATH']
 
@@ -147,12 +180,18 @@ class ArchARM(Arch):
     command_prefix = 'arm-linux-androideabi'
     platform_dir = 'arch-arm'
 
+    @property
+    def target(self):
+        target_data = self.command_prefix.split('-')
+        return '-'.join(
+            ['armv7a', 'none', target_data[1], target_data[2]])
+
 
 class ArchARMv7_a(ArchARM):
     arch = 'armeabi-v7a'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(ArchARMv7_a, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(ArchARMv7_a, self).get_env(with_flags_in_cc, clang=clang)
         env['CFLAGS'] = (env['CFLAGS'] +
                          (' -march=armv7-a -mfloat-abi=softfp '
                           '-mfpu=vfp -mthumb'))
@@ -166,8 +205,8 @@ class Archx86(Arch):
     command_prefix = 'i686-linux-android'
     platform_dir = 'arch-x86'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(Archx86, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(Archx86, self).get_env(with_flags_in_cc, clang=clang)
         env['CFLAGS'] = (env['CFLAGS'] +
                          ' -march=i686 -mtune=intel -mssse3 -mfpmath=sse -m32')
         env['CXXFLAGS'] = env['CFLAGS']
@@ -180,8 +219,8 @@ class Archx86_64(Arch):
     command_prefix = 'x86_64-linux-android'
     platform_dir = 'arch-x86_64'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(Archx86_64, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(Archx86_64, self).get_env(with_flags_in_cc, clang=clang)
         env['CFLAGS'] = (env['CFLAGS'] +
                          ' -march=x86-64 -msse4.2 -mpopcnt -m64 -mtune=intel')
         env['CXXFLAGS'] = env['CFLAGS']
@@ -194,8 +233,8 @@ class ArchAarch_64(Arch):
     command_prefix = 'aarch64-linux-android'
     platform_dir = 'arch-arm64'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(ArchAarch_64, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(ArchAarch_64, self).get_env(with_flags_in_cc, clang=clang)
         incpath = ' -I' + join(dirname(__file__), 'includes', 'arm64-v8a')
         env['EXTRA_CFLAGS'] = incpath
         env['CFLAGS'] += incpath

pythonforandroid/bootstrap.py

@@ -269,14 +269,13 @@ def strip_libraries(self, arch):
             return
         strip = sh.Command(strip)
 
-        if self.ctx.python_recipe.name == 'python2':
-            filens = shprint(sh.find, join(self.dist_dir, 'private'),
-                             join(self.dist_dir, 'libs'),
-                             '-iname', '*.so', _env=env).stdout.decode('utf-8')
-        else:
-            filens = shprint(sh.find, join(self.dist_dir, '_python_bundle', '_python_bundle', 'modules'),
-                             join(self.dist_dir, 'libs'),
-                             '-iname', '*.so', _env=env).stdout.decode('utf-8')
+        libs_dir = join(self.dist_dir, '_python_bundle',
+                        '_python_bundle', 'modules')
+        if self.ctx.python_recipe.name == 'python2legacy':
+            libs_dir = join(self.dist_dir, 'private')
+        filens = shprint(sh.find, libs_dir, join(self.dist_dir, 'libs'),
+                         '-iname', '*.so', _env=env).stdout.decode('utf-8')
+
         logger.info('Stripping libraries in private dir')
         for filen in filens.split('\n'):
             try:

pythonforandroid/bootstraps/common/build/build.py

@@ -46,7 +46,6 @@ def get_bootstrap_name():
 # Try to find a host version of Python that matches our ARM version.
 PYTHON = join(curdir, 'python-install', 'bin', 'python.host')
 if not exists(PYTHON):
-    print('Could not find hostpython, will not compile to .pyo (this is normal with python3)')
     PYTHON = None
 
 BLACKLIST_PATTERNS = [
@@ -151,7 +150,6 @@ def make_python_zip():
 
     if not exists('private'):
         print('No compiled python is present to zip, skipping.')
-        print('this should only be the case if you are using the CrystaX python')
         return
 
     global python_files

pythonforandroid/bootstraps/common/build/jni/application/src/start.c

@@ -104,6 +104,10 @@ int main(int argc, char *argv[]) {
   LOGP(env_argument);
   chdir(env_argument);
 
+#if PY_MAJOR_VERSION < 3
+  Py_NoSiteFlag=1;
+#endif
+
   Py_SetProgramName(L"android_python");
 
 #if PY_MAJOR_VERSION >= 3
@@ -144,10 +148,6 @@ int main(int argc, char *argv[]) {
     #if PY_MAJOR_VERSION >= 3
         wchar_t *wchar_paths = Py_DecodeLocale(paths, NULL);
         Py_SetPath(wchar_paths);
-    #else
-        char *wchar_paths = paths;
-        LOGP("Can't Py_SetPath in python2, so crystax python2 doesn't work yet");
-        exit(1);
     #endif
 
         LOGP("set wchar paths...");
@@ -161,6 +161,12 @@ int main(int argc, char *argv[]) {
   Py_Initialize();
 
 #if PY_MAJOR_VERSION < 3
+  // Can't Py_SetPath in python2 but we can set PySys_SetPath, which must
+  // be applied after Py_Initialize rather than before like Py_SetPath
+  #if PY_MICRO_VERSION >= 15
+    // Only for python native-build
+    PySys_SetPath(paths);
+  #endif
   PySys_SetArgv(argc, argv);
 #endif
 
@@ -183,7 +189,9 @@ int main(int argc, char *argv[]) {
    */
   PyRun_SimpleString("import sys, posix\n");
   if (dir_exists("lib")) {
-    /* If we built our own python, set up the paths correctly */
+    /* If we built our own python, set up the paths correctly.
+     * This is only the case if we are using the python2legacy recipe
+     */
     LOGP("Setting up python from ANDROID_APP_PATH");
     PyRun_SimpleString("private = posix.environ['ANDROID_APP_PATH']\n"
                        "argument = posix.environ['ANDROID_ARGUMENT']\n"

pythonforandroid/bootstraps/pygame/build/jni/application/Android.mk

@@ -18,9 +18,7 @@ LOCAL_CFLAGS := $(foreach D, $(APP_SUBDIRS), -I$(LOCAL_PATH)/$(D)) \
 				-I$(LOCAL_PATH)/../jpeg \
 				-I$(LOCAL_PATH)/../intl \
 				-I$(LOCAL_PATH)/.. \
-                -I$(LOCAL_PATH)/../../../../other_builds/$(PYTHON2_NAME)/$(ARCH)/python2/python-install/include/python2.7
-				# -I$(LOCAL_PATH)/../../../../python-install/include/python2.7
-				# -I$(LOCAL_PATH)/../../../build/python-install/include/python2.7
+                -I$(LOCAL_PATH)/../../../../other_builds/$(MK_PYTHON_INCLUDE_ROOT)
 
 
 LOCAL_CFLAGS += $(APPLICATION_ADDITIONAL_CFLAGS)
@@ -40,7 +38,7 @@ LOCAL_LDLIBS := -lpython2.7 -lGLESv1_CM -ldl -llog -lz
 
 # AND: Another hardcoded path that should be templated
 # AND: NOT TEMPALTED! We can use $ARCH
-LOCAL_LDFLAGS += -L$(LOCAL_PATH)/../../../../other_builds/$(PYTHON2_NAME)/$(ARCH)/python2/python-install/lib $(APPLICATION_ADDITIONAL_LDFLAGS)
+LOCAL_LDFLAGS += -L$(LOCAL_PATH)/../../../../other_builds/$(MK_PYTHON_LINK_ROOT) $(APPLICATION_ADDITIONAL_LDFLAGS)
 
 LIBS_WITH_LONG_SYMBOLS := $(strip $(shell \
 	for f in $(LOCAL_PATH)/../../libs/$ARCH/*.so ; do \