bpo-30386: Add a build infrastructure for Android.

Android/README.rst

@@ -0,0 +1,320 @@
+========================
+Python on Android README
+========================
+
+.. contents::
+
+This document provides a quick overview of the Python build system for Android
+on a linux platform [1]_.
+
+
+- The ``Android/makesetup`` script creates a Makefile and its command line
+  arguments are passed directly to Python ``configure``. Although the
+  ``--prefix`` command line argument may be omitted, it does not make much sense
+  as ``/usr/local`` does not exist on Android. One may run ``makesetup`` for
+  multiple (api, architecture) combinations in the same directory.
+
+  The targets of this Makefile allow to:
+
+  - Cross-compile Python with external libraries.
+  - Build a distribution to be copied on an Android device on an existing
+    Android application.
+  - Run the Python interpreter on the Android emulator using an adb_ shell, the
+    interpreter runs with readline enabled.
+  - Run a Python command on the Android emulator.
+  - Start a gdb remote debugging session of a python process running on the
+    emulator.
+  - Run the buildbottest.
+
+
+Requirements
+============
+
+- A linux platform [1]_
+- The Android NDK, android-ndk-r14 is required. The NDK is distributed by
+  Android as a zip file at `NDK downloads`_. The android-ndk-r14 needs 2.8 Gb of
+  disk space.
+- The Android SDK (see `Installation of the SDK`_) to run any emulator Makefile
+  target.  The SDK with the system images of the armv7, arm64, x86 and x86_64
+  architectures for API 24 requires about 11 Gb of disk space.
+- A Java 8 JRE is required by the SDK.
+- GNU make, find, xargs, md5sum, awk, sed, zip and unzip.
+- The native compiler of the build platform to build the native Python.
+- wget or curl for downloading the external libraries unless those files are
+  downloaded by other means and copied by hand to build/external-libraries.
+
+
+Environment variables
+=====================
+The ``Android/makesetup`` script uses the following environment variables:
+
+- ``ANDROID_NDK_ROOT``, the location where the Android NDK has been installed.
+  The default location is ``$HOME/android/android-ndk`` when this variable is
+  not set and this is where the buildbots expect to find the NDK.
+- ``ANDROID_API``, the target API level for the cross-compilation. This is also
+  the API level of the emulator AVD [2]_. The default value is 24.
+- ``ANDROID_ARCH``, the target architecture for the cross-compilation.  This is
+  also the architecture of the AVD. It may be armv7, arm64, x86 or x86_64. See
+  `ABI Management`_ for the corresponding ABIs. The default value is x86_64.
+
+The ``WITH_LIBRARIES`` variable is a comma separated list of the external
+libraries that are linked with extension modules when the build is done with the
+Makefile built by ``makesetup``. This list may include the libffi, ncurses,
+openssl, readline or sqlite library. When this environment variable is not set
+the readline and ncurses libraries are used.
+
+When building for the emulator, the following environment variables are used by
+``makesetup``:
+
+- ``ANDROID_SDK_ROOT``, the location where the Android SDK has been installed.
+  The default location is ``$HOME/android/android-sdk`` and this is where the
+  buildbots expect to find the SDK.
+
+- ``EMULATOR_CMD_LINE_OPTIONS``, the emulator command line options (see
+  `Start the Emulator from the Command Line`_). Some useful options:
+
+  * ``-use-system-libs``, use the system libstdc++ instead of the version
+    bundled with the emulator system. Use it if the emulator does not start.
+  * ``-no-window``, run the emulator on a server that has no display.
+
+- ``PYTHON_DEVPT_VERSION``, the current Python development version (e.g. 3.8).
+  This variable is used by the buildbots to allow for running simultaneous
+  emulator instances with both the current development version and the
+  maintenance version. It is not useful otherwise and when not set or
+  incorrectly set, simultaneous emulator instances can still be be run provided
+  that each one runs with a distinct architecture.
+
+The Makefile generated by makesetup is run according to the values of these
+environment variables at the time it has been generated.
+
+
+Makefile targets
+================
+Build targets
+^^^^^^^^^^^^^
+*build*
+  * Compile the native Python interpreter and cross-compile the external
+    libraries and python. This is the default target.
+
+*dist*
+  * Make a distribution consisting of:
+
+    - The machine-specific Python library zip file.
+    - The Python standard library zip file.
+
+*distclean*
+  * Kill the emulator if it is running.
+  * Make things clean, before making a distribution.
+  * Removes the DESTDIR directory where the external libraries have been
+    copied and where the cross-compiled Python has been installed by the
+    command ``make DESTDIR=$(PY_DESTDIR) install`` run on Python own Makefile.
+    The external libraries will not be rebuilt on the next make invocation.
+
+*clean*
+  Remove everything for the given (api, architecture) except the AVD.
+
+Emulator targets
+^^^^^^^^^^^^^^^^
+
+When the AVD is being created by one of the targets, it is not necessary to
+answer the following question printed on the screen at that time::
+
+    Do you wish to create a custom hardware profile? [no]
+
+Upon starting the emulator, python may fail to start with the error ``No module
+named 'encodings'`` because ``/sdcard`` is not mounted yet. This does not happen
+with the ``install`` target, but a user running the ``python`` or ``adb_shell``
+target must ensure that python is not run too early after the emulator has been
+started.
+
+*install*
+    Make a distribution, create the AVD if it does not exist, start the emulator
+    after having wiped out previous data and install the content of the two zip
+    files on the emulator. Then start an adb_ shell (see the ``adb_shell``
+    target description below).
+
+    Run ``make distclean install`` to get an install from scratch.
+
+*python*
+    Start the emulator if needed and run the python command defined by
+    ``PYTHON_ARGS``.  This variable is set on make command line or as an
+    environment variable when make is run. Quotes in the command are interpreted
+    both by the shell when interpreting the make command line and by make
+    itself, so they must be escaped properly such as in this example::
+
+        $ make python PYTHON_ARGS="-c 'print(\\\"Hello world.\\\")'"
+
+*adb_shell*
+    Start the emulator if needed and create an adb_ shell on the emulator.
+
+    At the first shell prompt a message is printed giving the shell command to
+    run to source a shell script that sets the environment variables needed to
+    run the Python interpreter. The script mainly does:
+
+    - Set ``PATH`` and ``LD_LIBRARY_PATH``.
+    - Set ``HOME`` to ``sys.exec_prefix``.
+    - Set miscellaneous stuff such as the terminal type, the terminal width and
+      the readline inputrc configuration file.
+    - Change the current directory to ``$HOME``.
+
+    After sourcing this script one can run the Python interpreter.
+
+*kill_emulator*
+    Kill the emulator. Useful when the emulator refuses to be shutdown from its
+    GUI or when there is no GUI.
+
+*avdclean*
+    Remove the AVD. This is the proper way to remove an AVD, do not just remove
+    the corresponding directory in the avd/ directory because Android maintains
+    also some information on the AVD in  ~/.android/avd.
+
+*gdb*
+    Start a gdb remote debugging session of a python process running on the
+    emulator. There must be a unique python process running on the emulator.
+
+    This requires that Python 2 is installed on the build platform and that
+    ``python2`` is found in the ``$PATH``.
+
+    The debugging session can be customized with the following variables set
+    on the command line (or as environment variables) to the value ``yes``
+    (for example ``GDB_PYTHON=yes make gdb``):
+
+    - ``GDB_PYTHON=yes``
+        Import the `libpython module`_ in gdb and get detailed information of
+        the PyObject(s) at the cost of speed.
+
+    - ``GDB_LOGGING=yes``
+        Setup logging in gdb and have the output of all the gdb commands also
+        redirected to ./gdb.log.
+
+    - ``GDB_SIGILL=yes``
+        Work around the problem that gdb fails with SIGILL in
+        ``__dl_notify_gdb_of_libraries()`` whenever a library is loaded when
+        debugging on the armv7 platforms.
+
+
+Using the emulator
+==================
+The ``makesetup`` script creates a Makefile. Running the Makefile requires that
+both the NDK and the SDK be installed (see `Installation of the SDK`_).  The
+`Makefile targets`_ section lists the available targets. For example, to build
+Python and install it on the emulator::
+
+    $ /path/to/python_src/Android/makesetup --with-pydebug && make install
+
+Another example, testing the socket module::
+
+    $ /path/to/python_src/Android/makesetup && make python PYTHON_ARGS="-m test test_socket"
+
+The first execution of this statement takes quite a while. It creates the
+Makefile for the given (api, architecture), as set by the ``$ANDROID_API`` and
+``$ANDROID_ARCH`` environment variables at the time the Makefile was generated,
+builds a native Python interpreter if it does not exist yet (used later for the
+cross-compilation of Python on Android) and builds the readline and ncurses
+external libraries for Android. It then cross-compiles Python for Android,
+linking extension modules with the external libraries and builds the zip
+distributions, creates an AVD, starts the emulator, installs the Python
+distribution on the emulator and finally runs the python command. Note that the
+Android emulators are very slow except for the x86 and x86_64 architectures when
+the processor of the build platform is itself of the x86 family.
+
+The next runs of::
+
+    $ make python PYTHON_ARGS="-m test test_socket"
+
+are much faster. The external libraries do not have to be rebuilt again, the AVD
+is already created and if there is no change in the source code then the native
+Python and Python on Android do not have to be rebuilt, but all the other steps
+enumerated above are run again except the Makefile creation.
+
+
+Installation of the SDK
+=======================
+The SDK provides the tools and system images to run the emulator. The emulator
+uses a specific AVD for each given (api, architecture) combination and system
+images must be added to the SDK for each one. One may install and manage the SDK
+with `Android Studio`_ but the prefered method is with the command line.
+
+The remaining part of this section describes the installation of the SDK and its
+management with the ``sdkmanager`` command line tool, through an example that
+installs the SDK and four system images for armv7, arm64, x86 and x86_64 at API
+24:
+
+- Download the basic Android command line tools by looking for a section named
+  *Get just the command line tools* at the end of the `Android Studio`_ page.
+  Unzip the downloaded file into the ``$ANDROID_SDK_ROOT`` (possibly not yet
+  created) directory::
+
+    $ unzip sdk-tools-linux-xxx.zip -d $ANDROID_SDK_ROOT
+
+  See the `sdkmanager manual`_.
+
+- List the installed packages (with an internet connection this gives also the
+  list of available packages)::
+
+    $ $ANDROID_SDK_ROOT/tools/bin/sdkmanager --list
+
+  The package names are truncated in the list output by this command, one must
+  add the ``--verbose`` option to the command to get the full package names.
+
+  Substitute ``';'`` in a package name with ``'/'`` to get the relative path to
+  ``ANDROID_SDK_ROOT`` where this package is installed.
+
+- Create a file named ``package_file`` that lists the packages to be installed
+  and whose content, following our example, is listed below. This file **must
+  not** contain **any empty line** or the command just fails with a cryptic
+  message::
+
+    build-tools;25.0.3
+    platform-tools
+    emulator
+    platforms;android-24
+    system-images;android-24;default;arm64-v8a
+    system-images;android-24;default;armeabi-v7a
+    system-images;android-24;default;x86
+    system-images;android-24;default;x86_64
+
+  A minimun installation consists of the first three packages in this list, plus
+  a ``platforms`` package for a given API and a system image in this API.
+
+- Install the packages::
+
+    $ $ANDROID_SDK_ROOT/tools/bin/sdkmanager --verbose --package_file=package_file
+
+- For reference, here is the output of ``sdkmanager --list`` after those
+  packages have been installed from scratch in december 2017::
+
+    Installed packages:
+      Path                              | Version | Description                    | Location
+      -------                           | ------- | -------                        | -------
+      build-tools;25.0.3                | 25.0.3  | Android SDK Build-Tools 25.0.3 | build-tools/25.0.3/
+      emulator                          | 26.1.4  | Android Emulator               | emulator/
+      patcher;v4                        | 1       | SDK Patch Applier v4           | patcher/v4/
+      platform-tools                    | 26.0.2  | Android SDK Platform-Tools     | platform-tools/
+      platforms;android-24              | 2       | Android SDK Platform 24        | platforms/android-24/
+      system-images;a...ult;arm64-v8a   | 7       | ARM 64 v8a System Image        | system-images/android-24/default/arm64-v8a/
+      system-images;a...ult;armeabi-v7a | 7       | ARM EABI v7a System Image      | system-images/android-24/default/armeabi-v7a/
+      system-images;a...ult;x86         | 8       | Intel x86 Atom System Image    | system-images/android-24/default/x86/
+      system-images;a...ult;x86_64      | 8       | Intel x86 Atom_64 System Image | system-images/android-24/default/x86_64/
+      tools                             | 26.1.1  | Android SDK Tools              | tools/
+
+
+.. [1] A 64-bit linux distribution capable of running 32-bit applications with
+   GNU C Library (glibc) 2.19 or later, see the `Android Studio`_ system
+   requirements section. It is not necessary to install Android Studio.
+
+.. [2] Android Virtual Device. This is the image run by the emulator. It is
+   specific to each (api, architecture) and it holds also the configuration of
+   the emulator.
+
+
+.. _termux: https://termux.com/
+.. _adb: https://developer.android.com/studio/command-line/adb.html
+.. _`libpython module`: https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py
+.. _`NDK downloads`: https://developer.android.com/ndk/downloads/index.html
+.. _`Android Studio`: https://developer.android.com/studio/index.html
+.. _`sdkmanager manual`: https://developer.android.com/studio/command-line/sdkmanager.html
+.. _`Start the Emulator from the Command Line`: https://developer.android.com/studio/run/emulator-commandline.html
+.. _`ABI Management`: https://developer.android.com/ndk/guides/abis.html
+
+.. vim:filetype=rst:tw=80:ts=8:sts=2:sw=2:et: