Unify most of the test apps into on device unit test app (#2046)

* 📦 Refactor (move) `PythonTestMixIn` into `tests.mixin`

* 📦 Unify most of the test apps into one app

So depending on the supplied recipes, a different test app will be ran.

The supported test apps are:
  - kivy
  - flask
  - no-gui

* 🏗️ Make test app `on_device_unit_tests` the default...

for our CI providers and `rebuild_updated_recipes.py` because we recently removed
the old test app that we made use of.

* 📝 Update docs

* ✅ Create a `x86_64` testapp for gh-actions

* 📝 Fix typo errors and...

Remove sentence regarding python2 compatibility...
because python2 it's almost at the end of his life

* ✅ Make a `flask` testapp...

So we can get an apk via gh-actions

Notes:
  - this way the reviewers can install the flask apk without making the build
  - these changes will be reverted in the next
    commit...since we want a kivy app as our default)

* ✅ Revert "Make a `flask` testapp..."

This reverts commit 0ed4d70.

* 🔥 Remove `orientation` kwarg...

in case that we detect that p4a will build an `service_only` app.

Note: if we don't do this we wil get errors once we try to create our apk with the target distribution

* ✅ Create a `x86` testapp for gh-actions

and change gh-actions job title (because it looks better)

* 🏗️ Rename test app

Because it looks better and is more readable.

* 📝 Fix a syntactical error at docstring

* 🚚 Rename `setup_test_app.py` to `setup.py`

Because reducing filename to `setup.py` will make our cli arguments shorter and source code cleaner

* 🔧 Disable gh-actions `fail-fast`

So our matrix gh-actions builds will not be cancelled if some arch fails
which is very useful to fix specific arch issues.

See also: #2073

* 💚 Add `testapps-with-numpy/%` & ...

make `testapps/%` support multiple archs

Notes: this completes the job started at 9ea53fc & afd4413

Author: opacam

.github/workflows/push.yml

@@ -1,4 +1,4 @@
-name: Unit tests & Build Testapp
+name: Unit tests & build apps
 
 on: ['push', 'pull_request']
 
@@ -42,25 +42,26 @@ jobs:
         make test
 
   build:
-    name: Build testapp
+    name: Unit test apk
     needs: [flake8]
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        build-arch: ['arm64-v8a', 'armeabi-v7a']
+        build-arch: ['arm64-v8a', 'armeabi-v7a', 'x86_64', 'x86']
     steps:
     - name: Checkout python-for-android
       uses: actions/checkout@v2
     - name: Pull docker image
       run: |
         make docker/pull
-    - name: Build apk for Python 3 ${{ matrix.build-arch }}
+    - name: Build apk Python 3 ${{ matrix.build-arch }}
       run: |
         mkdir -p apks
-        make docker/run/make/with-artifact/testapps/${{ matrix.build-arch }}
+        make docker/run/make/with-artifact/testapps-with-numpy/${{ matrix.build-arch }}
     - uses: actions/upload-artifact@v1
       with:
-        name: bdisttest_python3_sqlite_openssl_googlendk__${{ matrix.build-arch }}-debug-1.1.apk
+        name: bdist_test_app_unittests__${{ matrix.build-arch }}-debug-1.1.apk
         path: apks
 
   rebuild_updated_recipes:

.travis.yml

@@ -45,7 +45,7 @@ jobs:
       name: Python 3 arm64-v8a (with numpy)
       stage: build testapps
       before_script: make docker/pull
-      script: make docker/run/make/testapps/arm64-v8a
+      script: make docker/run/make/testapps-with-numpy/arm64-v8a
     - <<: *testapps
       name: Python 3 armeabi-v7a
       os: osx

Makefile

@@ -35,16 +35,18 @@ rebuild_updated_recipes: virtualenv
 	ANDROID_SDK_HOME=$(ANDROID_SDK_HOME) ANDROID_NDK_HOME=$(ANDROID_NDK_HOME) \
 	$(PYTHON) ci/rebuild_updated_recipes.py
 
-testapps/arm64-v8a: virtualenv
-	. $(ACTIVATE) && cd testapps/ && \
-    python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
+testapps-with-numpy/%: virtualenv
+	$(eval $@_APP_ARCH := $(shell basename $*))
+	. $(ACTIVATE) && cd testapps/on_device_unit_tests/ && \
+    python setup.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
     --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools,numpy \
-    --arch=arm64-v8a
+    --arch=$($@_APP_ARCH)
 
-testapps/armeabi-v7a: virtualenv
-	. $(ACTIVATE) && cd testapps/ && \
-    python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
-    --arch=armeabi-v7a
+testapps/%: virtualenv
+	$(eval $@_APP_ARCH := $(shell basename $*))
+	. $(ACTIVATE) && cd testapps/on_device_unit_tests/ && \
+    python setup.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
+    --arch=$($@_APP_ARCH)
 
 clean:
 	find . -type d -name "__pycache__" -exec rm -r {} +
@@ -72,10 +74,9 @@ docker/run/make/%: docker/build
 	docker run --rm --env-file=.env $(DOCKER_IMAGE) make $*
 
 docker/run/make/with-artifact/%: docker/build
-	$(eval $@_APP_NAME := bdisttest_python3_sqlite_openssl_googlendk)
 	$(eval $@_APP_ARCH := $(shell basename $*))
 	docker run --name p4a-latest --env-file=.env $(DOCKER_IMAGE) make $*
-	docker cp p4a-latest:/home/user/app/testapps/$($@_APP_NAME)__$($@_APP_ARCH)-debug-1.1-.apk ./apks
+	docker cp p4a-latest:/home/user/app/testapps/on_device_unit_tests/bdist_unit_tests_app__$($@_APP_ARCH)-debug-1.1-.apk ./apks
 	docker rm -fv p4a-latest
 
 docker/run/shell: docker/build

ci/rebuild_updated_recipes.py

@@ -52,16 +52,15 @@ def build(target_python, requirements):
     """
     if not requirements:
         return
-    testapp = 'setup_testapp_python3_sqlite_openssl.py'
     android_sdk_home = os.environ['ANDROID_SDK_HOME']
     android_ndk_home = os.environ['ANDROID_NDK_HOME']
     requirements.add(target_python.name)
     requirements = ','.join(requirements)
     logger.info('requirements: {}'.format(requirements))
-    with current_directory('testapps/'):
+    with current_directory('testapps/on_device_unit_tests/'):
         # iterates to stream the output
         for line in sh.python(
-                testapp, 'apk', '--sdk-dir', android_sdk_home,
+                'setup.py', 'apk', '--sdk-dir', android_sdk_home,
                 '--ndk-dir', android_ndk_home, '--requirements',
                 requirements, _err_to_out=True, _iter=True):
             print(line)

doc/source/testing_pull_requests.rst

@@ -103,30 +103,31 @@ Using python-for-android commands directly from the pull request files
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 - Enter inside the directory of the cloned repository in the above
-  step and run p4a command with proper args, eg::
+  step and run p4a command with proper args, eg (to test an modified
+  `pycryptodome` recipe)::
 
     .. codeblock:: bash
 
         cd p4a-feature-fix-numpy
         PYTHONPATH=. python3 -m pythonforandroid.toolchain apk \
-            --private=testapps/testapp_sqlite_openssl \
-            --dist-name=dist_test_app_python3_libs \
+            --private=testapps/on_device_unit_tests/test_app \
+            --dist-name=dist_unit_tests_app_pycryptodome \
             --package=org.kivy \
-            --name=test_app_python3_sqlite_openssl \
+            --name=unit_tests_app_pycryptodome \
             --version=0.1 \
-            --requirements=requests,peewee,sdl2,pyjnius,kivy,python3 \
+            --requirements=sdl2,pyjnius,kivy,python3,pycryptodome \
             --ndk-dir=/media/DEVEL/Android/android-ndk-r20 \
             --sdk-dir=/media/DEVEL/Android/android-sdk-linux \
             --android-api=27 \
             --arch=arm64-v8a \
-            --permission=INTERNET \
+            --permission=VIBRATE \
             --debug
 
 Things that you should know:
 
 
-    - The example above will build an testapp we will make use of the files of
-      the testapp named `testapp_sqlite_openssl.py` but we don't use the setup
+    - The example above will build an test app we will make use of the files of
+      the `on device unit tests` test app but we don't use the setup
       file to build it so we must tell python-for-android what we want via
       arguments
     - be sure to at least edit the following arguments when running the above
@@ -159,19 +160,19 @@ Installing python-for-android using the github's branch of the pull request
         cd p4a-feature-fix-numpy
         pip3 install . --upgrade --user
 
-- Now, go inside the `testapps` directory (we assume that you still are inside
-  the cloned repository)::
+- Now, go inside the `testapps/on_device_unit_tests` directory (we assume that
+  you still are inside the cloned repository)::
 
     .. codeblock:: bash
 
-        cd testapps
+        cd testapps/on_device_unit_tests
 
 - Run the build of the apk via the freshly installed copy of python-for-android
   by running a similar command than below::
 
     .. code-block:: bash
 
-        python3 setup_testapp_python3_sqlite_openssl.py apk \
+        python3 setup.py apk \
             --ndk-dir=/media/DEVEL/Android/android-ndk-r20 \
             --sdk-dir=/media/DEVEL/Android/android-sdk-linux \
             --android-api=27 \
@@ -182,8 +183,7 @@ Installing python-for-android using the github's branch of the pull request
 Things that you should know:
 
     - In the example above, we override some variables that are set in
-      `setup_testapp_python3_sqlite_openssl.py`, you could also override them
-      by editing this file
+      `setup.py`, you could also override them by editing this file
     - be sure to at least edit the following arguments when running the above
       command, since the default set in there it's unlikely that match your
       installation:

testapps/on_device_unit_tests/README.rst

@@ -4,18 +4,38 @@ On device unit tests
 This test app runs a set of unit tests, to help confirm that the
 python-for-android build is actually working properly.
 
+Also it's dynamic, because it will run one app or another depending on the
+supplied recipes at build time.
+
+It currently supports three app `modes`:
+  - `kivy app` (with sdl2 bootstrap): if kivy in recipes
+  - `flask app` (with webview bootstrap): if flask in recipes
+  - `no gui`: if neither of above cases is taken
+
 The main tests are for the recipes built in the apk. Each module (or
 other tool) is at least imported and subject to some basic check.
 
-This app is experimental, it doesn't yet support things like testing
-only the requirements you ask for (so if you build with requirements
-other than those specified, the tests may fail). It also has no gui
-yet, the results must be checked via logcat.
+This test app can be build via `setup.py` or via buildozer. In both
+cases it will build a basic kivy app with a set of tests defined via the
+`requirements` keyword (specified at build time).
+
+In case that you build the `test app with no-gui`, the unittests results must
+be checked via command `adb logcat` or some logging apk (you may need root
+permissions in your device to use such app).
+
+Building the app with python-for-android
+========================================
+
+You can use the provided file `setup.py`. Check our `Makefile
+<https://github.com/kivy/python-for-android/blob/develop/Makefile>`__ to guess
+how to build the test app, or also you can look at `testing pull requests documentation
+<https://github.com/kivy/python-for-android/blob/develop/doc/source/testing_pull_requests.rst>`__,
+which describes some of the methods that you can use to build the test app.
 
-Building the app
-================
+Building the app with buildozer
+===============================
 
-This app should be built using buildozer, which it also serves as a
+This app can be built using buildozer, which it also serves as a
 test for::
 
   $ buildozer android debug

testapps/on_device_unit_tests/buildozer.spec

@@ -13,7 +13,7 @@ package.domain = org.kivy
 source.dir = test_app
 
 # (list) Source files to include (let empty to include all the files)
-source.include_exts = py,png,jpg,kv,atlas
+source.include_exts = py,png,jpg,kv,atlas,html,css,otf,txt
 
 # (list) List of inclusions using pattern matching
 #source.include_patterns = assets/*,images/*.png
@@ -36,7 +36,7 @@ version = 0.1
 
 # (list) Application requirements
 # comma separated e.g. requirements = sqlite3,kivy
-requirements = python3,kivy,openssl,numpy,sqlite3
+requirements = python3,kivy,libffi,openssl,numpy,sqlite3
 
 # (str) Custom source folders for requirements
 # Sets custom source for any requirements with recipes
@@ -52,7 +52,7 @@ requirements = python3,kivy,openssl,numpy,sqlite3
 #icon.filename = %(source.dir)s/data/icon.png
 
 # (str) Supported orientation (one of landscape, sensorLandscape, portrait or all)
-orientation = portrait
+orientation = all
 
 # (list) List of service to declare
 #services = NAME:ENTRYPOINT_TO_PY,NAME2:ENTRYPOINT2_TO_PY

testapps/on_device_unit_tests/setup.py

@@ -0,0 +1,92 @@
+"""
+This is the `setup.py` file for the `on device unit test app`.
+
+In this module we can control how will be built our test app. Depending on
+our requirements we can build an kivy, flask or a non-gui app. We default to an
+kivy app, since the python-for-android project its a sister project of kivy.
+
+The parameter `requirements` is crucial to determine the unit tests we will
+perform with our app, so we must explicitly name the recipe we want to test
+and, of course, we should have the proper test for the given recipe at
+`tests.test_requirements.py` or nothing will be tested. We control our default
+app requirements via the dictionary `options`. Here you have some examples
+to build the supported app modes::
+
+  - kivy *basic*: `sqlite3,libffi,openssl,pyjnius,kivy,python3,requests`
+  - kivy *images/graphs*: `kivy,python3,numpy,matplotlib,Pillow`
+  - kivy *encryption*: `kivy,python3,cryptography,pycryptodome,scrypt,
+    m2crypto,pysha3`
+  - flask (with webview bootstrap): `sqlite3,libffi,openssl,pyjnius,flask,
+    python3,genericndkbuild`
+
+
+.. note:: just noting that, for the `kivy basic` app, we add the requirements:
+          `sqlite3,libffi,openssl` so this way we will trigger the unit tests
+          that we have for such recipes.
+
+.. tip:: to force `python-for-android` generate an `flask` app without using
+         the kwarg `bootstrap`, we add the recipe `genericndkbuild`, which will
+         trigger the `webview bootstrap` at build time.
+"""
+
+import os
+import sys
+from distutils.core import setup
+from setuptools import find_packages
+
+# define a basic test app, which can be override passing the proper args to cli
+options = {
+    'apk':
+        {
+            'requirements':
+                'sqlite3,libffi,openssl,pyjnius,kivy,python3,requests',
+            'android-api': 27,
+            'ndk-api': 21,
+            'dist-name': 'bdist_unit_tests_app',
+            'arch': 'armeabi-v7a',
+            'permissions': ['INTERNET', 'VIBRATE'],
+            'orientation': 'sensor',
+            'service': 'P4a_test_service:app_service.py',
+        }
+}
+
+# check if we overwrote the default test_app requirements via `cli`
+requirements = options['apk']['requirements'].rsplit(',')
+for n, arg in enumerate(sys.argv):
+    if arg == '--requirements':
+        print('found requirements')
+        requirements = sys.argv[n + 1].rsplit(',')
+        break
+
+# remove `orientation` in case that we don't detect a kivy or flask app,
+# since the `service_only` bootstrap does not support such argument
+if not ({'kivy', 'flask'} & set(requirements)):
+    options['apk'].pop('orientation')
+
+# write a file to let the test_app know which requirements we want to test
+# Note: later, when running the app, we will guess if we have the right test.
+app_requirements_txt = os.path.join(
+    os.path.split(__file__)[0],
+    'test_app',
+    'app_requirements.txt',
+)
+with open(app_requirements_txt, 'w') as requirements_file:
+    for req in requirements:
+        requirements_file.write(f'{req.split("==")[0]}\n')
+
+# run the install
+setup(
+    name='unit_tests_app',
+    version='1.1',
+    description='p4a on device unit test app',
+    author='Alexander Taylor, Pol Canelles',
+    author_email='[email protected], [email protected]',
+    packages=find_packages(),
+    options=options,
+    package_data={
+        'test_app': ['*.py', '*.kv', '*.txt'],
+        'test_app/static': ['*.png', '*.css', '*.otf'],
+        'test_app/templates': ['*.html'],
+        'test_app/tests': ['*.py'],
+    }
+)