Skip to content

Playing with on_device_unit_test_app #2046 - Episode II #2088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 30 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
e5cd6f4
:package: Refactor (move) `PythonTestMixIn` into `tests.mixin`
opacam Dec 11, 2019
52162ef
:package: Unify most of the test apps into one app
opacam Dec 15, 2019
1ce7d8c
:rocket: Make test app `on_device_unit_tests` the default...
opacam Dec 15, 2019
1ac5309
:pencil: Update docs
opacam Dec 15, 2019
f92376c
:rocket: Enhance gh-actions with a matrix for our testapps
opacam Dec 18, 2019
6989cb2
:pencil: Fix typo errors and...
opacam Dec 29, 2019
4dd643a
:white_check_mark: Make a `flask` testapp...
opacam Dec 29, 2019
ba86c92
:white_check_mark: Revert "Make a `flask` testapp..."
opacam Dec 29, 2019
eb8427e
:rocket: Remove `orientation` kwarg...
opacam Jan 2, 2020
ea402c1
:rocket: Build test app for all archs
opacam Jan 6, 2020
d108dd1
:rocket: Rename test app
opacam Jan 6, 2020
26c95f2
:pencil: Fix a syntactical error at docstring
opacam Jan 18, 2020
b54d8e4
:rocket: Rename `setup_test_app.py` to `setup.py`
opacam Jan 18, 2020
cbebd95
:rocket: Disable gh-actions `fail-fast`
opacam Mar 14, 2020
237de3f
:green_heart: Add `testapps/python3-with-numpy/%`
opacam Mar 14, 2020
e9c3e0a
:sparkles: Add `libbz2` recipe (libbz2.so)
opacam Mar 12, 2020
d404e4b
:white_check_mark: Add tests for `libbz2` recipe
opacam Mar 14, 2020
ee24e23
:sparkles: Add `liblzma` recipe (liblzma.so)
opacam Mar 14, 2020
184c6ef
:white_check_mark: Add tests for `liblzma` recipe
opacam Mar 14, 2020
5686941
:sparkles: Add `bz2` and `lzma` support to `Python3`
opacam Mar 14, 2020
a0c1a92
:sparkles: Add `pandas` recipe
opacam Mar 12, 2020
5da75de
:white_check_mark: Add tests for `pandas` recipe
opacam Mar 14, 2020
1a4deb7
:sparkles: Add `mplfinance` recipe
opacam Mar 12, 2020
9228336
:rocket: Add `KiwisolverTestCase` to `on_device_unit_tests_app`
opacam Mar 14, 2020
51a02c0
:rocket: Add `PandasTestCase` to `on_device_unit_tests_app`
opacam Mar 14, 2020
6e4972a
:rocket: Add `MplfinanceTestCase` to `on_device_unit_tests_app`
opacam Mar 14, 2020
33a8a9a
:green_heart: Add new dependency (`autopoint`)
opacam Mar 14, 2020
9d111d9
:bug: Remove `pandas` dependencies from `setup.py`
opacam Mar 15, 2020
fc0c6a6
:bug: Create install dir for `liblzma`
opacam Mar 15, 2020
47bbd9f
:construction: Pass the apk's requirements from the `CI` files
opacam Mar 15, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions .github/workflows/push.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Unit tests & Build Testapp
name: Unit tests & build apps

on: ['push', 'pull_request']

Expand Down Expand Up @@ -42,25 +42,28 @@ jobs:
make test

build:
name: Build testapp
name: Unit test apk
needs: [flake8]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
build-arch: ['arm64-v8a', 'armeabi-v7a']
build-arch: ['arm64-v8a', 'armeabi-v7a', 'x86_64', 'x86']
steps:
- name: Checkout python-for-android
uses: actions/checkout@v2
- name: Pull docker image
run: |
make docker/pull
- name: Build apk for Python 3 ${{ matrix.build-arch }}
- name: Build apk python3 ${{ matrix.build-arch }}
run: |
mkdir -p apks
make docker/run/make/with-artifact/testapps/python3/${{ matrix.build-arch }}
make docker/run/make/with-artifact/testapps/python3-reqs-arch/$REQUIREMENTS/${{ matrix.build-arch }}
env:
REQUIREMENTS: sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools,numpy,kiwisolver,pandas,matplotlib,mplfinance
- uses: actions/upload-artifact@v1
with:
name: bdisttest_python3_sqlite_openssl_googlendk__${{ matrix.build-arch }}-debug-1.1.apk
name: unit_tests_app__python3-${{ matrix.build-arch }}-debug-1.1.apk
path: apks

rebuild_updated_recipes:
Expand Down
4 changes: 3 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,10 @@ jobs:
- &testapps
name: Python 3 arm64-v8a (with numpy)
stage: build testapps
env:
- REQUIREMENTS=sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools,numpy
before_script: make docker/pull
script: make docker/run/make/testapps/python3/arm64-v8a
script: make docker/run/make/testapps/python3-reqs-arch/${REQUIREMENTS}/arm64-v8a
- <<: *testapps
name: Python 3 armeabi-v7a
os: osx
Expand Down
1 change: 1 addition & 0 deletions Dockerfile.py3
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ RUN dpkg --add-architecture i386 \
&& ${RETRY} apt -y install -qq --no-install-recommends \
autoconf \
automake \
autopoint \
build-essential \
ccache \
cmake \
Expand Down
42 changes: 21 additions & 21 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -35,21 +35,26 @@ rebuild_updated_recipes: virtualenv
ANDROID_SDK_HOME=$(ANDROID_SDK_HOME) ANDROID_NDK_HOME=$(ANDROID_NDK_HOME) \
$(PYTHON) ci/rebuild_updated_recipes.py

testapps/python2/armeabi-v7a: virtualenv
. $(ACTIVATE) && cd testapps/ && \
python setup_testapp_python2_sqlite_openssl.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
--requirements sdl2,pyjnius,kivy,python2,openssl,requests,sqlite3,setuptools

testapps/python3/arm64-v8a: virtualenv
. $(ACTIVATE) && cd testapps/ && \
python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
--requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools,numpy \
--arch=arm64-v8a

testapps/python3/armeabi-v7a: virtualenv
. $(ACTIVATE) && cd testapps/ && \
python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
--arch=armeabi-v7a
testapps/python2/%: virtualenv
$(eval $@_APP_ARCH := $(shell basename $*))
. $(ACTIVATE) && cd testapps/on_device_unit_tests/ && \
python setup.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
--requirements sdl2,pyjnius,kivy,python2,openssl,requests,sqlite3,setuptools \
--arch=$($@_APP_ARCH)

testapps/python3-reqs-arch/%: virtualenv
$(eval $@_APP_ARCH := $(shell basename $*))
$(eval $@_APP_REQS := $(shell basename $(dir $*)))
. $(ACTIVATE) && cd testapps/on_device_unit_tests/ && \
python setup.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
--requirements $($@_APP_REQS) \
--arch=$($@_APP_ARCH)

testapps/python3/%: virtualenv
$(eval $@_APP_ARCH := $(shell basename $*))
. $(ACTIVATE) && cd testapps/on_device_unit_tests/ && \
python setup.py apk --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
--arch=$($@_APP_ARCH)

clean:
find . -type d -name "__pycache__" -exec rm -r {} +
Expand Down Expand Up @@ -77,14 +82,9 @@ docker/run/make/%: docker/build
docker run --rm --env-file=.env $(DOCKER_IMAGE) make $*

docker/run/make/with-artifact/%: docker/build
ifeq (,$(findstring python3,$($*)))
$(eval $@_APP_NAME := bdisttest_python3_sqlite_openssl_googlendk)
else
$(eval $@_APP_NAME := bdisttest_python2_sqlite_openssl)
endif
$(eval $@_APP_ARCH := $(shell basename $*))
docker run --name p4a-latest --env-file=.env $(DOCKER_IMAGE) make $*
docker cp p4a-latest:/home/user/app/testapps/$($@_APP_NAME)__$($@_APP_ARCH)-debug-1.1-.apk ./apks
docker cp p4a-latest:/home/user/app/testapps/on_device_unit_tests/bdist_unit_tests_app__$($@_APP_ARCH)-debug-1.1-.apk ./apks
docker rm -fv p4a-latest

docker/run/shell: docker/build
Expand Down
4 changes: 4 additions & 0 deletions ci/makefiles/osx.mk
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ install_java:
brew tap adoptopenjdk/openjdk
brew cask install adoptopenjdk8
/usr/libexec/java_home -V
# install gettext (because it contains autopoint
# and we need that to build `liblzma` recipe)
brew install gettext
ln -s `brew ls gettext | grep bin/autopoint` /usr/local/bin

upgrade_cython:
pip3 install --upgrade Cython==0.28.6
Expand Down
7 changes: 2 additions & 5 deletions ci/rebuild_updated_recipes.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,18 +55,15 @@ def build(target_python, requirements):
"""
if not requirements:
return
testapp = 'setup_testapp_python2.py'
android_sdk_home = os.environ['ANDROID_SDK_HOME']
android_ndk_home = os.environ['ANDROID_NDK_HOME']
if target_python == TargetPython.python3:
testapp = 'setup_testapp_python3_sqlite_openssl.py'
requirements.add(target_python.name)
requirements = ','.join(requirements)
logger.info('requirements: {}'.format(requirements))
with current_directory('testapps/'):
with current_directory('testapps/on_device_unit_tests/'):
# iterates to stream the output
for line in sh.python(
testapp, 'apk', '--sdk-dir', android_sdk_home,
'setup.py', 'apk', '--sdk-dir', android_sdk_home,
'--ndk-dir', android_ndk_home, '--requirements',
requirements, _err_to_out=True, _iter=True):
print(line)
Expand Down
28 changes: 14 additions & 14 deletions doc/source/testing_pull_requests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -103,30 +103,31 @@ Using python-for-android commands directly from the pull request files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Enter inside the directory of the cloned repository in the above
step and run p4a command with proper args, eg::
step and run p4a command with proper args, eg (to test an modified
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an modified -> a modified

`pycryptodome` recipe)::

.. codeblock:: bash

cd p4a-feature-fix-numpy
PYTHONPATH=. python3 -m pythonforandroid.toolchain apk \
--private=testapps/testapp_sqlite_openssl \
--dist-name=dist_test_app_python3_libs \
--private=testapps/on_device_unit_tests/test_app \
--dist-name=dist_unit_tests_app_pycryptodome \
--package=org.kivy \
--name=test_app_python3_sqlite_openssl \
--name=unit_tests_app_pycryptodome \
--version=0.1 \
--requirements=requests,peewee,sdl2,pyjnius,kivy,python3 \
--requirements=sdl2,pyjnius,kivy,python3,pycryptodome \
--ndk-dir=/media/DEVEL/Android/android-ndk-r20 \
--sdk-dir=/media/DEVEL/Android/android-sdk-linux \
--android-api=27 \
--arch=arm64-v8a \
--permission=INTERNET \
--permission=VIBRATE \
--debug

Things that you should know:


- The example above will build an testapp we will make use of the files of
the testapp named `testapp_sqlite_openssl.py` but we don't use the setup
- The example above will build an test app we will make use of the files of
the `on device unit tests` test app but we don't use the setup
file to build it so we must tell python-for-android what we want via
arguments
- be sure to at least edit the following arguments when running the above
Expand Down Expand Up @@ -159,19 +160,19 @@ Installing python-for-android using the github's branch of the pull request
cd p4a-feature-fix-numpy
pip3 install . --upgrade --user

- Now, go inside the `testapps` directory (we assume that you still are inside
the cloned repository)::
- Now, go inside the `testapps/on_device_unit_tests` directory (we assume that
you still are inside the cloned repository)::

.. codeblock:: bash

cd testapps
cd testapps/on_device_unit_tests

- Run the build of the apk via the freshly installed copy of python-for-android
by running a similar command than below::

.. code-block:: bash

python3 setup_testapp_python3_sqlite_openssl.py apk \
python3 setup.py apk \
--ndk-dir=/media/DEVEL/Android/android-ndk-r20 \
--sdk-dir=/media/DEVEL/Android/android-sdk-linux \
--android-api=27 \
Expand All @@ -182,8 +183,7 @@ Installing python-for-android using the github's branch of the pull request
Things that you should know:

- In the example above, we override some variables that are set in
`setup_testapp_python3_sqlite_openssl.py`, you could also override them
by editing this file
`setup.py`, you could also override them by editing this file
- be sure to at least edit the following arguments when running the above
command, since the default set in there it's unlikely that match your
installation:
Expand Down
8 changes: 8 additions & 0 deletions pythonforandroid/python.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,14 @@ def add_flags(include_flags, link_dirs, link_libs):
add_flags(recipe.include_flags(arch),
recipe.link_dirs_flags(arch), recipe.link_libs_flags())

for library_name in {'libbz2', 'liblzma'}:
if library_name in self.ctx.recipe_build_order:
info(f'Activating flags for {library_name}')
recipe = Recipe.get_recipe(library_name, self.ctx)
add_flags(recipe.get_library_includes(arch),
recipe.get_library_ldflags(arch),
recipe.get_library_libs_flag())

# python build system contains hardcoded zlib version which prevents
# the build of zlib module, here we search for android's zlib version
# and sets the right flags, so python can be build with android's zlib
Expand Down
54 changes: 54 additions & 0 deletions pythonforandroid/recipes/libbz2/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
from multiprocessing import cpu_count

from pythonforandroid.archs import Arch
from pythonforandroid.recipe import Recipe
from pythonforandroid.logger import shprint
from pythonforandroid.util import current_directory
import sh


class LibBz2Recipe(Recipe):

# there is not an official release yet...
# so pinned to last commit at the time of writing
version = "1.0.8"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that doesn't look like a pinned commit to me 🤔

url = "https://sourceware.org/pub/bzip2/bzip2-{version}.tar.gz"
built_libraries = {"libbz2.so": ""}
patches = ["lib_android.patch"]

def build_arch(self, arch: Arch) -> None:
env = self.get_recipe_env(arch)
with current_directory(self.get_build_dir(arch.arch)):
shprint(
sh.make,
"-j",
str(cpu_count()),
f'CC={env["CC"]}',
f'AR={env["AR"]}',
f'RANLIB={env["RANLIB"]}',
"-f",
"Makefile-libbz2_so",
_env=env,
)

def get_library_includes(self, arch: Arch) -> str:
"""Returns a string with the appropriate `-I<lib directory>` to link
with the bz2 lib. This string is usually added to the environment
variable `CPPFLAGS`"""
return " -I" + self.get_build_dir(arch.arch)

def get_library_ldflags(self, arch: Arch) -> str:
"""Returns a string with the appropriate `-L<lib directory>` to link
with the bz2 lib. This string is usually added to the environment
variable `LDFLAGS`"""
return " -L" + self.get_build_dir(arch.arch)

@staticmethod
def get_library_libs_flag() -> str:
"""Returns a string with the appropriate `-l<lib>` flags to link with
the bz2 lib. This string is usually added to the environment
variable `LIBS`"""
return " -lbz2"


recipe = LibBz2Recipe()
29 changes: 29 additions & 0 deletions pythonforandroid/recipes/libbz2/lib_android.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
Set default compiler to `clang` and disable versioned shared library
--- bzip2-1.0.8/Makefile-libbz2_so.orig 2019-07-13 19:50:05.000000000 +0200
+++ bzip2-1.0.8/Makefile-libbz2_so 2020-03-13 20:10:32.336990786 +0100
@@ -22,7 +22,7 @@


SHELL=/bin/sh
-CC=gcc
+CC=clang
BIGFILES=-D_FILE_OFFSET_BITS=64
CFLAGS=-fpic -fPIC -Wall -Winline -O2 -g $(BIGFILES)

@@ -35,13 +35,11 @@ OBJS= blocksort.o \
bzlib.o

all: $(OBJS)
- $(CC) -shared -Wl,-soname -Wl,libbz2.so.1.0 -o libbz2.so.1.0.8 $(OBJS)
- $(CC) $(CFLAGS) -o bzip2-shared bzip2.c libbz2.so.1.0.8
- rm -f libbz2.so.1.0
- ln -s libbz2.so.1.0.8 libbz2.so.1.0
+ $(CC) -shared -Wl,-soname=libbz2.so -o libbz2.so $(OBJS)
+ $(CC) $(CFLAGS) -o bzip2-shared bzip2.c libbz2.so

clean:
- rm -f $(OBJS) bzip2.o libbz2.so.1.0.8 libbz2.so.1.0 bzip2-shared
+ rm -f $(OBJS) bzip2.o libbz2.so bzip2-shared

blocksort.o: blocksort.c
$(CC) $(CFLAGS) -c blocksort.c
71 changes: 71 additions & 0 deletions pythonforandroid/recipes/liblzma/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
from os.path import exists, join
from multiprocessing import cpu_count

from pythonforandroid.archs import Arch
from pythonforandroid.recipe import Recipe
from pythonforandroid.logger import shprint
from pythonforandroid.util import current_directory, ensure_dir
import sh


class LibLzmaRecipe(Recipe):

version = '5.2.4'
url = 'https://tukaani.org/xz/xz-{version}.tar.gz'
built_libraries = {'liblzma.so': 'install/lib'}

def build_arch(self, arch: Arch) -> None:
env = self.get_recipe_env(arch)
install_dir = join(self.get_build_dir(arch.arch), 'install')
with current_directory(self.get_build_dir(arch.arch)):
if not exists('configure'):
shprint(sh.Command('./autogen.sh'), _env=env)
shprint(sh.Command('autoreconf'), '-vif', _env=env)
shprint(sh.Command('./configure'),
'--host=' + arch.command_prefix,
'--prefix=' + install_dir,
'--disable-builddir',
'--disable-static',
'--enable-shared',

'--disable-xz',
'--disable-xzdec',
'--disable-lzmadec',
'--disable-lzmainfo',
'--disable-scripts',
'--disable-doc',

_env=env)
shprint(
sh.make, '-j', str(cpu_count()),
_env=env
)

ensure_dir('install')
shprint(sh.make, 'install', _env=env)

def get_library_includes(self, arch: Arch) -> str:
"""Returns a string with the appropriate `-I<lib directory>` to link
with the lzma lib. This string is usually added to the environment
variable `CPPFLAGS`"""
return " -I" + join(
self.get_build_dir(arch.arch), 'install', 'include',
)

def get_library_ldflags(self, arch: Arch) -> str:
"""Returns a string with the appropriate `-L<lib directory>` to link
with the lzma lib. This string is usually added to the environment
variable `LDFLAGS`"""
return " -L" + join(
self.get_build_dir(arch.arch), self.built_libraries['liblzma.so'],
)

@staticmethod
def get_library_libs_flag() -> str:
"""Returns a string with the appropriate `-l<lib>` flags to link with
the lzma lib. This string is usually added to the environment
variable `LIBS`"""
return " -llzma"


recipe = LibLzmaRecipe()
Loading