lxml recipe #1428

goffi-contrib · 2018-10-23T18:35:17Z

This PR fixes the small issues found in #1166.
It's a working recipe for lxml and its dependencies.

AndreMiras

Thanks for the contribution, this is a good start.
Can we start with the following:

fix all the linter issues, try make tox to list them
use git rebasing and make a single clear commit

Here are the few errors listed by the linter:

pep8 runtests: commands[0] | flake8 pythonforandroid/ tests/ ci/
pythonforandroid/recipes/libxml2/__init__.py:5:1: E302 expected 2 blank lines, found 1
pythonforandroid/recipes/libxml2/__init__.py:20:47: E241 multiple spaces after ','
pythonforandroid/recipes/libxml2/__init__.py:21:41: E241 multiple spaces after ','
pythonforandroid/recipes/libxml2/__init__.py:21:83: E241 multiple spaces after ','
pythonforandroid/recipes/libxml2/__init__.py:21:103: E241 multiple spaces after ','
pythonforandroid/recipes/libxml2/__init__.py:30:5: E303 too many blank lines (2)
pythonforandroid/recipes/libxml2/__init__.py:37:1: E305 expected 2 blank lines after class or function definition, found 1
pythonforandroid/recipes/levenshtein/__init__.py:4:1: E302 expected 2 blank lines, found 1
pythonforandroid/recipes/levenshtein/__init__.py:5:9: E225 missing whitespace around operator
pythonforandroid/recipes/levenshtein/__init__.py:14:26: F821 undefined name 'Recipe'
pythonforandroid/recipes/levenshtein/__init__.py:15:26: F821 undefined name 'Recipe'
pythonforandroid/recipes/levenshtein/__init__.py:21:1: E305 expected 2 blank lines after class or function definition, found 1
pythonforandroid/recipes/lxml/__init__.py:5:1: E302 expected 2 blank lines, found 1
pythonforandroid/recipes/lxml/__init__.py:11:45: E261 at least two spaces before inline comment
pythonforandroid/recipes/lxml/__init__.py:32:1: E305 expected 2 blank lines after class or function definition, found 1
pythonforandroid/recipes/libxslt/__init__.py:5:1: E302 expected 2 blank lines, found 1
pythonforandroid/recipes/libxslt/__init__.py:35:5: E303 too many blank lines (2)
pythonforandroid/recipes/libxslt/__init__.py:44:1: E305 expected 2 blank lines after class or function definition, found 1

See more on Travis: https://travis-ci.org/kivy/python-for-android/builds/445274361

Feel free to talk to us on Discord #dev channel so we can go through this together.

AndreMiras · 2018-10-23T18:49:52Z

pythonforandroid/recipes/levenshtein/__init__.py

+    def get_recipe_env(self, arch):
+        env = super(LevenshteinRecipe, self).get_recipe_env(arch)
+        libxslt_recipe = Recipe.get_recipe('libxslt', self.ctx)
+        libxml2_recipe = Recipe.get_recipe('libxml2', self.ctx)


I'm not sure how can these be working as Recipe is undefined 🤔

goffi-contrib · 2018-10-23T19:37:57Z

Hello @AndreMiras thanks for feedback. This PR is just a small correction over #1166, and I've kept the original commits. The original author is @Scefing, I'll wait to see if she is still interested in this patch, else I'll clean it up myself.

goffi-contrib · 2018-10-24T17:10:11Z

@ZachGoldberg @Scefing just pinging you to see if you wanna continue on this recipe :)

goffi-contrib · 2018-10-25T07:04:54Z

@AndreMiras I've updated the PR, as @ZachGoldberg and @Scefing don't look active anymore.
I've squashed initial commits in a first one, then a second one with my fixes. I've kept separate commits because of different authors.

About the undefined Recipe you have found in levenshtein recipe, I've fixed it. But I'm not sure what this recipe is used for, as it's not linked in lxml recipe or any dependency.

Anyway, this is hopefully OK now, I've tested it working OK 2 days ago. Thanks.

AndreMiras · 2018-10-25T07:23:10Z

Nice work @goffi-contrib !
As for levenshtein recipe if none of us know what's for, I would suggest to discard it then. It seems very specific and unmaintained since a while now. Last binary upload on pypi is from 2014. Last GitHub activity 2016. Also it links to python3 which doesn't yet exist (probably Travis will complain about it).

goffi-contrib · 2018-10-25T07:35:03Z

@AndreMiras yes I agree. I'll remove it tonight (CET), I've started my working day now :)

AndreMiras · 2018-10-25T07:44:38Z

Great, can't wait for it 👍
And the levenshtein recipe is indeed failing, but straight from the import:

Traceback (most recent call last):
  File "setup_testapp_python3.py", line 30, in <module>
    package_data={'testapp': ['*.py', '*.png']}
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/app/pythonforandroid/bdistapk.py", line 81, in run
    main()
  File "/app/pythonforandroid/toolchain.py", line 999, in main
    ToolchainCL()
  File "/app/pythonforandroid/toolchain.py", line 532, in __init__
    getattr(self, args.subparser_name.replace('-', '_'))(args)
  File "/app/pythonforandroid/toolchain.py", line 145, in wrapper_func
    build_dist_from_args(ctx, dist, args)
  File "/app/pythonforandroid/toolchain.py", line 166, in build_dist_from_args
    = get_recipe_order_and_bootstrap(ctx, dist.recipes, bs)
  File "/app/pythonforandroid/graph.py", line 112, in get_recipe_order_and_bootstrap
    name, ctx, orders=new_possible_orders)
  File "/app/pythonforandroid/graph.py", line 35, in recursively_collect_orders
    recipe = Recipe.get_recipe(name, ctx)
  File "/app/pythonforandroid/recipe.py", line 614, in get_recipe
    mod = import_recipe('pythonforandroid.recipes.{}'.format(name), recipe_file)
  File "/app/pythonforandroid/recipes/levenshtein/__init__.py", line 1, in <module>
    from pythonforandroid.toolchain import CompiledComponentsPythonRecipe
ImportError: cannot import name CompiledComponentsPythonRecipe

Full log: https://api.travis-ci.org/v3/job/446032065/log.txt

goffi-contrib · 2018-10-26T08:39:16Z

@AndreMiras I've removed levenshtein recipe, the CI is failing because of name resolution issue with gradle (maybe the server was down during the test?). We would need to restart it, but I'm not sure how I can do (I can still do a minor change to PR, but it would be better to just relaunch travis).

goffi-contrib · 2018-10-26T08:44:12Z

I've closed/reopened PR to trigger the travis build

AndreMiras · 2018-10-26T08:45:53Z

pythonforandroid/recipes/lxml/__init__.py

+class LXMLRecipe(CompiledComponentsPythonRecipe):
+    version = "3.6.0"
+    url = "https://pypi.python.org/packages/source/l/lxml/lxml-{version}.tar.gz"
+    depends = ["python2", "libxml2", "libxslt"]


So you have only tested it under python2?

yes, my project is not yet Python 3 compatible.

OK so it means, I either need to test it by hand, or try to make conditional builds also working with python2 😄

AndreMiras · 2018-10-26T08:52:05Z

Yes Travis is flaky often because of the network. But I usually restart the failing jobs myself to check their outcome.
The build I was most interested in is failing for different reasons however: https://api.travis-ci.org/v3/job/446531236/log.txt
Most interesting part of the log is here for records https://pastebin.com/KKE0b6jq
Basically the run command was:

  RAN: /app/venv/bin/python setup_testapp_python3.py apk --sdk-dir /opt/android/android-sdk --ndk-dir /opt/android/crystax-ndk --bootstrap sdl2 --requirements libxml2,lxml,python3crystax,libxslt

Also see:

('recipes modified:', set([u'libxml2', u'lxml', u'libxslt']))
('recipes to build:', set([u'libxml2', u'lxml', u'libxslt']))
('requirements:', u'libxml2,lxml,python3crystax,libxslt')

However your recipe seems to be python2 only which conditional builds doesn't support currently, see https://github.com/kivy/python-for-android/blob/b6ca832/ci/rebuild_updated_recipes.py
Do you know if your recipe could be working under python3crystax?

goffi-contrib · 2018-10-26T08:58:52Z

@AndreMiras I don't know about python3crystax, as I've said earlier, I've just fixed recipe from an other PR which is inactive now. I don't see why it would not work, but it would take too much time for me to try on python 3 now, I haven't python 3 app to test and I'm working in parallel on other tasks.

AndreMiras · 2018-10-26T09:04:30Z

No problem 👍
Just that I need to take a deeper look since some part of the CI would not be of any help.
So that will just require a bit more time

goffi-contrib · 2018-10-26T09:17:03Z

@AndreMiras most test are passing, beside the one with python3crystax. The other failing one is due to docker and smells like an other network issue (it can't find p4a image).
I'll try to add python3crystax in depends to see where it goes

AndreMiras · 2018-10-26T09:25:00Z

Yes other tests are passing because they try to compile core recipes and you didn't touch core recipes.
That's why I'm mainly interested in the conditional build because it's the one that tries to build only the recipes that got touched.
But since it's failing I'll need to test your pull request manually to make sure things are OK

goffi-contrib · 2018-10-26T09:31:02Z

Yes other tests are passing because they try to compile core recipes and you didn't touch core recipes.
That's why I'm mainly interested in the conditional build because it's the one that tries to build only the recipes that got touched.
But since it's failing I'll need to test your pull request manually to make sure things are OK

OK, I don't touch the PR for now then.

Falls back to python2 when tested recipes are python3crystax compatible. This is for instance the case on kivy#1428 where the recipes depend only on python2, hence couldn't be automatically tested. Also updated `vispy` recipe to handle the version string properly, pin it and clean comments. This recipe is a demo case scenario for the conditional build falling back on python2 since `vispy` is not yet compatible with python3crystax.

AndreMiras · 2018-10-29T20:28:53Z

You though I already forgot you? Nope 😄
I hate exhausting my laptop trying to compile ton of recipes so I made Travis deal with it for us 😉
In #1435 we're now handling python2 target (as a fallback).
Once/if it gets merged I can merge your PR. Well I will kinda give it a rebase/retry on my fork first.

goffi-contrib · 2018-10-30T06:35:21Z

You though I already forgot you? Nope smile
I hate exhausting my laptop trying to compile ton of recipes so I made Travis deal with it for us wink
In #1435 we're now handling python2 target (as a fallback).
Once/if it gets merged I can merge your PR. Well I will kinda give it a rebase/retry on my fork first.

he he, thanks for the update :)

Falls back to python2 when tested recipes are python3crystax compatible. This is for instance the case on kivy#1428 where the recipes depend only on python2, hence couldn't be automatically tested. Also updated `vispy` recipe to handle the version string properly, pin it and clean comments. This recipe is a demo case scenario for the conditional build falling back on python2 since `vispy` is not yet compatible with python3crystax. Last, removes the debug flag from the test apps used by the CI as it produces too many logs and halts Travis.

AndreMiras · 2018-11-04T00:23:31Z

Hi @goffi-contrib since #1435 got merged, do you mind rebasing your changes to master and push again so the CI picks it up? Otherwise I can give it a try locally, but it requires more effort 😛

This is a squash of many commits comming from kivy#1166 work by Zachary Goldberg (https://github.com/ZachGoldberg) and Elena Pereira (https://github.com/Scefing)

This patch fixes the recipes from PR kivy#1166. levenshtein recipe has been removed has it's not used anywhere and its causing compilation troubles.

goffi-contrib · 2018-11-04T17:23:34Z

Sure @AndreMiras , I've just done it.

AndreMiras · 2018-11-04T21:51:05Z

Great, seems like Travis is happy 👍
Thanks!

goffi-contrib mentioned this pull request Oct 23, 2018

Add lxml recipe and other recipes from ZachGoldberg fork #1166

Closed

AndreMiras requested changes Oct 23, 2018

View reviewed changes

goffi-contrib force-pushed the lxml_fix branch from 4fd0a6d to ca1d25a Compare October 25, 2018 07:00

goffi-contrib force-pushed the lxml_fix branch from ca1d25a to 5286676 Compare October 25, 2018 07:08

goffi-contrib force-pushed the lxml_fix branch from 5286676 to a311633 Compare October 26, 2018 06:21

goffi-contrib closed this Oct 26, 2018

goffi-contrib reopened this Oct 26, 2018

AndreMiras reviewed Oct 26, 2018

View reviewed changes

AndreMiras mentioned this pull request Oct 29, 2018

Conditional builds falls back on Python2 if needed #1435

Merged

ZachGoldberg and others added 2 commits November 4, 2018 18:22

lxml recipe

8fae315

This is a squash of many commits comming from kivy#1166 work by Zachary Goldberg (https://github.com/ZachGoldberg) and Elena Pereira (https://github.com/Scefing)

[lxml] recipe fix

d36cd9d

This patch fixes the recipes from PR kivy#1166. levenshtein recipe has been removed has it's not used anywhere and its causing compilation troubles.

goffi-contrib force-pushed the lxml_fix branch from a311633 to d36cd9d Compare November 4, 2018 17:22

AndreMiras approved these changes Nov 4, 2018

View reviewed changes

AndreMiras merged commit 13638ff into kivy:master Nov 4, 2018

This was referenced Nov 5, 2018

recipe for libxml2 #999

Closed

lxml recipe doesn't work with Python 3 #1445

Closed

lxml recipe #1428

lxml recipe #1428

Uh oh!

Conversation

goffi-contrib commented Oct 23, 2018

Uh oh!

AndreMiras left a comment

Choose a reason for hiding this comment

Uh oh!

AndreMiras Oct 23, 2018

Choose a reason for hiding this comment

Uh oh!

goffi-contrib commented Oct 23, 2018

Uh oh!

goffi-contrib commented Oct 24, 2018

Uh oh!

goffi-contrib commented Oct 25, 2018

Uh oh!

AndreMiras commented Oct 25, 2018

Uh oh!

goffi-contrib commented Oct 25, 2018

Uh oh!

AndreMiras commented Oct 25, 2018

Uh oh!

goffi-contrib commented Oct 26, 2018

Uh oh!

goffi-contrib commented Oct 26, 2018

Uh oh!

AndreMiras Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

goffi-contrib Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

AndreMiras Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

AndreMiras commented Oct 26, 2018

Uh oh!

goffi-contrib commented Oct 26, 2018

Uh oh!

AndreMiras commented Oct 26, 2018

Uh oh!

goffi-contrib commented Oct 26, 2018

Uh oh!

AndreMiras commented Oct 26, 2018

Uh oh!

goffi-contrib commented Oct 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndreMiras commented Oct 29, 2018

Uh oh!

goffi-contrib commented Oct 30, 2018

Uh oh!

AndreMiras commented Nov 4, 2018

Uh oh!

goffi-contrib commented Nov 4, 2018

Uh oh!

AndreMiras commented Nov 4, 2018

Uh oh!

Uh oh!

goffi-contrib commented Oct 26, 2018 •

edited

Loading