Support for PY37 without contextvars #138

pfreixes · 2018-04-10T04:45:26Z

This MR aligns the uvloop code to become compatible, not production
ready, with PY37 adding the following changes:

Support for the new API used behind the set_debug function.
Adds the context=None KW to those functions that will receive it
when uvloop runs with PY37.
Former asyncio.test_utils moved as an uvloop test resource.

New MR on top of that to enable contextvars should be expected.

This MR aligns the uvloop code to become compatible, not production ready, with PY37 adding the following changes: - Support for the new API used behind the `set_debug` function. - Adds the `context=None` kw to those functions that will receive it when uvloop runs with PY37. - Former asyncio.test_utils moved as an uvloop test resource.

pfreixes · 2018-04-10T04:48:20Z

tests/test_unix.py

@@ -362,6 +363,7 @@ def connection_lost(self, exc):
        self.assertIn(excs[0].__class__,
                      (BrokenPipeError, ConnectionResetError))

+    @unittest.skipUnless(sys.version_info < (3, 7), 'Python version must be < 3.7')


If you have some light about why this testing is failing with 3.7 please tell me.

pfreixes · 2018-04-10T04:51:30Z

uvloop/includes/stdlib.pxi

-# so we delete refs to all modules manually (except sys)
-del asyncio, concurrent, collections, errno
+# so we delete refs to all modules manually (except sys, asyncio)
+del concurrent, collections, errno


Just removed the asyncio to avoid losing the ability to call later the asyncio.coroutines.debug_wrapper when uvloop run on an interpreter different than 3.7.

If you believe that this is a red flag, let's find another way to make the _set_coroutine_debug ready to be executed by different Python versions.

We need to keep del asyncio. Otherwise uvloop would hold refs to Tasks longer than necessary (that's Cython's fault and these dels are a workaround).

You can use getattr:

cdef aio_debug_wrapper = getattr(asyncio.coroutines, 'debug_wrapper', None)

And just check if it's not None before you call it.

pfreixes · 2018-04-10T06:00:38Z

No way, Travis still failing one test gets frozen forever.

My gut feeling says that is because the older 3.7-dev CPython version pointing still to 3.7.0a4+. Tried to run the Travis step using Ubuntu 16.04 that must come with 3.7.0b1 [1] but looks like Travis does not have official and production-ready support for Xenial - WAT! - and keeps installing the Trusty version.

Any ideas?

[1] https://s3.amazonaws.com/travis-python-archives/binaries/ubuntu/16.04/x86_64/python-3.7-dev.tar.bz2

pfreixes · 2018-04-12T03:20:52Z

@1st1 ^^

1st1 · 2018-04-12T03:26:08Z

Yep, I saw this PR yesterday. Will take a look tomorrow. Thanks!

pfreixes · 2018-04-16T04:56:35Z

.ci/travis-install.sh

@@ -22,6 +22,7 @@ if [ "${TRAVIS_OS_NAME}" == "osx" ]; then
    brew outdated automake || brew upgrade automake --with-default-names
 fi

-pip install --upgrade pip wheel
+#pip install --upgrade pip


Getting this error with the Mac environment

You should consider upgrading via the 'pip install --upgrade pip' command. +pip install --upgrade setuptools Traceback (most recent call last): File "/Users/travis/.pyenv/versions/3.6.3/bin/pip", line 7, in <module> from pip import main ImportError: cannot import name 'main'

This is weird. Which version of pip did it upgrade to?

pfreixes · 2018-04-16T05:43:43Z

Finally got a green CI, not really happy with some of the patches. in any case, its something that can be used to start and obviously to get improve it. Feedback welcomed @1st1.

I know that you had some plans to give support for contextvars - the natural continuation after this PR. if you are in a rush with other stuff, I can make it.

1st1 · 2018-04-16T16:21:45Z

.travis.yml

+          sudo: required
+          # Travis has removed the py37 image from their repos
+          #  language: python
+          #  python: "3.7"


They seem to have "3.7-dev". Wouldn't that be enough?

Ideally I'd like to avoid using docker for running our tests, it would slow the CI down way too much.

1st1 · 2018-04-16T16:28:42Z

tests/test_udp.py

@@ -3,7 +3,7 @@
 import unittest
 import sys

-from asyncio import test_utils
+from tests import utils as test_utils


I don't like this "tests" import. Just move/copy the necessary functions to uvloop._testbase.

1st1 · 2018-04-16T16:43:03Z

if you are in a rush with other stuff, I can make it.

I think I'll work on it myself. My idea is to copy _asynciomodule.c to uvloop. Since it's my code I think it shouldn't take me too long to figure out how to do that correctly.

1st1 · 2018-04-16T16:43:41Z

This PR needs to land before I start working on contextvars though.

pfreixes · 2018-04-17T05:51:19Z

@1st1 Changes applied, about the issues with Pip and Python 3.7:

Travis CI uses an older 3.7-dev image, exactly 3.7.0a4+. And the 3.7 is no longer available. Since this breaking change [1] I would say that Python 3.7 is kinda broken for Travis. I hope that the Docker solution will be a temporary one, but right now is the only way to get a functional environment running an interpreter with a version beyond the alpha 4.

The Pip issue started with the recent 10.0.0 release and the Pyenv environments [2] [3]. Take a look to the last commit, just pins the PIP version for the Mac osx environment. I hope so again that this will be just a temporary solution.

[1] travis-ci/travis-ci#9069
[2] pypa/pip#5240
[3] pyenv/pyenv#1141

1st1 · 2018-04-18T18:27:40Z

Travis CI uses an older 3.7-dev image, exactly 3.7.0a4+. And the 3.7 is no longer available. Since this breaking change [1] I would say that Python 3.7 is kinda broken for Travis. I hope that the Docker solution will be a temporary one, but right now is the only way to get a functional environment running an interpreter with a version beyond the alpha 4.

Can you try installing python 3.7 using pyenv? It worked fine for me when we tried it for https://github.com/azure/azure-functions-python-worker/

pfreixes · 2018-05-06T07:12:08Z

@1st1 I didn't manage to reproduce the same issue in local. These have been the environments that ran the tests successfully in local:

Docker image with Ubuntu 14.04 (trusty) with a OpenSSL 1.1 and Python 3.7 b03 from Pyenv,
Docker image with Ubuntu 16.04 (xenial), with Python 3.7 b03 from Pyenv.
Docker image with Ubuntu 16.04 (xenial), with Python 3.7 b04 (used the same one that is used by Travis)

I've tried to reproduce the local environment following this guide but looks like that the Xenial is still experimental and there is no way to download the image used by Travis, or at least I wasn't able to find it. In any case, I've opened an issue to gather some feedback from Travis

Also, I've tried to get some information running the Travis CI, no luck and also the same issue: stuck in the same test test_create_connection_3 (test_tcp.Test_AIO_TCP). Worth mentioning that the xenial environment is quite slower than the trusty one, usually takes 10 minutes more to start! and even sometimes the job finishes while the worker is still compiling the package (timeout?).

So, let's wait for some feedback from the Travis team. But I wouldn't put so many hopes on it, taking into account that for them the Xenial is still experimental and not officially supported by Travis.

pfreixes · 2018-05-06T15:00:29Z

Definitely, something is wrong with Xenial and Travis

wumpus · 2018-05-18T20:15:18Z

I am using pyenv, python 3.7.0b4 on CentOS 7, and installed this branch -- success!

It would be nice for this to get released so more people could test 3.7-dev. I personally also use Travis/Xenial for CI so the remaining bug affects me, but ...

pfreixes · 2018-05-20T15:20:28Z

My proposal here is going for the Docker option that was proven as successful and TBH didn't add so much overhead in terms of time needed to run the tests.

Later if the xenial distro is officially supported by Travis and the tests can be run natively without using Docker its a matter of changing the configuration.

wumpus · 2018-05-20T18:07:35Z

I think we are agreeing that uvloop should not wait for TravisCI.

This MR aligns the uvloop code to become compatible, not production ready, with PY37 adding the following changes: * Support for the new API used behind the set_debug function. * Adds the context=None KW to those functions that will receive it when uvloop runs with PY37. * Former asyncio.test_utils moved as an uvloop test resource. -- Yury: the original PR for this change is #138. I stripped out all the CI bits and squashed all commits by hand. The CI will be addressed later.

1st1 · 2018-05-23T16:50:00Z

Closing this one in favour of #149. The new PR is a version of this one with removed CI tweaks.

This MR aligns the uvloop code to become compatible, not production ready, with PY37 adding the following changes: * Support for the new API used behind the set_debug function. * Adds the context=None KW to those functions that will receive it when uvloop runs with PY37. * Former asyncio.test_utils moved as an uvloop test resource. -- Yury: the original PR for this change is #138. I stripped out all the CI bits and squashed all commits by hand. The CI will be addressed later.

pfreixes · 2018-05-23T19:58:09Z

Thanks!

pfreixes commented Apr 10, 2018

View reviewed changes

pfreixes changed the title ~~Support for PY37 without contexvars~~ Support for PY37 without contextvars Apr 10, 2018

Testing with Ubuntu Xenial for PY37 step, comes with 3.7.0b1+

efc3a9e

pfreixes added 4 commits April 15, 2018 09:04

Run Python 3.7 Travis step using docker

1f01d3f

Dont relay on Travis PY37 environment

0d1dc9c

Fixes an issue testing the environment in the travis file

2b3b1c7

Disable pip upgrade at Travis CI

9ea9e29

pfreixes commented Apr 16, 2018

View reviewed changes

1st1 reviewed Apr 16, 2018

View reviewed changes

pfreixes added 7 commits April 17, 2018 06:10

Added back the explicit del asyncio and other minor stuff

6714d99

Temporary commit to get more info about Python env and Pip env

8421107

Test again the 3.7 Travis Python image

36407ac

Get rid of none functional Travis Python 3.7 configuration

9dfd99d

Do not upgrade pip for Travis CI

5cb302c

Travis CI Pins PIP to 9.X for OSX enviornment

f0375bc

Fixed invalid pined expresion

3a3e4cc

1st1 mentioned this pull request Apr 18, 2018

Removed use of asyncio.coroutines.debug_wrapper #127

Closed

drpoggi mentioned this pull request Apr 18, 2018

Test against 3.7-dev #124

Closed

Test py37 recipe using an external PPA

d1bdc50

pfreixes added 7 commits May 4, 2018 14:16

Test with trusty

8e6184c

Get network interfaces in Travis

8480b46

Python 3.7.0b4 back for xenial wihout pyenv?

8fe70b9

Trigger CI

8646ff4

Trigger CI 2

3bd621f

Run only problematic test

0f36879

Test without debug asyncio

5bdb238

pfreixes added 7 commits May 6, 2018 14:17

Test 3.6 against Xenial

09cffad

Test all Travis matrix

7638a59

Fix use of PIPENV for Mac OSX

8c3b28a

Test with only 3.5 and 3.7-dev

46480da

Only 3.7-dev

5338ae6

Test with PYENV=true PYTHON_VERSION=3.7.0b3

b3a9de1

Test with PYENV=true PYTHON_VERSION=3.7.0b3

7d7ac60

wumpus mentioned this pull request May 18, 2018

AttributeError: module 'asyncio.coroutines' has no attribute 'debug_wrapper' #126

Closed

pfreixes mentioned this pull request May 20, 2018

POC UVloop tracing, support for DNS traces #144

Closed

Test with PYENV=true PYTHON_VERSION=3.7.0b4

f7fc3e1

ghost mentioned this pull request May 21, 2018

Added optional uvloop support. dustinpianalto/Sebi-Machine#4

Closed

sakurai-youhei mentioned this pull request May 22, 2018

Replace asyncio.coroutines.debug_wrapper with CoroWrapper #147

Closed

1st1 mentioned this pull request May 23, 2018

Support for PY37 without contextvars. #149

Merged

1st1 closed this May 23, 2018

Support for PY37 without contextvars #138

Support for PY37 without contextvars #138

Uh oh!

Conversation

pfreixes commented Apr 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pfreixes commented Apr 10, 2018

Uh oh!

pfreixes commented Apr 12, 2018

Uh oh!

1st1 commented Apr 12, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pfreixes commented Apr 16, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1st1 commented Apr 16, 2018

Uh oh!

1st1 commented Apr 16, 2018

Uh oh!

pfreixes commented Apr 17, 2018

Uh oh!

1st1 commented Apr 18, 2018

Uh oh!

pfreixes commented May 6, 2018

Uh oh!

pfreixes commented May 6, 2018

Uh oh!

wumpus commented May 18, 2018

Uh oh!

pfreixes commented May 20, 2018

Uh oh!

wumpus commented May 20, 2018

Uh oh!

1st1 commented May 23, 2018

Uh oh!

pfreixes commented May 23, 2018

Uh oh!

Uh oh!

pfreixes commented Apr 10, 2018 •

edited

Loading