Skip to content

Commit 73a7e9b

Browse files
serhiy-storchakancoghlan
authored andcommitted
bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)
The current behaviour of yield expressions inside comprehensions and generator expressions is essentially an accident of implementation - it arises implicitly from the way the compiler handles yield expressions inside nested functions and generators. Since the current behaviour wasn't deliberately designed, and is inherently confusing, we're deprecating it, with no current plans to reintroduce it. Instead, our advice will be to use a named nested generator definition for cases where this behaviour is desired.
1 parent 6a89481 commit 73a7e9b

File tree

6 files changed

+134
-20
lines changed

6 files changed

+134
-20
lines changed

Doc/reference/expressions.rst

Lines changed: 53 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -183,8 +183,21 @@ by considering each of the :keyword:`for` or :keyword:`if` clauses a block,
183183
nesting from left to right, and evaluating the expression to produce an element
184184
each time the innermost block is reached.
185185

186-
Note that the comprehension is executed in a separate scope, so names assigned
187-
to in the target list don't "leak" into the enclosing scope.
186+
However, aside from the iterable expression in the leftmost :keyword:`for` clause,
187+
the comprehension is executed in a separate implicitly nested scope. This ensures
188+
that names assigned to in the target list don't "leak" into the enclosing scope.
189+
190+
The iterable expression in the leftmost :keyword:`for` clause is evaluated
191+
directly in the enclosing scope and then passed as an argument to the implictly
192+
nested scope. Subsequent :keyword:`for` clauses and any filter condition in the
193+
leftmost :keyword:`for` clause cannot be evaluated in the enclosing scope as
194+
they may depend on the values obtained from the leftmost iterable. For example:
195+
``[x*y for x in range(10) for y in range(x, x+10)]``.
196+
197+
To ensure the comprehension always results in a container of the appropriate
198+
type, ``yield`` and ``yield from`` expressions are prohibited in the implicitly
199+
nested scope (in Python 3.7, such expressions emit :exc:`DeprecationWarning`
200+
when compiled, in Python 3.8+ they will emit :exc:`SyntaxError`).
188201

189202
Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for`
190203
clause may be used to iterate over a :term:`asynchronous iterator`.
@@ -198,6 +211,13 @@ or :keyword:`await` expressions it is called an
198211
suspend the execution of the coroutine function in which it appears.
199212
See also :pep:`530`.
200213

214+
.. versionadded:: 3.6
215+
Asynchronous comprehensions were introduced.
216+
217+
.. deprecated:: 3.7
218+
``yield`` and ``yield from`` deprecated in the implicitly nested scope.
219+
220+
201221
.. _lists:
202222

203223
List displays
@@ -316,27 +336,42 @@ brackets or curly braces.
316336

317337
Variables used in the generator expression are evaluated lazily when the
318338
:meth:`~generator.__next__` method is called for the generator object (in the same
319-
fashion as normal generators). However, the leftmost :keyword:`for` clause is
320-
immediately evaluated, so that an error produced by it can be seen before any
321-
other possible error in the code that handles the generator expression.
322-
Subsequent :keyword:`for` clauses cannot be evaluated immediately since they
323-
may depend on the previous :keyword:`for` loop. For example: ``(x*y for x in
324-
range(10) for y in bar(x))``.
339+
fashion as normal generators). However, the iterable expression in the
340+
leftmost :keyword:`for` clause is immediately evaluated, so that an error
341+
produced by it will be emitted at the point where the generator expression
342+
is defined, rather than at the point where the first value is retrieved.
343+
Subsequent :keyword:`for` clauses and any filter condition in the leftmost
344+
:keyword:`for` clause cannot be evaluated in the enclosing scope as they may
345+
depend on the values obtained from the leftmost iterable. For example:
346+
``(x*y for x in range(10) for y in range(x, x+10))``.
325347

326348
The parentheses can be omitted on calls with only one argument. See section
327349
:ref:`calls` for details.
328350

351+
To avoid interfering with the expected operation of the generator expression
352+
itself, ``yield`` and ``yield from`` expressions are prohibited in the
353+
implicitly defined generator (in Python 3.7, such expressions emit
354+
:exc:`DeprecationWarning` when compiled, in Python 3.8+ they will emit
355+
:exc:`SyntaxError`).
356+
329357
If a generator expression contains either :keyword:`async for`
330358
clauses or :keyword:`await` expressions it is called an
331359
:dfn:`asynchronous generator expression`. An asynchronous generator
332360
expression returns a new asynchronous generator object,
333361
which is an asynchronous iterator (see :ref:`async-iterators`).
334362

363+
.. versionadded:: 3.6
364+
Asynchronous generator expressions were introduced.
365+
335366
.. versionchanged:: 3.7
336367
Prior to Python 3.7, asynchronous generator expressions could
337368
only appear in :keyword:`async def` coroutines. Starting
338369
with 3.7, any function can use asynchronous generator expressions.
339370

371+
.. deprecated:: 3.7
372+
``yield`` and ``yield from`` deprecated in the implicitly nested scope.
373+
374+
340375
.. _yieldexpr:
341376

342377
Yield expressions
@@ -364,6 +399,16 @@ coroutine function to be an asynchronous generator. For example::
364399
async def agen(): # defines an asynchronous generator function (PEP 525)
365400
yield 123
366401

402+
Due to their side effects on the containing scope, ``yield`` expressions
403+
are not permitted as part of the implicitly defined scopes used to
404+
implement comprehensions and generator expressions (in Python 3.7, such
405+
expressions emit :exc:`DeprecationWarning` when compiled, in Python 3.8+
406+
they will emit :exc:`SyntaxError`)..
407+
408+
.. deprecated:: 3.7
409+
Yield expressions deprecated in the implicitly nested scopes used to
410+
implement comprehensions and generator expressions.
411+
367412
Generator functions are described below, while asynchronous generator
368413
functions are described separately in section
369414
:ref:`asynchronous-generator-functions`.

Doc/whatsnew/3.7.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -570,6 +570,18 @@ Other CPython Implementation Changes
570570
Deprecated
571571
==========
572572

573+
* Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated
574+
in comprehensions and generator expressions (aside from the iterable expression
575+
in the leftmost :keyword:`for` clause). This ensures that comprehensions
576+
always immediately return a container of the appropriate type (rather than
577+
potentially returning a :term:`generator iterator` object), while generator
578+
expressions won't attempt to interleave their implicit output with the output
579+
from any explicit yield expressions.
580+
581+
In Python 3.7, such expressions emit :exc:`DeprecationWarning` when compiled,
582+
in Python 3.8+ they will emit :exc:`SyntaxError`. (Contributed by Serhiy
583+
Storchaka in :issue:`10544`.)
584+
573585
- Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with
574586
a macro if ``Py_LIMITED_API`` is not set or set to the value between
575587
``0x03050400`` and ``0x03060000`` (not including) or ``0x03060100`` or

Lib/test/test_generators.py

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1830,13 +1830,7 @@ def printsolution(self, x):
18301830
[None]
18311831
18321832
1833-
1834-
An obscene abuse of a yield expression within a generator expression:
1835-
1836-
>>> list((yield 21) for i in range(4))
1837-
[21, None, 21, None, 21, None, 21, None]
1838-
1839-
And a more sane, but still weird usage:
1833+
Yield is allowed only in the outermost iterable in generator expression:
18401834
18411835
>>> def f(): list(i for i in [(yield 26)])
18421836
>>> type(f())
@@ -2106,10 +2100,6 @@ def printsolution(self, x):
21062100
>>> type(f())
21072101
<class 'generator'>
21082102
2109-
>>> def f(): x=(i for i in (yield) if (yield))
2110-
>>> type(f())
2111-
<class 'generator'>
2112-
21132103
>>> def f(d): d[(yield "a")] = d[(yield "b")] = 27
21142104
>>> data = [1,2]
21152105
>>> g = f(data)

Lib/test/test_grammar.py

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -841,6 +841,41 @@ def g(): f((yield from ()), 1)
841841
# Check annotation refleak on SyntaxError
842842
check_syntax_error(self, "def g(a:(yield)): pass")
843843

844+
def test_yield_in_comprehensions(self):
845+
# Check yield in comprehensions
846+
def g(): [x for x in [(yield 1)]]
847+
def g(): [x for x in [(yield from ())]]
848+
849+
def check(code, warntext):
850+
with self.assertWarnsRegex(DeprecationWarning, warntext):
851+
compile(code, '<test string>', 'exec')
852+
import warnings
853+
with warnings.catch_warnings():
854+
warnings.filterwarnings('error', category=DeprecationWarning)
855+
with self.assertRaisesRegex(SyntaxError, warntext):
856+
compile(code, '<test string>', 'exec')
857+
858+
check("def g(): [(yield x) for x in ()]",
859+
"'yield' inside list comprehension")
860+
check("def g(): [x for x in () if not (yield x)]",
861+
"'yield' inside list comprehension")
862+
check("def g(): [y for x in () for y in [(yield x)]]",
863+
"'yield' inside list comprehension")
864+
check("def g(): {(yield x) for x in ()}",
865+
"'yield' inside set comprehension")
866+
check("def g(): {(yield x): x for x in ()}",
867+
"'yield' inside dict comprehension")
868+
check("def g(): {x: (yield x) for x in ()}",
869+
"'yield' inside dict comprehension")
870+
check("def g(): ((yield x) for x in ())",
871+
"'yield' inside generator expression")
872+
check("def g(): [(yield from x) for x in ()]",
873+
"'yield' inside list comprehension")
874+
check("class C: [(yield x) for x in ()]",
875+
"'yield' inside list comprehension")
876+
check("[(yield x) for x in ()]",
877+
"'yield' inside list comprehension")
878+
844879
def test_raise(self):
845880
# 'raise' test [',' test]
846881
try: raise RuntimeError('just testing')
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Yield expressions are now deprecated in comprehensions and generator
2+
expressions. They are still permitted in the definition of the outermost
3+
iterable, as that is evaluated directly in the enclosing scope.

Python/symtable.c

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1734,7 +1734,6 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e,
17341734
e->lineno, e->col_offset)) {
17351735
return 0;
17361736
}
1737-
st->st_cur->ste_generator = is_generator;
17381737
if (outermost->is_async) {
17391738
st->st_cur->ste_coroutine = 1;
17401739
}
@@ -1754,6 +1753,36 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e,
17541753
if (value)
17551754
VISIT(st, expr, value);
17561755
VISIT(st, expr, elt);
1756+
if (st->st_cur->ste_generator) {
1757+
PyObject *msg = PyUnicode_FromString(
1758+
(e->kind == ListComp_kind) ? "'yield' inside list comprehension" :
1759+
(e->kind == SetComp_kind) ? "'yield' inside set comprehension" :
1760+
(e->kind == DictComp_kind) ? "'yield' inside dict comprehension" :
1761+
"'yield' inside generator expression");
1762+
if (msg == NULL) {
1763+
symtable_exit_block(st, (void *)e);
1764+
return 0;
1765+
}
1766+
if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning,
1767+
msg, st->st_filename, st->st_cur->ste_lineno,
1768+
NULL, NULL) == -1)
1769+
{
1770+
if (PyErr_ExceptionMatches(PyExc_DeprecationWarning)) {
1771+
/* Replace the DeprecationWarning exception with a SyntaxError
1772+
to get a more accurate error report */
1773+
PyErr_Clear();
1774+
PyErr_SetObject(PyExc_SyntaxError, msg);
1775+
PyErr_SyntaxLocationObject(st->st_filename,
1776+
st->st_cur->ste_lineno,
1777+
st->st_cur->ste_col_offset);
1778+
}
1779+
Py_DECREF(msg);
1780+
symtable_exit_block(st, (void *)e);
1781+
return 0;
1782+
}
1783+
Py_DECREF(msg);
1784+
}
1785+
st->st_cur->ste_generator |= is_generator;
17571786
return symtable_exit_block(st, (void *)e);
17581787
}
17591788

0 commit comments

Comments
 (0)