Implement PEP 380 - 'yield from' (closes #11682)

Author: ncoghlan

Doc/library/dis.rst

@@ -431,6 +431,13 @@ the stack so that it is available for further iterations of the loop.
    Pops ``TOS`` and yields it from a :term:`generator`.
 
 
+.. opcode:: YIELD_FROM
+
+   Pops ``TOS`` and delegates to it as a subiterator from a :term:`generator`.
+
+   .. versionadded:: 3.3
+
+
 .. opcode:: IMPORT_STAR
 
    Loads all symbols not starting with ``'_'`` directly from the module TOS to the

Doc/library/exceptions.rst

@@ -250,7 +250,16 @@ The following exceptions are the exceptions that are usually raised.
 .. exception:: StopIteration
 
    Raised by built-in function :func:`next` and an :term:`iterator`\'s
-   :meth:`__next__` method to signal that there are no further values.
+   :meth:`__next__` method to signal that there are no further items to be
+   produced by the iterator.
+
+   The exception object has a single attribute :attr:`value`, which is
+   given as an argument when constructing the exception, and defaults
+   to :const:`None`.
+
+   When a generator function returns, a new :exc:`StopIteration` instance is
+   raised, and the value returned by the function is used as the
+   :attr:`value` parameter to the constructor of the exception.
 
 
 .. exception:: SyntaxError

Doc/reference/expressions.rst

@@ -318,7 +318,7 @@ Yield expressions
 
 .. productionlist::
    yield_atom: "(" `yield_expression` ")"
-   yield_expression: "yield" [`expression_list`]
+   yield_expression: "yield" [`expression_list` | "from" `expression`]
 
 The :keyword:`yield` expression is only used when defining a generator function,
 and can only be used in the body of a function definition.  Using a
@@ -336,7 +336,10 @@ the internal evaluation stack.  When the execution is resumed by calling one of
 the generator's methods, the function can proceed exactly as if the
 :keyword:`yield` expression was just another external call.  The value of the
 :keyword:`yield` expression after resuming depends on the method which resumed
-the execution.
+the execution. If :meth:`__next__` is used (typically via either a
+:keyword:`for` or the :func:`next` builtin) then the result is :const:`None`,
+otherwise, if :meth:`send` is used, then the result will be the value passed
+in to that method.
 
 .. index:: single: coroutine
 
@@ -346,12 +349,29 @@ suspended.  The only difference is that a generator function cannot control
 where should the execution continue after it yields; the control is always
 transferred to the generator's caller.
 
-The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a
+:keyword:`yield` expressions are allowed in the :keyword:`try` clause of a
 :keyword:`try` ...  :keyword:`finally` construct.  If the generator is not
 resumed before it is finalized (by reaching a zero reference count or by being
 garbage collected), the generator-iterator's :meth:`close` method will be
 called, allowing any pending :keyword:`finally` clauses to execute.
 
+When ``yield from expression`` is used, it treats the supplied expression as
+a subiterator. All values produced by that subiterator are passed directly
+to the caller of the current generator's methods. Any values passed in with
+:meth:`send` and any exceptions passed in with :meth:`throw` are passed to
+the underlying iterator if it has the appropriate methods. If this is not the
+case, then :meth:`send` will raise :exc:`AttributeError` or :exc:`TypeError`,
+while :meth:`throw` will just raise the passed in exception immediately.
+
+When the underlying iterator is complete, the :attr:`~StopIteration.value`
+attribute of the raised :exc:`StopIteration` instance becomes the value of
+the yield expression. It can be either set explicitly when raising
+:exc:`StopIteration`, or automatically when the sub-iterator is a generator
+(by returning a value from the sub-generator).
+
+The parentheses can be omitted when the :keyword:`yield` expression is the
+sole expression on the right hand side of an assignment statement.
+
 .. index:: object: generator
 
 The following generator's methods can be used to control the execution of a
@@ -444,6 +464,10 @@ generator functions::
       The proposal to enhance the API and syntax of generators, making them
       usable as simple coroutines.
 
+   :pep:`0380` - Syntax for Delegating to a Subgenerator
+      The proposal to introduce the :token:`yield_from` syntax, making delegation
+      to sub-generators easy.
+
 
 .. _primaries:
 

Doc/reference/simple_stmts.rst

@@ -425,10 +425,10 @@ When :keyword:`return` passes control out of a :keyword:`try` statement with a
 :keyword:`finally` clause, that :keyword:`finally` clause is executed before
 really leaving the function.
 
-In a generator function, the :keyword:`return` statement is not allowed to
-include an :token:`expression_list`.  In that context, a bare :keyword:`return`
-indicates that the generator is done and will cause :exc:`StopIteration` to be
-raised.
+In a generator function, the :keyword:`return` statement indicates that the
+generator is done and will cause :exc:`StopIteration` to be raised. The returned
+value (if any) is used as an argument to construct :exc:`StopIteration` and
+becomes the :attr:`StopIteration.value` attribute.
 
 
 .. _yield:
@@ -450,6 +450,7 @@ The :keyword:`yield` statement is only used when defining a generator function,
 and is only used in the body of the generator function. Using a :keyword:`yield`
 statement in a function definition is sufficient to cause that definition to
 create a generator function instead of a normal function.
+
 When a generator function is called, it returns an iterator known as a generator
 iterator, or more commonly, a generator.  The body of the generator function is
 executed by calling the :func:`next` function on the generator repeatedly until
@@ -469,14 +470,25 @@ resumed before it is finalized (by reaching a zero reference count or by being
 garbage collected), the generator-iterator's :meth:`close` method will be
 called, allowing any pending :keyword:`finally` clauses to execute.
 
+When ``yield from expression`` is used, it treats the supplied expression as
+a subiterator, producing values from it until the underlying iterator is
+exhausted.
+
+For full details of :keyword:`yield` semantics, refer to the :ref:`yieldexpr`
+section.
+
 .. seealso::
 
    :pep:`0255` - Simple Generators
       The proposal for adding generators and the :keyword:`yield` statement to Python.
 
    :pep:`0342` - Coroutines via Enhanced Generators
-      The proposal that, among other generator enhancements, proposed allowing
-      :keyword:`yield` to appear inside a :keyword:`try` ... :keyword:`finally` block.
+      The proposal to enhance the API and syntax of generators, making them
+      usable as simple coroutines.
+
+   :pep:`0380` - Syntax for Delegating to a Subgenerator
+      The proposal to introduce the :token:`yield_from` syntax, making delegation
+      to sub-generators easy.
 
 
 .. _raise:

Doc/whatsnew/3.3.rst

@@ -195,6 +195,22 @@ inspection of exception attributes::
         print("You are not allowed to read document.txt")
 
 
+PEP 380: Syntax for Delegating to a Subgenerator
+================================================
+
+PEP 380 adds the ``yield from`` expression, allowing a generator to delegate
+part of its operations to another generator. This allows a section of code
+containing 'yield' to be factored out and placed in another generator.
+Additionally, the subgenerator is allowed to return with a value, and the
+value is made available to the delegating generator.
+While designed primarily for use in delegating to a subgenerator, the ``yield
+from`` expression actually allows delegation to arbitrary subiterators.
+
+(Implementation by Greg Ewing, integrated into 3.3 by Renaud Blanch, Ryan
+Kelly and Nick Coghlan, documentation by Zbigniew Jędrzejewski-Szmek and
+Nick Coghlan)
+
+
 PEP 3155: Qualified name for classes and functions
 ==================================================
 
@@ -208,7 +224,6 @@ it provides better information about where they were actually defined, and
 how they might be accessible from the global scope.
 
 Example with (non-bound) methods::
-
    >>> class C:
    ...     def meth(self):
    ...         pass

Grammar/Grammar

@@ -121,12 +121,13 @@ arglist: (argument ',')* (argument [',']
                          |'**' test)
 # The reason that keywords are test nodes instead of NAME is that using NAME
 # results in an ambiguity. ast.c makes sure it's a NAME.
-argument: test [comp_for] | test '=' test  # Really [keyword '='] test
+argument: (test) [comp_for] | test '=' test  # Really [keyword '='] test
 comp_iter: comp_for | comp_if
 comp_for: 'for' exprlist 'in' or_test [comp_iter]
 comp_if: 'if' test_nocond [comp_iter]
 
 # not used in grammar, but may appear in "node" passed from Parser to Compiler
 encoding_decl: NAME
 
-yield_expr: 'yield' [testlist]
+yield_expr: 'yield' [yield_arg]
+yield_arg: 'from' test | testlist

Include/Python-ast.h

@@ -245,6 +245,7 @@ struct _expr {
                 } GeneratorExp;
 
                 struct {
+                        int is_from;
                         expr_ty value;
                 } Yield;
 
@@ -487,8 +488,9 @@ expr_ty _Py_DictComp(expr_ty key, expr_ty value, asdl_seq * generators, int
 #define GeneratorExp(a0, a1, a2, a3, a4) _Py_GeneratorExp(a0, a1, a2, a3, a4)
 expr_ty _Py_GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int
                          col_offset, PyArena *arena);
-#define Yield(a0, a1, a2, a3) _Py_Yield(a0, a1, a2, a3)
-expr_ty _Py_Yield(expr_ty value, int lineno, int col_offset, PyArena *arena);
+#define Yield(a0, a1, a2, a3, a4) _Py_Yield(a0, a1, a2, a3, a4)
+expr_ty _Py_Yield(int is_from, expr_ty value, int lineno, int col_offset,
+                  PyArena *arena);
 #define Compare(a0, a1, a2, a3, a4, a5) _Py_Compare(a0, a1, a2, a3, a4, a5)
 expr_ty _Py_Compare(expr_ty left, asdl_int_seq * ops, asdl_seq * comparators,
                     int lineno, int col_offset, PyArena *arena);

Include/frameobject.h

@@ -9,45 +9,46 @@ extern "C" {
 #endif
 
 typedef struct {
-    int b_type;			/* what kind of block this is */
-    int b_handler;		/* where to jump to find handler */
-    int b_level;		/* value stack level to pop to */
+    int b_type;                 /* what kind of block this is */
+    int b_handler;              /* where to jump to find handler */
+    int b_level;                /* value stack level to pop to */
 } PyTryBlock;
 
 typedef struct _frame {
     PyObject_VAR_HEAD
-    struct _frame *f_back;	/* previous frame, or NULL */
-    PyCodeObject *f_code;	/* code segment */
-    PyObject *f_builtins;	/* builtin symbol table (PyDictObject) */
-    PyObject *f_globals;	/* global symbol table (PyDictObject) */
-    PyObject *f_locals;		/* local symbol table (any mapping) */
-    PyObject **f_valuestack;	/* points after the last local */
+    struct _frame *f_back;      /* previous frame, or NULL */
+    PyCodeObject *f_code;       /* code segment */
+    PyObject *f_builtins;       /* builtin symbol table (PyDictObject) */
+    PyObject *f_globals;        /* global symbol table (PyDictObject) */
+    PyObject *f_locals;         /* local symbol table (any mapping) */
+    PyObject **f_valuestack;    /* points after the last local */
     /* Next free slot in f_valuestack.  Frame creation sets to f_valuestack.
        Frame evaluation usually NULLs it, but a frame that yields sets it
        to the current stack top. */
     PyObject **f_stacktop;
-    PyObject *f_trace;		/* Trace function */
-
-	/* In a generator, we need to be able to swap between the exception
-	   state inside the generator and the exception state of the calling
-	   frame (which shouldn't be impacted when the generator "yields"
-	   from an except handler).
-	   These three fields exist exactly for that, and are unused for
-	   non-generator frames. See the SAVE_EXC_STATE and SWAP_EXC_STATE
-	   macros in ceval.c for details of their use. */
+    PyObject *f_trace;          /* Trace function */
+    PyObject *f_yieldfrom;      /* Iterator being delegated to by yield from */
+
+        /* In a generator, we need to be able to swap between the exception
+           state inside the generator and the exception state of the calling
+           frame (which shouldn't be impacted when the generator "yields"
+           from an except handler).
+           These three fields exist exactly for that, and are unused for
+           non-generator frames. See the SAVE_EXC_STATE and SWAP_EXC_STATE
+           macros in ceval.c for details of their use. */
     PyObject *f_exc_type, *f_exc_value, *f_exc_traceback;
 
     PyThreadState *f_tstate;
-    int f_lasti;		/* Last instruction if called */
+    int f_lasti;                /* Last instruction if called */
     /* Call PyFrame_GetLineNumber() instead of reading this field
        directly.  As of 2.3 f_lineno is only valid when tracing is
        active (i.e. when f_trace is set).  At other times we use
        PyCode_Addr2Line to calculate the line from the current
        bytecode index. */
-    int f_lineno;		/* Current line number */
-    int f_iblock;		/* index in f_blockstack */
+    int f_lineno;               /* Current line number */
+    int f_iblock;               /* index in f_blockstack */
     PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */
-    PyObject *f_localsplus[1];	/* locals+stack, dynamically sized */
+    PyObject *f_localsplus[1];  /* locals+stack, dynamically sized */
 } PyFrameObject;
 
 

Include/genobject.h

@@ -11,20 +11,20 @@ extern "C" {
 struct _frame; /* Avoid including frameobject.h */
 
 typedef struct {
-	PyObject_HEAD
-	/* The gi_ prefix is intended to remind of generator-iterator. */
+        PyObject_HEAD
+        /* The gi_ prefix is intended to remind of generator-iterator. */
 
-	/* Note: gi_frame can be NULL if the generator is "finished" */
-	struct _frame *gi_frame;
+        /* Note: gi_frame can be NULL if the generator is "finished" */
+        struct _frame *gi_frame;
 
-	/* True if generator is being executed. */
-	int gi_running;
+        /* True if generator is being executed. */
+        int gi_running;
 
-	/* The code object backing the generator */
-	PyObject *gi_code;
+        /* The code object backing the generator */
+        PyObject *gi_code;
 
-	/* List of weak reference. */
-	PyObject *gi_weakreflist;
+        /* List of weak reference. */
+        PyObject *gi_weakreflist;
 } PyGenObject;
 
 PyAPI_DATA(PyTypeObject) PyGen_Type;
@@ -34,6 +34,7 @@ PyAPI_DATA(PyTypeObject) PyGen_Type;
 
 PyAPI_FUNC(PyObject *) PyGen_New(struct _frame *);
 PyAPI_FUNC(int) PyGen_NeedsFinalizing(PyGenObject *);
+PyAPI_FUNC(int) PyGen_FetchStopIterationValue(PyObject **);
 
 #ifdef __cplusplus
 }

Include/graminit.h

@@ -81,3 +81,4 @@
 #define comp_if 334
 #define encoding_decl 335
 #define yield_expr 336
+#define yield_arg 337