PEP 618: Third Draft (#1429)

Lots of improvements, and GvR is now BDFL-Delegate.

Author: brandtbucher

pep-0618.rst

@@ -4,12 +4,13 @@ Version: $Revision$
 Last-Modified: $Date$
 Author: Brandt Bucher <[email protected]>
 Sponsor: Antoine Pitrou <[email protected]>
+BDFL-Delegate: Guido van Rossum <[email protected]>
 Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 01-May-2020
-Python-Version: 3.9
-Post-History: 01-May-2020
+Python-Version: 3.10
+Post-History: 01-May-2020, 10-May-2020
 Resolution:
 
 
@@ -24,14 +25,15 @@ raised if one of the arguments is exhausted before the others.
 Motivation
 ==========
 
-Many Python users find that most of their ``zip`` usage involves
-iterables that *should* be of equal length.  Sometimes this invariant
-is proven true from the context of the surrounding code, but often the
-data being zipped is passed from the caller, sourced separately, or
-generated in some fashion.  In any of these cases, the default
-behavior of ``zip`` means that faulty refactoring or logic errors
-could easily result in silently losing data.  These bugs are not only
-difficult to diagnose, but difficult to even detect at all.
+It is clear from the author's personal experience and a `survey of the
+standard library <examples_>`_ that much (if not most) ``zip`` usage
+involves iterables that *must* be of equal length.  Sometimes this
+invariant is proven true from the context of the surrounding code, but
+often the data being zipped is passed from the caller, sourced
+separately, or generated in some fashion.  In any of these cases, the
+default behavior of ``zip`` means that faulty refactoring or logic
+errors could easily result in silently losing data.  These bugs are
+not only difficult to diagnose, but difficult to even detect at all.
 
 It is easy to come up with simple cases where this could be a problem.
 For example, the following code may work fine when ``items`` is a
@@ -40,16 +42,16 @@ if ``items`` is refactored by the caller to be a consumable iterator::
 
     def apply_calculations(items):
         transformed = transform(items)
-        for x, y in zip(items, transformed):
-            yield something(x, y)
+        for i, t in zip(items, transformed):
+            yield calculate(i, t)
 
 There are several other ways in which ``zip`` is commonly used.
 Idiomatic tricks are especially susceptible, because they are often
 employed by users who lack a complete understanding of how the code
 works.  One example is unpacking into ``zip`` to lazily "unzip" or
 "transpose" nested iterables::
 
-    >>> x = iter([iter([1, 2, 3]), iter(["one" "two" "three"])])
+    >>> x = [[1, 2, 3], ["one" "two" "three"]]
     >>> xt = list(zip(*x))
 
 Another is "chunking" data into equal-sized groups::
@@ -63,19 +65,19 @@ the second case, data with a length that is not a multiple of ``n`` is
 often an error as well.  However, both of these idioms will silently
 omit the tail-end items of malformed input.
 
-Perhaps most convincingly, the current use of ``zip`` in the
-standard-library ``ast`` module has created multiple bugs that
-`silently drop parts of malformed nodes
+Perhaps most convincingly, the use of ``zip`` in the standard-library
+``ast`` module created a bug in ``literal_eval`` which `silently
+dropped parts of malformed nodes
 <https://bugs.python.org/issue40355>`_::
 
     >>> from ast import Constant, Dict, literal_eval
     >>> nasty_dict = Dict(keys=[Constant(None)], values=[])
     >>> literal_eval(nasty_dict)  # Like eval("{None: }")
     {}
 
-In fact, the author has counted dozens of other call sites in Python's
-standard library and tooling where it would be appropriate to enable
-this new feature immediately.
+In fact, the author has `counted dozens of other call sites
+<examples_>`_ in Python's standard library and tooling where it
+would be appropriate to enable this new feature immediately.
 
 
 Rationale
@@ -93,10 +95,6 @@ functions which are typically called with compile-time constants:
 
 Many more exist in the standard library.
 
-A good rule of thumb is that "mode-switches" which change return types
-or significantly alter functionality are indeed an anti-pattern, while
-ones which enable or disable complementary checks or behavior are not.
-
 The idea and name for this new parameter were `originally proposed
 <https://mail.python.org/archives/list/[email protected]/message/6GFUADSQ5JTF7W7OGWF7XF2NH2XUTUQM>`_
 by Ram Rachum.  The thread received over 100 replies, with the
@@ -117,9 +115,6 @@ When the built-in ``zip`` is called with the keyword-only argument
 the arguments are exhausted at differing lengths.  This error will
 occur at the point when iteration would normally stop today.
 
-At most one additional item may be consumed from one of the iterators
-when compared to normal ``zip`` usage.
-
 
 Backward Compatibility
 ======================
@@ -133,7 +128,7 @@ Reference Implementation
 The author has drafted a `C implementation
 <https://github.com/python/cpython/compare/master...brandtbucher:zip-strict>`_.
 
-An approximate pure-Python translation is::
+An approximate Python translation is::
 
     def zip(*iterables, strict=False):
         if not iterables:
@@ -160,38 +155,71 @@ An approximate pure-Python translation is::
 Rejected Ideas
 ==============
 
-Add Additional Flavors Of ``zip`` To ``itertools``
-''''''''''''''''''''''''''''''''''''''''''''''''''
+Add ``itertools.zip_strict``
+----------------------------
+
+This is the alternative with the most support on the Python-Ideas
+mailing list, so it deserves do be discussed in some detail here.  It
+does not have any disqualifying flaws, and could work well enough as a
+substitute if this PEP is rejected.
+
+With that in mind, this section aims to outline why adding an optional
+parameter to ``zip`` is a smaller change that ultimately does a better
+job of solving the problems motivating this PEP.
 
-Adding ``zip_strict`` to itertools is a larger change with greater
-maintenance burden than the simple modification being proposed.
+
+Precedent
+'''''''''
 
 It seems that a great deal of the motivation driving this alternative
 is that ``zip_longest`` already exists in ``itertools``.  However,
-``zip_longest`` is really another beast entirely: it takes on the
-responsibility of filling in missing values, a problem neither of
-the other variants even have.  It also arguably has the most
-specialized behavior of the three (to the point of exposing a new
-``fillvalue`` parameter), so it makes sense that it would live in
-``itertools`` while ``zip`` grows in-place.
-
-Importing a drop-in replacement for a built-in also feels too heavy,
-especially just to check a tricky condition that should "always" be
-true.  The goal here is not just to provide a way to catch bugs, but
-to also make it easy (even tempting) for a user to enable the check
-whenever using ``zip`` at a call site with this property.
+``zip_longest`` is in many ways a much more complicated, specialized
+utility: it takes on the responsibility of filling in missing values,
+a job neither of the other variants needs to concern themselves with.
+
+If both ``zip`` and ``zip_longest`` lived alongside each other in
+``itertools`` or as builtins, then adding ``zip_strict`` in the same
+location would indeed be a much stronger argument.  However, the new
+"strict" variant is conceptually *much* closer to ``zip`` in interface
+and behavior than ``zip_longest``, while still not meeting the high
+bar of being its own builtin.  Given this situation, it seems most
+natural for ``zip`` to grow this new option in-place.
+
+
+Usability
+'''''''''
+
+If ``zip`` is capable of preventing this class of bug, it becomes much
+simpler for users to enable the check at call sites with this
+property.  Compare this with importing a drop-in replacement for a
+built-in utility, which feels somewhat heavy just to check a tricky
+condition that should "always" be true.
 
 Some have also argued that a new function buried in the standard
 library is somehow more "discoverable" than a keyword parameter on the
-built-in itself.  The author does not believe this to be true.
+built-in itself.  The author does not agree with this assessment.
+
+
+Maintenance Cost
+''''''''''''''''
+
+While implementation should only be a secondary concern when making
+usability improvements, it is important to recognize that adding a new
+utility is significantly more complicated than modifying an existing
+one.  The CPython implementation accompanying this PEP is simple and
+has no measurable performance impact on default ``zip`` behavior,
+while adding an entirely new utility to ``itertools`` would require
+either:
 
-Another proposed idiom, per-module shadowing of the built-in ``zip``
-with some subtly different variant from ``itertools``, is an
-anti-pattern that shouldn't be encouraged.
+- Duplicating much of the existing ``zip`` logic, as ``zip_longest``
+  already does.
+- Significantly refactoring either ``zip``, ``zip_longest``, or both
+  to share a common or inherited implementation (which may impact
+  performance).
 
 
 Add Several "Modes" To Switch Between
-'''''''''''''''''''''''''''''''''''''
+-------------------------------------
 
 This option only makes more sense than a binary flag if we anticipate
 having three or more modes. The "obvious" three choices for these
@@ -211,7 +239,7 @@ long-lived namesake utility in ``itertools``.
 
 
 Add A Method Or Alternate Constructor To The ``zip`` Type
-'''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+---------------------------------------------------------
 
 Consider the following two options, which have both been proposed::
 
@@ -235,11 +263,14 @@ nothing (which is the problem we are trying to avoid in the first
 place).
 
 This proposal is further complicated by the fact that CPython's actual
-``zip`` type is an undocumented implementation detail.
+``zip`` type is currently an undocumented implementation detail.  This
+means that choosing one of the above behaviors will effectively "lock
+in" the current implementation (or at least require it to be emulated)
+going forward.
 
 
 Change The Default Behavior Of ``zip``
-''''''''''''''''''''''''''''''''''''''
+--------------------------------------
 
 There is nothing "wrong" with the default behavior of ``zip``, since
 there are many cases where it is indeed the correct way to handle
@@ -251,15 +282,15 @@ the "extra" tail-end data is still needed.
 
 
 Accept A Callback To Handle Remaining Items
-'''''''''''''''''''''''''''''''''''''''''''
+-------------------------------------------
 
 While able to do basically anything a user could need, this solution
 makes handling the more common cases (like rejecting mismatched
 lengths) unnecessarily complicated and non-obvious.
 
 
-Raise An ``AssertionError`` Instead Of A ``ValueError``
-'''''''''''''''''''''''''''''''''''''''''''''''''''''''
+Raise An ``AssertionError``
+---------------------------
 
 There are no built-in functions or types that raise an
 ``AssertionError`` as part of their API.  Further, the `official
@@ -276,7 +307,7 @@ Users desiring a check that is disabled in optimized mode (like an
 
 
 Add A Similar Feature to ``map``
-''''''''''''''''''''''''''''''''
+--------------------------------
 
 This PEP does not propose any changes to ``map``, since the use of
 ``map`` with multiple iterable arguments is quite rare. However, this
@@ -291,7 +322,7 @@ debated here for ``zip``.
 
 
 Do Nothing
-''''''''''
+----------
 
 This option is perhaps the least attractive.
 
@@ -303,6 +334,39 @@ are evidence that it's *very* easy to fall into the sort of trap that
 this feature aims to avoid.
 
 
+References
+==========
+
+Examples
+--------
+
+.. note:: This listing is not exhaustive.
+
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/_pydecimal.py#L3394
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/_pydecimal.py#L3418
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/_pydecimal.py#L3435
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L94-L95
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1184
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1275
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1363
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1391
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/copy.py#L217
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/csv.py#L142
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/dis.py#L462
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/filecmp.py#L142
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/filecmp.py#L143
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/inspect.py#L1440
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/inspect.py#L2095
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/os.py#L510
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/plistlib.py#L577
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/tarfile.py#L1317
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/tarfile.py#L1323
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/tarfile.py#L1339
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/turtle.py#L3015
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/turtle.py#L3071
+- https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/turtle.py#L3901
+
+
 Copyright
 =========