Merge remote-tracking branch 'origin/master' into slotscache

Author: gvanrossum

.azure-pipelines/docs-steps.yml

@@ -12,7 +12,7 @@ steps:
   inputs:
     versionSpec: '>=3.6'
 
-- script: python -m pip install sphinx==3.2.1 blurb python-docs-theme
+- script: python -m pip install sphinx==2.2.0 blurb python-docs-theme
   displayName: 'Install build dependencies'
 
 - ${{ if ne(parameters.latex, 'true') }}:

.github/workflows/build.yml

@@ -63,6 +63,7 @@ jobs:
           # Build Python with the libpython dynamic library
           ./configure --with-pydebug --enable-shared
           make -j4 regen-all
+          make regen-stdlib-module-names
       - name: Check for changes
         run: |
           changes=$(git status --porcelain)

.travis.yml

@@ -172,6 +172,7 @@ before_script:
   - eval "$(pyenv init -)"
   - pyenv global 3.8
   - PYTHON_FOR_REGEN=python3.8 make -j4 regen-all
+  - make regen-stdlib-module-names
   - changes=`git status --porcelain`
   - |
       # Check for changes in regenerated files

Doc/c-api/init.rst

@@ -1192,7 +1192,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 
    .. versionadded:: 3.9
 
-.. c:function:: void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame);
+.. c:function:: void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame)
 
    Set the frame evaluation function.
 

Doc/c-api/memory.rst

@@ -92,6 +92,38 @@ for the I/O buffer escapes completely the Python memory manager.
    statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a
    new pymalloc object arena is created, and on shutdown.
 
+Allocator Domains
+=================
+
+All allocating functions belong to one of three different "domains" (see also
+:c:type`PyMemAllocatorDomain`). These domains represent different allocation
+strategies and are optimized for different purposes. The specific details on
+how every domain allocates memory or what internal functions each domain calls
+is considered an implementation detail, but for debugging purposes a simplified
+table can be found at :ref:`here <default-memory-allocators>`. There is no hard
+requirement to use the memory returned by the allocation functions belonging to
+a given domain for only the purposes hinted by that domain (although this is the
+recommended practice). For example, one could use the memory returned by
+:c:func:`PyMem_RawMalloc` for allocating Python objects or the memory returned
+by :c:func:`PyObject_Malloc` for allocating memory for buffers.
+
+The three allocation domains are:
+
+* Raw domain: intended for allocating memory for general-purpose memory
+  buffers where the allocation *must* go to the system allocator or where the
+  allocator can operate without the :term:`GIL`. The memory is requested directly
+  to the system.
+
+* "Mem" domain: intended for allocating memory for Python buffers and
+  general-purpose memory buffers where the allocation must be performed with
+  the :term:`GIL` held. The memory is taken from the Python private heap.
+
+* Object domain: intended for allocating memory belonging to Python objects. The
+  memory is taken from the Python private heap.
+
+When freeing memory previously allocated by the allocating functions belonging to a
+given domain,the matching specific deallocating functions must be used. For example,
+:c:func:`PyMem_Free` must be used to free memory allocated using :c:func:`PyMem_Malloc`.
 
 Raw Memory Interface
 ====================
@@ -272,6 +304,12 @@ The following function sets, modeled after the ANSI C standard, but specifying
 behavior when requesting zero bytes, are available for allocating and releasing
 memory from the Python heap.
 
+.. note::
+    There is no guarantee that the memory returned by these allocators can be
+    succesfully casted to a Python object when intercepting the allocating
+    functions in this domain by the methods described in
+    the :ref:`Customize Memory Allocators <customize-memory-allocators>` section.
+
 The :ref:`default object allocator <default-memory-allocators>` uses the
 :ref:`pymalloc memory allocator <pymalloc>`.
 
@@ -353,6 +391,7 @@ Legend:
 * ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`
 * "+ debug": with debug hooks installed by :c:func:`PyMem_SetupDebugHooks`
 
+.. _customize-memory-allocators:
 
 Customize Memory Allocators
 ===========================
@@ -601,4 +640,3 @@ heap, objects in Python are allocated and released with :c:func:`PyObject_New`,
 
 These will be explained in the next chapter on defining and implementing new
 object types in C.
-

Doc/conf.py

@@ -237,3 +237,5 @@
 # bpo-40204: Disable warnings on Sphinx 2 syntax of the C domain since the
 # documentation is built with -W (warnings treated as errors).
 c_warn_on_allowed_pre_v3 = False
+
+strip_signature_backslash = True

Doc/howto/clinic.rst

@@ -1206,6 +1206,68 @@ type for ``self``, it's best to create your own converter, subclassing
     [clinic start generated code]*/
 
 
+Using a "defining class" converter
+----------------------------------
+
+Argument Clinic facilitates gaining access to the defining class of a method.
+This is useful for :ref:`heap type <heap-types>` methods that need to fetch
+module level state.  Use :c:func:`PyType_FromModuleAndSpec` to associate a new
+heap type with a module.  You can now use :c:func:`PyType_GetModuleState` on
+the defining class to fetch the module state, for example from a module method.
+
+Example from ``Modules/zlibmodule.c``.  First, ``defining_class`` is added to
+the clinic input::
+
+    /*[clinic input]
+    zlib.Compress.compress
+
+      cls: defining_class
+      data: Py_buffer
+        Binary data to be compressed.
+      /
+
+
+After running the Argument Clinic tool, the following function signature is
+generated::
+
+    /*[clinic start generated code]*/
+    static PyObject *
+    zlib_Compress_compress_impl(compobject *self, PyTypeObject *cls,
+                                Py_buffer *data)
+    /*[clinic end generated code: output=6731b3f0ff357ca6 input=04d00f65ab01d260]*/
+
+
+The following code can now use ``PyType_GetModuleState(cls)`` to fetch the
+module state::
+
+    zlibstate *state = PyType_GetModuleState(cls);
+
+
+Each method may only have one argument using this converter, and it must appear
+after ``self``, or, if ``self`` is not used, as the first argument.  The argument
+will be of type ``PyTypeObject *``.  The argument will not appear in the
+``__text_signature__``.
+
+The ``defining_class`` converter is not compatible with ``__init__`` and ``__new__``
+methods, which cannot use the ``METH_METHOD`` convention.
+
+It is not possible to use ``defining_class`` with slot methods.  In order to
+fetch the module state from such methods, use ``_PyType_GetModuleByDef`` to
+look up the module and then :c:func:`PyModule_GetState` to fetch the module
+state.  Example from the ``setattro`` slot method in
+``Modules/_threadmodule.c``::
+
+    static int
+    local_setattro(localobject *self, PyObject *name, PyObject *v)
+    {
+        PyObject *module = _PyType_GetModuleByDef(Py_TYPE(self), &thread_module);
+        thread_module_state *state = get_thread_state(module);
+        ...
+    }
+
+
+See also :pep:`573`.
+
 
 Writing a custom converter
 --------------------------

Doc/library/asyncio-stream.rst

@@ -185,7 +185,7 @@ StreamReader
       can be read.  Use the :attr:`IncompleteReadError.partial`
       attribute to get the partially read data.
 
-   .. coroutinemethod:: readuntil(separator=b'\n')
+   .. coroutinemethod:: readuntil(separator=b'\\n')
 
       Read data from the stream until *separator* is found.
 

Doc/library/base64.rst

@@ -199,7 +199,7 @@ The modern interface provides:
    .. versionadded:: 3.4
 
 
-.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \t\n\r\v')
+.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \\t\\n\\r\\v')
 
    Decode the Ascii85 encoded :term:`bytes-like object` or ASCII string *b* and
    return the decoded :class:`bytes`.

Doc/library/curses.rst

@@ -220,11 +220,15 @@ The module :mod:`curses` defines the following functions:
    multiple devices, and *x*, *y*, *z* are the event's coordinates.  (*z* is
    currently unused.)  *bstate* is an integer value whose bits will be set to
    indicate the type of event, and will be the bitwise OR of one or more of the
-   following constants, where *n* is the button number from 1 to 4:
+   following constants, where *n* is the button number from 1 to 5:
    :const:`BUTTONn_PRESSED`, :const:`BUTTONn_RELEASED`, :const:`BUTTONn_CLICKED`,
    :const:`BUTTONn_DOUBLE_CLICKED`, :const:`BUTTONn_TRIPLE_CLICKED`,
    :const:`BUTTON_SHIFT`, :const:`BUTTON_CTRL`, :const:`BUTTON_ALT`.
 
+   .. versionchanged:: 3.10
+      The ``BUTTON5_*`` constants are now exposed if they are provided by the
+      underlying curses library.
+
 
 .. function:: getsyx()
 

Doc/library/difflib.rst

@@ -149,7 +149,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
    contains a good example of its use.
 
 
-.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n')
+.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n')
 
    Compare *a* and *b* (lists of strings); return a delta (a :term:`generator`
    generating the delta lines) in context diff format.
@@ -279,7 +279,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
       emu
 
 
-.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n')
+.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n')
 
    Compare *a* and *b* (lists of strings); return a delta (a :term:`generator`
    generating the delta lines) in unified diff format.
@@ -321,7 +321,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
 
    See :ref:`difflib-interface` for a more detailed example.
 
-.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\n')
+.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n')
 
    Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a
    sequence of delta lines (also bytes) in the format returned by *dfunc*.

Doc/library/dis.rst

@@ -742,7 +742,7 @@ iterations of the loop.
 
    This opcode performs several operations before a with block starts.  First,
    it loads :meth:`~object.__exit__` from the context manager and pushes it onto
-   the stack for later use by :opcode:`WITH_CLEANUP_START`.  Then,
+   the stack for later use by :opcode:`WITH_EXCEPT_START`.  Then,
    :meth:`~object.__enter__` is called, and a finally block pointing to *delta*
    is pushed.  Finally, the result of calling the ``__enter__()`` method is pushed onto
    the stack.  The next opcode will either ignore it (:opcode:`POP_TOP`), or

Doc/library/doctest.rst

@@ -719,51 +719,36 @@ above.
 An example's doctest directives modify doctest's behavior for that single
 example.  Use ``+`` to enable the named behavior, or ``-`` to disable it.
 
-For example, this test passes:
+For example, this test passes::
 
-.. doctest::
-   :no-trim-doctest-flags:
-
-   >>> print(list(range(20)))  # doctest: +NORMALIZE_WHITESPACE
+   >>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE
    [0,   1,  2,  3,  4,  5,  6,  7,  8,  9,
    10,  11, 12, 13, 14, 15, 16, 17, 18, 19]
 
 Without the directive it would fail, both because the actual output doesn't have
 two blanks before the single-digit list elements, and because the actual output
 is on a single line.  This test also passes, and also requires a directive to do
-so:
-
-.. doctest::
-   :no-trim-doctest-flags:
+so::
 
-   >>> print(list(range(20)))  # doctest: +ELLIPSIS
+   >>> print(list(range(20))) # doctest: +ELLIPSIS
    [0, 1, ..., 18, 19]
 
 Multiple directives can be used on a single physical line, separated by
-commas:
+commas::
 
-.. doctest::
-   :no-trim-doctest-flags:
-
-   >>> print(list(range(20)))  # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
+   >>> print(list(range(20))) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
    [0,    1, ...,   18,    19]
 
 If multiple directive comments are used for a single example, then they are
-combined:
-
-.. doctest::
-   :no-trim-doctest-flags:
+combined::
 
-   >>> print(list(range(20)))  # doctest: +ELLIPSIS
-   ...                         # doctest: +NORMALIZE_WHITESPACE
+   >>> print(list(range(20))) # doctest: +ELLIPSIS
+   ...                        # doctest: +NORMALIZE_WHITESPACE
    [0,    1, ...,   18,    19]
 
 As the previous example shows, you can add ``...`` lines to your example
 containing only directives.  This can be useful when an example is too long for
-a directive to comfortably fit on the same line:
-
-.. doctest::
-   :no-trim-doctest-flags:
+a directive to comfortably fit on the same line::
 
    >>> print(list(range(5)) + list(range(10, 20)) + list(range(30, 40)))
    ... # doctest: +ELLIPSIS
@@ -808,23 +793,18 @@ instead.  Another is to do ::
 
 There are others, but you get the idea.
 
-Another bad idea is to print things that embed an object address, like
-
-.. doctest::
+Another bad idea is to print things that embed an object address, like ::
 
-   >>> id(1.0) # certain to fail some of the time  # doctest: +SKIP
+   >>> id(1.0) # certain to fail some of the time
    7948648
    >>> class C: pass
-   >>> C()   # the default repr() for instances embeds an address  # doctest: +SKIP
-   <C object at 0x00AC18F0>
-
-The :const:`ELLIPSIS` directive gives a nice approach for the last example:
+   >>> C()   # the default repr() for instances embeds an address
+   <__main__.C instance at 0x00AC18F0>
 
-.. doctest::
-   :no-trim-doctest-flags:
+The :const:`ELLIPSIS` directive gives a nice approach for the last example::
 
-   >>> C()  # doctest: +ELLIPSIS
-   <C object at 0x...>
+   >>> C() #doctest: +ELLIPSIS
+   <__main__.C instance at 0x...>
 
 Floating-point numbers are also subject to small output variations across
 platforms, because Python defers to the platform C library for float formatting,

Doc/library/email.header.rst

@@ -116,7 +116,7 @@ Here is the :class:`Header` class description:
       if *s* is a byte string.
 
 
-   .. method:: encode(splitchars=';, \t', maxlinelen=None, linesep='\n')
+   .. method:: encode(splitchars=';, \\t', maxlinelen=None, linesep='\\n')
 
       Encode a message header into an RFC-compliant format, possibly wrapping
       long lines and encapsulating non-ASCII parts in base64 or quoted-printable

Doc/library/ensurepip.rst

@@ -48,7 +48,7 @@ The simplest possible invocation is::
 
 This invocation will install ``pip`` if it is not already installed,
 but otherwise does nothing. To ensure the installed version of ``pip``
-is at least as recent as the one bundled with ``ensurepip``, pass the
+is at least as recent as the one available in ``ensurepip``, pass the
 ``--upgrade`` option::
 
     python -m ensurepip --upgrade
@@ -86,7 +86,7 @@ Module API
 
 .. function:: version()
 
-   Returns a string specifying the bundled version of pip that will be
+   Returns a string specifying the available version of pip that will be
    installed when bootstrapping an environment.
 
 .. function:: bootstrap(root=None, upgrade=False, user=False, \
@@ -100,7 +100,7 @@ Module API
    for the current environment.
 
    *upgrade* indicates whether or not to upgrade an existing installation
-   of an earlier version of ``pip`` to the bundled version.
+   of an earlier version of ``pip`` to the available version.
 
    *user* indicates whether to use the user scheme rather than installing
    globally.