merge branch 3.5

Author: Anselm Kruis

.gitignore

@@ -1,7 +1,7 @@
 # Two-trick pony for OSX and other case insensitive file systems:
 # Ignore ./python binary on Unix but still look into ./Python/ directory.
 /python
-!/Python/**
+!/Python/
 *.cover
 *.o
 *.orig
@@ -69,8 +69,8 @@ config.status
 config.status.lineno
 core
 db_home
-config.log
-config.status
+.hg/
+ipch/
 libpython*.a
 libpython*.so*
 platform
@@ -93,3 +93,5 @@ htmlcov/
 Tools/msi/obj
 Tools/ssl/amd64
 Tools/ssl/win32
+.vs/
+.vscode/

.hgignore

@@ -94,7 +94,10 @@ htmlcov/
 *.gcda
 *.gcno
 *.gcov
+ipch/
 coverage.info
 Tools/msi/obj
 Tools/ssl/amd64
 Tools/ssl/win32
+.vs/
+.vscode/

Doc/c-api/arg.rst

@@ -32,8 +32,12 @@ Strings and buffers
 
 These formats allow accessing an object as a contiguous chunk of memory.
 You don't have to provide raw storage for the returned unicode or bytes
-area.  Also, you won't have to release any memory yourself, except with the
-``es``, ``es#``, ``et`` and ``et#`` formats.
+area.
+
+In general, when a format sets a pointer to a buffer, the buffer is
+managed by the corresponding Python object, and the buffer shares
+the lifetime of this object.  You won't have to release any memory yourself.
+The only exceptions are ``es``, ``es#``, ``et`` and ``et#``.
 
 However, when a :c:type:`Py_buffer` structure gets filled, the underlying
 buffer is locked so that the caller can subsequently use the buffer even
@@ -44,6 +48,11 @@ in any early abort case).
 
 Unless otherwise stated, buffers are not NUL-terminated.
 
+Some formats require a read-only :term:`bytes-like object`, and set a
+pointer instead of a buffer structure.  They work by checking that
+the object's :c:member:`PyBufferProcs.bf_releasebuffer` field is *NULL*,
+which disallows mutable objects such as :class:`bytearray`.
+
 .. note::
 
    For all ``#`` variants of formats (``s#``, ``y#``, etc.), the type of
@@ -59,7 +68,7 @@ Unless otherwise stated, buffers are not NUL-terminated.
    Convert a Unicode object to a C pointer to a character string.
    A pointer to an existing string is stored in the character pointer
    variable whose address you pass.  The C string is NUL-terminated.
-   The Python string must not contain embedded null characters; if it does,
+   The Python string must not contain embedded null code points; if it does,
    a :exc:`ValueError` exception is raised. Unicode objects are converted
    to C strings using ``'utf-8'`` encoding. If this conversion fails, a
    :exc:`UnicodeError` is raised.
@@ -72,7 +81,7 @@ Unless otherwise stated, buffers are not NUL-terminated.
       as *converter*.
 
    .. versionchanged:: 3.5
-      Previously, :exc:`TypeError` was raised when embedded null characters
+      Previously, :exc:`TypeError` was raised when embedded null code points
       were encountered in the Python string.
 
 ``s*`` (:class:`str` or :term:`bytes-like object`) [Py_buffer]
@@ -82,8 +91,8 @@ Unless otherwise stated, buffers are not NUL-terminated.
    Unicode objects are converted to C strings using ``'utf-8'`` encoding.
 
 ``s#`` (:class:`str`, read-only :term:`bytes-like object`) [const char \*, int or :c:type:`Py_ssize_t`]
-   Like ``s*``, except that it doesn't accept mutable bytes-like objects
-   such as :class:`bytearray`.  The result is stored into two C variables,
+   Like ``s*``, except that it doesn't accept mutable objects.
+   The result is stored into two C variables,
    the first one a pointer to a C string, the second one its length.
    The string may contain embedded null bytes. Unicode objects are converted
    to C strings using ``'utf-8'`` encoding.
@@ -135,21 +144,17 @@ Unless otherwise stated, buffers are not NUL-terminated.
    pointer variable, which will be filled with the pointer to an existing
    Unicode buffer.  Please note that the width of a :c:type:`Py_UNICODE`
    character depends on compilation options (it is either 16 or 32 bits).
-   The Python string must not contain embedded null characters; if it does,
+   The Python string must not contain embedded null code points; if it does,
    a :exc:`ValueError` exception is raised.
 
-   .. note::
-      Since ``u`` doesn't give you back the length of the string, and it
-      may contain embedded NUL characters, it is recommended to use ``u#``
-      or ``U`` instead.
-
    .. versionchanged:: 3.5
-      Previously, :exc:`TypeError` was raised when embedded null characters
+      Previously, :exc:`TypeError` was raised when embedded null code points
       were encountered in the Python string.
 
 ``u#`` (:class:`str`) [Py_UNICODE \*, int]
    This variant on ``u`` stores into two C variables, the first one a pointer to a
-   Unicode data buffer, the second one its length.
+   Unicode data buffer, the second one its length.  This variant allows
+   null code points.
 
 ``Z`` (:class:`str` or ``None``) [Py_UNICODE \*]
    Like ``u``, but the Python object may also be ``None``, in which case the

Doc/c-api/import.rst

@@ -207,13 +207,13 @@ Importing Modules
 
 .. c:function:: PyObject* PyImport_GetImporter(PyObject *path)
 
-   Return an importer object for a :data:`sys.path`/:attr:`pkg.__path__` item
+   Return a finder object for a :data:`sys.path`/:attr:`pkg.__path__` item
    *path*, possibly by fetching it from the :data:`sys.path_importer_cache`
    dict.  If it wasn't yet cached, traverse :data:`sys.path_hooks` until a hook
    is found that can handle the path item.  Return ``None`` if no hook could;
-   this tells our caller it should fall back to the built-in import mechanism.
-   Cache the result in :data:`sys.path_importer_cache`.  Return a new reference
-   to the importer object.
+   this tells our caller that the :term:`path based finder` could not find a
+   finder for this path item. Cache the result in :data:`sys.path_importer_cache`.
+   Return a new reference to the finder object.
 
 
 .. c:function:: void _PyImport_Init()

Doc/c-api/init.rst

@@ -37,6 +37,10 @@ Initializing and finalizing the interpreter
    (without calling :c:func:`Py_Finalize` first).  There is no return value; it is a
    fatal error if the initialization fails.
 
+   .. note::
+      On Windows, changes the console mode from ``O_TEXT`` to ``O_BINARY``, which will
+      also affect non-Python uses of the console using the C Runtime.
+
 
 .. c:function:: void Py_InitializeEx(int initsigs)
 

Doc/c-api/module.rst

@@ -59,10 +59,13 @@ Module Objects
    .. index:: single: __dict__ (module attribute)
 
    Return the dictionary object that implements *module*'s namespace; this object
-   is the same as the :attr:`~object.__dict__` attribute of the module object.  This
-   function never fails.  It is recommended extensions use other
-   :c:func:`PyModule_\*` and :c:func:`PyObject_\*` functions rather than directly
-   manipulate a module's :attr:`~object.__dict__`.
+   is the same as the :attr:`~object.__dict__` attribute of the module object.
+   If *module* is not a module object (or a subtype of a module object),
+   :exc:`SystemError` is raised and *NULL* is returned.
+
+   It is recommended extensions use other :c:func:`PyModule_\*` and
+   :c:func:`PyObject_\*` functions rather than directly manipulate a module's
+   :attr:`~object.__dict__`.
 
 
 .. c:function:: PyObject* PyModule_GetNameObject(PyObject *module)
@@ -321,7 +324,7 @@ The available slot types are:
    :c:type:`PyModule_Type`. Any type can be used, as long as it supports
    setting and getting import-related attributes.
    However, only ``PyModule_Type`` instances may be returned if the
-   ``PyModuleDef`` has non-*NULL* ``m_methods``, ``m_traverse``, ``m_clear``,
+   ``PyModuleDef`` has non-*NULL* ``m_traverse``, ``m_clear``,
    ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``.
 
 .. c:var:: Py_mod_exec

Doc/c-api/veryhigh.rst

@@ -219,9 +219,10 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* PyRun_StringFlags(const char *str, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
 
    Execute Python source code from *str* in the context specified by the
-   dictionaries *globals* and *locals* with the compiler flags specified by
-   *flags*.  The parameter *start* specifies the start token that should be used to
-   parse the source code.
+   objects *globals* and *locals* with the compiler flags specified by
+   *flags*.  *globals* must be a dictionary; *locals* can be any object
+   that implements the mapping protocol.  The parameter *start* specifies
+   the start token that should be used to parse the source code.
 
    Returns the result of executing the code as a Python object, or *NULL* if an
    exception was raised.
@@ -295,16 +296,16 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* PyEval_EvalCode(PyObject *co, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyEval_EvalCodeEx`, with just
-   the code object, and the dictionaries of global and local variables.
-   The other arguments are set to *NULL*.
+   the code object, and global and local variables.  The other arguments are
+   set to *NULL*.
 
 
 .. c:function:: PyObject* PyEval_EvalCodeEx(PyObject *co, PyObject *globals, PyObject *locals, PyObject **args, int argcount, PyObject **kws, int kwcount, PyObject **defs, int defcount, PyObject *closure)
 
    Evaluate a precompiled code object, given a particular environment for its
-   evaluation.  This environment consists of dictionaries of global and local
-   variables, arrays of arguments, keywords and defaults, and a closure tuple of
-   cells.
+   evaluation.  This environment consists of a dictionary of global variables,
+   a mapping object of local variables, arrays of arguments, keywords and
+   defaults, and a closure tuple of cells.
 
 
 .. c:type:: PyFrameObject

Doc/conf.py

@@ -76,7 +76,10 @@
 
 # Custom sidebar templates, filenames relative to this file.
 html_sidebars = {
-    'index': 'indexsidebar.html',
+    # Defaults taken from http://www.sphinx-doc.org/en/stable/config.html#confval-html_sidebars
+    # Removes the quick search block
+    '**': ['localtoc.html', 'relations.html', 'customsourcelink.html'],
+    'index': ['indexsidebar.html'],
 }
 
 # Additional templates that should be rendered to pages.

Doc/distutils/examples.rst

@@ -245,7 +245,9 @@ Let's take an example with a simple script::
 
     setup(name='foobar')
 
-Running the ``check`` command will display some warnings::
+Running the ``check`` command will display some warnings:
+
+.. code-block:: shell-session
 
     $ python setup.py check
     running check
@@ -274,7 +276,9 @@ For example, if the :file:`setup.py` script is changed like this::
         url='http://example.com', long_description=desc)
 
 Where the long description is broken, ``check`` will be able to detect it
-by using the :mod:`docutils` parser::
+by using the :mod:`docutils` parser:
+
+.. code-block:: shell-session
 
     $ python setup.py check --restructuredtext
     running check
@@ -286,7 +290,9 @@ Reading the metadata
 
 The :func:`distutils.core.setup` function provides a command-line interface
 that allows you to query the metadata fields of a project through the
-``setup.py`` script of a given project::
+``setup.py`` script of a given project:
+
+.. code-block:: shell-session
 
     $ python setup.py --name
     distribute

Doc/distutils/packageindex.rst

@@ -233,7 +233,9 @@ in the root of the package besides :file:`setup.py`.
 
 To prevent registering broken reStructuredText content, you can use the
 :program:`rst2html` program that is provided by the :mod:`docutils` package and
-check the ``long_description`` from the command line::
+check the ``long_description`` from the command line:
+
+.. code-block:: shell-session
 
     $ python setup.py --long-description | rst2html.py > output.html
 

Doc/distutils/sourcedist.rst

@@ -133,7 +133,9 @@ described above does not apply in this case.
 
 The manifest template has one command per line, where each command specifies a
 set of files to include or exclude from the source distribution.  For an
-example, again we turn to the Distutils' own manifest template::
+example, again we turn to the Distutils' own manifest template:
+
+.. code-block:: none
 
    include *.txt
    recursive-include examples *.txt *.py

Doc/extending/building.rst

@@ -55,7 +55,9 @@ Since distutils also supports creation of binary packages, users don't
 necessarily need a compiler and distutils to install the extension.
 
 A distutils package contains a driver script, :file:`setup.py`. This is a plain
-Python file, which, in the most simple case, could look like this::
+Python file, which, in the most simple case, could look like this:
+
+.. code-block:: python3
 
    from distutils.core import setup, Extension
 
@@ -96,7 +98,9 @@ file, :file:`demo.c`.
 
 In many cases, building an extension is more complex, since additional
 preprocessor defines and libraries may be needed. This is demonstrated in the
-example below. ::
+example below.
+
+.. code-block:: python3
 
    from distutils.core import setup, Extension
 
@@ -161,4 +165,3 @@ commands can be used to do so. ::
    python setup.py bdist_wininst
    python setup.py bdist_rpm
    python setup.py bdist_dumb
-

Doc/extending/embedding.rst

@@ -155,7 +155,9 @@ script, such as:
            c = c + b
        return c
 
-then the result should be::
+then the result should be:
+
+.. code-block:: shell-session
 
    $ call multiply multiply 3 2
    Will compute 3 times 2
@@ -289,16 +291,20 @@ available).  This script has several options, of which the following will
 be directly useful to you:
 
 * ``pythonX.Y-config --cflags`` will give you the recommended flags when
-  compiling::
+  compiling:
+
+  .. code-block:: shell-session
 
-   $ /opt/bin/python3.4-config --cflags
-   -I/opt/include/python3.4m -I/opt/include/python3.4m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
+     $ /opt/bin/python3.4-config --cflags
+     -I/opt/include/python3.4m -I/opt/include/python3.4m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
 
 * ``pythonX.Y-config --ldflags`` will give you the recommended flags when
-  linking::
+  linking:
+
+  .. code-block:: shell-session
 
-   $ /opt/bin/python3.4-config --ldflags
-   -L/opt/lib/python3.4/config-3.4m -lpthread -ldl -lutil -lm -lpython3.4m -Xlinker -export-dynamic
+     $ /opt/bin/python3.4-config --ldflags
+     -L/opt/lib/python3.4/config-3.4m -lpthread -ldl -lutil -lm -lpython3.4m -Xlinker -export-dynamic
 
 .. note::
    To avoid confusion between several Python installations (and especially

Doc/extending/extending.rst

@@ -792,7 +792,9 @@ the format string is empty, it returns ``None``; if it contains exactly one
 format unit, it returns whatever object is described by that format unit.  To
 force it to return a tuple of size 0 or one, parenthesize the format string.
 
-Examples (to the left the call, to the right the resulting Python value)::
+Examples (to the left the call, to the right the resulting Python value):
+
+.. code-block:: none
 
    Py_BuildValue("")                        None
    Py_BuildValue("i", 123)                  123
@@ -1348,4 +1350,3 @@ code distribution).
 
 .. [#] These guarantees don't hold when you use the "old" style calling convention ---
    this is still found in much existing code.
-

Doc/extending/newtypes.rst

@@ -52,11 +52,15 @@ The first bit that will be new is::
    } noddy_NoddyObject;
 
 This is what a Noddy object will contain---in this case, nothing more than what
-every Python object contains---a refcount and a pointer to a type object.
-These are the fields the ``PyObject_HEAD`` macro brings in.  The reason for the
-macro is to standardize the layout and to enable special debugging fields in
-debug builds.  Note that there is no semicolon after the ``PyObject_HEAD``
-macro; one is included in the macro definition.  Be wary of adding one by
+every Python object contains---a field called ``ob_base`` of type
+:c:type:`PyObject`.  :c:type:`PyObject` in turn, contains an ``ob_refcnt``
+field and a pointer to a type object.  These can be accessed using the macros
+:c:macro:`Py_REFCNT` and :c:macro:`Py_TYPE` respectively.  These are the fields
+the :c:macro:`PyObject_HEAD` macro brings in.  The reason for the macro is to
+standardize the layout and to enable special debugging fields in debug builds.
+
+Note that there is no semicolon after the :c:macro:`PyObject_HEAD` macro;
+one is included in the macro definition.  Be wary of adding one by
 accident; it's easy to do from habit, and your compiler might not complain,
 but someone else's probably will!  (On Windows, MSVC is known to call this an
 error and refuse to compile the code.)
@@ -209,7 +213,9 @@ That's it!  All that remains is to build it; put the above code in a file called
    setup(name="noddy", version="1.0",
          ext_modules=[Extension("noddy", ["noddy.c"])])
 
-in a file called :file:`setup.py`; then typing ::
+in a file called :file:`setup.py`; then typing
+
+.. code-block:: shell-session
 
    $ python setup.py build
 
@@ -1513,4 +1519,3 @@ might be something like the following::
 .. [#] Even in the third version, we aren't guaranteed to avoid cycles.  Instances of
    string subclasses are allowed and string subclasses could allow cycles even if
    normal strings don't.
-