Merge branch 'main' into gh-94808/improve-coverage-pyobject-print

Author: MonadChains

Doc/data/stable_abi.dat

@@ -8,10 +8,11 @@ Python support for the Linux ``perf`` profiler
 
 :author: Pablo Galindo
 
-The Linux ``perf`` profiler is a very powerful tool that allows you to profile and
-obtain information about the performance of your application. ``perf`` also has
-a very vibrant ecosystem of tools that aid with the analysis of the data that it
-produces.
+`The Linux perf profiler <https://perf.wiki.kernel.org>`_
+is a very powerful tool that allows you to profile and obtain
+information about the performance of your application.
+``perf`` also has a very vibrant ecosystem of tools
+that aid with the analysis of the data that it produces.
 
 The main problem with using the ``perf`` profiler with Python applications is that
 ``perf`` only allows to get information about native symbols, this is, the names of
@@ -25,7 +26,7 @@ fly before the execution of every Python function and it will teach ``perf`` the
 relationship between this piece of code and the associated Python function using
 `perf map files`_.
 
-.. warning::
+.. note::
 
     Support for the ``perf`` profiler is only currently available for Linux on
     selected architectures. Check the output of the configure build step or
@@ -51,11 +52,11 @@ For example, consider the following script:
     if __name__ == "__main__":
         baz(1000000)
 
-We can run perf to sample CPU stack traces at 9999 Hertz:
+We can run ``perf`` to sample CPU stack traces at 9999 Hertz::
 
     $ perf record -F 9999 -g -o perf.data python my_script.py
 
-Then we can use perf report to analyze the data:
+Then we can use ``perf`` report to analyze the data:
 
 .. code-block:: shell-session
 
@@ -101,7 +102,7 @@ As you can see here, the Python functions are not shown in the output, only ``_P
 functions use the same C function to evaluate bytecode so we cannot know which Python function corresponds to which
 bytecode-evaluating function.
 
-Instead, if we run the same experiment with perf support activated we get:
+Instead, if we run the same experiment with ``perf`` support enabled we get:
 
 .. code-block:: shell-session
 
@@ -147,52 +148,58 @@ Instead, if we run the same experiment with perf support activated we get:
 
 
 
-Enabling perf profiling mode
-----------------------------
+How to enable ``perf`` profiling support
+----------------------------------------
 
-There are two main ways to activate the perf profiling mode. If you want it to be
-active since the start of the Python interpreter, you can use the ``-Xperf`` option:
+``perf`` profiling support can either be enabled from the start using
+the environment variable :envvar:`PYTHONPERFSUPPORT` or the
+:option:`-X perf <-X>` option,
+or dynamically using :func:`sys.activate_stack_trampoline` and
+:func:`sys.deactivate_stack_trampoline`.
 
-    $ python -Xperf my_script.py
+The :mod:`!sys` functions take precedence over the :option:`!-X` option,
+the :option:`!-X` option takes precedence over the environment variable.
 
-You can also set the :envvar:`PYTHONPERFSUPPORT` to a nonzero value to actiavate perf
-profiling mode globally.
+Example, using the environment variable::
 
-There is also support for dynamically activating and deactivating the perf
-profiling mode by using the APIs in the :mod:`sys` module:
+   $ PYTHONPERFSUPPORT=1
+   $ python script.py
+   $ perf report -g -i perf.data
 
-.. code-block:: python
-
-    import sys
-    sys.activate_stack_trampoline("perf")
+Example, using the :option:`!-X` option::
 
-    # Run some code with Perf profiling active
+   $ python -X perf script.py
+   $ perf report -g -i perf.data
 
-    sys.deactivate_stack_trampoline()
+Example, using the :mod:`sys` APIs in file :file:`example.py`:
 
-    # Perf profiling is not active anymore
+.. code-block:: python
 
-These APIs can be handy if you want to activate/deactivate profiling mode in
-response to a signal or other communication mechanism with your process.
+   import sys
 
+   sys.activate_stack_trampoline("perf")
+   do_profiled_stuff()
+   sys.deactivate_stack_trampoline()
 
+   non_profiled_stuff()
 
-Now we can analyze the data with ``perf report``:
+...then::
 
-    $ perf report -g -i perf.data
+   $ python ./example.py
+   $ perf report -g -i perf.data
 
 
 How to obtain the best results
--------------------------------
+------------------------------
 
 For the best results, Python should be compiled with
 ``CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"`` as this allows
 profilers to unwind using only the frame pointer and not on DWARF debug
-information. This is because as the code that is interposed to allow perf
+information. This is because as the code that is interposed to allow ``perf``
 support is dynamically generated it doesn't have any DWARF debugging information
 available.
 
-You can check if you system has been compiled with this flag by running:
+You can check if your system has been compiled with this flag by running::
 
     $ python -m sysconfig | grep 'no-omit-frame-pointer'
 

Doc/howto/perf_profiling.rst

@@ -413,6 +413,15 @@ The Python compiler currently generates the following bytecode instructions.
    Removes the top-of-stack (TOS) item.
 
 
+.. opcode:: END_FOR
+
+   Removes the top two values from the stack.
+   Equivalent to POP_TOP; POP_TOP.
+   Used to clean up at the end of loops, hence the name.
+
+   .. versionadded:: 3.12
+
+
 .. opcode:: COPY (i)
 
    Push the *i*-th item to the top of the stack. The item is not removed from its
@@ -1088,9 +1097,11 @@ iterations of the loop.
 
    TOS is an :term:`iterator`.  Call its :meth:`~iterator.__next__` method.  If
    this yields a new value, push it on the stack (leaving the iterator below
-   it).  If the iterator indicates it is exhausted, TOS is popped, and the byte
+   it).  If the iterator indicates it is exhausted then the byte
    code counter is incremented by *delta*.
 
+   .. versionchanged:: 3.12
+      Up until 3.11 the iterator was popped when it was exhausted.
 
 .. opcode:: LOAD_GLOBAL (namei)
 

Doc/library/dis.rst

@@ -1555,6 +1555,38 @@ always available.
       This function has been added on a provisional basis (see :pep:`411`
       for details.)  Use it only for debugging purposes.
 
+.. function:: activate_stack_trampoline(backend, /)
+
+   Activate the stack profiler trampoline *backend*.
+   The only supported backend is ``"perf"``.
+
+   .. availability:: Linux.
+
+   .. versionadded:: 3.12
+
+   .. seealso::
+
+      * :ref:`perf_profiling`
+      * https://perf.wiki.kernel.org
+
+.. function:: deactivate_stack_trampoline()
+
+   Deactivate the current stack profiler trampoline backend.
+
+   If no stack profiler is activated, this function has no effect.
+
+   .. availability:: Linux.
+
+   .. versionadded:: 3.12
+
+.. function:: is_stack_trampoline_active()
+
+   Return ``True`` if a stack profiler trampoline is active.
+
+   .. availability:: Linux.
+
+   .. versionadded:: 3.12
+
 .. function:: _enablelegacywindowsfsencoding()
 
    Changes the :term:`filesystem encoding and error handler` to 'mbcs' and

Doc/library/sys.rst

@@ -154,7 +154,7 @@ tuple may or may not yield the same object).
    single: , (comma)
 
 Note that tuples are not formed by the parentheses, but rather by use of the
-comma operator.  The exception is the empty tuple, for which parentheses *are*
+comma.  The exception is the empty tuple, for which parentheses *are*
 required --- allowing unparenthesized "nothing" in expressions would cause
 ambiguities and allow common typos to pass uncaught.
 

Doc/reference/expressions.rst

@@ -538,12 +538,11 @@ Miscellaneous options
      development (running from the source tree) then the default is "off".
      Note that the "importlib_bootstrap" and "importlib_bootstrap_external"
      frozen modules are always used, even if this flag is set to "off".
-   * ``-X perf`` to activate compatibility mode with the ``perf`` profiler.
-     When this option is activated, the Linux ``perf`` profiler will be able to
+   * ``-X perf`` enables support for the Linux ``perf`` profiler.
+     When this option is provided, the ``perf`` profiler will be able to
      report Python calls. This option is only available on some platforms and
      will do nothing if is not supported on the current system. The default value
-     is "off". See also :envvar:`PYTHONPERFSUPPORT` and :ref:`perf_profiling`
-     for more information.
+     is "off". See also :envvar:`PYTHONPERFSUPPORT` and :ref:`perf_profiling`.
 
    It also allows passing arbitrary values and retrieving them through the
    :data:`sys._xoptions` dictionary.
@@ -1048,9 +1047,13 @@ conflict.
 
 .. envvar:: PYTHONPERFSUPPORT
 
-   If this variable is set to a nonzero value, it activates compatibility mode
-   with the ``perf`` profiler so Python calls can be detected by it. See the
-   :ref:`perf_profiling` section for more information.
+   If this variable is set to a nonzero value, it enables support for
+   the Linux ``perf`` profiler so Python calls can be detected by it.
+
+   If set to ``0``, disable Linux ``perf`` profiler support.
+
+   See also the :option:`-X perf <-X>` command-line option
+   and :ref:`perf_profiling`.
 
    .. versionadded:: 3.12
 

Doc/using/cmdline.rst

@@ -74,6 +74,15 @@ Important deprecations, removals or restrictions:
 New Features
 ============
 
+* Add :ref:`perf_profiling` through the new
+  environment variable :envvar:`PYTHONPERFSUPPORT`,
+  the new command-line option :option:`-X perf <-X>`,
+  as well as the new :func:`sys.activate_stack_trampoline`,
+  :func:`sys.deactivate_stack_trampoline`,
+  and :func:`sys.is_stack_trampoline_active` APIs.
+  (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes
+  with contributions from Gregory P. Smith [Google] and Mark Shannon
+  in :gh:`96123`.)
 
 
 Other Language Changes
@@ -194,6 +203,19 @@ tempfile
 The :class:`tempfile.NamedTemporaryFile` function has a new optional parameter
 *delete_on_close* (Contributed by Evgeny Zorin in :gh:`58451`.)
 
+sys
+---
+
+* Add :func:`sys.activate_stack_trampoline` and
+  :func:`sys.deactivate_stack_trampoline` for activating and deactivating
+  stack profiler trampolines,
+  and :func:`sys.is_stack_trampoline_active` for querying if stack profiler
+  trampolines are active.
+  (Contributed by Pablo Galindo and Christian Heimes
+  with contributions from Gregory P. Smith [Google] and Mark Shannon
+  in :gh:`96123`.)
+
+
 Optimizations
 =============
 

Doc/whatsnew/3.12.rst

@@ -238,6 +238,22 @@ PyAPI_FUNC(Py_ssize_t) PyVectorcall_NARGS(size_t nargsf);
    "tuple" and keyword arguments "dict". "dict" may also be NULL */
 PyAPI_FUNC(PyObject *) PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *dict);
 
+#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030C0000
+#define PY_VECTORCALL_ARGUMENTS_OFFSET \
+    (_Py_STATIC_CAST(size_t, 1) << (8 * sizeof(size_t) - 1))
+
+/* Perform a PEP 590-style vector call on 'callable' */
+PyAPI_FUNC(PyObject *) PyObject_Vectorcall(
+    PyObject *callable,
+    PyObject *const *args,
+    size_t nargsf,
+    PyObject *kwnames);
+
+/* Call the method 'name' on args[0] with arguments in args[1..nargsf-1]. */
+PyAPI_FUNC(PyObject *) PyObject_VectorcallMethod(
+    PyObject *name, PyObject *const *args,
+    size_t nargsf, PyObject *kwnames);
+#endif
 
 /* Implemented elsewhere:
 

Include/abstract.h

@@ -50,9 +50,6 @@ PyAPI_FUNC(PyObject *) _PyObject_MakeTpCall(
     PyObject *const *args, Py_ssize_t nargs,
     PyObject *keywords);
 
-#define PY_VECTORCALL_ARGUMENTS_OFFSET \
-    (_Py_STATIC_CAST(size_t, 1) << (8 * sizeof(size_t) - 1))
-
 // PyVectorcall_NARGS() is exported as a function for the stable ABI.
 // Here (when we are not using the stable ABI), the name is overridden to
 // call a static inline function for best performance.
@@ -65,12 +62,6 @@ _PyVectorcall_NARGS(size_t n)
 
 PyAPI_FUNC(vectorcallfunc) PyVectorcall_Function(PyObject *callable);
 
-PyAPI_FUNC(PyObject *) PyObject_Vectorcall(
-    PyObject *callable,
-    PyObject *const *args,
-    size_t nargsf,
-    PyObject *kwnames);
-
 // Backwards compatibility aliases for API that was provisional in Python 3.8
 #define _PyObject_Vectorcall PyObject_Vectorcall
 #define _PyObject_VectorcallMethod PyObject_VectorcallMethod
@@ -96,10 +87,6 @@ PyAPI_FUNC(PyObject *) _PyObject_FastCall(
 
 PyAPI_FUNC(PyObject *) PyObject_CallOneArg(PyObject *func, PyObject *arg);
 
-PyAPI_FUNC(PyObject *) PyObject_VectorcallMethod(
-    PyObject *name, PyObject *const *args,
-    size_t nargsf, PyObject *kwnames);
-
 static inline PyObject *
 PyObject_CallMethodNoArgs(PyObject *self, PyObject *name)
 {