bpo-45640: [docs] Tokens are now clickable (GH-29260)

Co-authored-by: Łukasz Langa <[email protected]>

Author: Arthur-Milchior

Doc/reference/compound_stmts.rst

@@ -418,8 +418,8 @@ usage patterns to be encapsulated for convenient reuse.
 
 The execution of the :keyword:`with` statement with one "item" proceeds as follows:
 
-#. The context expression (the expression given in the :token:`with_item`) is
-   evaluated to obtain a context manager.
+#. The context expression (the expression given in the
+   :token:`~python-grammar:with_item`) is evaluated to obtain a context manager.
 
 #. The context manager's :meth:`__enter__` is loaded for later use.
 
@@ -797,7 +797,8 @@ Syntax:
    capture_pattern: !'_' NAME
 
 A single underscore ``_`` is not a capture pattern (this is what ``!'_'``
-expresses). It is instead treated as a :token:`wildcard_pattern`.
+expresses). It is instead treated as a
+:token:`~python-grammar:wildcard_pattern`.
 
 In a given pattern, a given name can only be bound once.  E.g.
 ``case x, x: ...`` is invalid while ``case [x] | x: ...`` is allowed.
@@ -1182,9 +1183,9 @@ is roughly equivalent to ::
 except that the original function is not temporarily bound to the name ``func``.
 
 .. versionchanged:: 3.9
-   Functions may be decorated with any valid :token:`assignment_expression`.
-   Previously, the grammar was much more restrictive; see :pep:`614` for
-   details.
+   Functions may be decorated with any valid
+   :token:`~python-grammar:assignment_expression`. Previously, the grammar was
+   much more restrictive; see :pep:`614` for details.
 
 .. index::
    triple: default; parameter; value
@@ -1360,9 +1361,9 @@ The evaluation rules for the decorator expressions are the same as for function
 decorators.  The result is then bound to the class name.
 
 .. versionchanged:: 3.9
-   Classes may be decorated with any valid :token:`assignment_expression`.
-   Previously, the grammar was much more restrictive; see :pep:`614` for
-   details.
+   Classes may be decorated with any valid
+   :token:`~python-grammar:assignment_expression`. Previously, the grammar was
+   much more restrictive; see :pep:`614` for details.
 
 **Programmer's note:** Variables defined in the class definition are class
 attributes; they are shared by instances.  Instance attributes can be set in a

Doc/reference/expressions.rst

@@ -450,21 +450,20 @@ functions are described separately in section
 :ref:`asynchronous-generator-functions`.
 
 When a generator function is called, it returns an iterator known as a
-generator.  That generator then controls the execution of the generator function.
-The execution starts when one of the generator's methods is called.  At that
-time, the execution proceeds to the first yield expression, where it is
-suspended again, returning the value of :token:`expression_list` to the generator's
-caller.  By suspended, we mean that all local state is retained, including the
-current bindings of local variables, the instruction pointer, the internal
-evaluation stack, and the state of any exception handling.  When the execution
-is resumed by calling one of the
-generator's methods, the function can proceed exactly as if the yield expression
-were just another external call.  The value of the yield expression after
-resuming depends on the method which resumed the execution.  If
-:meth:`~generator.__next__` is used (typically via either a :keyword:`for` or
-the :func:`next` builtin) then the result is :const:`None`.  Otherwise, if
-:meth:`~generator.send` is used, then the result will be the value passed in to
-that method.
+generator.  That generator then controls the execution of the generator
+function.  The execution starts when one of the generator's methods is called.
+At that time, the execution proceeds to the first yield expression, where it is
+suspended again, returning the value of :token:`~python-grammar:expression_list`
+to the generator's caller.  By suspended, we mean that all local state is
+retained, including the current bindings of local variables, the instruction
+pointer, the internal evaluation stack, and the state of any exception handling.
+When the execution is resumed by calling one of the generator's methods, the
+function can proceed exactly as if the yield expression were just another
+external call.  The value of the yield expression after resuming depends on the
+method which resumed the execution.  If :meth:`~generator.__next__` is used
+(typically via either a :keyword:`for` or the :func:`next` builtin) then the
+result is :const:`None`.  Otherwise, if :meth:`~generator.send` is used, then
+the result will be the value passed in to that method.
 
 .. index:: single: coroutine
 
@@ -514,8 +513,8 @@ on the right hand side of an assignment statement.
       usable as simple coroutines.
 
    :pep:`380` - Syntax for Delegating to a Subgenerator
-      The proposal to introduce the :token:`yield_from` syntax, making delegation
-      to subgenerators easy.
+      The proposal to introduce the :token:`~python-grammar:yield_from` syntax,
+      making delegation to subgenerators easy.
 
    :pep:`525` - Asynchronous Generators
       The proposal that expanded on :pep:`492` by adding generator capabilities to
@@ -543,9 +542,9 @@ is already executing raises a :exc:`ValueError` exception.
    :meth:`~generator.__next__` method, the current yield expression always
    evaluates to :const:`None`.  The execution then continues to the next yield
    expression, where the generator is suspended again, and the value of the
-   :token:`expression_list` is returned to :meth:`__next__`'s caller.  If the
-   generator exits without yielding another value, a :exc:`StopIteration`
-   exception is raised.
+   :token:`~python-grammar:expression_list` is returned to :meth:`__next__`'s
+   caller.  If the generator exits without yielding another value, a
+   :exc:`StopIteration` exception is raised.
 
    This method is normally called implicitly, e.g. by a :keyword:`for` loop, or
    by the built-in :func:`next` function.
@@ -634,21 +633,20 @@ An asynchronous generator object is typically used in an
 :keyword:`async for` statement in a coroutine function analogously to
 how a generator object would be used in a :keyword:`for` statement.
 
-Calling one of the asynchronous generator's methods returns an
-:term:`awaitable` object, and the execution starts when this object
-is awaited on. At that time, the execution proceeds to the first yield
-expression, where it is suspended again, returning the value of
-:token:`expression_list` to the awaiting coroutine. As with a generator,
-suspension means that all local state is retained, including the
-current bindings of local variables, the instruction pointer, the internal
-evaluation stack, and the state of any exception handling.  When the execution
-is resumed by awaiting on the next object returned by the asynchronous
-generator's methods, the function can proceed exactly as if the yield
-expression were just another external call. The value of the yield expression
-after resuming depends on the method which resumed the execution.  If
+Calling one of the asynchronous generator's methods returns an :term:`awaitable`
+object, and the execution starts when this object is awaited on. At that time,
+the execution proceeds to the first yield expression, where it is suspended
+again, returning the value of :token:`~python-grammar:expression_list` to the
+awaiting coroutine. As with a generator, suspension means that all local state
+is retained, including the current bindings of local variables, the instruction
+pointer, the internal evaluation stack, and the state of any exception handling.
+When the execution is resumed by awaiting on the next object returned by the
+asynchronous generator's methods, the function can proceed exactly as if the
+yield expression were just another external call. The value of the yield
+expression after resuming depends on the method which resumed the execution.  If
 :meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if
-:meth:`~agen.asend` is used, then the result will be the value passed in to
-that method.
+:meth:`~agen.asend` is used, then the result will be the value passed in to that
+method.
 
 If an asynchronous generator happens to exit early by :keyword:`break`, the caller
 task being cancelled, or other exceptions, the generator's async cleanup code
@@ -700,10 +698,10 @@ which are used to control the execution of a generator function.
    Returns an awaitable which when run starts to execute the asynchronous
    generator or resumes it at the last executed yield expression.  When an
    asynchronous generator function is resumed with an :meth:`~agen.__anext__`
-   method, the current yield expression always evaluates to :const:`None` in
-   the returned awaitable, which when run will continue to the next yield
-   expression. The value of the :token:`expression_list` of the yield
-   expression is the value of the :exc:`StopIteration` exception raised by
+   method, the current yield expression always evaluates to :const:`None` in the
+   returned awaitable, which when run will continue to the next yield
+   expression. The value of the :token:`~python-grammar:expression_list` of the
+   yield expression is the value of the :exc:`StopIteration` exception raised by
    the completing coroutine.  If the asynchronous generator exits without
    yielding another value, the awaitable instead raises a
    :exc:`StopAsyncIteration` exception, signalling that the asynchronous
@@ -1706,8 +1704,9 @@ Assignment expressions
    assignment_expression: [`identifier` ":="] `expression`
 
 An assignment expression (sometimes also called a "named expression" or
-"walrus") assigns an :token:`expression` to an :token:`identifier`, while also
-returning the value of the :token:`expression`.
+"walrus") assigns an :token:`~python-grammar:expression` to an
+:token:`~python-grammar:identifier`, while also returning the value of the
+:token:`~python-grammar:expression`.
 
 One common use case is when handling matched regular expressions:
 

Doc/reference/lexical_analysis.rst

@@ -469,10 +469,10 @@ String literals are described by the following lexical definitions:
    bytesescapeseq: "\" <any ASCII character>
 
 One syntactic restriction not indicated by these productions is that whitespace
-is not allowed between the :token:`stringprefix` or :token:`bytesprefix` and the
-rest of the literal. The source character set is defined by the encoding
-declaration; it is UTF-8 if no encoding declaration is given in the source file;
-see section :ref:`encodings`.
+is not allowed between the :token:`~python-grammar:stringprefix` or
+:token:`~python-grammar:bytesprefix` and the rest of the literal. The source
+character set is defined by the encoding declaration; it is UTF-8 if no encoding
+declaration is given in the source file; see section :ref:`encodings`.
 
 .. index:: triple-quoted string, Unicode Consortium, raw string
    single: """; string literal

Misc/NEWS.d/next/Documentation/2021-11-18-16-44-12.bpo-45640.lSpc2A.rst

@@ -0,0 +1,3 @@
+Properly marked-up grammar tokens in the documentation are now clickable and
+take you to the definition of a given piece of grammar.  Patch by Arthur
+Milchior.