[3.8] bpo-38234: Backport init path config changes from master (GH-16423)

* bpo-38234: Py_SetPath() uses the program full path (GH-16357)

Py_SetPath() now sets sys.executable to the program full path
(Py_GetProgramFullPath()), rather than to the program name
(Py_GetProgramName()).

Fix also memory leaks in pathconfig_set_from_config().

(cherry picked from commit 1ce152a)

* bpo-38234: Add tests for Python init path config (GH-16358)


(cherry picked from commit bb6bf7d)

* bpo-38234: test_embed: test pyvenv.cfg and pybuilddir.txt (GH-16366)

Add test_init_pybuilddir() and test_init_pyvenv_cfg() to test_embed
to test pyvenv.cfg and pybuilddir.txt configuration files.

Fix sysconfig._generate_posix_vars(): pybuilddir.txt uses UTF-8
encoding, not ASCII.

(cherry picked from commit 52ad33a)

* bpo-38234: Cleanup getpath.c (GH-16367)

* search_for_prefix() directly calls reduce() if found is greater
  than 0.
* Add calculate_pybuilddir() subfunction.
* search_for_prefix(): add path string buffer for readability.
* Fix some error handling code paths: release resources on error.
* calculate_read_pyenv(): rename tmpbuffer to filename.
* test.pythoninfo now also logs windows.dll_path

(cherry picked from commit 221fd84)

* bpo-38234: Fix test_embed pathconfig tests (GH-16390)

bpo-38234: On macOS and FreeBSD, the temporary directory can be
symbolic link. For example, /tmp can be a symbolic link to /var/tmp.
Call realpath() to resolve all symbolic links.

(cherry picked from commit 00508a7)

* bpo-38234: Add test_init_setpath_config() to test_embed (GH-16402)

* Add test_embed.test_init_setpath_config(): test Py_SetPath()
  with PyConfig.
* test_init_setpath() and test_init_setpythonhome() no longer call
  Py_SetProgramName(), but use the default program name.
* _PyPathConfig: isolated, site_import  and base_executable
  fields are now only available on Windows.
* If executable is set explicitly in the configuration, ignore
  calculated base_executable: _PyConfig_InitPathConfig() copies
  executable to base_executable.
* Complete path config documentation.

(cherry picked from commit 8bf39b6)

* bpo-38234: Complete init config documentation (GH-16404)


(cherry picked from commit 88feaec)

* bpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406)

Explicitly preinitializes with a Python preconfiguration to avoid
Py_SetPath() implicit preinitialization with a compat
preconfiguration.

Fix also test_init_setpath() and test_init_setpythonhome() on macOS:
use self.test_exe as the executable (and base_executable), rather
than shutil.which('python3').

(cherry picked from commit 49d99f0)

* bpo-38234: Py_Initialize() sets global path configuration (GH-16421)

* Py_InitializeFromConfig() now writes PyConfig path configuration to
  the global path configuration (_Py_path_config).
* Add test_embed.test_get_pathconfig().
* Fix typo in _PyWideStringList_Join().

(cherry picked from commit 12f2f17)

Author: vstinner

Doc/c-api/init.rst

@@ -472,8 +472,8 @@ Process-wide parameters
    dependent delimiter character, which is ``':'`` on Unix and Mac OS X, ``';'``
    on Windows.
 
-   This also causes :data:`sys.executable` to be set only to the raw program
-   name (see :c:func:`Py_SetProgramName`) and for :data:`sys.prefix` and
+   This also causes :data:`sys.executable` to be set to the program
+   full path (see :c:func:`Py_GetProgramFullPath`) and for :data:`sys.prefix` and
    :data:`sys.exec_prefix` to be empty.  It is up to the caller to modify these
    if required after calling :c:func:`Py_Initialize`.
 
@@ -483,6 +483,10 @@ Process-wide parameters
    The path argument is copied internally, so the caller may free it after the
    call completes.
 
+   .. versionchanged:: 3.8
+      The program full path is now used for :data:`sys.executable`, instead
+      of the program name.
+
 
 .. c:function:: const char* Py_GetVersion()
 

Doc/c-api/init_config.rst

@@ -241,6 +241,7 @@ PyPreConfig
       locale to decide if it should be coerced.
 
    .. c:member:: int coerce_c_locale_warn
+
       If non-zero, emit a warning if the C locale is coerced.
 
    .. c:member:: int dev_mode
@@ -300,7 +301,7 @@ For :ref:`Python Configuration <init-python-config>`
 (:c:func:`PyPreConfig_InitPythonConfig`), if Python is initialized with
 command line arguments, the command line arguments must also be passed to
 preinitialize Python, since they have an effect on the pre-configuration
-like encodings. For example, the :option:`-X` ``utf8`` command line option
+like encodings. For example, the :option:`-X utf8 <-X>` command line option
 enables the UTF-8 Mode.
 
 ``PyMem_SetAllocator()`` can be called after :c:func:`Py_PreInitialize` and
@@ -464,7 +465,7 @@ PyConfig
 
    .. c:member:: int dev_mode
 
-      Development mode: see :option:`-X` ``dev``.
+      Development mode: see :option:`-X dev <-X>`.
 
    .. c:member:: int dump_refs
 
@@ -482,7 +483,7 @@ PyConfig
 
    .. c:member:: int faulthandler
 
-      If non-zero, call :func:`faulthandler.enable`.
+      If non-zero, call :func:`faulthandler.enable` at startup.
 
    .. c:member:: wchar_t* filesystem_encoding
 
@@ -504,6 +505,9 @@ PyConfig
 
       Python home directory.
 
+      Initialized from :envvar:`PYTHONHOME` environment variable value by
+      default.
+
    .. c:member:: int import_time
 
       If non-zero, profile import time.
@@ -561,7 +565,7 @@ PyConfig
 
       :data:`sys.path`. If :c:member:`~PyConfig.module_search_paths_set` is
       equal to 0, the :c:member:`~PyConfig.module_search_paths` is overridden
-      by the function computing the :ref:`Path Configuration
+      by the function calculating the :ref:`Path Configuration
       <init-path-config>`.
 
    .. c:member:: int optimization_level
@@ -586,49 +590,56 @@ PyConfig
 
    .. c:member:: int pathconfig_warnings
 
-      If equal to 0, suppress warnings when computing the path configuration
-      (Unix only, Windows does not log any warning). Otherwise, warnings are
-      written into ``stderr``.
+      If equal to 0, suppress warnings when calculating the :ref:`Path
+      Configuration <init-path-config>` (Unix only, Windows does not log any
+      warning). Otherwise, warnings are written into ``stderr``.
 
    .. c:member:: wchar_t* prefix
 
       :data:`sys.prefix`.
 
    .. c:member:: wchar_t* program_name
 
-      Program name.
+      Program name. Used to initialize :c:member:`~PyConfig.executable`, and in
+      early error messages.
 
    .. c:member:: wchar_t* pycache_prefix
 
-      ``.pyc`` cache prefix.
+      :data:`sys.pycache_prefix`: ``.pyc`` cache prefix.
+
+      If NULL, :data:`sys.pycache_prefix` is set to ``None``.
 
    .. c:member:: int quiet
 
       Quiet mode. For example, don't display the copyright and version messages
-      even in interactive mode.
+      in interactive mode.
 
    .. c:member:: wchar_t* run_command
 
-      ``python3 -c COMMAND`` argument.
+      ``python3 -c COMMAND`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: wchar_t* run_filename
 
-      ``python3 FILENAME`` argument.
+      ``python3 FILENAME`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: wchar_t* run_module
 
-      ``python3 -m MODULE`` argument.
+      ``python3 -m MODULE`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: int show_alloc_count
 
       Show allocation counts at exit?
 
+      Set to 1 by :option:`-X showalloccount <-X>` command line option.
+
       Need a special Python build with ``COUNT_ALLOCS`` macro defined.
 
    .. c:member:: int show_ref_count
 
       Show total reference count at exit?
 
+      Set to 1 by :option:`-X showrefcount <-X>` command line option.
+
       Need a debug build of Python (``Py_REF_DEBUG`` macro must be defined).
 
    .. c:member:: int site_import
@@ -647,7 +658,7 @@ PyConfig
 
    .. c:member:: int tracemalloc
 
-      If non-zero, call :func:`tracemalloc.start`.
+      If non-zero, call :func:`tracemalloc.start` at startup.
 
    .. c:member:: int use_environment
 
@@ -669,6 +680,9 @@ PyConfig
 
       If non-zero, write ``.pyc`` files.
 
+      :data:`sys.dont_write_bytecode` is initialized to the inverted value of
+      :c:member:`~PyConfig.write_bytecode`.
+
    .. c:member:: PyWideStringList xoptions
 
       :data:`sys._xoptions`.
@@ -694,8 +708,8 @@ Function to initialize Python:
 The caller is responsible to handle exceptions (error or exit) using
 :c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
 
-``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
-``PyImport_ExtendInittab()`` is used: they must be set or called after Python
+If ``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
+``PyImport_ExtendInittab()`` are used, they must be set or called after Python
 preinitialization and before the Python initialization.
 
 Example setting the program name::
@@ -760,7 +774,7 @@ configuration, and then override some parameters::
 
         /* Append our custom search path to sys.path */
         status = PyWideStringList_Append(&config.module_search_paths,
-                                      L"/path/to/more/modules");
+                                         L"/path/to/more/modules");
         if (PyStatus_Exception(status)) {
             goto done;
         }
@@ -791,9 +805,9 @@ isolate Python from the system. For example, to embed Python into an
 application.
 
 This configuration ignores global configuration variables, environments
-variables and command line arguments (:c:member:`PyConfig.argv` is not parsed).
-The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left
-unchanged by default.
+variables, command line arguments (:c:member:`PyConfig.argv` is not parsed)
+and user site directory. The C standard streams (ex: ``stdout``) and the
+LC_CTYPE locale are left unchanged. Signal handlers are not installed.
 
 Configuration files are still used with this configuration. Set the
 :ref:`Path Configuration <init-path-config>` ("output fields") to ignore these
@@ -864,37 +878,46 @@ Path Configuration
 
 :c:type:`PyConfig` contains multiple fields for the path configuration:
 
-* Path configuration input fields:
+* Path configuration inputs:
 
   * :c:member:`PyConfig.home`
   * :c:member:`PyConfig.pathconfig_warnings`
   * :c:member:`PyConfig.program_name`
   * :c:member:`PyConfig.pythonpath_env`
+  * current working directory: to get absolute paths
+  * ``PATH`` environment variable to get the program full path
+    (from :c:member:`PyConfig.program_name`)
+  * ``__PYVENV_LAUNCHER__`` environment variable
+  * (Windows only) Application paths in the registry under
+    "Software\Python\PythonCore\X.Y\PythonPath" of HKEY_CURRENT_USER and
+    HKEY_LOCAL_MACHINE (where X.Y is the Python version).
 
 * Path configuration output fields:
 
+  * :c:member:`PyConfig.base_exec_prefix`
   * :c:member:`PyConfig.base_executable`
+  * :c:member:`PyConfig.base_prefix`
   * :c:member:`PyConfig.exec_prefix`
   * :c:member:`PyConfig.executable`
-  * :c:member:`PyConfig.prefix`
   * :c:member:`PyConfig.module_search_paths_set`,
     :c:member:`PyConfig.module_search_paths`
+  * :c:member:`PyConfig.prefix`
 
-If at least one "output field" is not set, Python computes the path
+If at least one "output field" is not set, Python calculates the path
 configuration to fill unset fields. If
 :c:member:`~PyConfig.module_search_paths_set` is equal to 0,
 :c:member:`~PyConfig.module_search_paths` is overridden and
 :c:member:`~PyConfig.module_search_paths_set` is set to 1.
 
-It is possible to completely ignore the function computing the default
+It is possible to completely ignore the function calculating the default
 path configuration by setting explicitly all path configuration output
 fields listed above. A string is considered as set even if it is non-empty.
 ``module_search_paths`` is considered as set if
 ``module_search_paths_set`` is set to 1. In this case, path
 configuration input fields are ignored as well.
 
 Set :c:member:`~PyConfig.pathconfig_warnings` to 0 to suppress warnings when
-computing the path configuration (Unix only, Windows does not log any warning).
+calculating the path configuration (Unix only, Windows does not log any warning).
 
 If :c:member:`~PyConfig.base_prefix` or :c:member:`~PyConfig.base_exec_prefix`
 fields are not set, they inherit their value from :c:member:`~PyConfig.prefix`
@@ -961,7 +984,7 @@ initialization, the core feature of the :pep:`432`:
   * Builtin exceptions;
   * Builtin and frozen modules;
   * The :mod:`sys` module is only partially initialized
-    (ex: :data:`sys.path` doesn't exist yet);
+    (ex: :data:`sys.path` doesn't exist yet).
 
 * "Main" initialization phase, Python is fully initialized:
 
@@ -987,9 +1010,9 @@ No module is imported during the "Core" phase and the ``importlib`` module is
 not configured: the :ref:`Path Configuration <init-path-config>` is only
 applied during the "Main" phase. It may allow to customize Python in Python to
 override or tune the :ref:`Path Configuration <init-path-config>`, maybe
-install a custom sys.meta_path importer or an import hook, etc.
+install a custom :data:`sys.meta_path` importer or an import hook, etc.
 
-It may become possible to compute the :ref:`Path Configuration
+It may become possible to calculatin the :ref:`Path Configuration
 <init-path-config>` in Python, after the Core phase and before the Main phase,
 which is one of the :pep:`432` motivation.
 

Doc/whatsnew/3.8.rst

@@ -1347,6 +1347,11 @@ Build and C API Changes
   parameter for indicating the number of positional-only arguments.
   (Contributed by Pablo Galindo in :issue:`37221`.)
 
+* :c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full
+  path (:c:func:`Py_GetProgramFullPath`) rather than to the program name
+  (:c:func:`Py_GetProgramName`).
+  (Contributed by Victor Stinner in :issue:`38234`.)
+
 
 Deprecated
 ==========

Include/internal/pycore_pathconfig.h

@@ -19,19 +19,26 @@ typedef struct _PyPathConfig {
     wchar_t *program_name;
     /* Set by Py_SetPythonHome() or PYTHONHOME environment variable */
     wchar_t *home;
+#ifdef MS_WINDOWS
     /* isolated and site_import are used to set Py_IsolatedFlag and
        Py_NoSiteFlag flags on Windows in read_pth_file(). These fields
        are ignored when their value are equal to -1 (unset). */
     int isolated;
     int site_import;
     /* Set when a venv is detected */
     wchar_t *base_executable;
+#endif
 } _PyPathConfig;
 
-#define _PyPathConfig_INIT \
-    {.module_search_path = NULL, \
-     .isolated = -1, \
-     .site_import = -1}
+#ifdef MS_WINDOWS
+#  define _PyPathConfig_INIT \
+      {.module_search_path = NULL, \
+       .isolated = -1, \
+       .site_import = -1}
+#else
+#  define _PyPathConfig_INIT \
+      {.module_search_path = NULL}
+#endif
 /* Note: _PyPathConfig_INIT sets other fields to 0/NULL */
 
 PyAPI_DATA(_PyPathConfig) _Py_path_config;
@@ -59,7 +66,7 @@ extern int _Py_FindEnvConfigValue(
 extern wchar_t* _Py_GetDLLPath(void);
 #endif
 
-extern PyStatus _PyPathConfig_Init(void);
+extern PyStatus _PyConfig_WritePathConfig(const PyConfig *config);
 extern void _Py_DumpPathConfig(PyThreadState *tstate);
 
 #ifdef __cplusplus

Lib/sysconfig.py

@@ -412,7 +412,7 @@ def _generate_posix_vars():
         pprint.pprint(vars, stream=f)
 
     # Create file used for sys.path fixup -- see Modules/getpath.c
-    with open('pybuilddir.txt', 'w', encoding='ascii') as f:
+    with open('pybuilddir.txt', 'w', encoding='utf8') as f:
         f.write(pybuilddir)
 
 def _init_posix(vars):