Skip to content

Commit 48d9262

Browse files
authored
bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes (GH-31034)
Define *posix_venv* and *nt_venv* sysconfig installation schemes to be used for bootstrapping new virtual environments. Add *venv* sysconfig installation scheme to get the appropriate one of the above. The schemes are identical to the pre-existing *posix_prefix* and *nt* install schemes. The venv module now uses the *venv* scheme to create new virtual environments instead of hardcoding the paths depending only on the platform. Downstream Python distributors customizing the *posix_prefix* or *nt* install scheme in a way that is not compatible with the install scheme used in virtual environments are encouraged not to customize the *venv* schemes. When Python itself runs in a virtual environment, sysconfig.get_default_scheme and sysconfig.get_preferred_scheme with `key="prefix"` returns *venv*.
1 parent cd44afc commit 48d9262

File tree

8 files changed

+211
-16
lines changed

8 files changed

+211
-16
lines changed

Doc/library/sysconfig.rst

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ Every new component that is installed using :mod:`distutils` or a
7373
Distutils-based system will follow the same scheme to copy its file in the right
7474
places.
7575

76-
Python currently supports six schemes:
76+
Python currently supports nine schemes:
7777

7878
- *posix_prefix*: scheme for POSIX platforms like Linux or macOS. This is
7979
the default scheme used when Python or a component is installed.
@@ -83,8 +83,14 @@ Python currently supports six schemes:
8383
- *posix_user*: scheme for POSIX platforms used when a component is installed
8484
through Distutils and the *user* option is used. This scheme defines paths
8585
located under the user home directory.
86+
- *posix_venv*: scheme for :mod:`Python virtual environments <venv>` on POSIX
87+
platforms; by default it is the same as *posix_prefix* .
8688
- *nt*: scheme for NT platforms like Windows.
8789
- *nt_user*: scheme for NT platforms, when the *user* option is used.
90+
- *nt_venv*: scheme for :mod:`Python virtual environments <venv>` on NT
91+
platforms; by default it is the same as *nt* .
92+
- *venv*: a scheme with values from ether *posix_venv* or *nt_venv* depending
93+
on the platform Python runs on
8894
- *osx_framework_user*: scheme for macOS, when the *user* option is used.
8995

9096
Each scheme is itself composed of a series of paths and each path has a unique
@@ -119,6 +125,9 @@ identifier. Python currently uses eight paths:
119125
This function was previously named ``_get_default_scheme()`` and
120126
considered an implementation detail.
121127

128+
.. versionchanged:: 3.11
129+
When Python runs from a virtual environment,
130+
the *venv* scheme is returned.
122131

123132
.. function:: get_preferred_scheme(key)
124133

@@ -132,6 +141,10 @@ identifier. Python currently uses eight paths:
132141

133142
.. versionadded:: 3.10
134143

144+
.. versionchanged:: 3.11
145+
When Python runs from a virtual environment and ``key="prefix"``,
146+
the *venv* scheme is returned.
147+
135148

136149
.. function:: _get_preferred_schemes()
137150

Doc/library/venv.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,11 @@ creation according to their needs, the :class:`EnvBuilder` class.
177177
``clear=True``, contents of the environment directory will be cleared
178178
and then all necessary subdirectories will be recreated.
179179

180+
.. versionchanged:: 3.11
181+
The *venv*
182+
:ref:`sysconfig installation scheme <installation_paths>`
183+
is used to construct the paths of the created directories.
184+
180185
.. method:: create_configuration(context)
181186

182187
Creates the ``pyvenv.cfg`` configuration file in the environment.

Doc/whatsnew/3.11.rst

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -361,6 +361,24 @@ sys
361361
(equivalent to ``sys.exc_info()[1]``).
362362
(Contributed by Irit Katriel in :issue:`46328`.)
363363

364+
365+
sysconfig
366+
---------
367+
368+
* Two new :ref:`installation schemes <installation_paths>`
369+
(*posix_venv*, *nt_venv* and *venv*) were added and are used when Python
370+
creates new virtual environments or when it is running from a virtual
371+
environment.
372+
The first two schemes (*posix_venv* and *nt_venv*) are OS-specific
373+
for non-Windows and Windows, the *venv* is essentially an alias to one of
374+
them according to the OS Python runs on.
375+
This is useful for downstream distributors who modify
376+
:func:`sysconfig.get_preferred_scheme`.
377+
Third party code that creates new virtual environments should use the new
378+
*venv* installation scheme to determine the paths, as does :mod:`venv`.
379+
(Contributed by Miro Hrončok in :issue:`45413`.)
380+
381+
364382
threading
365383
---------
366384

@@ -395,6 +413,20 @@ unicodedata
395413
* The Unicode database has been updated to version 14.0.0. (:issue:`45190`).
396414

397415

416+
venv
417+
----
418+
419+
* When new Python virtual environments are created, the *venv*
420+
:ref:`sysconfig installation scheme <installation_paths>` is used
421+
to determine the paths inside the environment.
422+
When Python runs in a virtual environment, the same installation scheme
423+
is the default.
424+
That means that downstream distributors can change the default sysconfig install
425+
scheme without changing behavior of virtual environments.
426+
Third party code that also creates new virtual environments should do the same.
427+
(Contributed by Miro Hrončok in :issue:`45413`.)
428+
429+
398430
fcntl
399431
-----
400432

Lib/sysconfig.py

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,53 @@
5656
'scripts': '{base}/Scripts',
5757
'data': '{base}',
5858
},
59+
# Downstream distributors can overwrite the default install scheme.
60+
# This is done to support downstream modifications where distributors change
61+
# the installation layout (eg. different site-packages directory).
62+
# So, distributors will change the default scheme to one that correctly
63+
# represents their layout.
64+
# This presents an issue for projects/people that need to bootstrap virtual
65+
# environments, like virtualenv. As distributors might now be customizing
66+
# the default install scheme, there is no guarantee that the information
67+
# returned by sysconfig.get_default_scheme/get_paths is correct for
68+
# a virtual environment, the only guarantee we have is that it is correct
69+
# for the *current* environment. When bootstrapping a virtual environment,
70+
# we need to know its layout, so that we can place the files in the
71+
# correct locations.
72+
# The "*_venv" install scheme is a scheme to bootstrap virtual environments,
73+
# essentially identical to the default posix_prefix/nt schemes.
74+
# Downstream distributors who patch posix_prefix/nt scheme are encouraged to
75+
# leave the following schemes unchanged
76+
'posix_venv': {
77+
'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
78+
'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
79+
'purelib': '{base}/lib/python{py_version_short}/site-packages',
80+
'platlib': '{platbase}/{platlibdir}/python{py_version_short}/site-packages',
81+
'include':
82+
'{installed_base}/include/python{py_version_short}{abiflags}',
83+
'platinclude':
84+
'{installed_platbase}/include/python{py_version_short}{abiflags}',
85+
'scripts': '{base}/bin',
86+
'data': '{base}',
87+
},
88+
'nt_venv': {
89+
'stdlib': '{installed_base}/Lib',
90+
'platstdlib': '{base}/Lib',
91+
'purelib': '{base}/Lib/site-packages',
92+
'platlib': '{base}/Lib/site-packages',
93+
'include': '{installed_base}/Include',
94+
'platinclude': '{installed_base}/Include',
95+
'scripts': '{base}/Scripts',
96+
'data': '{base}',
97+
},
5998
}
6099

100+
# For the OS-native venv scheme, we essentially provide an alias:
101+
if os.name == 'nt':
102+
_INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['nt_venv']
103+
else:
104+
_INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['posix_venv']
105+
61106

62107
# NOTE: site.py has copy of this function.
63108
# Sync it when modify this function.
@@ -251,6 +296,8 @@ def _get_preferred_schemes():
251296

252297

253298
def get_preferred_scheme(key):
299+
if key == 'prefix' and sys.prefix != sys.base_prefix:
300+
return 'venv'
254301
scheme = _get_preferred_schemes()[key]
255302
if scheme not in _INSTALL_SCHEMES:
256303
raise ValueError(

Lib/test/test_sysconfig.py

Lines changed: 67 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,72 @@ def test_get_preferred_schemes(self):
139139
self.assertIsInstance(schemes, dict)
140140
self.assertEqual(set(schemes), expected_schemes)
141141

142+
def test_posix_venv_scheme(self):
143+
# The following directories were hardcoded in the venv module
144+
# before bpo-45413, here we assert the posix_venv scheme does not regress
145+
binpath = 'bin'
146+
incpath = 'include'
147+
libpath = os.path.join('lib',
148+
'python%d.%d' % sys.version_info[:2],
149+
'site-packages')
150+
151+
# Resolve the paths in prefix
152+
binpath = os.path.join(sys.prefix, binpath)
153+
incpath = os.path.join(sys.prefix, incpath)
154+
libpath = os.path.join(sys.prefix, libpath)
155+
156+
self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='posix_venv'))
157+
self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='posix_venv'))
158+
159+
# The include directory on POSIX isn't exactly the same as before,
160+
# but it is "within"
161+
sysconfig_includedir = sysconfig.get_path('include', scheme='posix_venv')
162+
self.assertTrue(sysconfig_includedir.startswith(incpath + os.sep))
163+
164+
def test_nt_venv_scheme(self):
165+
# The following directories were hardcoded in the venv module
166+
# before bpo-45413, here we assert the posix_venv scheme does not regress
167+
binpath = 'Scripts'
168+
incpath = 'Include'
169+
libpath = os.path.join('Lib', 'site-packages')
170+
171+
# Resolve the paths in prefix
172+
binpath = os.path.join(sys.prefix, binpath)
173+
incpath = os.path.join(sys.prefix, incpath)
174+
libpath = os.path.join(sys.prefix, libpath)
175+
176+
self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='nt_venv'))
177+
self.assertEqual(incpath, sysconfig.get_path('include', scheme='nt_venv'))
178+
self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='nt_venv'))
179+
180+
def test_venv_scheme(self):
181+
if sys.platform == 'win32':
182+
self.assertEqual(
183+
sysconfig.get_path('scripts', scheme='venv'),
184+
sysconfig.get_path('scripts', scheme='nt_venv')
185+
)
186+
self.assertEqual(
187+
sysconfig.get_path('include', scheme='venv'),
188+
sysconfig.get_path('include', scheme='nt_venv')
189+
)
190+
self.assertEqual(
191+
sysconfig.get_path('purelib', scheme='venv'),
192+
sysconfig.get_path('purelib', scheme='nt_venv')
193+
)
194+
else:
195+
self.assertEqual(
196+
sysconfig.get_path('scripts', scheme='venv'),
197+
sysconfig.get_path('scripts', scheme='posix_venv')
198+
)
199+
self.assertEqual(
200+
sysconfig.get_path('include', scheme='venv'),
201+
sysconfig.get_path('include', scheme='posix_venv')
202+
)
203+
self.assertEqual(
204+
sysconfig.get_path('purelib', scheme='venv'),
205+
sysconfig.get_path('purelib', scheme='posix_venv')
206+
)
207+
142208
def test_get_config_vars(self):
143209
cvars = get_config_vars()
144210
self.assertIsInstance(cvars, dict)
@@ -267,7 +333,7 @@ def test_get_config_h_filename(self):
267333
self.assertTrue(os.path.isfile(config_h), config_h)
268334

269335
def test_get_scheme_names(self):
270-
wanted = ['nt', 'posix_home', 'posix_prefix']
336+
wanted = ['nt', 'posix_home', 'posix_prefix', 'posix_venv', 'nt_venv', 'venv']
271337
if HAS_USER_BASE:
272338
wanted.extend(['nt_user', 'osx_framework_user', 'posix_user'])
273339
self.assertEqual(get_scheme_names(), tuple(sorted(wanted)))

Lib/test/test_venv.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -236,6 +236,20 @@ def test_prefixes(self):
236236
out, err = check_output(cmd)
237237
self.assertEqual(out.strip(), expected.encode(), prefix)
238238

239+
@requireVenvCreate
240+
def test_sysconfig_preferred_and_default_scheme(self):
241+
"""
242+
Test that the sysconfig preferred(prefix) and default scheme is venv.
243+
"""
244+
rmtree(self.env_dir)
245+
self.run_with_capture(venv.create, self.env_dir)
246+
envpy = os.path.join(self.env_dir, self.bindir, self.exe)
247+
cmd = [envpy, '-c', None]
248+
for call in ('get_preferred_scheme("prefix")', 'get_default_scheme()'):
249+
cmd[2] = 'import sysconfig; print(sysconfig.%s)' % call
250+
out, err = check_output(cmd)
251+
self.assertEqual(out.strip(), b'venv', err)
252+
239253
if sys.platform == 'win32':
240254
ENV_SUBDIRS = (
241255
('Scripts',),

Lib/venv/__init__.py

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,15 @@ def clear_directory(self, path):
9393
elif os.path.isdir(fn):
9494
shutil.rmtree(fn)
9595

96+
def _venv_path(self, env_dir, name):
97+
vars = {
98+
'base': env_dir,
99+
'platbase': env_dir,
100+
'installed_base': env_dir,
101+
'installed_platbase': env_dir,
102+
}
103+
return sysconfig.get_path(name, scheme='venv', vars=vars)
104+
96105
def ensure_directories(self, env_dir):
97106
"""
98107
Create the directories for the environment.
@@ -120,27 +129,21 @@ def create_if_needed(d):
120129
context.executable = executable
121130
context.python_dir = dirname
122131
context.python_exe = exename
123-
if sys.platform == 'win32':
124-
binname = 'Scripts'
125-
incpath = 'Include'
126-
libpath = os.path.join(env_dir, 'Lib', 'site-packages')
127-
else:
128-
binname = 'bin'
129-
incpath = 'include'
130-
libpath = os.path.join(env_dir, 'lib',
131-
'python%d.%d' % sys.version_info[:2],
132-
'site-packages')
133-
context.inc_path = path = os.path.join(env_dir, incpath)
134-
create_if_needed(path)
132+
binpath = self._venv_path(env_dir, 'scripts')
133+
incpath = self._venv_path(env_dir, 'include')
134+
libpath = self._venv_path(env_dir, 'purelib')
135+
136+
context.inc_path = incpath
137+
create_if_needed(incpath)
135138
create_if_needed(libpath)
136139
# Issue 21197: create lib64 as a symlink to lib on 64-bit non-OS X POSIX
137140
if ((sys.maxsize > 2**32) and (os.name == 'posix') and
138141
(sys.platform != 'darwin')):
139142
link_path = os.path.join(env_dir, 'lib64')
140143
if not os.path.exists(link_path): # Issue #21643
141144
os.symlink('lib', link_path)
142-
context.bin_path = binpath = os.path.join(env_dir, binname)
143-
context.bin_name = binname
145+
context.bin_path = binpath
146+
context.bin_name = os.path.relpath(binpath, env_dir)
144147
context.env_exe = os.path.join(binpath, exename)
145148
create_if_needed(binpath)
146149
# Assign and update the command to use when launching the newly created
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Define *posix_venv* and *nt_venv*
2+
:ref:`sysconfig installation schemes <installation_paths>`
3+
to be used for bootstrapping new virtual environments.
4+
Add *venv* sysconfig installation scheme to get the appropriate one of the above.
5+
The schemes are identical to the pre-existing
6+
*posix_prefix* and *nt* install schemes.
7+
The :mod:`venv` module now uses the *venv* scheme to create new virtual environments
8+
instead of hardcoding the paths depending only on the platform. Downstream
9+
Python distributors customizing the *posix_prefix* or *nt* install
10+
scheme in a way that is not compatible with the install scheme used in
11+
virtual environments are encouraged not to customize the *venv* schemes.
12+
When Python itself runs in a virtual environment,
13+
:func:`sysconfig.get_default_scheme` and
14+
:func:`sysconfig.get_preferred_scheme` with ``key="prefix"`` returns
15+
*venv*.

0 commit comments

Comments
 (0)