bpo-37412: Fix os.getcwd() for long path on Windows

[vstinner]

• Fix test for integer overflow.
• Add an unit test.

https://bugs.python.org/issue37412

[vstinner]

Oh. I first used min_len = 1000 to avoid ENAMETOOLONG, but then I realized that min_len = 1000 is not enough to test the regression fixed by this change. So I modified my change to use min_len = 2000.

[vstinner]

@eryksun: Would you mind to review my PR?

[vstinner]

Oh, test_getcwd_long_path() failed on AppVeyor with:

FileNotFoundError: [WinError 206] The filename or extension is too long: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmpqlshuxlz\\python_test_dir_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

mkdir() fails on a path of 251 characters. On my Windows 10 VM, the test works as expected.

pythoninfo says:

platform.platform: Windows-10-10.0.14393-SP0
sys.windowsversion: sys.getwindowsversion(major=10, minor=0, build=14393, platform=2, service_pack='')

[eryksun]

The CRT in Windows doesn't use ENAMETOOLONG. In the Windows API, depending on the context, a path that's too long fails with either ERROR_PATH_NOT_FOUND (3) or ERROR_FILENAME_EXCED_RANGE (206). Both get mapped to the errno value ENOENT.

[vstinner]

I modified my PR to catch also FileNotFoundError. The test stops on such error: we reached the maximum path length.

[vstinner]

The CRT in Windows doesn't use ENAMETOOLONG.

"if exc.errno == errno.ENAMETOOLONG:" is for Linux ;-)

[eryksun]

Oh, test_getcwd_long_path() failed on AppVeyor with:

FileNotFoundError: [WinError 206] The filename or extension is too long: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmpqlshuxlz\\python_test_dir_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

mkdir() fails on a path of 251 characters. On my Windows 10 VM, the test works as expected.

The classic DOS path length limit is MAX_PATH (260) characters. In practice it's 259 characters when opening or creating a file, not counting the null terminator; or 258 characters when setting the current directory, not counting the trailing path separator and null terminator; or 246 or 247 characters when creating a directory, leaving space for a path separator, 12-character short filename (DOS 8.3), and the null terminator.

UNC paths (e.g. "\\localhost\C$\Temp") have the same length limit as DOS drive-letter paths. Normalized device paths that start with "\\.\" also have the same limit. Unless we're in Windows 10 with long paths enabled, only non-normalized "\\?\" device paths are extended to the native limit of 32,767 characters. Since "\\?\" paths aren't normalized, they can only use backslash as the path separator. Also, for Windows 7 and older versions, "\\?\" paths must be Unicode because the routine that decodes from ANSI to Unicode is limited to MAX_PATH in older versions of Windows.

That said, "\\?\" paths are not supported for the current directory. The only way the current directory can exceed MAX_PATH - 2 is to enable long-path support in Windows 10, which removes the MAX_PATH limit in most cases. This is configured by a registry DWORD value named "LongPathsEnabled" in the key "HKLM\SYSTEM\CurrentControlSet\Control\FileSystem". The application manifest also has to enable the "longPathAware" setting. Python 3.6+ enables this in the python[w].exe embedded manifest.

There's no documented way to check whether long paths are enabled in the current process. Using public APIs, we can only check indirectly, e.g. by trying to create a long path in the temp directory. We can check this directly if we're willing to use the undocumented function RtlAreLongPathsEnabled. The following code is from an answer I wrote on Stack Overflow:

import ctypes
ntdll = ctypes.WinDLL('ntdll')

if hasattr(ntdll, 'RtlAreLongPathsEnabled'):

    ntdll.RtlAreLongPathsEnabled.restype = ctypes.c_ubyte
    ntdll.RtlAreLongPathsEnabled.argtypes = ()

    def are_long_paths_enabled():
        return bool(ntdll.RtlAreLongPathsEnabled())

else:

    def are_long_paths_enabled():
        return False

(FYI, a notable exception with long-path support is that CreateProcessW does not allow a long path as the current directory of a child process, neither implicitly inherited nor explicitly set. This causes subprocess.Popen to mysteriously fail with an invalid-parameter error if a long path is set as the current directory.)

[vstinner]

We can check this directly if we're willing to use the undocumented function RtlAreLongPathsEnabled.

I'm not sure that it's worth it to attempt to get the exact maximum path. My design of my test is more "try to build a long path, if we cannot, stop the test". We have enough buildbots to hope that one has long paths enabled ;-)

[eryksun]

I'm not sure that it's worth it to attempt to get the exact maximum path. My design of my test is more "try to build a long path, if we cannot, stop the test". We have enough buildbots to hope that one has long paths enabled ;-)

My comment was intended to provide a general background in order to understand why a Python 3.6+ process doesn't always support long paths in Windows 10 (depends on the system LongPathsEnabled policy at startup) and why, with long paths not enabled, the system failed to create a directory when the fully-qualified path was only 251 characters (the path length of a new directory is limited to 247 characters, unless long paths are enabled or it's a "\\?\" path).

[vstinner]

@eryksun: So what do you think of my test, is it good enough to check for the regression?

I'm not sure about using "\?", "\localhost\C$\Temp" or "\.". They might be used to create a new sub-directory (mkdir), but I don't see how they could be used to get the current directory: os.getcwd() (GetCurrentDirectoryW) takes no argument. So I understand that it's fine to stop the test when we fail to build a subdirectory even if we didn't reach min_len (2000 characters).

We have enough buildbots to hope that one has long paths enabled ;-)

I wrote PR #14434 to check if it's the case :-)

[vstinner]

Oh, the test failed on Windows jobs of Azure Pipelines:

ERROR: test_getcwd_long_path (test.test_os.MiscTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\s\lib\test\test_os.py", line 130, in test_getcwd_long_path
    os.chdir(path)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\tmp5tfbc984\\python_test_dir_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\python_test_dir_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

chdir() failed on a path of 452 characters:

>>> len("'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\tmp5tfbc984\\python_test_dir_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\python_test_dir_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'")
452

whereas the previous iteration with 250 characters was fine:

>>> len("'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\tmp5tfbc984\\python_test_dir_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
250

[vstinner]

Oh, the test failed on Windows jobs of Azure Pipelines (...)
chdir() failed on a path of 452 characters: ...

According to PR #14434 (RtlAreLongPathsEnabled), long paths are enabled on this CI.

[eryksun]

It seems the os.chdir call needs to be in the try block. Though it's weird that CreateDirectoryW succeeds but SetCurrentDirectoryW fails. If long paths are not enabled, then CreateDirectoryW is limited to 247 characters unless we use a "\\?\" path, and SetCurrentDirectoryW is limited to 258 characters regardless of the path type. Maybe it's a bug in Windows 10 1607 (build 14393). Long-path support was added in this version.

[vstinner]

Ok, the test now pass on AppVeyor and Azure Pipelines (win32, win64).

The macOS job of Azure Pipelines failed, but it's unrelated: https://bugs.python.org/issue37245

[vstinner]

I rebased my PR, squashed commits, I fixed a typo in test comment, and I elaborated the test comment.

[eryksun]

To be clear, UNC paths have the same length limit as DOS drive paths. "\\?\" paths remove the limit because they bypass normalization, but these are not UNC paths.

[vstinner]

Hum. I rephased the comment in a more generic way.

"On Windows, MAX_PATH is defined as 260 characters, but Windows supports longer path if longer paths support is enabled."

[miss-islington]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

[bedevere-bot]

GH-14451 is a backport of this pull request to the 3.8 branch.

[vstinner]

The test pass on all CI, I fixed the test comment.

I decided to not wait for @eryksun just to make sure that the fix is merged before the weekend, otherwise I would forget and there is an incoming 3.8beta2 release ;-)