stubtest: Fix wrong assumption about relative path (#12282)

Fixes #11019

`run_stubtest` creates temp directory and prepend `sys.path` with relative path (dot `.`) wrongly assuming that the dot will be resolved to absolute path on *every* import attempt.

But in Python dot(`.`) in sys.path is actually resolved by PathFinder and cached in `sys.path_importer_cache` like:
```
sys.path_importer_cache['.']
FileFinder('/somepath/.')
```
later calls for `find_module` return None and import of `test_module` fails. This resulted in only the first test in stubtest's suite passed in non-pytest-xdist environments.

This issue was hidden with bug or feature in pytest-xdist < 2.3.0:
pytest-dev/pytest-xdist#421

It was fixed in pytest-xdist 2.3.0:
pytest-dev/pytest-xdist#667

- sys.path for pytest-xdist < 2.3.0
  `'.', 'project_path', ''`

- sys.path for pytest-xdist >= 2.3.0 or without xdist
  `'.', 'project_path'`

In Python for denoting cwd the empty path `''` can be used as a special case, but for readability `sys.path` is prepended with resolved absolute path of temp directory. Also it's essential to restore back `sys.path` after a test to not break subsequent tests.

Signed-off-by: Stanislav Levin <[email protected]>

Author: stanislavlevin

mypy/test/teststubtest.py

@@ -16,13 +16,20 @@
 
 
 @contextlib.contextmanager
-def use_tmp_dir() -> Iterator[None]:
+def use_tmp_dir(mod_name: str) -> Iterator[None]:
     current = os.getcwd()
+    current_syspath = sys.path[:]
     with tempfile.TemporaryDirectory() as tmp:
         try:
             os.chdir(tmp)
+            if sys.path[0] != tmp:
+                sys.path.insert(0, tmp)
             yield
         finally:
+            sys.path = current_syspath[:]
+            if mod_name in sys.modules:
+                del sys.modules[mod_name]
+
             os.chdir(current)
 
 
@@ -92,7 +99,7 @@ def staticmethod(f: T) -> T: ...
 def run_stubtest(
     stub: str, runtime: str, options: List[str], config_file: Optional[str] = None,
 ) -> str:
-    with use_tmp_dir():
+    with use_tmp_dir(TEST_MODULE_NAME):
         with open("builtins.pyi", "w") as f:
             f.write(stubtest_builtins_stub)
         with open("typing.pyi", "w") as f:
@@ -105,11 +112,6 @@ def run_stubtest(
             with open("{}_config.ini".format(TEST_MODULE_NAME), "w") as f:
                 f.write(config_file)
             options = options + ["--mypy-config-file", "{}_config.ini".format(TEST_MODULE_NAME)]
-        if sys.path[0] != ".":
-            sys.path.insert(0, ".")
-        if TEST_MODULE_NAME in sys.modules:
-            del sys.modules[TEST_MODULE_NAME]
-
         output = io.StringIO()
         with contextlib.redirect_stdout(output):
             test_stubs(

test-requirements.txt

@@ -7,8 +7,8 @@ flake8-pyi>=20.5
 lxml>=4.4.0; python_version<'3.11'
 psutil>=4.0
 # pytest 6.2.3 does not support Python 3.10
-pytest>=6.2.4,<7.0.0
-pytest-xdist>=1.34.0,<2.0.0
+pytest>=6.2.4
+pytest-xdist>=1.34.0
 pytest-forked>=1.3.0,<2.0.0
 pytest-cov>=2.10.0,<3.0.0
 py>=1.5.2