Skip to content

gh-121973: Fix flaky test_pyrepl tests #122140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 23, 2024
Merged

Conversation

colesbury
Copy link
Contributor

@colesbury colesbury commented Jul 22, 2024

This fixes the flakiness in:

  • test_inspect_keeps_globals_from_inspected_file
  • test_inspect_keeps_globals_from_inspected_module

The output already includes newlines. Adding newlines for every entry in the output list introduces non-determinism because it added '\n' in places where stdout is flushed or some buffer becomes full.

The regex also needed to be updated because pyrepl includes control characters -- the visible output on each line doesn't immediately follow a newline character.

This fixes the flakiness in:
* test_inspect_keeps_globals_from_inspected_file
* test_inspect_keeps_globals_from_inspected_module

The output already includes newlines. Adding newlines for every entry in
the output list introduces non-determinism because it added '\n' in
places where stdout is flushed or some buffer becomes full.

The regex also needed to be updated because pyrepl includes control
characters -- the visible output on each line doesn't immediately follow
a newline character.
@colesbury colesbury added tests Tests in the Lib/test dir needs backport to 3.13 bugs and security fixes topic-repl Related to the interactive shell labels Jul 22, 2024
@colesbury colesbury requested a review from AlexWaygood July 22, 2024 20:55
@colesbury
Copy link
Contributor Author

!buildbot s390x Fedora Refleaks

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @colesbury for commit d23344e 🤖

The command will test the builders whose names match following regular expression: s390x Fedora Refleaks

The builders matched are:

  • s390x Fedora Refleaks PR

@ambv
Copy link
Contributor

ambv commented Jul 23, 2024

The output already includes newlines.

Good observation!

Adding newlines for every entry in the output list introduces non-determinism (...)

That looks like it's surfacing a different problem then, because run_repl executes the subprocess with -u (unbuffered). So there shouldn't be non-determinism here. But sure, removing the spurious newlines makes sense regardless, so let's do that.

The regex also needed to be updated (...)

Without the extra newlines that may be so. But the new regex is a bit weird. There are no braces in the output, that's an f-string. It's 2am now, I'll come up with something in the morning.

@ambv
Copy link
Contributor

ambv commented Jul 23, 2024

(maybe something as simple as a word boundary would work, i.e. \b instead of [^{{]; but I'll test it in the morning)

@colesbury
Copy link
Contributor Author

I think -u just disables Python's buffering in TextIOWrapper. It doesn't make things deterministic. Sometimes the read() boundaries match the newlines, but if once process is slower you might read multiple lines in a single os.read(). Or you might be limited by the 1024 bytes in the os.read() call.

There are braces in the output because the "output" also contains the input, which contains an f-string that we don't want to match. That's apparently how ptys work -- I don't make the rules!

A nicer regex would be welcome.

@ambv
Copy link
Contributor

ambv commented Jul 23, 2024

That's apparently how ptys work

Ah, of course, you need to show user input in the terminal, of course. It works as regular output because of possible cursor changes, control characters, syntax highlighting, and stuff.

In any case, I added a caret before the variable name in the output so we could match it. That way if we ever want to query a variable bar but there's foobar in the output, it won't match incorrectly.

@ambv
Copy link
Contributor

ambv commented Jul 23, 2024

!buildbot s390x Fedora Refleaks

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @ambv for commit 1c4be95 🤖

The command will test the builders whose names match following regular expression: s390x Fedora Refleaks

The builders matched are:

  • s390x Fedora Refleaks PR

@ambv
Copy link
Contributor

ambv commented Jul 23, 2024

Refleaks buildbot's happy with it, and so am I.

@ambv ambv merged commit 2c1b1e7 into python:main Jul 23, 2024
32 checks passed
@miss-islington-app
Copy link

Thanks @colesbury for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 23, 2024
This fixes the flakiness in:
* test_inspect_keeps_globals_from_inspected_file
* test_inspect_keeps_globals_from_inspected_module

The output already includes newlines. Adding newlines for every entry in
the output list introduces non-determinism because it added '\n' in
places where stdout is flushed or some buffer becomes full.

The regex also needed to be updated because pyrepl includes control
characters -- the visible output on each line doesn't immediately follow
a newline character.

(cherry picked from commit 2c1b1e7)

Co-authored-by: Sam Gross <[email protected]>
Co-authored-by: Łukasz Langa <[email protected]>
@bedevere-app
Copy link

bedevere-app bot commented Jul 23, 2024

GH-122173 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jul 23, 2024
ambv added a commit that referenced this pull request Jul 23, 2024
This fixes the flakiness in:
* test_inspect_keeps_globals_from_inspected_file
* test_inspect_keeps_globals_from_inspected_module

The output already includes newlines. Adding newlines for every entry in
the output list introduces non-determinism because it added '\n' in
places where stdout is flushed or some buffer becomes full.

The regex also needed to be updated because pyrepl includes control
characters -- the visible output on each line doesn't immediately follow
a newline character.

(cherry picked from commit 2c1b1e7)

Co-authored-by: Sam Gross <[email protected]>
Co-authored-by: Łukasz Langa <[email protected]>
@colesbury colesbury deleted the gh-121973-pyrepl-flaky branch July 23, 2024 14:36
nohlson pushed a commit to nohlson/cpython that referenced this pull request Jul 24, 2024
This fixes the flakiness in:
* test_inspect_keeps_globals_from_inspected_file
* test_inspect_keeps_globals_from_inspected_module

The output already includes newlines. Adding newlines for every entry in
the output list introduces non-determinism because it added '\n' in
places where stdout is flushed or some buffer becomes full.

The regex also needed to be updated because pyrepl includes control
characters -- the visible output on each line doesn't immediately follow
a newline character.

Co-authored-by: Łukasz Langa <[email protected]>
nohlson pushed a commit to nohlson/cpython that referenced this pull request Jul 24, 2024
This fixes the flakiness in:
* test_inspect_keeps_globals_from_inspected_file
* test_inspect_keeps_globals_from_inspected_module

The output already includes newlines. Adding newlines for every entry in
the output list introduces non-determinism because it added '\n' in
places where stdout is flushed or some buffer becomes full.

The regex also needed to be updated because pyrepl includes control
characters -- the visible output on each line doesn't immediately follow
a newline character.

Co-authored-by: Łukasz Langa <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip news tests Tests in the Lib/test dir topic-repl Related to the interactive shell
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants