bpo-33735: Fix test_multiprocessing random failure #8059

vstinner · 2018-07-03T11:06:17Z

When hunting for memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.

Modify test_imap_unordered() to be closer to test_imap():

Only test 10 numbers instead of 100: it's a pool of 4 processes, so
10 is enough to test at least one number per process
Use chunksize=100 instead of chunksize=53 to mimick test_imap()

https://bugs.python.org/issue33735

When hunting for memory leaks using -R 3:3, test_imap_unordered() of test_multiprocessing leaks randomly a few memory blocks. It is a false alarm: when testing using -R 3:20 for example, no leak is detected. Modify test_imap_unordered() to be closer to test_imap(): * Only test 10 numbers instead of 100: it's a pool of 4 processes, so 10 is enough to test at least one number per process * Use chunksize=100 instead of chunksize=53 to mimick test_imap()

vstinner · 2018-07-03T11:10:43Z

I validated on my laptop that the change fixes the false alarm.

I modified locally regrtest to make it stricter:

diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py
index 6724488fcf..a3c50e21e0 100644
--- a/Lib/test/libregrtest/refleak.py
+++ b/Lib/test/libregrtest/refleak.py
@@ -101,7 +101,7 @@ def dash_R(the_module, test, indirect_test, huntrleaks):
     failed = False
     for deltas, item_name, checker in [
         (rc_deltas, 'references', check_rc_deltas),
-        (alloc_deltas, 'memory blocks', check_rc_deltas),
+        (alloc_deltas, 'memory blocks', check_fd_deltas),
         (fd_deltas, 'file descriptors', check_fd_deltas)
     ]:
         # ignore warmup runs

On master, I can easily reproduce the issue on my idle laptop:

$ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:20
Run tests sequentially
0:00:00 load avg: 0.78 [1/1] test_multiprocessing_spawn
beginning 23 repetitions
12345678901234567890123
.......................
test_multiprocessing_spawn leaked [5, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=8
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 11 sec 112 ms
Tests result: FAILURE

With the change, the bug goes away:

vstinner@apu$ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:20
Run tests sequentially
0:00:00 load avg: 0.92 [1/1] test_multiprocessing_spawn
beginning 23 repetitions
12345678901234567890123
.......................
test_multiprocessing_spawn leaked [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=1
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 9 sec 967 ms
Tests result: FAILURE

The Refleak buildbots use -R 3:3, so only the first 3 numbers matter. Without the change, the first 3 numbers are non-zero, whereas they only one is non-zero with the change.

vstinner · 2018-07-03T11:16:58Z

I also validated on the Gentoo Refleak that the change fixes the false alarm.

master branch: 2 failures on 5 runs.

pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3
Run tests sequentially
0:00:00 load avg: 0.41 [1/1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
test_multiprocessing_spawn leaked [2, 2, 1] memory blocks, sum=5
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 6 sec 539 ms
Tests result: FAILURE

With the change: 0 failure on 5 runs.

I also ran the following command with the change:

pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3 -F
Run tests sequentially
0:00:00 load avg: 2.81 [  1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
(...)
0:02:59 load avg: 5.17 [ 25] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
0:03:09 load avg: 5.45 [ 26] test_multiprocessing_spawn
beginning 6 repetitions
123456
^C
== Tests result: INTERRUPTED ==

Test suite interrupted by signal SIGINT.
0 tests omitted:


25 tests OK.

Total duration: 3 min 10 sec
Tests result: INTERRUPTED

I interrupted the test after 25 success runs.

miss-islington · 2018-07-03T11:20:38Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 2.7, 3.6, 3.7.
🐍🍒⛏🤖

bedevere-bot · 2018-07-03T11:21:58Z

GH-8060 is a backport of this pull request to the 3.7 branch.

When hunting memory leaks using -R 3:3, test_imap_unordered() of test_multiprocessing leaks randomly a few memory blocks. It is a false alarm: when testing using -R 3:20 for example, no leak is detected. Modify test_imap_unordered() to be closer to test_imap(): * Only test 10 numbers instead of 1000: it's a pool of 4 processes, so 10 is enough to test at least one number per process * Use chunksize=100 instead of chunksize=53 to mimick test_imap() (cherry picked from commit 23401fb) Co-authored-by: Victor Stinner <[email protected]>

miss-islington · 2018-07-03T11:22:42Z

Sorry, @vstinner, I could not cleanly backport this to 2.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 23401fb960bb94e6ea62d2999527968d53d3fc65 2.7

bedevere-bot · 2018-07-03T11:23:24Z

GH-8061 is a backport of this pull request to the 2.7 branch.

bedevere-bot · 2018-07-03T11:23:48Z

GH-8062 is a backport of this pull request to the 3.6 branch.

When hunting memory leaks using -R 3:3, test_imap_unordered() of test_multiprocessing leaks randomly a few memory blocks. It is a false alarm: when testing using -R 3:20 for example, no leak is detected. Modify test_imap_unordered() to be closer to test_imap(): * Only test 10 numbers instead of 1000: it's a pool of 4 processes, so 10 is enough to test at least one number per process * Use chunksize=100 instead of chunksize=53 to mimick test_imap() (cherry picked from commit 23401fb) Co-authored-by: Victor Stinner <[email protected]>

When hunting memory leaks using -R 3:3, test_imap_unordered() of test_multiprocessing leaks randomly a few memory blocks. It is a false alarm: when testing using -R 3:20 for example, no leak is detected. Modify test_imap_unordered() to be closer to test_imap(): * Only test 10 numbers instead of 1000: it's a pool of 4 processes, so 10 is enough to test at least one number per process * Use chunksize=100 instead of chunksize=53 to mimick test_imap() (cherry picked from commit 23401fb)

When hunting memory leaks using -R 3:3, test_imap_unordered() of test_multiprocessing leaks randomly a few memory blocks. It is a false alarm: when testing using -R 3:20 for example, no leak is detected. Modify test_imap_unordered() to be closer to test_imap(): * Only test 10 numbers instead of 1000: it's a pool of 4 processes, so 10 is enough to test at least one number per process * Use chunksize=100 instead of chunksize=53 to mimick test_imap() (cherry picked from commit 23401fb) Co-authored-by: Victor Stinner <[email protected]>

vstinner added needs backport to 3.6 labels Jul 3, 2018

the-knights-who-say-ni added the CLA signed label Jul 3, 2018

bedevere-bot added the awaiting merge label Jul 3, 2018

vstinner added the skip news label Jul 3, 2018

vstinner added the needs backport to 2.7 label Jul 3, 2018

vstinner merged commit 23401fb into python:master Jul 3, 2018

bedevere-bot removed the awaiting merge label Jul 3, 2018

vstinner deleted the mp_imap_unordered branch July 3, 2018 11:20

bedevere-bot removed the needs backport to 3.7 label Jul 3, 2018

miss-islington assigned vstinner Jul 3, 2018

bedevere-bot removed the needs backport to 2.7 label Jul 3, 2018

bedevere-bot removed the needs backport to 3.6 label Jul 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-33735: Fix test_multiprocessing random failure #8059

bpo-33735: Fix test_multiprocessing random failure #8059

Uh oh!

vstinner commented Jul 3, 2018 •

edited by bedevere-bot

Loading

Uh oh!

vstinner commented Jul 3, 2018

Uh oh!

vstinner commented Jul 3, 2018 •

edited

Loading

Uh oh!

miss-islington commented Jul 3, 2018

Uh oh!

bedevere-bot commented Jul 3, 2018

Uh oh!

miss-islington commented Jul 3, 2018

Uh oh!

bedevere-bot commented Jul 3, 2018

Uh oh!

bedevere-bot commented Jul 3, 2018

Uh oh!

Uh oh!

Uh oh!

bpo-33735: Fix test_multiprocessing random failure #8059

bpo-33735: Fix test_multiprocessing random failure #8059

Uh oh!

Conversation

vstinner commented Jul 3, 2018 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Jul 3, 2018

Uh oh!

vstinner commented Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miss-islington commented Jul 3, 2018

Uh oh!

bedevere-bot commented Jul 3, 2018

Uh oh!

miss-islington commented Jul 3, 2018

Uh oh!

bedevere-bot commented Jul 3, 2018

Uh oh!

bedevere-bot commented Jul 3, 2018

Uh oh!

Uh oh!

vstinner commented Jul 3, 2018 •

edited by bedevere-bot

Loading

vstinner commented Jul 3, 2018 •

edited

Loading