Skip to content

bpo-33078 - FIX queue size on pickling error #6119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Mar 21, 2018
Merged

bpo-33078 - FIX queue size on pickling error #6119

merged 4 commits into from
Mar 21, 2018

Conversation

tomMoral
Copy link
Contributor

@tomMoral tomMoral commented Mar 15, 2018 •
edited by bedevere-bot
Loading

The Queue._feed does not properly handle the size of the Queue on errors. This can lead to a situation where the Queue is considered as Full when it is empty. Here is a reproducing script:

import multiprocessing as mp

q = mp.Queue(1)


class FailPickle():
    def __reduce__(self):
        raise ValueError()

q.put(FailPickle())
print("Queue is full:", q.full())
q.put(0)
print(f"Got result: {q.get()}")

This PR fixes this behavior.

https://bugs.python.org/issue33078

@tomMoral tomMoral requested a review from rhettinger as a code owner March 15, 2018 06:42
pitrou
pitrou requested changes Mar 20, 2018
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! A few comments below.

Lib/multiprocessing/queues.py Outdated
@@ -255,6 +255,7 @@ def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
info('error in queue thread: %s', e)
return
else:
queue_sem.release()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment explaining why you need to adjust the semaphore's value?

Lib/test/_test_multiprocessing.py Outdated
q.put(NotSerializable())
q.put(True)
# bpo-30595: use a timeout of 1 second for slow buildbots
self.assertTrue(q.get(timeout=1.0))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should you add a check for q.qsize() as well?

Misc/NEWS.d/next/Library/2018-03-15-07-38-00.bpo-33078.RmjUF5.rst Outdated
@@ -0,0 +1,2 @@
Fix the size handeling in multiprocessing.Queue when a pickling error
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: "handling".

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pitrou
pitrou reviewed Mar 21, 2018
View reviewed changes
Lib/test/_test_multiprocessing.py
q = self.Queue(maxsize=1)
q.put(NotSerializable())
q.put(True)
self.assertEqual(q.qsize(), 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this check correct? Depending on when the worker thread triggers, it might return 2, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is correct because maxsize=1 so the second q.put should return only once the failure has been triggered.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point, thank you.

@pitrou pitrou merged commit e2f33ad into python:master Mar 21, 2018
@miss-islington
Copy link
Contributor

Thanks @tomMoral for the PR, and @pitrou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

@bedevere-bot
Copy link

GH-6178 is a backport of this pull request to the 3.7 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 21, 2018
@tomMoral @miss-islington
bpo-33078 - Fix queue size on pickling error (pythonGH-6119)
0e919f9 
(cherry picked from commit e2f33ad)

Co-authored-by: Thomas Moreau <[email protected]>
pitrou pushed a commit that referenced this pull request Mar 21, 2018
@miss-islington @tomMoral @pitrou
bpo-33078 - Fix queue size on pickling error (GH-6119) (GH-6178)
bb5b529 
(cherry picked from commit e2f33ad)

Co-authored-by: Thomas Moreau <[email protected]>
@tomMoral tomMoral mentioned this pull request Mar 21, 2018
Merged
jo2y pushed a commit to jo2y/cpython that referenced this pull request Mar 23, 2018
@tomMoral @jo2y
bpo-33078 - Fix queue size on pickling error (pythonGH-6119)
7249ba0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pitrou pitrou pitrou approved these changes

@rhettinger rhettinger Awaiting requested review from rhettinger

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@tomMoral @bedevere-bot @miss-islington @pitrou @the-knights-who-say-ni