Skip to content

bpo-35589: Prevent buffer copy in sock_sendall() #11418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 16, 2019
Merged

bpo-35589: Prevent buffer copy in sock_sendall() #11418

merged 3 commits into from
May 16, 2019

Conversation

asvetlov
Copy link
Contributor

@asvetlov asvetlov commented Jan 2, 2019 •
edited by bedevere-bot
Loading

No NEWs is needed since the problem was introduced on master only and never released.

https://bugs.python.org/issue35589

@asvetlov asvetlov requested a review from 1st1 as a code owner January 2, 2019 21:29
eamanu
eamanu approved these changes Jan 3, 2019
View reviewed changes
Copy link
Contributor

@eamanu eamanu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eamanu
Copy link
Contributor

eamanu commented Jan 3, 2019

Some tests are not required?

@asvetlov
Copy link
Contributor Author

asvetlov commented Jan 3, 2019

There are tests for sending large data already.
An explicit test for memory consumption is not robust.
We can add a mocked test but the test is useless almost: it checks mocks correctness only :)

1st1
1st1 reviewed Jan 3, 2019
View reviewed changes
Lib/asyncio/selector_events.py Outdated

fut = self.create_future()
fd = sock.fileno()
fut.add_done_callback(
functools.partial(self._sock_write_done, fd))
self.add_writer(fd, self._sock_sendall, fut, sock, data)
# use a trick with a list in closure to store a mutable state
self.add_writer(fd, self._sock_sendall, fut, sock, data, [n])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But why do we want to have a mutable state here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot pass just n -- need either a mutable object or re-add a writer after every sock.send() call until all data is sent.

methane
methane approved these changes Feb 5, 2019
View reviewed changes
Copy link
Member

@methane methane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Theorically speaking, memoryview should be closed after using.
But it is ignored often. Unlike file.close(), not closing memoryview is not a real problem on PyPy.

I want portable version of os.pwrite(fd, buf, start), instead of abusing memoryview though...

@miss-islington miss-islington merged commit 6e78900 into python:master May 16, 2019
@asvetlov
Copy link
Contributor Author

Sorry, forgot to merge it 3 months ago

@asvetlov asvetlov deleted the fast-sendall branch May 16, 2019 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@methane methane methane approved these changes

@1st1 1st1 1st1 approved these changes

@eamanu eamanu eamanu approved these changes

Assignees
No one assigned
Labels
skip news topic-asyncio
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@asvetlov @eamanu @methane @1st1 @the-knights-who-say-ni @bedevere-bot @miss-islington