Skip to content

bpo-35721: Close socket pair if Popen in _UnixSubprocessTransport fails #11553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 20, 2019
Merged

bpo-35721: Close socket pair if Popen in _UnixSubprocessTransport fails #11553

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions Lib/asyncio/unix_events.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -757,12 +757,18 @@ def _start(self, args, shell, stdin, stdout, stderr, bufsize, **kwargs):
# other end). Notably this is needed on AIX, and works
# just fine on other platforms.
stdin, stdin_w = socket.socketpair()
self._proc = subprocess.Popen(
args, shell=shell, stdin=stdin, stdout=stdout, stderr=stderr,
universal_newlines=False, bufsize=bufsize, **kwargs)
if stdin_w is not None:
stdin.close()
self._proc.stdin = open(stdin_w.detach(), 'wb', buffering=bufsize)
try:
self._proc = subprocess.Popen(
args, shell=shell, stdin=stdin, stdout=stdout, stderr=stderr,
universal_newlines=False, bufsize=bufsize, **kwargs)
if stdin_w is not None:
stdin.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move the line before open() call.
It saves 1 open file descriptor.
Not a very big deal but why not?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I even understand this code :) What's it for? Why does monkeypatching self._proc.stdin work?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1st1 please read a code comment at the begin of the method.
AIX requires to make stdin a socket pair (I love all Unix flavors and their differences).
proc.stdin should be a writable file-like object.
open() creates a file object from write-fd of created socket pair.

It exists in asyncio starting from very old times.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asvetlov: Good point. I suppose since open(stdin_w.detach(), ...) cannot fail I should have kept the order.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1st1 please read a code comment at the begin of the method.

Yeah, that's what I did first. The comment doesn't explain why we pass the read end of the pipe to Popen initially (I see that we then monkey-patch it to the write end).

It exists in asyncio starting from very old times.

I've no problem with the code or this PR, but I wish that the comment is a bit more elaborate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1st1 you know, I'm bad in comment writing, sorry.
Would you propose exact text?

self._proc.stdin = open(stdin_w.detach(), 'wb', buffering=bufsize)
stdin_w = None
finally:
if stdin_w is not None:
stdin.close()
stdin_w.close()


class AbstractChildWatcher:
Expand Down
17 changes: 13 additions & 4 deletions Lib/test/test_asyncio/test_subprocess.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,9 +468,7 @@ async def kill_running():
isinstance(self, SubprocessFastWatcherTests)):
asyncio.get_child_watcher()._callbacks.clear()

def test_popen_error(self):
# Issue #24763: check that the subprocess transport is closed
# when BaseSubprocessTransport fails
def _test_popen_error(self, stdin):
if sys.platform == 'win32':
target = 'asyncio.windows_utils.Popen'
else:
Expand All @@ -480,12 +478,23 @@ def test_popen_error(self):
popen.side_effect = exc

create = asyncio.create_subprocess_exec(sys.executable, '-c',
'pass', loop=self.loop)
'pass', stdin=stdin,
loop=self.loop)
with warnings.catch_warnings(record=True) as warns:
with self.assertRaises(exc):
self.loop.run_until_complete(create)
self.assertEqual(warns, [])

def test_popen_error(self):
# Issue #24763: check that the subprocess transport is closed
# when BaseSubprocessTransport fails
self._test_popen_error(stdin=None)

def test_popen_error_with_stdin_pipe(self):
# Issue #35721: check that newly created socket pair is closed when
# Popen fails
self._test_popen_error(stdin=subprocess.PIPE)

def test_read_stdout_after_process_exit(self):

async def execute():
Expand Down
1 change: 1 addition & 0 deletions Misc/ACKS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,6 +483,7 @@ Florian Festi
John Feuerstein
Carl Feynman
Vincent Fiack
Niklas Fiekas
Anastasia Filatova
Tomer Filiba
Segev Finer
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Library/2019-01-18-16-23-00.bpo-35721.d8djAJ.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Fix :meth:`asyncio.SelectorEventLoop.subprocess_exec()` leaks file descriptors
if ``Popen`` fails and called with ``stdin=subprocess.PIPE``.
Patch by Niklas Fiekas.