rdar://38123690 (overlarge batches) and rdar://37865437 (task queue exec errors) #14976
Conversation
|
@swift-ci please test |
|
Build failed |
include/swift/Driver/ToolChain.h
Outdated
| @@ -29,6 +29,9 @@ namespace driver { | |||
| class Job; | |||
| class OutputInfo; | |||
|
|
|||
| /// The limit for passing a list of files on the command line. | |||
| const size_t TOO_MANY_FILES = 128; | |||
There was a problem hiding this comment.
Please make this static, there's no need for it to have a stable address.
lib/Driver/Compilation.cpp
Outdated
| // error" at some stage before even calling us with a process exit / | ||
| // signal (or else a poll failed); unfortunately the task causing it | ||
| // was dropped on the floor and we have no way to recover it here, | ||
| // so we report a very poor, generic error. |
There was a problem hiding this comment.
Better than nothing, and I'm glad we caught it. Maybe this should be a FIXME, though?
graydon
force-pushed
the
rdar-38123690-task-queue-exec-errors
branch
from
d8faf61 to
3c8c151
Compare
|
@jrose-apple I'm not convinced TOO_MANY_FILES is great either; have rewritten this with a slightly-more-meaningful test of whether the overall command-line length is too long, with a rebatching cycle (at twice the number of partitions / half the batch size) if it fails. |
|
@swift-ci please test |
|
Build failed |
|
Build failed |
graydon
force-pushed
the
rdar-38123690-task-queue-exec-errors
branch
from
3c8c151 to
6c9029e
Compare
|
@swift-ci please test |
|
Build failed |
|
Build failed |
This fixes another assert @davidungar ran into testing batch mode, this time due to overlong command lines / overlarge batches (to be fixed when we move to passing OFMs, but not fixed yet). For the time being we just cap at total/TOO_MANY_FILES, which might still not be small enough given the auxiliary file amplification, but it's enough to pass a credible test, and none of this is especially precise in the existing code either (I think the actual limit arises from colliding with ARG_MAX, which we're definitely not measuring!)
Along the way, fixes (by necessity) the cause of an assert @ddunbar hit, wherein the driver doesn't trap exec-failure properly; there's not a lot we can recover in these cases, but we can do better than throwing an unrelated assert (or not trapping at all in release builds). The overlong args were triggering exec failure, but then so do all sorts of things, such as exec'ing a nonexistent binary.
rdar://37865437
rdar://38123690