Skip to content

Adjust batch partition count calculation #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

kastiglione
Copy link
Contributor

This mirrors the batch partition calculation change in swiftlang/swift#28972. See below for an explanation.


While exploring how to optimize global build times, I noticed that the batch sizing logic wasn't producing multiple batches until a module had 50 files. Based on @graydon's detailed batch calculation comment (which says "cap the batch size ... at ... 25"), and based on the variable names ("SizeLimit"), I believe the batch calculation implementation is off.

The simple example is a module of 26 files. With the existing batch calculation, the result is a 26/25=1 batch. However this is larger than the DefaultSizeLimit. This calculation continues to yield 1 for modules up to 49 files. Only at modules of 50 files does the batch calculation produce two batches.

Here's a table showing the effects of the current partition count calculation:

Num Files Num Batches Min Batch Size Max Batch Size
1-49 1 1 49
50-74 2 25 37
75-99 3 25 33
100-124 4 25 31

This table shows that the max batch size is inversely proportional with the number of files. Smaller modules have a higher max batch size, and as a module gets larger, its max batch size trends toward 25. It also shows that after 50, the minimum batch size is consistently 25.

With this proposed change to the batch count calculation, the math is now:

Num Files Num Batches Min Batch Size Max Batch Size
1-25 1 1 25
26-50 2 13 25
51-75 3 17 25
76-100 4 19 25
101-125 5 21 25

This table shows that the batch size is in fact capped to 25, independent of how many files are in the module. It also show the minimum batch size increases with module size.

@DougGregor
Copy link
Member

@swift-ci please test

@DougGregor DougGregor merged commit 3eafea1 into swiftlang:master Jan 8, 2020
@kastiglione kastiglione deleted the dl/adjust-batch-partition-count-calculation branch January 30, 2020 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants