Delay processing fresh SCCs until needed #2110
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, the process_graph function would always process SCCs, fresh and stale, as they were encountered. This commit modifies this process so instead, we batch and defer processing fresh SCCs until we encounter a stale SCCs. If we do not end up encountering any stale SCCs at all, we throw away the queued fresh SCCs without having to load any of the data files at all. Since loading the data file of fresh SCCs does take a small amount of time, this change will make mypy slightly faster, particularly in the case where there are very few to no files to recheck, and the corresponding SCCs occur early in the sorted list of SCCs. As an example, when running mypy against itself with a fully warm cache, the time it takes to finish typechecking is now approximately 0.3 to 0.4 seconds -- previously, it was 1.3 to 1.4.
|
Nice! Can you add some extra logging at all the new branch points? E.g. when putting an SCC into the queue, and when the queue has to be processed. Also at the end, log the queue size so we know if it made any difference. This also makes me think that sometimes we can keep things in the queue because the stale thing we've found doesn't depend on them. But that's probably better done as a TODO for now. |
This commit adds some logging when fresh SCCs are queued and handled and cleans up some of the logic associated with logging fresh SCCs.
|
Ok, added some logging and a TODO note. (I tried implementing something yesterday that would try and load only a subset of the queued SCCs yesterday, but I ran into a few bugs and ultimately decided to just go with this version rather then grappling with it more.) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, the process_graph function would always process SCCs, fresh and stale, as they were encountered.
This commit modifies this process so instead, we batch and defer processing fresh SCCs until we encounter a stale SCCs. If we do not end up encountering any stale SCCs at all, we throw away the queued fresh SCCs without having to load any of the data files at all.
Since loading the data file of fresh SCCs does take a small amount of time, this change will make mypy slightly faster, particularly in the case where there are very few to no files to recheck, and the corresponding SCCs occur early in the sorted list of SCCs.
As an example, when running mypy against itself with a fully warm cache, the time it takes to finish typechecking is now approximately 0.3 to 0.4 seconds -- previously, it was 1.3 to 1.4.