Modify cache file formats to shrink overall cache size #2108
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit modifies several mypy nodes that tend to contain a large number of usually-false boolean flags to store a compressed version of those flags. More precisely, rather then storing each flag and its boolean value as a key-value pair in the output JSON, we now store a list of the flags that are known to be true at time of serialization. When deserializing, we iterate over the list of flags and set each attribute to True, overriding the default value of False. This does not result in any significant speedups, but does end up reducing the size of the cache by a non-trivial factor.
This commit modifies the serialization logic of Argument to stop saving the type_annotation data. After some investigation, it appears as if that data is never actually used once the cache data is loaded by mypy, and removing it would help shrink the cache size by a respectable margin.
This commit modifies the serialization logic and some of the surrounding code for FuncDefs so we no longer serialize the full argument list -- instead, we just stick to serializing the argument names and the argument kinds. As with the other commits, this does not really improve speed, but does end up shrinking the cache size by a fair amount. It would have been nice to avoid storing the arg kinds and arg names altogether since the information can usually be found within the corresponding callable type, but unfortunately, a FuncDef is not guaranteed to actually have a type in all instances -- sometimes, the type is None. This is a problem since we need the argument names and kinds to be able to infer a fallback type later on in mypy.
| @@ -587,6 +595,11 @@ class Var(SymbolNode, Statement): | |||
| # parse for some reason (eg a silenced module) | |||
| is_suppressed_import = False | |||
|
|
|||
| FLAGS = [ | |||
| 'is_self', 'is_ready', 'is_initialized_in_class', 'is_staticmethod', | |||
| 'is_classmethod', 'is_property', 'is_settable_property', 'is_suppressed_import' | |||
There was a problem hiding this comment.
It still hasn't become second nature for you to add a trailing comma to a list like this, when the close bracket is on the next line? (Don't push a fix, just noticing. :-)
There was a problem hiding this comment.
Old habits die hard, I guess :(
I think I typically remember when modifying an existing list of item, since the trailing comma is already there, but it hasn't quite sunken in for new lists yet.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes a small collection of improvements to the format of the data cache files, targeting primarily the
Var,Argument,FuncDef, andTypeInfoclasses.More specifically, this pull request:
Var,FuncDef, andTypeInfotype_commentsfield fromArgumentargumentsinsideFuncDef.These changes together don't appear to have a statistically significant impact on the speed of mypy when it's running in incremental mode, but do end up shrinking the average cache size by approximately 45% to 50%.