Merge branch 'master' into refine-enum-branch-analysis

Author: Michael0x2a

docs/source/command_line.rst

@@ -622,6 +622,19 @@ Miscellaneous
     have many scripts that import a large package, the behavior enabled
     by this flag is often more convenient.)
 
+``--new-semantic-analyzer``
+    This flag switches to an improved, experimental implementation of
+    the *semantic analyzer* (the part of mypy that binds Python
+    names to definitions). The old and the new semantic analyzers
+    mostly behave identically. The new semantic analyzer is better at
+    handling import cycles and forward references to definitions. It
+    also fixes inconsistencies between the daemon and non-daemon modes,
+    and it detects additional error conditions.
+
+    Likely, the next mypy release will use the new semantic analyzer by
+    default, and the old semantic analyzer will be removed in the next
+    release after that.
+
 .. _PEP 420: https://www.python.org/dev/peps/pep-0420/
 
 .. _PEP 561: https://www.python.org/dev/peps/pep-0561/

docs/source/common_issues.rst

@@ -456,8 +456,8 @@ to see the types of all local variables at once. Example:
    b = 'one'
    reveal_locals()
    # Revealed local types are:
-   # a: builtins.int
-   # b: builtins.str
+   #     a: builtins.int
+   #     b: builtins.str
 .. note::
 
    ``reveal_type`` and ``reveal_locals`` are only understood by mypy and

docs/source/config_file.rst

@@ -52,15 +52,16 @@ characters.
 
   .. _config-precedence:
 
-  When options conflict, the precedence order for the configuration sections is:
-    1. Sections with concrete module names (``foo.bar``)
-    2. Sections with "unstructured" wildcard patterns (``foo.*.baz``),
+  When options conflict, the precedence order for configuration is:
+    1. :ref:`Inline configuration <inline-config>` in the source file
+    2. Sections with concrete module names (``foo.bar``)
+    3. Sections with "unstructured" wildcard patterns (``foo.*.baz``),
        with sections later in the configuration file overriding
        sections earlier.
-    3. Sections with "well-structured" wildcard patterns
+    4. Sections with "well-structured" wildcard patterns
        (``foo.bar.*``), with more specific overriding more general.
-    4. Command line options.
-    5. Top-level configuration file options.
+    5. Command line options.
+    6. Top-level configuration file options.
 
 The difference in precedence order between "structured" patterns (by
 specificity) and "unstructured" patterns (by order in the file) is
@@ -143,6 +144,10 @@ options take precedence. This lets you set global defaults and override them on
 module-by-module basis. If multiple pattern sections match a module, :ref:`the options from the
 most specific section are used where they disagree <config-precedence>`.
 
+Options that take a boolean value may be inverted by adding ``no_`` to
+their name or by (when applicable) swapping their prefix from
+``disallow`` to ``allow`` (and vice versa).
+
 .. _config-file-import-discovery-per-module:
 
 Import discovery
@@ -294,7 +299,7 @@ Miscellaneous strictness flags
     Allows variables to be redefined with an arbitrary type, as long as the redefinition
     is in the same block and nesting level as the original definition.
 
-``implicit-reexport`` (bool, default True)
+``implicit_reexport`` (bool, default True)
     By default, imported values to a module are treated as exported and mypy allows
     other modules to import them. When false, mypy will not re-export unless
     the item is imported using from-as. Note that mypy treats stub files as if this
@@ -311,6 +316,18 @@ Miscellaneous strictness flags
    Prohibit equality checks, identity checks, and container checks between
    non-overlapping types.
 
+Platform configuration
+----------------------
+
+``always_true`` (comma-separated list of strings)
+    Specifies a list of variables that mypy will treat as
+    compile-time constants that are always true.
+
+``always_false`` (comma-separated list of strings)
+    Specifies a list of variables that mypy will treat as
+    compile-time constants that are always false.
+
+
 Global-only options
 *******************
 
@@ -372,14 +389,6 @@ section of the command line docs.
     The default is the current platform as revealed by Python's
     ``sys.platform`` variable.
 
-``always_true`` (comma-separated list of strings)
-    Specifies a list of variables that mypy will treat as
-    compile-time constants that are always true.
-
-``always_false`` (comma-separated list of strings)
-    Specifies a list of variables that mypy will treat as
-    compile-time constants that are always false.
-
 
 Incremental mode
 ----------------
@@ -456,3 +465,7 @@ Miscellaneous
 
 ``verbosity`` (integer, default 0)
     Controls how much debug output will be generated.  Higher numbers are more verbose.
+
+``new_semantic_analyzer`` (bool, default False)
+    Enables the experimental new semantic analyzer.
+    (See :ref:`The mypy command line <command-line>` for more information.)

docs/source/extending_mypy.rst

@@ -235,3 +235,30 @@ insert some entries there) before the class body is analyzed.
 module. It is called before semantic analysis. For example, this can
 be used if a library has dependencies that are dynamically loaded
 based on configuration information.
+
+Supporting the new semantic analyzer
+************************************
+
+Support for the new semantic analyzer (enabled through
+``--new-semantic-analyzer``) requires some changes to plugins. Here is
+a short summary of the most important changes:
+
+* The order of processing AST nodes is different. Code outside
+  functions is processed first, and functions and methods are
+  processed afterwards.
+
+* Each AST node can be processed multiple times to resolve forward
+  references.  The same plugin hook may be called multiple times, so
+  they need to be idempotent.
+
+* The ``anal_type()`` API method returns ``None`` if some part of
+  the type is not available yet due to forward references, for example.
+
+* When looking up symbols, you may encounter *placeholder nodes* that
+  are used for names that haven't been fully processed yet. You'll
+  generally want to request another semantic analysis iteration by
+  *deferring* in that case.
+
+See the docstring at the top of
+`mypy/plugin.py <https://github.com/python/mypy/blob/master/mypy/plugin.py>`_
+for more details.

docs/source/index.rst

@@ -54,6 +54,7 @@ Mypy is a static type checker for Python 3 and Python 2.7.
    running_mypy
    command_line
    config_file
+   inline_config
    mypy_daemon
    installed_packages
    extending_mypy

docs/source/inline_config.rst

@@ -0,0 +1,43 @@
+.. _inline-config:
+
+Inline configuration
+====================
+
+Mypy supports setting per-file configuration options inside files themselves
+using ``# mypy:`` comments. For example:
+
+.. code-block:: python
+
+  # mypy: disallow-any-generics
+
+Inline configuration comments take precedence over all other
+configuration mechanisms.
+
+Configuration comment format
+****************************
+
+Flags correspond to :ref:`config file flags <config-file>` but allow
+hyphens to be substituted for underscores.
+
+Values are specified using ``=``, but ``= True`` may be omitted:
+
+.. code-block:: python
+
+  # mypy: disallow-any-generics
+  # mypy: always-true=FOO
+
+Multiple flags can be separated by commas or placed on separate
+lines. To include a comma as part of an option's value, place the
+value inside quotes:
+
+.. code-block:: python
+
+  # mypy: disallow-untyped-defs, always-false="FOO,BAR"
+
+Like in the configuration file, options that take a boolean value may be
+inverted by adding ``no-`` to their name or by (when applicable)
+swapping their prefix from ``disallow`` to ``allow`` (and vice versa):
+
+.. code-block:: python
+
+  # mypy: allow-untyped-defs, no-strict-optional

mypy/binder.py

@@ -1,10 +1,8 @@
-from typing import Dict, List, Set, Iterator, Union, Optional, Tuple, cast
 from contextlib import contextmanager
 from collections import defaultdict
 
-MYPY = False
-if MYPY:
-    from typing import DefaultDict
+from typing import Dict, List, Set, Iterator, Union, Optional, Tuple, cast
+from typing_extensions import DefaultDict
 
 from mypy.types import Type, AnyType, PartialType, UnionType, TypeOfAny, NoneType
 from mypy.subtypes import is_subtype
@@ -34,11 +32,16 @@ def __init__(self) -> None:
         self.types = {}  # type: Dict[Key, Type]
         self.unreachable = False
 
+        # Should be set only if we're entering a frame where it's not
+        # possible to accurately determine whether or not contained
+        # statements will be unreachable or not.
+        #
+        # Long-term, we should improve mypy to the point where we no longer
+        # need this field.
+        self.suppress_unreachable_warnings = False
+
 
-if MYPY:
-    # This is the type of stored assignments for union type rvalues.
-    # We use 'if MYPY: ...' since typing-3.5.1 does not have 'DefaultDict'
-    Assigns = DefaultDict[Expression, List[Tuple[Type, Optional[Type]]]]
+Assigns = DefaultDict[Expression, List[Tuple[Type, Optional[Type]]]]
 
 
 class ConditionalTypeBinder:
@@ -136,6 +139,9 @@ def put(self, expr: Expression, typ: Type) -> None:
     def unreachable(self) -> None:
         self.frames[-1].unreachable = True
 
+    def suppress_unreachable_warnings(self) -> None:
+        self.frames[-1].suppress_unreachable_warnings = True
+
     def get(self, expr: Expression) -> Optional[Type]:
         key = literal_hash(expr)
         assert key is not None, 'Internal error: binder tried to get non-literal'
@@ -146,6 +152,10 @@ def is_unreachable(self) -> bool:
         # this traversal on every statement?
         return any(f.unreachable for f in self.frames)
 
+    def is_unreachable_warning_suppressed(self) -> bool:
+        # TODO: See todo in 'is_unreachable'
+        return any(f.suppress_unreachable_warnings for f in self.frames)
+
     def cleanse(self, expr: Expression) -> None:
         """Remove all references to a Node from the binder."""
         key = literal_hash(expr)

mypy/build.py

@@ -25,11 +25,7 @@
 
 from typing import (AbstractSet, Any, Dict, Iterable, Iterator, List,
                     Mapping, NamedTuple, Optional, Set, Tuple, Union, Callable, TextIO)
-MYPY = False
-if MYPY:
-    from typing import ClassVar
-    from typing_extensions import Final
-
+from typing_extensions import ClassVar, Final, TYPE_CHECKING
 from mypy_extensions import TypedDict
 
 from mypy.nodes import MypyFile, ImportBase, Import, ImportFrom, ImportAll, SymbolTable
@@ -45,7 +41,7 @@
 from mypy.util import (
     DecodeError, decode_python_encoding, is_sub_path, get_mypy_comments, module_prefix
 )
-if MYPY:
+if TYPE_CHECKING:
     from mypy.report import Reports  # Avoid unconditional slow import
 from mypy import moduleinfo
 from mypy.fixup import fixup_module
@@ -117,7 +113,7 @@ def is_source(self, file: MypyFile) -> bool:
             return True
         elif file._fullname in self.source_modules:
             return True
-        elif file.path is None and self.source_text_present:
+        elif self.source_text_present:
             return True
         else:
             return False
@@ -576,6 +572,10 @@ def __init__(self, data_dir: str,
         self.plugins_snapshot = plugins_snapshot
         self.old_plugins_snapshot = read_plugins_snapshot(self)
         self.quickstart_state = read_quickstart_file(options, self.stdout)
+        # Fine grained targets (module top levels and top level functions) processed by
+        # the semantic analyzer, used only for testing. Currently used only by the new
+        # semantic analyzer.
+        self.processed_targets = []  # type: List[str]
 
     def dump_stats(self) -> None:
         self.log("Stats:")
@@ -1020,12 +1020,16 @@ def read_deps_cache(manager: BuildManager,
 def _load_json_file(file: str, manager: BuildManager,
                     log_sucess: str, log_error: str) -> Optional[Dict[str, Any]]:
     """A simple helper to read a JSON file with logging."""
+    t0 = time.time()
     try:
         data = manager.metastore.read(file)
     except IOError:
         manager.log(log_error + file)
         return None
-    manager.trace(log_sucess + data.rstrip())
+    manager.add_stats(metastore_read_time=time.time() - t0)
+    # Only bother to compute the log message if we are logging it, since it could be big
+    if manager.verbosity() >= 2:
+        manager.trace(log_sucess + data.rstrip())
     try:
         result = json.loads(data)
     except ValueError:  # TODO: JSONDecodeError in 3.5
@@ -1109,14 +1113,18 @@ def find_cache_meta(id: str, path: str, manager: BuildManager) -> Optional[Cache
     meta = _load_json_file(meta_json, manager,
                            log_sucess='Meta {} '.format(id),
                            log_error='Could not load cache for {}: '.format(id))
+    t1 = time.time()
     if meta is None:
         return None
     if not isinstance(meta, dict):
         manager.log('Could not load cache for {}: meta cache is not a dict: {}'
                     .format(id, repr(meta)))
         return None
     m = cache_meta_from_dict(meta, data_json)
-    manager.add_stats(load_meta_time=time.time() - t0)
+    t2 = time.time()
+    manager.add_stats(load_meta_time=t2 - t0,
+                      load_meta_load_time=t1 - t0,
+                      load_meta_from_dict_time=t2 - t1)
 
     # Don't check for path match, that is dealt with in validate_meta().
     if (m.id != id or
@@ -1876,20 +1884,17 @@ def patch_dependency_parents(self) -> None:
         details.
 
         However, this patching process can occur after `a` has been parsed and
-        serialized during increment mode. Consequently, we need to repeat this
+        serialized during incremental mode. Consequently, we need to repeat this
         patch when deserializing a cached file.
 
         This function should be called only when processing fresh SCCs -- the
         semantic analyzer will perform this patch for us when processing stale
         SCCs.
         """
-        Analyzer = Union[SemanticAnalyzerPass2, NewSemanticAnalyzer]  # noqa
-        if self.manager.options.new_semantic_analyzer:
-            analyzer = self.manager.new_semantic_analyzer  # type: Analyzer
-        else:
+        if not self.manager.options.new_semantic_analyzer:
             analyzer = self.manager.semantic_analyzer
-        for dep in self.dependencies:
-            analyzer.add_submodules_to_parent_modules(dep, True)
+            for dep in self.dependencies:
+                analyzer.add_submodules_to_parent_modules(dep, True)
 
     def fix_suppressed_dependencies(self, graph: Graph) -> None:
         """Corrects whether dependencies are considered stale in silent mode.