Skip to content

Delay processing fresh SCCs until needed #2110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 8, 2016
Merged

Delay processing fresh SCCs until needed #2110

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a62db03
Delay processing fresh SCCs until needed
MichaelLeeDBX Sep 8, 2016
560e50d
Add (and clean up) logging for processing SCCs
MichaelLeeDBX Sep 8, 2016
332d229
Add TODO for possible improvements to queue processing
MichaelLeeDBX Sep 8, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 29 additions & 6 deletions mypy/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1550,6 +1550,9 @@ def process_graph(graph: Graph, manager: BuildManager) -> None:
sccs = sorted_components(graph)
manager.log("Found %d SCCs; largest has %d nodes" %
(len(sccs), max(len(scc) for scc in sccs)))

fresh_scc_queue = [] # type: List[List[str]]

# We're processing SCCs from leaves (those without further
# dependencies) to roots (those from which everything else can be
# reached).
Expand Down Expand Up @@ -1627,16 +1630,36 @@ def process_graph(graph: Graph, manager: BuildManager) -> None:
fresh_msg += " with stale deps (%s)" % " ".join(sorted(stale_deps))
else:
fresh_msg = "stale due to deps (%s)" % " ".join(sorted(stale_deps))
if len(scc) == 1:
manager.log("Processing SCC singleton (%s) as %s" % (" ".join(scc), fresh_msg))
else:
manager.log("Processing SCC of size %d (%s) as %s" %
(len(scc), " ".join(scc), fresh_msg))

scc_str = " ".join(scc)
if fresh:
process_fresh_scc(graph, scc)
manager.log("Queuing fresh SCC (%s)" % scc_str)
fresh_scc_queue.append(scc)
else:
if len(fresh_scc_queue) > 0:
manager.log("Processing the last {} queued SCCs".format(len(fresh_scc_queue)))
# Defer processing fresh SCCs until we actually run into a stale SCC
# and need the earlier modules to be loaded.
#
# Note that `process_graph` may end with us not having processed every
# single fresh SCC. This is intentional -- we don't need those modules
# loaded if there are no more stale SCCs to be rechecked.
#
# TODO: see if it's possible to determine if we need to process only a
# _subset_ of the past SCCs instead of having to process them all.
for prev_scc in fresh_scc_queue:
process_fresh_scc(graph, prev_scc)
fresh_scc_queue = []
size = len(scc)
if size == 1:
manager.log("Processing SCC singleton (%s) as %s" % (scc_str, fresh_msg))
else:
manager.log("Processing SCC of size %d (%s) as %s" % (size, scc_str, fresh_msg))
process_stale_scc(graph, scc)

sccs_left = len(fresh_scc_queue)
manager.log("{} fresh SCCs left in queue (and will remain unprocessed)".format(sccs_left))


def order_ascc(graph: Graph, ascc: AbstractSet[str], pri_max: int = PRI_ALL) -> List[str]:
"""Come up with the ideal processing order within an SCC.
Expand Down