[mypyc] Support separate compilation #7636
Conversation
|
(I haven't squashed the commit history yet because it was useful for me in revising this PR and might be useful to me in addressing feedback. But it probably isn't useful to anybody reviewing this, because I made some changes in early commits that I undid in later commits and because github bizarrely insists on sorting commits by their timestamp.) |
There was a problem hiding this comment.
Great, incremental compilation is going to be a big win, and necessary for scaling to larger codebases.
This is the first pass of my code review. I'm mostly requesting additional documentation and comments. I'll do a more detailed pass after the doc updates are in, so that I can also check that the docs match the code.
There was a problem hiding this comment.
This is the second batch of my review. My main remaining confusion is about how the exported things table works in detail (primarily when importing a module) and what sort of library files we generate in all the different scenarios (1 lib per compiled module + some number of separate shared libs?). There are individual comments below that ask for clarification about these things.
|
OK I think I have addressed all the feedback. There was also some function renaming. One major pair of substantive changes went in, which is to generate group names from single-module groups in a legible manner and to restore namegen module name shortening. |
msullivan
force-pushed
the
tweak-mangling
branch
from
d47609d to
7290159
Compare
…t to do it by SCC
mypyc/emitmodule.py
Outdated
| @@ -104,32 +177,62 @@ def encode_bytes_as_c_string(b: bytes) -> Tuple[str, int]: | |||
| return '"{}"'.format(escaped), len(b) | |||
|
|
|||
|
|
|||
| class ModuleGenerator: | |||
| def pointerize(s: str, name: str) -> str: | |||
| """Given C type and a variable name, return a declaration of a pointer to it.""" | |||
There was a problem hiding this comment.
I'm not sure if the docstring matches the implementation. It looks like s should have both a type and a variable name, since we are replacing name in it. Also, is it possible that there will be multiple matches to replace? I assume that the variable name needs to be sufficiently unique.
|
Oh and there is a build failure. |
This adds support for separate compilation to mypyc.
A
separateargument is added tomypycify. IfTrue, mypyc places everymodule its own separate shared library. Otherwise it can take a list of groups
of source files that should be placed in a shared library together.
The shared libraries communication with each other using the C API's intended
mechanism for communicating between C extension modules: Capsules.
Each library creates a table of pointers to all of its exported data and stores a pointer
to it in a capsule stored as a module attribute. When a library is loaded, it
loads the linking table capsules from all its dependencies and copies them
into a local copy of the table. (To eliminate the need for a pointer indirection when
accessing it.)
This adds a test mode that will run all multi-module run tests in separate compilation mode.
I also manually tested mypy itself compiled in separate compilation mode.
This supports a limited form of incremental compilation already: only modules
that changed or had a header they depend on change will be recompiled by the
C compiler. The entire project still needs to go through the entire mypy/mypyc
front-end and middle-end, however. I expect to have a PR that adds support for
hooking into incremental mode next week.
This is progress on mypyc/mypyc#682.