[mypyc] Small improvements to developer docs (#9714)

Document tagged integers and some other updates.

Author: JukkaL

mypyc/doc/dev-intro.md

@@ -231,11 +231,25 @@ Installing a released version of mypy using `pip` (which is compiled)
 and using `dmypy` (mypy daemon) is a much, much faster way to type
 check mypyc during development.
 
-## Overview of Generated C
+## Value Representation
+
+Mypyc uses a tagged pointer representation for values of type `int`
+(`CPyTagged`), `char` for booleans, and C structs for tuples. For most
+other objects mypyc uses the CPython `PyObject *`.
+
+Python integers that fit in 31/63 bits (depending on whether we are on
+a 32-bit or 64-bit platform) are represented as C integers
+(`CPyTagged`) shifted left by 1. Integers that don't fit in this
+representation are represented as pointers to a `PyObject *` (this is
+always a Python `int` object) with the least significant bit
+set. Tagged integer operations are defined in `mypyc/lib-rt/int_ops.c`
+and `mypyc/lib-rt/CPy.h`.
 
-Mypyc uses a tagged pointer representation for integers, `char` for
-booleans, and C structs for tuples. For most other objects mypyc uses
-the CPython `PyObject *`.
+There are also low-level integer types, such as `int32` (see
+`mypyc.ir.rtypes`), that don't use the tagged representation. These
+types are not exposed to users, but they are used in generated code.
+
+## Overview of Generated C
 
 Mypyc compiles a function into two functions, a native function and
 a wrapper function:
@@ -261,10 +275,8 @@ insert a runtime type check (an unbox or a cast operation), since
 Python lists can contain arbitrary objects.
 
 The generated code uses various helpers defined in
-`mypyc/lib-rt/CPy.h`.  The header must only contain static functions,
-since it is included in many files. `mypyc/lib-rt/CPy.c` contains
-definitions that must only occur once, but really most of `CPy.h`
-should be moved into it.
+`mypyc/lib-rt/CPy.h`. The implementations are in various `.c` files
+under `mypyc/lib-rt`.
 
 ## Inspecting Generated C
 
@@ -298,10 +310,10 @@ also write tests that test the generated IR, however.
 ### Tests that compile and run code
 
 Test cases that compile and run code are located in
-`test-data/run*.test` and the test runner is in `mypyc.test.test_run`.
-The code to compile comes after `[case test<name>]`. The code gets
-saved into the file `native.py`, and it gets compiled into the module
-`native`.
+`mypyc/test-data/run*.test` and the test runner is in
+`mypyc.test.test_run`.  The code to compile comes after `[case
+test<name>]`. The code gets saved into the file `native.py`, and it
+gets compiled into the module `native`.
 
 Each test case uses a non-compiled Python driver that imports the
 `native` module and typically calls some compiled functions. Some
@@ -312,8 +324,10 @@ driver just calls each module-level function that is prefixed with
 `test_` and reports any uncaught exceptions as failures. (Failure to
 build or a segfault also count as failures.) `testStringOps` in
 `mypyc/test-data/run-strings.test` is an example of a test that uses
-the default driver. You should usually use the default driver. It's
-the simplest way to write most tests.
+the default driver.
+
+You should usually use the default driver (don't include
+`driver.py`). It's the simplest way to write most tests.
 
 Here's an example test case that uses the default driver:
 
@@ -412,23 +426,22 @@ If you add an operation that compiles into a lot of C code, you may
 also want to add a C helper function for the operation to make the
 generated code smaller. Here is how to do this:
 
-* Add the operation to `mypyc/lib-rt/CPy.h`. Usually defining a static
-  function is the right thing to do, but feel free to also define
-  inline functions for very simple and performance-critical
-  operations. We avoid macros since they are error-prone.
+* Declare the operation in `mypyc/lib-rt/CPy.h`. We avoid macros, and
+  we generally avoid inline functions to make it easier to target
+  additional backends in the future.
 
 * Consider adding a unit test for your C helper in `mypyc/lib-rt/test_capi.cc`.
   We use
   [Google Test](https://github.com/google/googletest) for writing
   tests in C++. The framework is included in the repository under the
   directory `googletest/`. The C unit tests are run as part of the
-  pytest test suite (`test_c_unit_tests`).
+  pytest test suite (`test_c_unit_test`).
 
 ### Adding a Specialized Primitive Operation
 
 Mypyc speeds up operations on primitive types such as `list` and `int`
 by having primitive operations specialized for specific types. These
-operations are defined in `mypyc.primitives` (and
+operations are declared in `mypyc.primitives` (and
 `mypyc/lib-rt/CPy.h`).  For example, `mypyc.primitives.list_ops`
 contains primitives that target list objects.
 
@@ -487,7 +500,7 @@ operations, and so on. You likely also want to add some faster,
 specialized primitive operations for the type (see Adding a
 Specialized Primitive Operation above for how to do this).
 
-Add a test case to `mypyc/test-data/run.test` to test compilation and
+Add a test case to `mypyc/test-data/run*.test` to test compilation and
 running compiled code. Ideas for things to test:
 
 * Test using the type as an argument.
@@ -523,7 +536,9 @@ about how to do this.
 
 * Feel free to open GitHub issues with questions if you need help when
   contributing, or ask questions in existing issues. Note that we only
-  support contributors. Mypyc is not (yet) an end-user product.
+  support contributors. Mypyc is not (yet) an end-user product. You
+  can also ask questions in our Gitter chat
+  (https://gitter.im/mypyc-dev/community).
 
 ## Undocumented Workflows
 

mypyc/lib-rt/mypyc_util.h

@@ -31,7 +31,15 @@
 // Here just for consistency
 #define CPy_XDECREF(p) Py_XDECREF(p)
 
+// Tagged integer -- our representation of Python 'int' objects.
+// Small enough integers are represented as unboxed integers (shifted
+// left by 1); larger integers (larger than 63 bits on a 64-bit
+// platform) are stored as a tagged pointer (PyObject *)
+// representing a Python int object, with the lowest bit set.
+// Tagged integers are always normalized. A small integer *must not*
+// have the tag bit set.
 typedef size_t CPyTagged;
+
 typedef size_t CPyPtr;
 
 #define CPY_INT_BITS (CHAR_BIT * sizeof(CPyTagged))
@@ -42,6 +50,7 @@ typedef size_t CPyPtr;
 
 typedef PyObject CPyModule;
 
+// Tag bit used for long integers
 #define CPY_INT_TAG 1
 
 typedef void (*CPyVTableItem)(void);