bpo-38392: PyObject_GC_Track() validates object in debug mode

[vstinner]

In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object to ensure that all object attributes visited by tp_traverse()
are initialized.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.

https://bugs.python.org/issue38392

[vstinner]

This change is backward incompatible and I'm not sure that it's a good idea, so I marked is a WIP. See https://bugs.python.org/issue38392 for the rationale an discussion.

[serhiy-storchaka]

LGTM. Please add a NEWS entry.

[vstinner]

LGTM. Please add a NEWS entry.

Should I add an entry in What's New in Python 3.9?

[serhiy-storchaka]

This change does not change the documented behavior. So I think that an entry in NEWS would be enough.

If you used PyObject_GC_Track() properly, nothing will changed for you. If you used it improperly, you should use it properly at first place.

[vstinner]

I modified my PR: if a bug is detected, _PyObject_ASSERT_FAILED_MSG() now gets the object passed to PyObject_GC_Track(). This object is more likely to help the developer debugging an issue, rather than displaying the visited object which is invalid and may only be displayed as "" without any further information.

I reintroduced https://bugs.python.org/issue33803 bug with this patch:

diff --git a/Python/hamt.c b/Python/hamt.c
index 28b4e1ef6c..d7dd555d15 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -2478,8 +2478,6 @@ hamt_alloc(void)
     if (o == NULL) {
         return NULL;
     }
-    o->h_count = 0;
-    o->h_root = NULL;
     o->h_weakreflist = NULL;
     PyObject_GC_Track(o);
     return o;

The hamt bug is immediately detected. Example (with tracemalloc enabled):

$ ./python -X tracemalloc=10 -m test -v test_context
(...)
0:00:00 load avg: 0.92 [1/1] test_context
test_context_copy_1 (test.test_context.ContextTest) ...
Modules/gcmodule.c:1930: visit_validate: Assertion failed: PyObject_GC_Track() object is not valid
Memory block allocated at (most recent call first):
  File "/home/vstinner/python/master/Lib/test/test_context.py", line 322
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 616
  (...)

object address  : 0x7feb39ce19b0
object refcount : 1
object type     : 0x7f5ea0
object type name: hamt
object repr     : <hamt object at 0x7feb39ce19b0>
Fatal Python error: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007feb48083740 (most recent call first):
  File "/home/vstinner/python/master/Lib/test/test_context.py", line 322 in test_context_copy_1
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 616 in _callTestMethod
  (...)
Aborted (core dumped)

• The bug is detected
• The object type and it's representation are logged
• The Python traceback where the bug occurred is logged
• The Python traceback where the object has been allocated is logged (this feature needs tracemalloc)

Note, when tracemalloc is disabled (default), this reminder is logged:

Enable tracemalloc to get the memory block allocation traceback