bpo-38965: Fix stuck in test_stack_overflow tests.

[marxin]

After update to GCC10, the testcase is stuck and
a pragma needs to be added.

https://bugs.python.org/issue38965

[vstinner]

Would it be possible to modify stack_overflow() so GCC couldn't turn the tail call into a loop?

[marxin]

Would it be possible to modify stack_overflow() so GCC couldn't turn the tail call into a loop?

Well, based on the discussion for https://bugs.python.org/issue23654
I would do the same for GCC, just disable optimizations.

[marxin]

An alternative fix would be someting like:

diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c
index d1280532ae..7a48a39be4 100644
--- a/Modules/faulthandler.c
+++ b/Modules/faulthandler.c
@@ -1167,6 +1167,9 @@ faulthandler_fatal_error_py(PyObject *self, PyObject *args)
     * a loop. */
 #  pragma intel optimization_level 0
 #endif
+
+unsigned char *buffer_ptr;
+
 static
 uintptr_t
 stack_overflow(uintptr_t min_sp, uintptr_t max_sp, size_t *depth)
@@ -1179,6 +1182,7 @@ stack_overflow(uintptr_t min_sp, uintptr_t max_sp, size_t *depth)
         return sp;
     buffer[0] = 1;
     buffer[4095] = 0;
+    buffer_ptr = &buffer[0];
     return stack_overflow(min_sp, max_sp, depth);
 }

@vstinner What do you prefer?

[vstinner]

If "+ buffer_ptr = &buffer[0];" prevents compiler specific pragma and fix the bug, yes: I prefer code working on any compiler ;-)

[vstinner]

I proposed PR #17467 which is a different approach. @marxin: would you be able to test it on GCC 10?

[marxin]

I proposed PR #17467 which is a different approach. @marxin: would you be able to test it on GCC 10?

Yes, I can confirm that it works fine with GCC 10. That means the function recurses and one will reach a stack overflow.