bpo-37603: tok_nextc() now also updates multi_line_start on REALLOC()

[vstinner]

Fix Python parser crash: when the tokenizer grows its internal buffer
using REALLOC(), update also tok->multi_line_start.

https://bugs.python.org/issue37603

[vstinner]

I don't know well this parser/tokenizer code of Python, so I'm not confident in my change. At least, it fix https://bugs.python.org/issue37603 and all tests pass.

cc @asottile

[asottile]

I have (what I believe to be) an equivalent PR here: https://github.com/python/cpython/pull/14433/files

[vstinner]

I have (what I believe to be) an equivalent PR here: https://github.com/python/cpython/pull/14433/files

Your change is "tok->multi_line_start = tok->buf + cur_multi_line_start; ". It's different than mine. I have no idea which change is correct :-)

[asottile]

let me check out your branch and see if it fixes both issues :)

[asottile]

yeah this PR doesn't pass the test from the other PR:

FAILED (failures=1)
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/asottile/workspace/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/asottile/workspace/cpython/Lib/test/test_cmd_line_script.py", line 719, in <module>
    test_main()
  File "/home/asottile/workspace/cpython/Lib/test/test_cmd_line_script.py", line 715, in test_main
    support.run_unittest(CmdLineTest)
  File "/home/asottile/workspace/cpython/Lib/test/support/__init__.py", line 2083, in run_unittest
    _run_suite(suite)
  File "/home/asottile/workspace/cpython/Lib/test/support/__init__.py", line 2002, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/test/test_cmd_line_script.py", line 621, in test_syntaxerror_multi_line_fstring
    self.assertEqual(
AssertionError: Lists differ: [b'  [18 chars]'    ^', b'SyntaxError: f-string: empty expres[13 chars]wed'] != [b'  [18 chars]'          ^', b'SyntaxError: f-string: empty [19 chars]wed']

First differing element 1:
b'    ^'
b'          ^'

  [b'    foo = f"""{}',
-  b'    ^',
+  b'          ^',
?    ++++++

   b'SyntaxError: f-string: empty expression not allowed']

Notably it ends up with a misplaced syntax error pointer:

# my branch
$ ./python ../t.py
  File "/home/asottile/workspace/cpython/../t.py", line 1
    foo = f"""{}
          ^
SyntaxError: f-string: empty expression not allowed
# this branch
$ ./python  ../t.py 
  File "/home/asottile/workspace/cpython/../t.py", line 1
    foo = f"""{}
    ^
SyntaxError: f-string: empty expression not allowed

[vstinner]

My PR is wrong, PR #14433 seems to be the correct fix and includes a test.