[LIT][E2E] Fix LIT hang when executing non-existent files

[ayylol]

Fixes #16351

In our containers if an exception is not immediately caught lit will hang on the following call

llvm/llvm/utils/lit/lit/run.py

Line 93 in 4adaef0

pool.join()

This can occur when using the internal lit shell and trying to run a program that does not exist. In this case _executeShCmd will throw an internal shell error, which will not be caught by the function directly calling it, executeShCmd, rather it is caught one function higher in the call stack in executeScriptInternal. Because that exception is percolated up the call stack instead of being immediately caught lit will hang. This patch changes the location where we catch this exception to executeShCmd instead to avoid this.

Previously to avoid this we would use the external lit shell. However this introduces some differences in how we need to write tests (i.e., needing to add --crash for certain not calls), it slightly changes how test output is printed (all in one block, rather than separated by RUN: lines), and it messes up the path to executables when running on Windows (all \ were interpreted as escapes for the next characters leading to trying to execute a non-existent file). This pr also changes the E2E tests to always use the internal lit shell.

For more background on what causes this hang see:
https://stackoverflow.com/questions/15314189/python-multiprocessing-pool-hangs-at-join
https://bugs.python.org/issue9400
python/cpython#53646

[ayylol]

Test run with dummy test that would previously hang when using the internal shell:
https://github.com/intel/llvm/actions/runs/13925966650

On PVC, and multiple other jobs running took less than the timeout. Meaning the hang is fixed.

Failed Tests (1):
  SYCL :: dummy_test.cpp


Testing Time: 181.56s

[ayylol]

Note: This fix is in upstream llvm-lit code so I've opened llvm/llvm-project#131881 as well with the same changes to TestRunner.py. However I was only able to reproduce this in our own containers. Also, these changes are necessary to enable the E2E test splitting on windows (#17335), since running the tests with the external shell messes up the paths of the executables on windows.

[sarnex]

fix seems fine even if its just locally

[ayylol]

@intel/llvm-gatekeepers this is ready to merge.

Note, these changes correspond to upstream LLVM pr llvm/llvm-project#131881, but merging here first will unblock progress on #17335.

Once the upstream pr is merged I will cherry-pick those changes into intel/llvm.