[Utils][SPIR-V] Adding spirv-sim to LLVM

[Keenuts]

2nd submission

The buildbots are using python 3.8, and some type annotations I was using are only available starting 3.9. The last commit on the pile is the additional changes compared to the original submission #104020.

Original text:

Currently, the testing infrastructure for SPIR-V is based on FileCheck. Those tests are great to check some level of codegen, but when the test needs check both the CFG layout and the content of each basic-block, things becomes messy.

Because the CHECK/CHECK-DAG/CHECK-NEXT state is limited, it is sometimes hard to catch the good block: if 2 basic blocks have similar instructions, FileCheck can match the wrong one.

Cross-lane interaction can be a bit difficult to understand, and writting a FileCheck test that is strong enough to catch bad CFG transforms while not being broken everytime some unrelated codegen part changes is hard.

And lastly, the spirv-val tooling we have checks that the generated SPIR-V respects the spec, not that it is correct in regards to the source IR.

For those reasons, I believe the best way to test the structurizer is to:

run spirv-val to make sure the CFG respects the spec.
simulate the function to validate result for each lane, making sure the generated code is correct.
This simulator has no other dependencies than core python. It also only supports a very limited set of instructions as we can test most features through control-flow and some basic cross-lane interactions.

As-is, the added tests are just a harness for the simulator itself. If this gets merged, the structurizer PR will benefit from this as I'll be able to add extensive testing using this.

[joker-eph]

Currently, the testing infrastructure for SPIR-V is based on FileCheck. Those tests are great to check some level of codegen, but when the test needs check both the CFG layout and the content of each basic-block, things becomes messy.

Because the CHECK/CHECK-DAG/CHECK-NEXT state is limited, it is sometimes hard to catch the good block: if 2 basic blocks have similar instructions, FileCheck can match the wrong one.

Cross-lane interaction can be a bit difficult to understand, and writting a FileCheck test that is strong enough to catch bad CFG transforms while not being broken everytime some unrelated codegen part changes is hard.

I have quite some concerns with this line of arguments: this seems to go against the way virtually all of LLVM is currently tested.
While this seems like a valuable discussion to have, I see this as a major departure from the current testing approach, and as such this needs a clear RFC on the Discourse forum about the kind of testing we expect in LLVM (unit-testing vs integration tests) and all the associated tradeoffs.

In the meantime, please don't consider this as a valid alternative to unit-test / FileCheck tests.

(if this RFC / discussion already happened and I missed it, then please just point me to the discussion :) )

[Keenuts]

Currently, the testing infrastructure for SPIR-V is based on FileCheck. Those tests are great to check some level of codegen, but when the test needs check both the CFG layout and the content of each basic-block, things becomes messy.
Because the CHECK/CHECK-DAG/CHECK-NEXT state is limited, it is sometimes hard to catch the good block: if 2 basic blocks have similar instructions, FileCheck can match the wrong one.
Cross-lane interaction can be a bit difficult to understand, and writting a FileCheck test that is strong enough to catch bad CFG transforms while not being broken everytime some unrelated codegen part changes is hard.

I have quite some concerns with this line of arguments: this seems to go against the way virtually all of LLVM is currently tested. While this seems like a valuable discussion to have, I see this as a major departure from the current testing approach, and as such this needs a clear RFC on the Discourse forum about the kind of testing we expect in LLVM (unit-testing vs integration tests) and all the associated tradeoffs.

In the meantime, please don't consider this as a valid alternative to unit-test / FileCheck tests.

(if this RFC / discussion already happened and I missed it, then please just point me to the discussion :) )

Hi Mehdi!
There was no discourse thread, we discussed that in the LLVM SPIR-V backend meetings (each monday).
Yes, this is a different method, and I can see why some would disagree. I'll open a discourse thread so we can have a discussion in a more suited place then.

(FYI @michalpaszkowski @s-perron )

[Keenuts]

@joker-eph
I opened an RFC (https://discourse.llvm.org/t/rfc-spir-v-simulator/81168)

Additional node: do keep in mind that this is an additional tool to what we have today.
Today we already use FileCheck and spirv-val for the SPIR-V backend. This just come in addition to all that, to test cases for which FileCheck has IMO strong limits.
The goal is not to rely only on spirv-sim to test the SPIR-V backend.

[joker-eph]

Thanks for the quick follow-up!

I understand the positioning, but this is still a departure: for example the X86 backend could easily just use lli and the JIT to run a bunch of similar tests, but this was deemed not desirable.

[Keenuts]

I understand the positioning, but this is still a departure: for example the X86 backend could easily just use lli and the JIT to run a bunch of similar tests, but this was deemed not desirable.

Do you have some thread/context on why this was deemed not desirable? I'd be interested in reading about this.

[joker-eph]

I'd have to do more archive search on [email protected] (or [email protected]), but some info:

• https://llvm.org/docs/TestingGuide.html#test-suite : note how we intentionally kept the e2e test-suite separated from the compiler, and consider that changes to the compiler should have unit-tests independent of the test-suite.
• here is the most recent thread on the topic I believe: https://lists.llvm.org/pipermail/llvm-dev/2019-October/thread.html#135739
• here is a long thread that may show how we approach such changes in the project: https://lists.llvm.org/pipermail/llvm-dev/2016-February/095300.html

This was discussed a few time before that of course over the lifetime of the project, it's just hard to find all the discussions on the topic.