Improve test failures with RuleTester and AVA

[jfmengels]

When developping and running the tests, I often get the following:

As you can see and imagine, this error message is really not helpful, and I often have to uncomment all the tests to find out which one fails. I thought this was the normal behavior of RuleTester, but then I started contributing to https://github.com/benmosher/eslint-plugin-import, where the errors are a bit clearer

I have the same problem with an other eslint plugin repo of mine, so it would be pretty neat if this could be improved. Do you guys know of ways to improve it?

[twada]

Stop writing everything in a single test function. It's an Assertion Roulette antipattern.

To improve, see test/prefer-power-assert.js

[sindresorhus]

@jfmengels See: avajs/ava#764 (comment)

[jfmengels]

Man, RuleTester was really built with Mocha in mind...

versus

Even if you wrap every single one of them in a seperate test, and add avajs/ava#764 (comment) (though that will help a lot with the kind of error I've shown here), the built-in feedback is much clearer (but you lose the transpiling)... Meh :(

[sindresorhus]

How is the Mocha output clearer when avajs/ava#764 (comment) is fixed? What can we improve?

Also keep in mind that you can use mocha reporters with the --tap option and tap-mocha-reporter.

[jfmengels]

This it the output with the mocha reporter (for those interested):

1. Since in AVA this is all one big test (ok, we could split them up as @twada suggested, but that is some horrible DX when RuleTester gives you a very easy time), so it stops at the first failing test, and you don't know how many tests actually failed. When I am developing, I often add a new failing case, then try to fix it, but when fixing and making the tests fail, I have no idea if it's still the new one that breaks or if I broke a new one (have to comment tests out...)
2. More importantly: You can see which ones fail, and where they are in the code. For some reason it doesn't show you the line (neither does AVA here), but you can at least tell around where it is compared to the others ("ok, it's the last test") and you have the test code in the title to help search (copy + find in your editor).
3. You have the test code in the report, meaning you can already see which cases doesn't work just from the report.

My main issues for now:

• (will be fixed by Show expected and actual if they exist on the AssertionError object. ava#764 (comment)) When the error message is slightly off, I have to go console.log the error message in the rule to see the difference (happens more often that I'd like)
• To find the failing tests, I have to comment out the tests one by one (or half by half, binary search), and I have to do this a whole lot.

[sindresorhus]

This it the output with the mocha reporter (for those interested):

You're using it incorrectly. Should be:

$ ava test/no-unknown-modifiers.js --tap | tap-mocha-reporter

[jfmengels]

True (I do need to specify the reporter too actually, so I've used the default on spec), but it's not much better. The problem comes ultimately from the fact that we declare it all in one test.

[sindresorhus]

So how does Mocha manage to display each RuleTester assert separately?

[jfmengels]

https://github.com/eslint/eslint/blob/master/lib%2Ftesters%2Frule-tester.js#L436-L452

It calls mocha's describe and it to create tests on the fly, then leaves mocha do the rest. I'll try and see if I can't hack it a bit to create AVA tests.