Add 'reproduction-order' flag for printing spec order in summary

[maschwenk]

When running specs using the random ordering, it can often be useful to see the example order that the seed generated.

This can be useful when you are doing a bisect over a large number of failing specs yet you know the first failing example should allow you to reproduce the order-dependent failure with smaller bisect-able bundle. With a large amount of failures, bisect tends to churn for hours and hours.

In the past I had used a combination of --fail-fast plus the --format json flag to parse out the actual spec ordering after the random seeding had taken place. The alternative is running in documentation mode to infer which examples ran in which order (kinda terrible).

Other folks have ran into similar issues and their solutions are similarly hacky

I plan to write specs/documentation if this seems like a palatable change.

Example output

rspec spec/speccy_the_spectacle.rb --order random --reproduction-order

Randomized with seed 21471
...

Finished in 0.21412 seconds (files took 7.1 seconds to load)
3 examples, 0 failures

Reproduce this spec order with:
rspec --seed ordered ./spec/speccy_the_spectacle.rb:50 ./spec/speccy_the_spectacle.rb:10 ./spec/speccy_the_spectacle.rb:20

[myronmarston]

Thanks for the feature idea (and taking a stab at implementing it yourself)!

We consider CLI options to be valuable "real estate" and want to avoid an explosion of rarely used options (which makes it hard for users to find what they really want via rspec --help). I'm not yet convinced this is a common enough need to add to RSpec itself, particularly given this can be accomplished with a custom formatter, with no need to modify RSpec itself.

In addition, you can already see the order a particular seed generates (without running the specs themselves) by using --dry-run. While it doesn't print out the Reproduce this spec order with: bit you've added, it should be noted that we've seem problems with two many arguments (see #2145) so I'm not sure that the printed command will necessarily always work for large suites.

This can be useful when you are doing a bisect over a large number of failing specs yet you know the first failing example should allow you to reproduce the order-dependent failure with smaller bisect-able bundle.

It's not clear to me how you would ever know this. The source of order dependency problems is often a passing example.

With a large amount of failures, bisect tends to churn for hours and hours.

If you're simply trying to make --bisect faster you'll be interested in a change coming in the next release. See #2511 for details, but the TL;DR is that bisect should be noticeably faster in the next release.

[maschwenk]

It's not clear to me how you would ever know this. The source of order dependency problems is often a passing example.

An example of this would be some kind of const lookup issue. There might be 20 successful runs and then a single bad lookup which then causes the entire bundle after that first failure to fail in the exact same way. So really, you just want to figure out the 20 passing specs + 1 first instance of the constant lookup error and run rpsec --order defined first_spec.rb:10 first_spec.rb:5 for the sequence of specs.

We consider CLI options to be valuable "real estate" and want to avoid an explosion of rarely used options

Figured this was the case. I'll try your suggestion of using a custom formatter + --dry-run. I have actually seen this use case discussed in a bunch of other places but I understand your reasoning.

[myronmarston]

I'll try your suggestion of using a custom formatter + --dry-run.

Great. I'll close this then. Thanks again!