Skip to content

Spec errors reporting incomplete source code #15832

@straight-shoota

Description

@straight-shoota

When an expectation fails, the reported failure includes an extract from the source code which caused the failure, together with a message what exactly failed. The source code serves to identify the test case and how it works, possibly helping find the reason for the failure.

For example, the expectation (1 + 1).should eq 1 generates the following output:

     Failure/Error: (1 + 1).should eq 1

       Expected: 1
            got: 2

However, this only works well for expectations that fit on a single line.
Granted, this is by far the most common use case, but sometimes expectations cover multiple lines.

Most notably, expect_raises typically has the raising code in a block on a separate line.

expect_raises RuntimeError, "foobar" do
  1 + 1
end

The resulting source excerpt shows only the first line and thus omits the code that's actually being tested.

     Failure/Error: expect_raises RuntimeError, "foobar" do

       Expected RuntimeError but nothing was raised

The source code does not provide any useful information besides what the failure message already states. It's just duplication noise.
It would be much more helpful to show 1 + 1 as the offending source code.

Additional to expect_raises there can be other cases where a normal expectation spans multiple lines.

A simplified and extreme example for that:

[
  1
].should eq [
] of Int32

The resulting source excerpt is completely useless:

     Failure/Error: [

       Expected: []
            got: [1]

The spec report should handle this better and provide meaningful information in the source excerpt.

We read the source code from the original file, based on the location information associated with the expectation by courtesy of __FILE__ and __LINE__. Similarly, the magic constant __END_LINE__ would provide the line number of the end location and thus would allow us to read all lines of a multi-line expectation.

This should work at least for the expected value because it's an argument of the should call. But the start line of the actual value is not trivially available, unfortunately.

There might be some downside if the expectation is extremely long, because that would impact readability of the spec report. Perhaps we'd want to add some upper limit on the number of printed lines?

For expect_raises in particular, the wrapping call itself often takes a considerable amount of space and does not provide any meaningful information.
Ideally, the report would leave that out and only show the body of the block. I don't think we can easily retrieve the start and end location of the block. But maybe we can use a heuristic to figure out whether the first and last line of an expect_raises call are safe to strip.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions