Add --fail-no-examples option: fail if no RSpec examples #2302

xmik · 2016-07-24T15:42:15Z

I added a new command line option: --fail-no-examples. It defaults to false, so default behavior is not changed. But, if this option is set and there are no RSpec examples to be run, exit status will be 1 (so far it was always 0).

Motivation

We use RSpec rake task on CI. Having updated RSpec from 3.3.0 to 3.5.0, there was some change in that rake task code. Without changing anything, the rake task resulted in running 0 examples and exit status was 0. Jobs were passed on CI, but I'd prefer if they were not.

myronmarston · 2016-07-24T18:51:19Z

Having updated RSpec from 3.3.0 to 3.5.0, there was some change in that rake task code.

Was there a regression in our rake task? If so, it'd be good for us to know about it so we can fix it.

Anyhow, if we're going to add this, I think it would be better to add it to RSpec::Core::Configuration (so it can be set via RSpec.configure) rather than adding it as a CLI option. We intentionally limit what config options are exposed via the CLI. If we exposed everything, rspec --help would be an overwhelming wall of text. We just want options that are the main ones users commonly want to run differently in a one-off manner. This sounds like an option that you'd want to permanently set for your CI environment, not something for a one-off run. If we added it to RSpec.configure, you could set it in your CI environment like so:

# spec/spec_helper.rb
RSpec.configure do |c|
  if ENV['CI']
    c.fail_if_no_examples = true
  end
end

# .rspec
--require spec_helper

(The latter is necessary to ensure spec_helper.rb is always loaded. Simply requiring it from spec files won't help if no spec files are loaded!)

Thoughts?

xmik · 2016-07-24T19:48:21Z

Yes, you are right, I want to set this option permanently and setting it by RSpec::Core::Configuration is enough. I'm going to rebase this PR so that there is no CLI option, only RSpec::Core::Configuration one.

As it comes for rake task potential regression, I opened an issue: #2303 .

JonRowe · 2016-07-24T23:55:30Z

lib/rspec/core/configuration.rb

@@ -199,6 +199,10 @@ def only_failures_but_not_configured?
      # The exit code to return if there are any failures (default: 1).
      add_setting :failure_exit_code

+      # @macro add_setting
+      # Whether or not to fail when there are no RSpec examples (default: false).
+      add_setting :fail_no_examples


Currently the build is failing due to a drop in test coverage, I believe this needs a test covering it to fix that :)

xmik · 2016-07-25T17:19:08Z

I've just rebased this PR. The option is named: fail_if_no_examples (as @myronmarston wrote in usage example) and added better tests, but let me know if not enough. I tried to test exit status in unit tests, but it always resulted in ruby errors (my fault), so I left exit status tested only in integration tests (cucumber).

And thanks for that comment on .rspec file, I understood that later.

myronmarston · 2016-07-25T17:23:53Z

lib/rspec/core/runner.rb

@@ -110,6 +110,9 @@ def setup(err, out)
      def run_specs(example_groups)
        @configuration.reporter.report(@world.example_count(example_groups)) do |reporter|
          @configuration.with_suite_hooks do
+            if @world.example_count(example_groups) == 0 && @configuration.fail_if_no_examples


@world.example_count(example_groups) is being called above on line 111 and then again here on line 113. Instead of calling it twice, can you call it once, store the value in a local variable, and then use it in two places?

Yes, of course! I was experimenting here and forgot.

xmik · 2016-07-26T11:57:02Z

I updated the PR with your remarks.

Edit: The low level tests are now in: spec/integration/fail_if_no_examples_spec.rb.

JonRowe · 2016-07-27T09:58:28Z

features/configuration/fail_if_no_examples.feature

@@ -0,0 +1,46 @@
+Feature: fail if no examples
+
+  Use the `fail_if_no_examples` option to make RSpec exit with exit status 1


Technically RSpec will exit with the configured error failure status, so prehaps this should read:

to make RSpec exit with a failure status (by default 1)

which also avoids the double exit :)

myronmarston · 2016-07-27T16:52:25Z

@xmik this is looking really close and is very well done!

xmik · 2016-07-27T19:39:35Z

Thanks @myronmarston and @JonRowe for all the remarks. I incorporated them. It's nice to see that you care for RSpec and keep the code clean. Let me know if you want me to squash commits or correct something else.

myronmarston · 2016-07-27T20:12:39Z

features/configuration/fail_if_no_examples.feature

@@ -0,0 +1,31 @@
+Feature: fail if no examples
+
+  Use the `fail_if_no_examples` option to make RSpec exit with a failure status (by default 1) if there are 0 examples. Using this option, it is recommended to add a `--require spec_helper` option to `.rspec` file to ensure the `fail_if_no_examples` option is set even if no spec files are loaded.


Suggestion: s/if there are 0 examples/if there are no examples/.

I think it reads more naturally and aligns with the config option name.

myronmarston · 2016-07-27T20:17:58Z

Let me know if you want me to squash commits or correct something else.

Please do squash it, then I'll gladly merge it :).

Exit with a failure status if no RSpec examples. Default behavior not changed. CLI option not implemented.

xmik · 2016-07-28T10:29:32Z

s/if there are 0 examples/if there are no examples/ incorporated and commits squashed.

JonRowe · 2016-07-28T11:20:31Z

Thanks @xmik

[ci skip]

* Add fail_if_no_examples option: exit 1 if no RSpec examples, Default behavior not changed. CLI option not implemented. * Add fail_if_no_examples option: better tests * Add fail_if_no_examples option: support custom failure_exit_code and correct cucumber test to be better rendered on relish

[ci skip]

JonRowe reviewed Jul 24, 2016
View reviewed changes

xmik force-pushed the fail_on_no_examples branch from e68d17f to 322dec2 Compare July 25, 2016 17:09

myronmarston reviewed Jul 25, 2016
View reviewed changes

JonRowe reviewed Jul 27, 2016
View reviewed changes

myronmarston reviewed Jul 27, 2016
View reviewed changes

Add fail_if_no_examples option

fea5c47

Exit with a failure status if no RSpec examples. Default behavior not changed. CLI option not implemented.

xmik force-pushed the fail_on_no_examples branch from 6c9e8d8 to fea5c47 Compare July 28, 2016 10:25

JonRowe merged commit aec4852 into rspec:master Jul 28, 2016

JonRowe added a commit that referenced this pull request Jul 28, 2016

changelog for #2302

eac94a6

myronmarston added a commit that referenced this pull request Jul 28, 2016

Revise changelog for #2302.

ad60091

[ci skip]

MatheusRich pushed a commit to MatheusRich/rspec-core that referenced this pull request Oct 30, 2020

changelog for rspec#2302

e2c7c62

MatheusRich pushed a commit to MatheusRich/rspec-core that referenced this pull request Oct 30, 2020

Revise changelog for rspec#2302.

76afb25

[ci skip]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add --fail-no-examples option: fail if no RSpec examples #2302

Add --fail-no-examples option: fail if no RSpec examples #2302

Uh oh!

xmik commented Jul 24, 2016

Uh oh!

myronmarston commented Jul 24, 2016

Uh oh!

xmik commented Jul 24, 2016

Uh oh!

JonRowe Jul 24, 2016

Uh oh!

xmik commented Jul 25, 2016

Uh oh!

myronmarston Jul 25, 2016

Uh oh!

xmik Jul 25, 2016

Uh oh!

xmik commented Jul 26, 2016 •

edited

Loading

Uh oh!

JonRowe Jul 27, 2016

Uh oh!

myronmarston commented Jul 27, 2016

Uh oh!

xmik commented Jul 27, 2016

Uh oh!

myronmarston Jul 27, 2016

Uh oh!

myronmarston commented Jul 27, 2016

Uh oh!

xmik commented Jul 28, 2016

Uh oh!

JonRowe commented Jul 28, 2016

Uh oh!

Uh oh!

		@@ -0,0 +1,46 @@
		Feature: fail if no examples

		Use the `fail_if_no_examples` option to make RSpec exit with exit status 1

		@@ -0,0 +1,31 @@
		Feature: fail if no examples

		Use the `fail_if_no_examples` option to make RSpec exit with a failure status (by default 1) if there are 0 examples. Using this option, it is recommended to add a `--require spec_helper` option to `.rspec` file to ensure the `fail_if_no_examples` option is set even if no spec files are loaded.

Uh oh!

Add --fail-no-examples option: fail if no RSpec examples #2302

Add --fail-no-examples option: fail if no RSpec examples #2302

Uh oh!

Conversation

xmik commented Jul 24, 2016

Motivation

Uh oh!

myronmarston commented Jul 24, 2016

Uh oh!

xmik commented Jul 24, 2016

Uh oh!

JonRowe Jul 24, 2016

Choose a reason for hiding this comment

Uh oh!

xmik commented Jul 25, 2016

Uh oh!

myronmarston Jul 25, 2016

Choose a reason for hiding this comment

Uh oh!

xmik Jul 25, 2016

Choose a reason for hiding this comment

Uh oh!

xmik commented Jul 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonRowe Jul 27, 2016

Choose a reason for hiding this comment

Uh oh!

myronmarston commented Jul 27, 2016

Uh oh!

xmik commented Jul 27, 2016

Uh oh!

myronmarston Jul 27, 2016

Choose a reason for hiding this comment

Uh oh!

myronmarston commented Jul 27, 2016

Uh oh!

xmik commented Jul 28, 2016

Uh oh!

JonRowe commented Jul 28, 2016

Uh oh!

Uh oh!

xmik commented Jul 26, 2016 •

edited

Loading