Merge pull request #1888 from rspec/rerun-failures

Add new --only-failures CLI option

Author: myronmarston

.gitignore

@@ -19,3 +19,4 @@ Gemfile-custom
 .idea
 bundle
 .rspec-local
+spec/examples.txt

Changelog.md

@@ -15,6 +15,15 @@ Enhancements:
   where the location isn't unique. (Myron Marston, #1884)
 * Use the example id in the rerun command printed for failed examples
   when the location is not unique. (Myron Marston, #1884)
+* Add `config.example_status_persistence_file_path` option, which is
+  used to persist the last run status of each example. (Myron Marston, #1888)
+* Add `:last_run_status` metadata to each example, which indicates what
+  happened the last time an example ran. (Myron Marston, #1888)
+* Add `--only-failures` CLI option which filters to only the examples
+  that failed the last time they ran. (Myron Marston, #1888)
+* Add `--next-failure` CLI option which allows you to repeatedly focus
+  on just one of the currently failing examples, then move on to the
+  next failure, etc. (Myron Marston, #1888)
 
 Bug Fixes:
 

appveyor.yml

@@ -25,7 +25,7 @@ install:
   - cinst ansicon
 
 test_script:
-  - bundle exec rspec
+  - bundle exec rspec --backtrace
 
 environment:
   matrix:

features/command_line/only_failures.feature

@@ -0,0 +1,109 @@
+Feature: Only Failures
+
+  The `--only-failures` option filters what examples are run so that only those that failed the last time they ran are executed. To use this option, you first have to configure `config.example_status_persistence_file_path`, which RSpec will use to store the status of each example the last time it ran.
+
+  There's also a `--next-failure` option, which is shorthand for `--only-failures --fail-fast --order defined`. It allows you to repeatedly focus on just one of the currently failing examples, then move on to the next failure, etc.
+
+  Either of these options can be combined with another a directory or file name; RSpec will run just the failures from the set of loaded examples.
+
+  Background:
+    Given a file named "spec/spec_helper.rb" with:
+      """ruby
+      RSpec.configure do |c|
+        c.example_status_persistence_file_path = "examples.txt"
+      end
+      """
+    And a file named ".rspec" with:
+      """
+      --require spec_helper
+      --order random
+      --format documentation
+      """
+    And a file named "spec/array_spec.rb" with:
+      """ruby
+      RSpec.describe 'Array' do
+        it "checks for inclusion of 1" do
+          expect([1, 2]).to include(1)
+        end
+
+        it "checks for inclusion of 2" do
+          expect([1, 2]).to include(2)
+        end
+
+        it "checks for inclusion of 3" do
+          expect([1, 2]).to include(3) # failure
+        end
+      end
+      """
+    And a file named "spec/string_spec.rb" with:
+      """ruby
+      RSpec.describe 'String' do
+        it "checks for inclusion of 'foo'" do
+          expect("food").to include('foo')
+        end
+
+        it "checks for inclusion of 'bar'" do
+          expect("food").to include('bar') # failure
+        end
+
+        it "checks for inclusion of 'baz'" do
+          expect("bazzy").to include('baz')
+        end
+
+        it "checks for inclusion of 'foobar'" do
+          expect("food").to include('foobar') # failure
+        end
+      end
+      """
+    And a file named "spec/passing_spec.rb" with:
+      """ruby
+      puts "Loading passing_spec.rb"
+
+      RSpec.describe "A passing spec" do
+        it "passes" do
+          expect(1).to eq(1)
+        end
+      end
+      """
+    And I have run `rspec` once, resulting in "8 examples, 3 failures"
+
+  Scenario: Running `rspec --only-failures` loads only spec files with failures and runs only the failures
+    When I run `rspec --only-failures`
+    Then the output from "rspec --only-failures" should contain "3 examples, 3 failures"
+     And the output from "rspec --only-failures" should not contain "Loading passing_spec.rb"
+
+  Scenario: Combine `--only-failures` with a file name
+    When I run `rspec spec/array_spec.rb --only-failures`
+    Then the output should contain "1 example, 1 failure"
+    When I run `rspec spec/string_spec.rb --only-failures`
+    Then the output should contain "2 examples, 2 failures"
+
+  Scenario: Use `--next-failure` to repeatedly run a single failure
+    When I run `rspec --next-failure`
+    Then the output should contain "1 example, 1 failure"
+     And the output should contain "checks for inclusion of 3"
+
+    When I fix "spec/array_spec.rb" by replacing "to include(3)" with "not_to include(3)"
+     And I run `rspec --next-failure`
+    Then the output should contain "2 examples, 1 failure"
+     And the output should contain "checks for inclusion of 3"
+     And the output should contain "checks for inclusion of 'bar'"
+
+    When I fix "spec/string_spec.rb" by replacing "to include('bar')" with "not_to include('bar')"
+     And I run `rspec --next-failure`
+    Then the output should contain "2 examples, 1 failure"
+     And the output should contain "checks for inclusion of 'bar'"
+     And the output should contain "checks for inclusion of 'foobar'"
+
+    When I fix "spec/string_spec.rb" by replacing "to include('foobar')" with "not_to include('foobar')"
+     And I run `rspec --next-failure`
+    Then the output should contain "1 example, 0 failures"
+     And the output should contain "checks for inclusion of 'foobar'"
+
+    When I run `rspec --next-failure`
+    Then the output should contain "All examples were filtered out"
+
+  Scenario: Clear error given when using `--only-failures` without configuring `example_status_persistence_file_path`
+    Given I have not configured `example_status_persistence_file_path`
+     When I run `rspec --only-failures`
+     Then it should fail with "To use `--only-failures`, you must first set `config.example_status_persistence_file_path`."

features/step_definitions/additional_cli_steps.rb

@@ -124,3 +124,29 @@
 When /^I create "([^"]*)" with the following content:$/ do |file_name, content|
   write_file(file_name, content)
 end
+
+Given(/^I have run `([^`]*)` once, resulting in "([^"]*)"$/) do |command, output_snippet|
+  step %Q{I run `#{command}`}
+  step %Q{the output from "#{command}" should contain "#{output_snippet}"}
+end
+
+When(/^I fix "(.*?)" by replacing "(.*?)" with "(.*?)"$/) do |file_name, original, replacement|
+  in_current_dir do
+    contents = File.read(file_name)
+    expect(contents).to include(original)
+    fixed = contents.sub(original, replacement)
+    File.open(file_name, "w") { |f| f.write(fixed) }
+  end
+end
+
+Then(/^it should fail with "(.*?)"$/) do |snippet|
+  assert_failing_with(snippet)
+end
+
+Given(/^I have not configured `example_status_persistence_file_path`$/) do
+  in_current_dir do
+    return unless File.exist?("spec/spec_helper.rb")
+    return unless File.read("spec/spec_helper.rb").include?("example_status_persistence_file_path")
+    File.open("spec/spec_helper.rb", "w") { |f| f.write("") }
+  end
+end

lib/rspec/core.rb

@@ -146,6 +146,8 @@ def self.world
 
   # Namespace for the rspec-core code.
   module Core
+    autoload :ExampleStatusPersister, "rspec/core/example_status_persister"
+
     # @private
     # This avoids issues with reporting time caused by examples that
     # change the value/meaning of Time.now without properly restoring

lib/rspec/core/configuration.rb

@@ -154,6 +154,33 @@ def deprecation_stream=(value)
         end
       end
 
+      # @macro define_reader
+      # The file path to use for persisting example statuses. Necessary for the
+      # `--only-failures` and `--next-failures` CLI options.
+      #
+      # @overload example_status_persistence_file_path
+      #   @return [String] the file path
+      # @overload example_status_persistence_file_path=(value)
+      #   @param value [String] the file path
+      define_reader :example_status_persistence_file_path
+
+      # Sets the file path to use for persisting example statuses. Necessary for the
+      # `--only-failures` and `--next-failures` CLI options.
+      def example_status_persistence_file_path=(value)
+        @example_status_persistence_file_path = value
+        clear_values_derived_from_example_status_persistence_file_path
+      end
+
+      # @macro define_reader
+      # Indicates if the `--only-failures` (or `--next-failure`) flag is being used.
+      define_reader :only_failures
+      alias_method :only_failures?, :only_failures
+
+      # @private
+      def only_failures_but_not_configured?
+        only_failures? && !example_status_persistence_file_path
+      end
+
       # @macro add_setting
       # Clean up and exit after the first failure (default: `false`).
       add_setting :fail_fast
@@ -342,6 +369,9 @@ def initialize
       def force(hash)
         ordering_manager.force(hash)
         @preferred_options.merge!(hash)
+
+        return unless hash.key?(:example_status_persistence_file_path)
+        clear_values_derived_from_example_status_persistence_file_path
       end
 
       # @private
@@ -792,7 +822,11 @@ def profile_examples
       # @private
       def files_or_directories_to_run=(*files)
         files = files.flatten
-        files << default_path if (command == 'rspec' || Runner.running_in_drb?) && default_path && files.empty?
+
+        if (command == 'rspec' || Runner.running_in_drb?) && default_path && files.empty?
+          files << default_path
+        end
+
         @files_or_directories_to_run = files
         @files_to_run = nil
       end
@@ -803,6 +837,40 @@ def files_to_run
         @files_to_run ||= get_files_to_run(@files_or_directories_to_run)
       end
 
+      # @private
+      def last_run_statuses
+        @last_run_statuses ||= Hash.new(UNKNOWN_STATUS).tap do |statuses|
+          if (path = example_status_persistence_file_path)
+            begin
+              ExampleStatusPersister.load_from(path).inject(statuses) do |hash, example|
+                hash[example.fetch(:example_id)] = example.fetch(:status)
+                hash
+              end
+            rescue SystemCallError => e
+              RSpec.warning "Could not read from #{path.inspect} (configured as " \
+                            "`config.example_status_persistence_file_path`) due " \
+                            "to a system error: #{e.inspect}. Please check that " \
+                            "the config option is set to an accessible, valid " \
+                            "file path", :call_site => nil
+            end
+          end
+        end
+      end
+
+      # @private
+      UNKNOWN_STATUS = "unknown".freeze
+
+      # @private
+      FAILED_STATUS = "failed".freeze
+
+      # @private
+      def spec_files_with_failures
+        @spec_files_with_failures ||= last_run_statuses.inject(Set.new) do |files, (id, status)|
+          files << id.split(ON_SQUARE_BRACKETS).first if status == FAILED_STATUS
+          files
+        end.to_a
+      end
+
       # Creates a method that delegates to `example` including the submitted
       # `args`. Used internally to add variants of `example` like `pending`:
       # @param name [String] example name alias
@@ -1560,10 +1628,15 @@ def run_hooks_with(hooks, hook_context)
       end
 
       def get_files_to_run(paths)
-        FlatMap.flat_map(paths_to_check(paths)) do |path|
+        files = FlatMap.flat_map(paths_to_check(paths)) do |path|
           path = path.gsub(File::ALT_SEPARATOR, File::SEPARATOR) if File::ALT_SEPARATOR
           File.directory?(path) ? gather_directories(path) : extract_location(path)
         end.sort.uniq
+
+        return files unless only_failures?
+        relative_files = files.map { |f| Metadata.relative_path(File.expand_path f) }
+        intersection = (relative_files & spec_files_with_failures.to_a)
+        intersection.empty? ? files : intersection
       end
 
       def paths_to_check(paths)
@@ -1678,6 +1751,11 @@ def update_pattern_attr(name, value)
         instance_variable_set(:"@#{name}", value)
         @files_to_run = nil
       end
+
+      def clear_values_derived_from_example_status_persistence_file_path
+        @last_run_statuses = nil
+        @spec_files_with_failures = nil
+      end
     end
     # rubocop:enable Style/ClassLength
   end

lib/rspec/core/configuration_options.rb

@@ -81,7 +81,7 @@ def order(keys)
 
         # `files_or_directories_to_run` uses `default_path` so it must be
         # set before it.
-        :default_path,
+        :default_path, :only_failures,
 
         # These must be set before `requires` to support checking
         # `config.files_to_run` from within `spec_helper.rb` when a

lib/rspec/core/example.rb

@@ -115,7 +115,7 @@ def rerun_argument
       # @return [String] the unique id of this example. Pass
       #   this at the command line to re-run this exact example.
       def id
-        Metadata.id_from(metadata)
+        @id ||= Metadata.id_from(metadata)
       end
 
       # @attr_reader
@@ -160,6 +160,11 @@ def initialize(example_group_class, description, user_metadata, example_block=ni
           description, example_block
         )
 
+        # This should perhaps be done in `Metadata::ExampleHash.create`,
+        # but the logic there has no knowledge of `RSpec.world` and we
+        # want to keep it that way. It's easier to just assign it here.
+        @metadata[:last_run_status] = RSpec.configuration.last_run_statuses[id]
+
         @example_group_instance = @exception = nil
         @clock = RSpec::Core::Time
         @reporter = RSpec::Core::NullReporter