Basic recursive bisect formatter

This modifies BisectProgressFormatter to more closely model the recursive
bisection algorithm, fixing the associated BisectCoordinator specs.

Author: urbanautomaton

features/command_line/bisect.feature

@@ -54,12 +54,12 @@ Feature: Bisect
       Bisect started using options: "--seed 1234"
       Running suite to find failures... (0.16755 seconds)
       Starting bisect with 1 failing example and 9 non-failing examples.
+      Checking that failures are order-dependent... failure is order-dependent
 
-      Round 1: searching for 5 non-failing examples (of 9) to ignore: .. (0.30166 seconds)
-      Round 2: searching for 3 non-failing examples (of 5) to ignore: .. (0.30306 seconds)
-      Round 3: searching for 2 non-failing examples (of 3) to ignore: .. (0.33292 seconds)
-      Round 4: searching for 1 non-failing example (of 2) to ignore: . (0.16476 seconds)
-      Round 5: searching for 1 non-failing example (of 1) to ignore: . (0.15329 seconds)
+      Bisecting over non-failing examples 1-9 .. ignoring examples 6-9
+      Bisecting over non-failing examples 1-5 .. ignoring examples 4-5
+      Bisecting over non-failing examples 1-3 .. ignoring example 3
+      Bisecting over non-failing examples 1-2 . ignoring example 1
       Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.26 seconds.
 
       The minimal reproduction command is:
@@ -75,10 +75,11 @@ Feature: Bisect
       Bisect started using options: "--seed 1234"
       Running suite to find failures... (0.17102 seconds)
       Starting bisect with 1 failing example and 9 non-failing examples.
+      Checking that failures are order-dependent... failure is order-dependent
 
-      Round 1: searching for 5 non-failing examples (of 9) to ignore: .. (0.32943 seconds)
-      Round 2: searching for 3 non-failing examples (of 5) to ignore: .. (0.3154 seconds)
-      Round 3: searching for 2 non-failing examples (of 3) to ignore: ..
+      Bisecting over non-failing examples 1-9 .. ignoring examples 6-9
+      Bisecting over non-failing examples 1-5 .. ignoring examples 4-5
+      Bisecting over non-failing examples 1-3 .. ignoring example 3
 
       Bisect aborted!
 
@@ -106,8 +107,10 @@ Feature: Bisect
           - ./spec/calculator_7_spec.rb[1:1]
           - ./spec/calculator_8_spec.rb[1:1]
           - ./spec/calculator_9_spec.rb[1:1]
+      Checking that failures are order-dependent..
+       - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (n.nnnn seconds) failure is order-dependent
 
-      Round 1: searching for 5 non-failing examples (of 9) to ignore:
+      Bisecting over non-failing examples 1-9
        - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 (0.15302 seconds)
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.19708 seconds)
        - Examples we can safely ignore (4):
@@ -121,8 +124,7 @@ Feature: Bisect
           - ./spec/calculator_3_spec.rb[1:1]
           - ./spec/calculator_4_spec.rb[1:1]
           - ./spec/calculator_5_spec.rb[1:1]
-       - Round finished (0.35172 seconds)
-      Round 2: searching for 3 non-failing examples (of 5) to ignore:
+      Bisecting over non-failing examples 1-5
        - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.15836 seconds)
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.19065 seconds)
        - Examples we can safely ignore (2):
@@ -132,26 +134,20 @@ Feature: Bisect
           - ./spec/calculator_10_spec.rb[1:1]
           - ./spec/calculator_2_spec.rb[1:1]
           - ./spec/calculator_3_spec.rb[1:1]
-       - Round finished (0.35022 seconds)
-      Round 3: searching for 2 non-failing examples (of 3) to ignore:
+      Bisecting over non-failing examples 1-3
        - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 (0.21028 seconds)
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.1975 seconds)
        - Examples we can safely ignore (1):
           - ./spec/calculator_2_spec.rb[1:1]
        - Remaining non-failing examples (2):
           - ./spec/calculator_10_spec.rb[1:1]
           - ./spec/calculator_3_spec.rb[1:1]
-       - Round finished (0.40882 seconds)
-      Round 4: searching for 1 non-failing example (of 2) to ignore:
+      Bisecting over non-failing examples 1-2
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.17173 seconds)
        - Examples we can safely ignore (1):
           - ./spec/calculator_3_spec.rb[1:1]
        - Remaining non-failing examples (1):
           - ./spec/calculator_10_spec.rb[1:1]
-       - Round finished (0.17234 seconds)
-      Round 5: searching for 1 non-failing example (of 1) to ignore:
-       - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.18279 seconds)
-       - Round finished (0.18312 seconds)
       Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.47 seconds.
 
       The minimal reproduction command is:

features/support/send_sigint_during_bisect.rb

@@ -5,8 +5,8 @@ module RSpec::Core::Formatters
   BisectProgressFormatter = Class.new(remove_const :BisectProgressFormatter) do
     RSpec::Core::Formatters.register self
 
-    def bisect_round_finished(notification)
-      return super unless notification.round == 3
+    def bisect_round_started(notification)
+      return super unless notification.candidates_range == (1..2)
 
       Process.kill("INT", Process.pid)
       # Process.kill is not a synchronous call, so to ensure the output

lib/rspec/core/bisect/example_minimizer.rb

@@ -20,35 +20,37 @@ def find_minimal_repro
             bisect_over(non_failing_example_ids)
           end
 
-          notify(:bisect_complete, :round => 10, :duration => duration,
+          notify(:bisect_complete, :duration => duration,
                                    :original_non_failing_count => non_failing_example_ids.size,
                                    :remaining_count => remaining_ids.size)
 
           remaining_ids + failed_example_ids
         end
 
         def bisect_over(candidate_ids)
+          notify(:bisect_dependency_check_started)
           if get_expected_failures_for?([])
+            notify(:bisect_dependency_check_failed)
             self.eliminated_ids = non_failing_example_ids
             return
           end
+          notify(:bisect_dependency_check_passed)
 
           bisect_over_recursive(candidate_ids)
         end
 
         def bisect_over_recursive(candidate_ids)
           return if candidate_ids.one?
 
-          slice_size = (candidate_ids.length / 2.0).ceil
-          lhs, rhs = candidate_ids.each_slice(slice_size).to_a
-
           notify(
             :bisect_round_started,
-            :round => 10,
-            :subset_size => slice_size,
-            :remaining_count => candidate_ids.size
+            :candidates_range => candidates_range(candidate_ids),
+            :candidates_count => candidate_ids.size
           )
 
+          slice_size = (candidate_ids.length / 2.0).ceil
+          lhs, rhs = candidate_ids.each_slice(slice_size).to_a
+
           ids_to_ignore = [lhs, rhs].find do |ids|
             get_expected_failures_for?(remaining_ids - ids)
           end
@@ -58,6 +60,7 @@ def bisect_over_recursive(candidate_ids)
             notify(
               :bisect_ignoring_ids,
               :ids_to_ignore => ids_to_ignore,
+              :ignore_range => candidates_range(ids_to_ignore),
               :remaining_ids => remaining_ids
             )
             bisect_over_recursive(candidate_ids - ids_to_ignore)
@@ -82,6 +85,13 @@ def repro_command_for_currently_needed_ids
 
       private
 
+        def candidates_range(ids)
+          Range.new(
+            non_failing_example_ids.find_index(ids.first) + 1,
+            non_failing_example_ids.find_index(ids.last) + 1
+          )
+        end
+
         def prep
           notify(:bisect_starting, :original_cli_args => runner.original_cli_args)
 

lib/rspec/core/formatters/bisect_progress_formatter.rb

@@ -10,7 +10,9 @@ class BisectProgressFormatter < BaseTextFormatter
         Formatters.register self, :bisect_starting, :bisect_original_run_complete,
                             :bisect_round_started, :bisect_individual_run_complete,
                             :bisect_round_finished, :bisect_complete, :bisect_repro_command,
-                            :bisect_failed, :bisect_aborted
+                            :bisect_failed, :bisect_aborted, :bisect_ignoring_ids,
+                            :bisect_dependency_check_started, :bisect_dependency_check_passed,
+                            :bisect_dependency_check_failed
 
         def bisect_starting(notification)
           options = notification.original_cli_args.join(' ')
@@ -26,18 +28,38 @@ def bisect_original_run_complete(notification)
           output.puts "Starting bisect with #{failures} and #{non_failures}."
         end
 
+        def bisect_dependency_check_started(notification)
+          output.print "Checking that failures are order-dependent.."
+        end
+
+        def bisect_dependency_check_passed(notification)
+          output.puts " failure is order-dependent"
+        end
+
+        def bisect_dependency_check_failed(notification)
+          output.puts " failure is not order-dependent!"
+        end
+
         def bisect_round_started(notification, include_trailing_space=true)
-          search_desc = Helpers.pluralize(
-            notification.subset_size, "non-failing example"
-          )
+          range_desc = [
+            notification.candidates_range.begin,
+            notification.candidates_range.end,
+          ].join("-")
 
-          output.print "\nRound #{notification.round}: searching for #{search_desc}" \
-                       " (of #{notification.remaining_count}) to ignore:"
+          output.print "\nBisecting over non-failing examples #{range_desc}"
           output.print " " if include_trailing_space
         end
 
-        def bisect_round_finished(notification)
-          output.print " (#{Helpers.format_duration(notification.duration)})"
+        def bisect_ignoring_ids(notification)
+          ignore_desc =
+            if notification.ignore_range.one?
+              "example #{notification.ignore_range.begin}"
+            else
+              "examples #{notification.ignore_range.begin}-" \
+                "#{notification.ignore_range.end}"
+            end
+
+          output.print " ignoring #{ignore_desc}"
         end
 
         def bisect_individual_run_complete(_)

spec/rspec/core/bisect/coordinator_spec.rb

@@ -34,10 +34,10 @@ def find_minimal_repro(output, formatter=Formatters::BisectProgressFormatter)
         |Bisect started using options: ""
         |Running suite to find failures... (n.nnnn seconds)
         |Starting bisect with 2 failing examples and 6 non-failing examples.
+        |Checking that failures are order-dependent... failure is order-dependent
         |
-        |Round 1: searching for 3 non-failing examples (of 6) to ignore: .. (n.nnnn seconds)
-        |Round 2: searching for 2 non-failing examples (of 3) to ignore: . (n.nnnn seconds)
-        |Round 3: searching for 1 non-failing example (of 1) to ignore: . (n.nnnn seconds)
+        |Bisecting over non-failing examples 1-6 .. ignoring examples 4-6
+        |Bisecting over non-failing examples 1-3 . ignoring examples 1-2
         |Bisect complete! Reduced necessary non-failing examples from 6 to 1 in n.nnnn seconds.
         |
         |The minimal reproduction command is:
@@ -63,8 +63,10 @@ def find_minimal_repro(output, formatter=Formatters::BisectProgressFormatter)
         |    - 6.rb[1:1]
         |    - 7.rb[1:1]
         |    - 8.rb[1:1]
+        |Checking that failures are order-dependent..
+        | - Running: rspec 2.rb[1:1] 5.rb[1:1] (n.nnnn seconds) failure is order-dependent
         |
-        |Round 1: searching for 3 non-failing examples (of 6) to ignore:
+        |Bisecting over non-failing examples 1-6
         | - Running: rspec 2.rb[1:1] 5.rb[1:1] 6.rb[1:1] 7.rb[1:1] 8.rb[1:1] (n.nnnn seconds)
         | - Running: rspec 1.rb[1:1] 2.rb[1:1] 3.rb[1:1] 4.rb[1:1] 5.rb[1:1] (n.nnnn seconds)
         | - Examples we can safely ignore (3):
@@ -75,18 +77,13 @@ def find_minimal_repro(output, formatter=Formatters::BisectProgressFormatter)
         |    - 1.rb[1:1]
         |    - 3.rb[1:1]
         |    - 4.rb[1:1]
-        | - Round finished (n.nnnn seconds)
-        |Round 2: searching for 2 non-failing examples (of 3) to ignore:
+        |Bisecting over non-failing examples 1-3
         | - Running: rspec 2.rb[1:1] 4.rb[1:1] 5.rb[1:1] (n.nnnn seconds)
         | - Examples we can safely ignore (2):
         |    - 1.rb[1:1]
         |    - 3.rb[1:1]
         | - Remaining non-failing examples (1):
         |    - 4.rb[1:1]
-        | - Round finished (n.nnnn seconds)
-        |Round 3: searching for 1 non-failing example (of 1) to ignore:
-        | - Running: rspec 2.rb[1:1] 5.rb[1:1] (n.nnnn seconds)
-        | - Round finished (n.nnnn seconds)
         |Bisect complete! Reduced necessary non-failing examples from 6 to 1 in n.nnnn seconds.
         |
         |The minimal reproduction command is:
@@ -99,8 +96,8 @@ def find_minimal_repro(output, formatter=Formatters::BisectProgressFormatter)
         Class.new(Formatters::BisectProgressFormatter) do
           Formatters.register self
 
-          def bisect_round_finished(notification)
-            return super unless notification.round == 2
+          def bisect_round_started(notification)
+            return super unless notification.candidates_range == (1..3)
 
             Process.kill("INT", Process.pid)
             # Process.kill is not a synchronous call, so to ensure the output
@@ -128,14 +125,14 @@ def bisect_round_finished(notification)
           |Bisect started using options: ""
           |Running suite to find failures... (n.nnnn seconds)
           |Starting bisect with 2 failing examples and 6 non-failing examples.
+          |Checking that failures are order-dependent... failure is order-dependent
           |
-          |Round 1: searching for 3 non-failing examples (of 6) to ignore: .. (n.nnnn seconds)
-          |Round 2: searching for 2 non-failing examples (of 3) to ignore: .
+          |Bisecting over non-failing examples 1-6 .. ignoring examples 4-6
           |
           |Bisect aborted!
           |
           |The most minimal reproduction command discovered so far is:
-          |  rspec 2.rb[1:1] 4.rb[1:1] 5.rb[1:1]
+          |  rspec 1.rb[1:1] 2.rb[1:1] 3.rb[1:1] 4.rb[1:1] 5.rb[1:1]
         EOS
       end
     end