Merge pull request #1997 from urbanautomaton/recursive-bisect-strategy

Recursive bisect strategy

Author: myronmarston

features/command_line/bisect.feature

@@ -54,12 +54,12 @@ Feature: Bisect
       Bisect started using options: "--seed 1234"
       Running suite to find failures... (0.16755 seconds)
       Starting bisect with 1 failing example and 9 non-failing examples.
+      Checking that failure(s) are order-dependent... failure appears to be order-dependent
 
-      Round 1: searching for 5 non-failing examples (of 9) to ignore: .. (0.30166 seconds)
-      Round 2: searching for 3 non-failing examples (of 5) to ignore: .. (0.30306 seconds)
-      Round 3: searching for 2 non-failing examples (of 3) to ignore: .. (0.33292 seconds)
-      Round 4: searching for 1 non-failing example (of 2) to ignore: . (0.16476 seconds)
-      Round 5: searching for 1 non-failing example (of 1) to ignore: . (0.15329 seconds)
+      Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9 (0.30166 seconds)
+      Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5 (0.30306 seconds)
+      Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.33292 seconds)
+      Round 4: bisecting over non-failing examples 1-2 . ignoring example 1 (0.16476 seconds)
       Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.26 seconds.
 
       The minimal reproduction command is:
@@ -75,10 +75,11 @@ Feature: Bisect
       Bisect started using options: "--seed 1234"
       Running suite to find failures... (0.17102 seconds)
       Starting bisect with 1 failing example and 9 non-failing examples.
+      Checking that failure(s) are order-dependent... failure appears to be order-dependent
 
-      Round 1: searching for 5 non-failing examples (of 9) to ignore: .. (0.32943 seconds)
-      Round 2: searching for 3 non-failing examples (of 5) to ignore: .. (0.3154 seconds)
-      Round 3: searching for 2 non-failing examples (of 3) to ignore: ..
+      Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9 (0.32943 seconds)
+      Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5 (0.3154 seconds)
+      Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.2175 seconds)
 
       Bisect aborted!
 
@@ -106,8 +107,10 @@ Feature: Bisect
           - ./spec/calculator_7_spec.rb[1:1]
           - ./spec/calculator_8_spec.rb[1:1]
           - ./spec/calculator_9_spec.rb[1:1]
-
-      Round 1: searching for 5 non-failing examples (of 9) to ignore:
+      Checking that failure(s) are order-dependent..
+       - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (n.nnnn seconds)
+       - Failure appears to be order-dependent
+      Round 1: bisecting over non-failing examples 1-9
        - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 (0.15302 seconds)
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.19708 seconds)
        - Examples we can safely ignore (4):
@@ -121,8 +124,7 @@ Feature: Bisect
           - ./spec/calculator_3_spec.rb[1:1]
           - ./spec/calculator_4_spec.rb[1:1]
           - ./spec/calculator_5_spec.rb[1:1]
-       - Round finished (0.35172 seconds)
-      Round 2: searching for 3 non-failing examples (of 5) to ignore:
+      Round 2: bisecting over non-failing examples 1-5
        - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.15836 seconds)
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.19065 seconds)
        - Examples we can safely ignore (2):
@@ -132,26 +134,20 @@ Feature: Bisect
           - ./spec/calculator_10_spec.rb[1:1]
           - ./spec/calculator_2_spec.rb[1:1]
           - ./spec/calculator_3_spec.rb[1:1]
-       - Round finished (0.35022 seconds)
-      Round 3: searching for 2 non-failing examples (of 3) to ignore:
+      Round 3: bisecting over non-failing examples 1-3
        - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 (0.21028 seconds)
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.1975 seconds)
        - Examples we can safely ignore (1):
           - ./spec/calculator_2_spec.rb[1:1]
        - Remaining non-failing examples (2):
           - ./spec/calculator_10_spec.rb[1:1]
           - ./spec/calculator_3_spec.rb[1:1]
-       - Round finished (0.40882 seconds)
-      Round 4: searching for 1 non-failing example (of 2) to ignore:
+      Round 4: bisecting over non-failing examples 1-2
        - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.17173 seconds)
        - Examples we can safely ignore (1):
           - ./spec/calculator_3_spec.rb[1:1]
        - Remaining non-failing examples (1):
           - ./spec/calculator_10_spec.rb[1:1]
-       - Round finished (0.17234 seconds)
-      Round 5: searching for 1 non-failing example (of 1) to ignore:
-       - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.18279 seconds)
-       - Round finished (0.18312 seconds)
       Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.47 seconds.
 
       The minimal reproduction command is:

features/support/send_sigint_during_bisect.rb

@@ -5,8 +5,8 @@ module RSpec::Core::Formatters
   BisectProgressFormatter = Class.new(remove_const :BisectProgressFormatter) do
     RSpec::Core::Formatters.register self
 
-    def bisect_round_finished(notification)
-      return super unless notification.round == 3
+    def bisect_round_started(notification)
+      return super unless @round_count == 3
 
       Process.kill("INT", Process.pid)
       # Process.kill is not a synchronous call, so to ensure the output

lib/rspec/core/bisect/example_minimizer.rb

@@ -1,5 +1,3 @@
-RSpec::Support.require_rspec_core "bisect/subset_enumerator"
-
 module RSpec
   module Core
     module Bisect
@@ -18,20 +16,65 @@ def initialize(runner, reporter)
         def find_minimal_repro
           prep
 
-          self.remaining_ids = non_failing_example_ids
+          _, duration = track_duration do
+            bisect(non_failing_example_ids)
+          end
+
+          notify(:bisect_complete, :duration => duration,
+                                   :original_non_failing_count => non_failing_example_ids.size,
+                                   :remaining_count => remaining_ids.size)
+
+          remaining_ids + failed_example_ids
+        end
+
+        def bisect(candidate_ids)
+          notify(:bisect_dependency_check_started)
+          if get_expected_failures_for?([])
+            notify(:bisect_dependency_check_failed)
+            self.remaining_ids = []
+            return
+          end
+          notify(:bisect_dependency_check_passed)
+
+          bisect_over(candidate_ids)
+        end
+
+        def bisect_over(candidate_ids)
+          return if candidate_ids.one?
+
+          notify(
+            :bisect_round_started,
+            :candidate_range => example_range(candidate_ids),
+            :candidates_count => candidate_ids.size
+          )
 
-          each_bisect_round do |subsets|
-            ids_to_ignore = subsets.find do |ids|
+          slice_size = (candidate_ids.length / 2.0).ceil
+          lhs, rhs = candidate_ids.each_slice(slice_size).to_a
+
+          ids_to_ignore, duration = track_duration do
+            [lhs, rhs].find do |ids|
               get_expected_failures_for?(remaining_ids - ids)
             end
+          end
 
-            next :done unless ids_to_ignore
-
+          if ids_to_ignore
             self.remaining_ids -= ids_to_ignore
-            notify(:bisect_ignoring_ids, :ids_to_ignore => ids_to_ignore, :remaining_ids => remaining_ids)
+            notify(
+              :bisect_round_ignoring_ids,
+              :ids_to_ignore => ids_to_ignore,
+              :ignore_range => example_range(ids_to_ignore),
+              :remaining_ids => remaining_ids,
+              :duration => duration
+            )
+            bisect_over(candidate_ids - ids_to_ignore)
+          else
+            notify(
+              :bisect_round_detected_multiple_culprits,
+              :duration => duration
+            )
+            bisect_over(lhs)
+            bisect_over(rhs)
           end
-
-          currently_needed_ids
         end
 
         def currently_needed_ids
@@ -43,15 +86,35 @@ def repro_command_for_currently_needed_ids
           "(Not yet enough information to provide any repro command)"
         end
 
+        # @private
+        # Convenience class for describing a subset of the candidate examples
+        ExampleRange = Struct.new(:start, :finish) do
+          def description
+            if start == finish
+              "example #{start}"
+            else
+              "examples #{start}-#{finish}"
+            end
+          end
+        end
+
       private
 
+        def example_range(ids)
+          ExampleRange.new(
+            non_failing_example_ids.find_index(ids.first) + 1,
+            non_failing_example_ids.find_index(ids.last) + 1
+          )
+        end
+
         def prep
           notify(:bisect_starting, :original_cli_args => runner.original_cli_args)
 
           _, duration = track_duration do
             original_results    = runner.original_results
             @all_example_ids    = original_results.all_example_ids
             @failed_example_ids = original_results.failed_example_ids
+            @remaining_ids      = non_failing_example_ids
           end
 
           if @failed_example_ids.empty?
@@ -70,7 +133,11 @@ def non_failing_example_ids
 
         def get_expected_failures_for?(ids)
           ids_to_run = ids + failed_example_ids
-          notify(:bisect_individual_run_start, :command => runner.repro_command_from(ids_to_run))
+          notify(
+            :bisect_individual_run_start,
+            :command => runner.repro_command_from(ids_to_run),
+            :ids_to_run => ids_to_run
+          )
 
           results, duration = track_duration { runner.run(ids_to_run) }
           notify(:bisect_individual_run_complete, :duration => duration, :results => results)
@@ -79,34 +146,6 @@ def get_expected_failures_for?(ids)
           (failed_example_ids & results.failed_example_ids) == failed_example_ids
         end
 
-        INFINITY = (1.0 / 0) # 1.8.7 doesn't define Float::INFINITY so we define our own...
-
-        def each_bisect_round(&block)
-          last_round, duration = track_duration do
-            1.upto(INFINITY) do |round|
-              break if :done == bisect_round(round, &block)
-            end
-          end
-
-          notify(:bisect_complete, :round => last_round, :duration => duration,
-                                   :original_non_failing_count => non_failing_example_ids.size,
-                                   :remaining_count => remaining_ids.size)
-        end
-
-        def bisect_round(round)
-          value, duration = track_duration do
-            subsets = SubsetEnumerator.new(remaining_ids)
-            notify(:bisect_round_started, :round => round,
-                                          :subset_size => subsets.subset_size,
-                                          :remaining_count => remaining_ids.size)
-
-            yield subsets
-          end
-
-          notify(:bisect_round_finished, :duration => duration, :round => round)
-          value
-        end
-
         def track_duration
           start = ::RSpec::Core::Time.now
           [yield, ::RSpec::Core::Time.now - start]