Allow filters of the same type to be combined

When filters are applied, we require all specified filters to approve
an item in order for it to match. When configuration is combined from
multiple sources, this can make it extremely difficult to integrate
filters into an existing list.

We would like to extend an existing passlist, blocklist, or
combination, but that's an impractical transformation on general YAML
(the filters are lists, and the only sane transformation at the
generic level is to concatenate lists).

Specifying two passlists, blocklists or combinations does not have the
desired effect, because of the `all` logic governing multiple filters,
such that we get the intersection of the specifications, not the
union. That is, e.g. for two passlists, only items on both lists can
match.

This commit adds a `combine` function to filters, such that they can
opt to absorb multiple declarations of the same type. When a second
passlist is encountered in the same context, the items in the second
specification are passed to the object made for the first. It can then
treat the specification as extending its own passlist.

The FilterFactory has been updated to use this combine logic.

Signed-off-by: Ed Smith <[email protected]>

Author: eds-collabora

kernelci/config/base.py

@@ -80,6 +80,31 @@ def match(self, **kw):
         """Return True if the given *kw* keywords match the filter."""
         raise NotImplementedError("Filter.match() is not implemented")
 
+    def combine(self, items):
+        """Try to avoid making a new filter if we can make a combined
+        filter matching both our existing data and the new items.
+
+        The *items* can be any data used to filter configurations.
+        Return True if we can combine, False otherwise.
+        """
+        return False
+
+
+def _merge_filter_lists(old, update):
+    """Merge the items for a Blocklist or Passlist.
+
+    *old*    is the items from the existing filter list that we
+             have already created.
+
+    *update* are the items of a new filter list, of the same type as
+             *old*, loaded from another configuration source. We are
+             going represent this second filter list declaration by
+             updating *old*, rather than by having a new object to
+             represent it.
+    """
+    for key, value in update.items():
+        old.setdefault(key, list()).extend(value)
+
 
 class Blocklist(Filter):
     """Blocklist filter to discard certain configurations.
@@ -99,6 +124,10 @@ def match(self, **kw):
 
         return True
 
+    def combine(self, items):
+        _merge_filter_lists(self._items, items)
+        return True
+
 
 class Passlist(Filter):
     """Passlist filter to only accept certain configurations.
@@ -118,6 +147,10 @@ def match(self, **kw):
 
         return True
 
+    def combine(self, items):
+        _merge_filter_lists(self._items, items)
+        return True
+
 
 class Regex(Filter):
     """Regex filter to only accept certain configurations.
@@ -156,6 +189,14 @@ def match(self, **kw):
         filter_values = tuple(kw.get(k) for k in self._keys)
         return filter_values in self._values
 
+    def combine(self, items):
+        keys = tuple(items['keys'])
+        if keys != self._keys:
+            return False
+
+        self._values.extend([tuple(values) for values in items['values']])
+        return True
+
 
 class FilterFactory(YAMLObject):
     """Factory to create filters from YAML data."""
@@ -171,10 +212,20 @@ class FilterFactory(YAMLObject):
     def from_yaml(cls, filter_params):
         """Iterate through the YAML filters and return Filter objects."""
         filter_list = []
+        filters = {}
+
         for f in filter_params:
             for filter_type, items in f.items():
-                filter_cls = cls._classes[filter_type]
-                filter_list.append(filter_cls(items))
+                for g in filters.get(filter_type, []):
+                    if g.combine(items):
+                        break
+                else:
+                    filter_cls = cls._classes[filter_type]
+                    filter_instance = filter_cls(items)
+                    filters.setdefault(filter_type, list()).append(
+                        filter_instance)
+                    filter_list.append(filter_instance)
+
         return filter_list
 
     @classmethod