reindex on mixed type/object axis fails

[gdementen]

>>> arr = ndtest(Axis(np.array([0, "a"], dtype=object), "value"))
>>> arr.reindex("value", [0, "a", 1])
TypeError: '<' not supported between instances of 'str' and 'int'

Note that users usually get a mixed type axis via with_total.

I think the underlying error is that Axis.index does not work for numpy array with mixed dtype

>>> ax.index(np.array([0, "a"]))
--> 895                 return array_lookup2(_seq_group_to_name(key), self._sorted_keys, self._sorted_values)
c:\users\gdm\devel\larray\larray\core\axis.py in _sorted_keys(self)
--> 144             keys, _ = self._update_key_values()
c:\users\gdm\devel\larray\larray\core\axis.py in _update_key_values(self)
--> 133             sorted_keys, sorted_values = tuple(zip(*sorted(mapping.items())))
TypeError: '<' not supported between instances of 'str' and 'int'

But we should get rid of these _sorted_keys/_sorted_values anyway (because that's O(N log N) instead of O(N)) so the fix will come for free "someday". But we should add a test for this issue nevertheless.

[gdementen]

related to #909

[alixdamman]

Why do we need to sort the keys and values anyway ?

[gdementen]

Why do we need to sort the keys and values anyway ?

It is a performance optimization. It is the only way I found to make the value-to-index transformation using only numpy operations (and thus at C speed). There are two problems with this approach:

• it has a non-negligible setup cost, so it is only worth it if we do those transformations several times for the same axis. I am unsure it is the case in practice but I assume(d) it is on average.
• it is very fast when the axis is small (the usual case) but is O(N log N), so grows more than linearly than the simpler options with the size of the axis.

I would like to revisit this someday. Ideally we should go for a simpler option (algorithmically speaking) using a mapping but with a small part implemented in C (probably using khash like Pandas and LIAM2).