pandas-dev · jreback · Apr 27, 2020 · Apr 22, 2020 · Apr 23, 2020 · Apr 24, 2020
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
@@ -659,6 +659,8 @@ Other
 - Bug in :meth:`Series.map` not raising on invalid ``na_action`` (:issue:`32815`)
 - Bug in :meth:`DataFrame.__dir__` caused a segfault when using unicode surrogates in a column name (:issue:`25509`)
 - Bug in :meth:`DataFrame.plot.scatter` caused an error when plotting variable marker sizes (:issue:`32904`)
+- Propagate Python exceptions from :func:`_lib.c_is_list_like` (#33721)
+-
 
 .. ---------------------------------------------------------------------------
 

diff --git a/pandas/_libs/lib.pxd b/pandas/_libs/lib.pxd
@@ -1 +1 @@
-cdef bint c_is_list_like(object, bint)
+cdef bint c_is_list_like(object, bint) except -1
diff --git a/pandas/_libs/lib.pyx b/pandas/_libs/lib.pyx
@@ -985,7 +985,7 @@ def is_list_like(obj: object, allow_sets: bool = True) -> bool:
     return c_is_list_like(obj, allow_sets)
 
 
-cdef inline bint c_is_list_like(object obj, bint allow_sets):
+cdef inline bint c_is_list_like(object obj, bint allow_sets) except -1:
     return (
         isinstance(obj, abc.Iterable)
         # we do not count strings/unicode/bytes as list-like
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		cdef bint c_is_list_like(object, bint)
		cdef bint c_is_list_like(object, bint) except -1
Copy link Member WillAyd Apr 22, 2020 Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Hmm don't think a bint can ever return -1. The only thing I think you could except here is NULL but would really have to refactor to do that Is there a particular issue you are trying to solve? Copy link Contributor Author shwina Apr 22, 2020 • edited Loading Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Thanks for reviewing. The ignored exception led to a segfault in https://github.com/rapidsai/cudf that I'm unable to reproduce at the moment since we've since refactored our code a bit -- apologies. However, we can demonstrate the issue by raising artificially within `c_is_list_like` -- I believe it's valid to assume that Python exceptions can originate from within this function. Without the `except -1`: cdef inline bint c_is_list_like(object obj, bint allow_sets): raise ZeroDivisionError() return ( isinstance(obj, abc.Iterable) # we do not count strings/unicode/bytes as list-like and not isinstance(obj, (str, bytes)) # exclude zero-dimensional numpy arrays, effectively scalars and not (util.is_array(obj) and obj.ndim == 0) # exclude sets if allow_sets is False and not (allow_sets is False and isinstance(obj, abc.Set)) ) >>> import pandas as pd >>> pd.api.types.is_list_like([]) ZeroDivisionError Exception ignored in: 'pandas._libs.lib.c_is_list_like' ZeroDivisionError: False With the `except -1`: cdef inline bint c_is_list_like(object obj, bint allow_sets) except -1: raise ZeroDivisionError() return ( isinstance(obj, abc.Iterable) # we do not count strings/unicode/bytes as list-like and not isinstance(obj, (str, bytes)) # exclude zero-dimensional numpy arrays, effectively scalars and not (util.is_array(obj) and obj.ndim == 0) # exclude sets if allow_sets is False and not (allow_sets is False and isinstance(obj, abc.Set)) ) >>> import pandas as pd >>>pd.api.types.is_list_like([]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pandas/_libs/lib.pyx", line 985, in pandas._libs.lib.is_list_like return c_is_list_like(obj, allow_sets) File "pandas/_libs/lib.pyx", line 989, in pandas._libs.lib.c_is_list_like raise ZeroDivisionError() ZeroDivisionError (`-1` seems to be a valid value for `bint`) Copy link Member jbrockmendel Apr 22, 2020 Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. The ignored exception led to a segfault Any idea what value was being passed? Its hard to imagine what could cause this to raise unless you specifically rigged it to make isinstance checks fail Copy link Contributor Author shwina Apr 22, 2020 Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Aha - so after digging up some old code, it looks the source of the problem was a `RecursionError`, which happened to be thrown within `isinstance` here. Copy link Contributor Author shwina Apr 22, 2020 • edited Loading Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Minimal repro (segfaults for me): `In [1]: import pandas as pd In [2]: def foo(): ...: pd.api.types.is_list_like("test") ...: foo()` Copy link Contributor jreback Apr 23, 2020 Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. -1 is appropriate here as it signals cython that there maybe is an exception to check.