Skip to content

BUG: infer_dtype with decimal/complex #37176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 18, 2020
Merged

BUG: infer_dtype with decimal/complex #37176

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
848b155
BUG: infer_dtype with decimal/complex
jbrockmendel Oct 16, 2020
686d528
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 17, 2020
4597993
implement is_decimal_array
jbrockmendel Oct 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions pandas/_libs/lib.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1414,10 +1414,12 @@ def infer_dtype(value: object, skipna: bool = True) -> str:
return "time"

elif is_decimal(val):
return "decimal"
if all(is_decimal(x) for x in values):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you create is_decimal_array to have the same style

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you actually want to use the validator e.g as a raw NaN would not work here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you expand on this? do you expect something other than Decimal objects to work here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NaNs are valid as well here; but more important is that every validator has the same pattern. a is_* function e.g. is_decimal_array, which should call here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to use the is_* pattern.

since decimal has Decimal("nan"), i wouldnt expect to accept np.nan here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since decimal has Decimal("nan"), i wouldnt expect to accept np.nan here

yeah ok, though i think we could actually accept np.nan as well and infer a decimal if that is the case (as we do this for all other dtypes). can you open an issue.

return "decimal"

elif is_complex(val):
return "complex"
if is_complex_array(values):
return "complex"

elif util.is_float_object(val):
if is_float_array(values):
Expand Down Expand Up @@ -1702,6 +1704,23 @@ cpdef bint is_float_array(ndarray values):
return validator.validate(values)


cdef class ComplexValidator(Validator):
cdef inline bint is_value_typed(self, object value) except -1:
return (
util.is_complex_object(value)
or (util.is_float_object(value) and is_nan(value))
)

cdef inline bint is_array_typed(self) except -1:
return issubclass(self.dtype.type, np.complexfloating)


cdef bint is_complex_array(ndarray values):
cdef:
ComplexValidator validator = ComplexValidator(len(values), values.dtype)
return validator.validate(values)


cdef class StringValidator(Validator):
cdef inline bint is_value_typed(self, object value) except -1:
return isinstance(value, str)
Expand Down Expand Up @@ -2546,8 +2565,6 @@ def fast_multiget(dict mapping, ndarray keys, default=np.nan):
# kludge, for Series
return np.empty(0, dtype='f8')

keys = getattr(keys, 'values', keys)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated cleanup

for i in range(n):
val = keys[i]
if val in mapping:
Expand Down
6 changes: 6 additions & 0 deletions pandas/tests/dtypes/test_inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,6 +709,9 @@ def test_decimals(self):
result = lib.infer_dtype(arr, skipna=True)
assert result == "mixed"

result = lib.infer_dtype(arr[::-1], skipna=True)
assert result == "mixed"

arr = np.array([Decimal(1), Decimal("NaN"), Decimal(3)])
result = lib.infer_dtype(arr, skipna=True)
assert result == "decimal"
Expand All @@ -729,6 +732,9 @@ def test_complex(self, skipna):
result = lib.infer_dtype(arr, skipna=skipna)
assert result == "mixed"

result = lib.infer_dtype(arr[::-1], skipna=skipna)
assert result == "mixed"

# gets cast to complex on array construction
arr = np.array([1, np.nan, 1 + 1j])
result = lib.infer_dtype(arr, skipna=skipna)
Expand Down