Skip to content

BUG: repr of np.datetime64('NaT') in Series/DataFrame with dtype object #25445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 30, 2019
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.25.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ Sparse
Other
^^^^^

-
- Bug in :class:`Series` and :class:`DataFrame` repr where ``np.datetime64('NaT')`` and ``np.timedelta64('NaT')`` with ``dtype=object`` would be represented as ``NaN`` (:issue:`25445`)
-
-

Expand Down
12 changes: 12 additions & 0 deletions pandas/_libs/tslibs/nattype.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,18 @@ _nat_scalar_rules[Py_GE] = False
# ----------------------------------------------------------------------


cpdef bint is_np_nat(x):
"""Compat check for np.datetime('NaT')"""
try:
return np.isnat(x)
except AttributeError:
# numpy 1.12 compat
return str(x) == 'NaT'
except TypeError:
# np.isnat only defined for datetime, timedelta
return False


def _make_nan_func(func_name, doc):
def f(*args, **kwargs):
return np.nan
Expand Down
3 changes: 2 additions & 1 deletion pandas/io/formats/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from pandas._libs import lib
from pandas._libs.tslib import format_array_from_datetime
from pandas._libs.tslibs import NaT, Timedelta, Timestamp, iNaT
from pandas._libs.tslibs.nattype import is_np_nat
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be removed

from pandas.compat import StringIO, lzip, map, u, zip

from pandas.core.dtypes.common import (
Expand Down Expand Up @@ -946,7 +947,7 @@ def _format(x):
if self.na_rep is not None and is_scalar(x) and isna(x):
if x is None:
return 'None'
elif x is NaT:
elif x is NaT or is_np_nat(x):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use is_null_datetimelike instead of rolling your own

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_null_datetimelike is too permissive here (accepts np.nan and None)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want yet another NaT detection routine.

Copy link
Member Author

@mroeschke mroeschke Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_null_datetimelike doesn't just detect NaT though. It checks if any args are None, np.nan, NaT, iNaT

return 'NaT'
return self.na_rep
elif isinstance(x, PandasObject):
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/frame/test_repr_info.py
Original file line number Diff line number Diff line change
Expand Up @@ -521,3 +521,11 @@ def test_repr_categorical_dates_periods(self):

df = DataFrame({'dt': Categorical(dt), 'p': Categorical(p)})
assert repr(df) == exp

@pytest.mark.parametrize('arg', [np.datetime64, np.timedelta64])
@pytest.mark.parametrize('box, expected', [
[Series, '0 NaT\ndtype: object'],
[DataFrame, ' 0\n0 NaT']])
def test_repr_np_nat_with_object(self, arg, box, expected):
result = repr(box([arg('NaT')], dtype=object))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add the issue number

assert result == expected