Skip to content

ENH: support of pandas.DataFrame.hist for datetime data #36287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Oct 10, 2020
Merged

ENH: support of pandas.DataFrame.hist for datetime data #36287

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
e8989e6
ENH: support of pandas.DataFrame.hist for datetime data
onshek Sep 11, 2020
ed86887
Revert "ENH: support of pandas.DataFrame.hist for datetime data"
onshek Sep 13, 2020
9dc96cf
ENH: support of pandas.DataFrame.hist for datetime data
onshek Sep 13, 2020
602c59b
REF: reformat imports
onshek Sep 13, 2020
17c182f
UPD: changed as requested
onshek Sep 14, 2020
aad3433
Merge branch 'master' into issue32590-Histogram-or-kde-from-datetime-…
onshek Sep 14, 2020
2cfc979
FIX: recover whatsnew note v1.2.0
onshek Sep 14, 2020
7569a31
UPD: delete an extra line
onshek Sep 14, 2020
35cccd6
UPD: explicit is better than implicit
onshek Sep 15, 2020
bd7f450
UPD: hist support for type of datetime64 and datetimetz
onshek Sep 17, 2020
731f097
Merge branch 'master' into issue32590-Histogram-or-kde-from-datetime-…
onshek Sep 17, 2020
3a99250
UPD: modify and reformat related tests
onshek Sep 20, 2020
e9fa7ec
UPD: reformat code
onshek Oct 4, 2020
1c098c8
Merge branch 'master' into issue32590-Histogram-or-kde-from-datetime-…
onshek Oct 4, 2020
b24df4d
UPD: focus on main issues
onshek Oct 4, 2020
b02c017
UPD: recover # GH 9351
onshek Oct 4, 2020
315d77e
UPD: recover a blank line
onshek Oct 4, 2020
c8c7460
UPD: reformat the blank line
onshek Oct 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions pandas/plotting/_matplotlib/hist.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
from pandas.core.dtypes.generic import ABCDataFrame, ABCIndexClass
from pandas.core.dtypes.missing import isna, remove_na_arraylike

from pandas.core.reshape.concat import concat

from pandas.io.formats.printing import pprint_thing
from pandas.plotting._matplotlib.core import LinePlot, MPLPlot
from pandas.plotting._matplotlib.tools import (
Expand Down Expand Up @@ -417,11 +419,21 @@ def hist_frame(
if not isinstance(column, (list, np.ndarray, ABCIndexClass)):
column = [column]
data = data[column]
data = data._get_numeric_data()
# GH32590
columns_copy = data.columns
numeric_data = data._get_numeric_data()
datetime_data = data.select_dtypes(include="datetime64[ns]")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should only datetime type be included? what about timedelta? could you pls check if timedelta should also be supported?

if so you could do something like:

include_type = ['datetime', 'timedelta']
datetime_data = data.select_dtypes(include=include_type)

or you could even do

include_type = ['datetime', 'timedelta', np.number]
data = data.select_dtypes(include=include_type)

then probably you could avoid using data._get_numeric_data() and concat.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with the first point.
Would it be ok to use np.number instead of _get_numeric_data()? I'm not sure about this.
I'll have a check on both.

Copy link
Contributor Author

@onshek onshek Sep 15, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@charlesdong1991 @jreback timedelta is not supported even in pd.Series.hist but, we can convert both timedelta and datetime[ns] into np.int64 then keep original data=data._get_numeric_data().
As to the second point, this minimal exsample shows df.select_dtypes(include=np.number) diffs from df._get_numeric_data():

import numpy as np
from pandas import DataFrame, to_datetime
from datetime import timedelta

df = DataFrame({"a": np.random.randn(10),
                "b": [timedelta(np.random.randn()) for _ in range(10)],          
                "c": to_datetime(np.random.randint(1582800000000000000, 1583500000000000000, 10, dtype=np.int64)),})

df.select_dtypes(include=np.number)
a	b
0	0.848384	-1 days +07:39:32.366134
1	-0.184510	-1 days +14:42:46.719730
2	-0.835072	1 days 06:49:21.386804
3	-0.026554	0 days 22:18:48.433275
4	-2.365708	-1 days +08:52:09.934553
5	0.974325	1 days 05:30:18.644021
6	0.135194	1 days 05:06:18.969120
7	1.802466	1 days 07:00:26.538467
8	-1.509265	0 days 00:19:41.677979
9	-0.752476	-1 days +09:55:45.548658

df._get_numeric_data()
a
0	0.848384
1	-0.184510
2	-0.835072
3	-0.026554
4	-2.365708
5	0.974325
6	0.135194
7	1.802466
8	-1.509265
9	-0.752476

Any thought?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, timedelta is treated as numbers since it represents interval, probably _get_numeric_data does some other processingss to filter them out, then I think you could keep _get_numeric_data and concat in this PR!

data = concat([numeric_data, datetime_data], axis=1)
naxes = len(data.columns)

if naxes == 0:
raise ValueError("hist method requires numerical columns, nothing to plot.")
raise ValueError(
"hist method requires numerical or datetime columns, nothing to plot."
)
else:
data = data.reindex(
columns=[name for name in columns_copy if name in data.columns]
)

fig, axes = create_subplots(
naxes=naxes,
Expand Down
21 changes: 16 additions & 5 deletions pandas/tests/plotting/test_hist_method.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

import pandas.util._test_decorators as td

from pandas import DataFrame, Index, Series
from pandas import DataFrame, Index, Series, to_datetime
import pandas._testing as tm
from pandas.tests.plotting.common import TestPlotBase, _check_plot_works

Expand Down Expand Up @@ -225,12 +225,23 @@ def test_hist_df_legacy(self):
ser.hist(foo="bar")

@pytest.mark.slow
def test_hist_non_numerical_raises(self):
# gh-10444
df = DataFrame(np.random.rand(10, 2))
def test_hist_non_numerical_or_datetime_raises(self):
# gh-10444, GH32590
df = DataFrame(
{
"a": np.random.rand(10),
"b": np.random.rand(10),
"c": to_datetime(
np.random.randint(1582800000000000000, 1583500000000000000, 10)
),
"d": to_datetime(
np.random.randint(1582800000000000000, 1583500000000000000, 10)
),
}
)
df_o = df.astype(object)

msg = "hist method requires numerical columns, nothing to plot."
msg = "hist method requires numerical or datetime columns, nothing to plot."
with pytest.raises(ValueError, match=msg):
df_o.hist()

Expand Down