Skip to content

BUG: Mitigate division with zero in roll_var #42459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jul 28, 2021
Merged

BUG: Mitigate division with zero in roll_var #42459

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
11d702d
BUG: Mitigate division with zero in roll_var
benHeid Jul 9, 2021
113e35b
Add tests with different rolling_arguments
benHeid Jul 12, 2021
0307665
Fix linting errors (Line to long)
benHeid Jul 12, 2021
6ac50a5
Remove autoformatted changes.
benHeid Jul 12, 2021
ba6fa38
Update parameters
benHeid Jul 12, 2021
2f0085e
Remove autoformatted changes again..
benHeid Jul 12, 2021
0310b53
Fix linting errors.
benHeid Jul 23, 2021
9797d20
Fix linting error
benHeid Jul 23, 2021
d39569f
Add comment in the release notes
benHeid Jul 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v1.4.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ Plotting
Groupby/resample/rolling
^^^^^^^^^^^^^^^^^^^^^^^^
- Bug in :meth:`Series.rolling.apply`, :meth:`DataFrame.rolling.apply`, :meth:`Series.expanding.apply` and :meth:`DataFrame.expanding.apply` with ``engine="numba"`` where ``*args`` were being cached with the user passed function (:issue:`42287`)
-
- Bug in :meth:`DataFrame.groupby.rolling.var` would calculate the rolling variance only on the first group (:issue:`42442`)

Reshaping
^^^^^^^^^
Expand Down
5 changes: 4 additions & 1 deletion pandas/_libs/window/aggregations.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,10 @@ cdef inline void add_var(float64_t val, float64_t *nobs, float64_t *mean_x,
t = y - mean_x[0]
compensation[0] = t + mean_x[0] - y
delta = t
mean_x[0] = mean_x[0] + delta / nobs[0]
if nobs[0]:
mean_x[0] = mean_x[0] + delta / nobs[0]
else:
mean_x[0] = 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this zero? and not NaN

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that if nobs is zero a new group starts, so I reset mean_x to zero. Analog to the initialization of mean_x in roll_var (compare with the initalization in line 349). However, I am not sure if this is correct.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Off the top of my head, this would also need testing when min_periods is 0 or len(array) as well to check if this is correct

ssqdm_x[0] = ssqdm_x[0] + (val - prev_mean) * (val - mean_x[0])


Expand Down
25 changes: 25 additions & 0 deletions pandas/tests/window/test_groupby.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -695,6 +695,31 @@ def test_groupby_rolling_object_doesnt_affect_groupby_apply(self):
assert not g.mutated
assert not g.grouper.mutated

@pytest.mark.parametrize(
("window", "min_periods", "closed", "expected"),
[
(2, 0, "left", [None, 0.0, 1.0, 1.0, None, 0.0, 1.0, 1.0]),
(2, 2, "left", [None, None, 1.0, 1.0, None, None, 1.0, 1.0]),
(4, 4, "left", [None, None, None, None, None, None, None, None]),
(4, 4, "right", [None, None, None, 5.0, None, None, None, 5.0]),
],
)
def test_groupby_rolling_var(self, window, min_periods, closed, expected):
df = DataFrame([1, 2, 3, 4, 5, 6, 7, 8])
result = (
df.groupby([1, 2, 1, 2, 1, 2, 1, 2])
.rolling(window=window, min_periods=min_periods, closed=closed)
.var(0)
)
expected_result = DataFrame(
np.array(expected, dtype="float64"),
index=MultiIndex(
levels=[[1, 2], [0, 1, 2, 3, 4, 5, 6, 7]],
codes=[[0, 0, 0, 0, 1, 1, 1, 1], [0, 2, 4, 6, 1, 3, 5, 7]],
),
)
tm.assert_frame_equal(result, expected_result)

@pytest.mark.parametrize(
"columns", [MultiIndex.from_tuples([("A", ""), ("B", "C")]), ["A", "B"]]
)
Expand Down