Skip to content

Fix overflow error in cartesian_product #15265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Fix overflow error in cartesian_product #15265

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b196878
Fix overflow error in cartesian_product
david-hoffman Jan 30, 2017
7aeee85
Added tests for large numbers
david-hoffman Jan 30, 2017
47a6c6c
Update test so that it will actually run on "normal" machine
david-hoffman Jan 30, 2017
d54583e
Remove `test_large_input` because it's too big
david-hoffman Jan 30, 2017
c9c8d5e
Update v0.19.2.txt
david-hoffman Jan 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions pandas/tools/tests/test_util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ def test_invalid_input(self):
msg = "Input must be a list-like of list-likes"
for X in invalid_inputs:
tm.assertRaisesRegexp(TypeError, msg, cartesian_product, X=X)

def test_large_input(self):
# test failure of large inputs on windows OS
X = np.arange(65536)
Y = np.arange(65535)
result1, result2 = cartesian_product([X, Y])
expected_size = X.size * Y.size
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, this is still quite large.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't think of another way to cause the error without adding checking code to the cartesian_product function itself, e.g.

if np.any(cumprodX < 0):
    raise RuntimeError(...)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, let's take the test out (just verify what you were doing works on windows).

pls add a whatsnew note and can merge

note that this is still buggy, because we do the cartesian product in the first place (this is #14942), so I think this will still break for you (just in a different place now).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I switched to an altered version of numpy's histogramdd for this problem.

tm.assert_equal(result1.size, expected_size)
tm.assert_equal(result2.size, expected_size)


class TestLocaleUtils(tm.TestCase):
Expand Down
2 changes: 1 addition & 1 deletion pandas/tools/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ def cartesian_product(X):
if len(X) == 0:
return []

lenX = np.fromiter((len(x) for x in X), dtype=int)
lenX = np.fromiter((len(x) for x in X), dtype=np.intp)
cumprodX = np.cumproduct(lenX)

a = np.roll(cumprodX, 1)
Expand Down