COMPAT: remove compat.scipy

[clham]

No description provided.

[jreback]

is this used anywhere? might be older code

[jorisvandenbossche]

This is only an internal function (copied exactly from scipy https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L4482), and the only use is in the function above (https://github.com/pydata/pandas/blob/master/pandas/compat/scipy.py#L35), which in turn is only used in some tests I think.
So I don't if it is worth to adapt this.

[jreback]

@clham if you'd like to make fastsort internal to rankdata would be ok. these routines are really just to check that the pandas routines track scipy and are not used anywhere outside the test suite AFAIK.

@jorisvandenbossche think we just ought to move them to pandas.util.testing? (and take out compat.scipy)

[jorisvandenbossche]

@jreback maybe that's a good idea, since they are only for testing to put them there.

[jreback]

@clham if you want to do a PR to:

put these functions into pandas.util.testing (and fold fastsort into rankdata), then change the tests to use this (essentially remove compat.scipy)

[jorisvandenbossche]

@jreback the percentileofscore is still used in pandas/stats/misc (although I don't know if that last is really used)

[jreback]

@jorisvandenbossche hmm, I see that.

ok well percentileRank should be changed to percentile_rank (and I don't see it used anywhere)

ok..then just let's take out rankdata and fastsort (and put in testing)

I would even be in favor of making percentile_rank requiring scipy; we use it in a lot of places (and mainting compat is something we shouldn't really be doing)

what do you think?

[jorisvandenbossche]

@jreback I wouldn't change the name of percentileRank, it is nowhere used or publicized in pandas, but if there would be someone using it, not worth the trouble to change it I think (unless we want to add it to the public/publicized api, we should rather deprecate/remove it I think).

But requiring scipy does not seem a problem, then compat.scipy can just be removed.

[jreback]

ok @clham

so idea is to completely remove compat.scipy

• move fastsort inside rankdata
• move rankdata to pandas.util.testing, maybe rename to compat_rankdata
• change tests in test_frame/test_series/test_tseries to use compat_rankdata rather than rankdata
• change percentileRank to use scipy directly (not sure this is tested at all, btw)

hopefully you can submit a PR for this!

[clham]

Sure. I'll see what I can do.

[clham]

@jreback

I'm confused here.

1. Why can't we convert rankdata to use scipy directly?
2. percentilerank can't use scipy directly, because the current version of scipy removed the options for tiebreaking.

Was this a typo, or am I missing the ball? I'm too new to assume much at this point.

(also, percentileRank behaves erratically when fed a 2d(+) dataframe, but that is an issue for after everything is moved around)

[jreback]

@clham you can try removing rankdata entirely (and just import from scipy). that might just work. (i think the original intent was to allow the tests to work w/o scipy). but its included on almost all builds now. you might need to use _skip_if_no_scipy() in that particular function where its tested (in test_series.py), so it skips that build