Skip to content

PERF: faster dir() calls #37450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Oct 29, 2020
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.2.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,7 @@ Performance improvements
- Performance improvement in :meth:`RollingGroupby.count` (:issue:`35625`)
- Small performance decrease to :meth:`Rolling.min` and :meth:`Rolling.max` for fixed windows (:issue:`36567`)
- Reduced peak memory usage in :meth:`DataFrame.to_pickle` when using ``protocol=5`` in python 3.8+ (:issue:`34244`)
- faster ``dir`` calls when many index labels, e.g. ``dir(ser)`` (:issue:`37450`)
- Performance improvement in :class:`ExpandingGroupby` (:issue:`37064`)

.. ---------------------------------------------------------------------------
Expand Down
9 changes: 1 addition & 8 deletions pandas/core/accessor.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,7 @@ def _dir_additions(self) -> Set[str]:
"""
Add additional __dir__ for this object.
"""
rv = set()
for accessor in self._accessors:
try:
getattr(self, accessor)
rv.add(accessor)
except AttributeError:
pass
return rv
return {accessor for accessor in self._accessors if hasattr(self, accessor)}

def __dir__(self) -> List[str]:
"""
Expand Down
6 changes: 1 addition & 5 deletions pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -5439,11 +5439,7 @@ def _dir_additions(self) -> Set[str]:
add the string-like attributes from the info_axis.
If info_axis is a MultiIndex, it's first level values are used.
"""
additions = {
c
for c in self._info_axis.unique(level=0)[:100]
if isinstance(c, str) and c.isidentifier()
}
additions = self._info_axis._dir_additions_for_owner
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of adding all of this code, why can't you slap a cache_readonly here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not possible, because the axes (index/columns) can be changed. However, the axis labels are immutable, so it's safe to use cache_readonly there.

I've changed the PR to avoid the addition in child classes.

return super()._dir_additions().union(additions)

# ----------------------------------------------------------------------
Expand Down
13 changes: 13 additions & 0 deletions pandas/core/indexes/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
NewType,
Optional,
Sequence,
Set,
Tuple,
TypeVar,
Union,
Expand Down Expand Up @@ -568,6 +569,18 @@ def _engine(self):
target_values = self._get_engine_target()
return self._engine_type(lambda: target_values, len(self))

@cache_readonly
def _dir_additions_for_owner(self) -> Set[str_t]:
"""
Add the string-like attributes from this index to the owners' dir output.
If this is a MultiIndex, it's first level values are used.
"""
return {
c
for c in self.unique(level=0)[:100]
if isinstance(c, str) and c.isidentifier()
}

# --------------------------------------------------------------------
# Array-Like Methods

Expand Down
6 changes: 5 additions & 1 deletion pandas/core/indexes/datetimelike.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Base and utility classes for tseries type pandas objects.
"""
from datetime import datetime, tzinfo
from typing import TYPE_CHECKING, Any, List, Optional, TypeVar, Union, cast
from typing import TYPE_CHECKING, Any, List, Optional, Set, TypeVar, Union, cast

import numpy as np

Expand Down Expand Up @@ -105,6 +105,10 @@ class DatetimeIndexOpsMixin(ExtensionIndex):
def _is_all_dates(self) -> bool:
return True

@cache_readonly
def _dir_additions_for_owner(self) -> Set[str]:
return set()

# ------------------------------------------------------------------------
# Abstract data attributes

Expand Down
6 changes: 5 additions & 1 deletion pandas/core/indexes/interval.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from functools import wraps
from operator import le, lt
import textwrap
from typing import TYPE_CHECKING, Any, List, Optional, Tuple, Union, cast
from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple, Union, cast

import numpy as np

Expand Down Expand Up @@ -371,6 +371,10 @@ def __contains__(self, key: Any) -> bool:
def _multiindex(self) -> MultiIndex:
return MultiIndex.from_arrays([self.left, self.right], names=["left", "right"])

@cache_readonly
def _dir_additions_for_owner(self) -> Set[str]:
return set()

@cache_readonly
def values(self) -> IntervalArray:
"""
Expand Down
6 changes: 5 additions & 1 deletion pandas/core/indexes/numeric.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from typing import Any
from typing import Any, Set

import numpy as np

Expand Down Expand Up @@ -184,6 +184,10 @@ def _union(self, other, sort):
else:
return super()._union(other, sort)

@cache_readonly
def _dir_additions_for_owner(self) -> Set[str]:
return set()


_num_index_shared_docs[
"class_descr"
Expand Down