REF: avoid special-casing in SelectN#45956

jreback merged 2 commits intopandas-dev:mainpandas-dev/pandas:mainfrom

jbrockmendel:ref-not-specialjbrockmendel/pandas:ref-not-specialCopy head branch name to clipboard