Address requested changes

Author: Albert Villanova del Moral

doc/source/whatsnew/v0.20.0.txt

@@ -228,6 +228,7 @@ Other enhancements
 - ``pd.TimedeltaIndex`` now has a custom datetick formatter specifically designed for nanosecond level precision (:issue:`8711`)
 - ``pd.types.concat.union_categoricals`` gained the ``ignore_ordered`` argument to allow ignoring the ordered attribute of unioned categoricals (:issue:`13410`). See the :ref:`categorical union docs <categorical.union>` for more information.
 - ``pandas.io.json.json_normalize()`` with an empty ``list`` will return an empty ``DataFrame`` (:issue:`15534`)
+- ``Index.intersection()`` accepts parameter ``sort`` (:issue:`15582`)
 
 .. _ISO 8601 duration: https://en.wikipedia.org/wiki/ISO_8601#Durations
 
@@ -567,52 +568,135 @@ New Behavior:
 
 .. _whatsnew_0200.api_breaking.index_order:
 
-Index order after DataFrame inner join or Index intersection
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Index order after inner join due to Index intersection
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The ``DataFrame`` inner join and the ``Index`` intersection, now preserve the
-order of the calling's Index (left) instead of the other's Index (right)
-(:issue:`15582`)
+The ``Index.intersection`` now preserves the order of the calling Index (left)
+instead of the other Index (right) (:issue:`15582`). This affects the inner
+joins (methods ``Index.join``, ``DataFrame.join``, ``DataFrame.merge`` and
+``pd.merge``) and the alignments with inner join (methods ``Series.align`` and
+``DataFrame.align``).
 
-Previous Behavior:
+- ``Index.intersection`` and ``Index.join``
 
-.. code-block:: ipython
-   In [2]: df1 = pd.DataFrame({'a': [20, 10, 0]}, index=[2, 1, 0])
+  .. ipython:: python
 
-   In [3]: df2 = pd.DataFrame({'b': [100, 200, 300]}, index=[1, 2, 3])
+     idx1 = pd.Index([2, 1, 0])
+     idx1
+     idx2 = pd.Index([1, 2, 3])
+     idx2
 
-   In [4]: df1.join(df2, how='inner')
-   Out[4]:
-       a    b
-   1  10  100
-   2  20  200
+  Previous Behavior:
 
-   In [5]: idx1 = pd.Index([5, 3, 2, 4, 1])
+  .. code-block:: ipython
 
-   In [6]: idx2 = pd.Index([4, 7, 6, 5, 3])
+     In [4]: idx1.intersection(idx2)
+     Out[4]: Int64Index([1, 2], dtype='int64')
 
-   In [7]: idx1.intersection(idx2)
-   Out[7]: Int64Index([4, 5, 3], dtype='int64')
+     In [5]: idx1.join(idx2, how='inner')
+     Out[5]: Int64Index([1, 2], dtype='int64')
 
-New Behavior:
+  New Behavior:
 
-.. code-block:: ipython
-   In [2]: df1 = pd.DataFrame({'a': [20, 10, 0]}, index=[2, 1, 0])
+  .. ipython:: python
+
+     idx1.intersection(idx2)
+
+     idx1.join(idx2, how='inner')
+
+- ``Series.align``
+
+  .. ipython:: python
+
+     s1 = pd.Series([20, 10, 0], index=[2, 1, 0])
+     s1
+     s2 = pd.Series([100, 200, 300], index=[1, 2, 3])
+     s2
+
+  Previous Behavior:
+
+  .. code-block:: ipython
 
-   In [3]: df2 = pd.DataFrame({'b': [100, 200, 300]}, index=[1, 2, 3])
+     In [4]: (res1, res2) = s1.align(s2, join='inner')
+
+     In [5]: res1
+     Out[5]:
+     1    10
+     2    20
+     dtype: int64
+
+     In [6]: res2
+     Out[6]:
+     1    100
+     2    200
+     dtype: int64
+
+  New Behavior:
+
+  .. ipython:: python
+
+     (res1, res2) = s1.align(s2, join='inner')
+     res1
+     res2
+
+- ``DataFrame.join``, ``DataFrame.merge`` and ``pd.merge``
+
+  .. ipython:: python
+
+     df1 = pd.DataFrame({'a': [20, 10, 0]}, index=[2, 1, 0])
+     df1
+     df2 = pd.DataFrame({'b': [100, 200, 300]}, index=[1, 2, 3])
+     df2
+
+  Previous Behavior:
+
+  .. code-block:: ipython
+
+     In [4]: df1.join(df2, how='inner')
+     Out[4]:
+         a    b
+     1  10  100
+     2  20  200
+
+     In [5]: df1.merge(df2, how='inner', left_index=True, right_index=True)
+     Out[5]:
+         a    b
+     1  10  100
+     2  20  200
+
+     In [6]: pd.merge(df1, df2, how='inner', left_index=True, right_index=True)
+     Out[6]:
+         a    b
+     1  10  100
+     2  20  200
+
+     In [7]: (res1, res2) = df1.align(df2, axis=0, join='inner')
+
+     In [8]: res1
+     Out[8]:
+         a
+     1  10
+     2  20
+
+     In [9]: res2
+     Out[9]:
+          b
+     1  100
+     2  200
+
+  New Behavior:
+
+  .. ipython:: python
 
-   In [4]: df1.join(df2, how='inner')
-   Out[4]:
-       a    b
-   2  20  200
-   1  10  100
+     df1.join(df2, how='inner')
 
-   In [5]: idx1 = pd.Index([5, 3, 2, 4, 1])
+     df1.merge(df2, how='inner', left_index=True, right_index=True)
 
-   In [6]: idx2 = pd.Index([4, 7, 6, 5, 3])
+     pd.merge(df1, df2, how='inner', left_index=True, right_index=True)
 
-   In [7]: idx1.intersection(idx2)
-   Out[7]: Int64Index([5, 3, 4], dtype='int64')
+     (res1, res2) = df1.align(df2, axis=0, join='inner')
+     res1
+     res2
 
 
 .. _whatsnew_0200.api:
@@ -820,3 +904,4 @@ Bug Fixes
 - Bug in ``pd.melt()`` where passing a tuple value for ``value_vars`` caused a ``TypeError`` (:issue:`15348`)
 - Bug in ``.eval()`` which caused multiline evals to fail with local variables not on the first line (:issue:`15342`)
 - Bug in ``pd.read_msgpack`` which did not allow to load dataframe with an index of type ``CategoricalIndex`` (:issue:`15487`)
+- Bug with ``sort=True`` in ``DataFrame.join``, ``DataFrame.merge`` and ``pd.merge`` when joining on index (:issue:`15582`)

pandas/core/frame.py

@@ -129,7 +129,8 @@
     * left: use only keys from left frame (SQL: left outer join)
     * right: use only keys from right frame (SQL: right outer join)
     * outer: use union of keys from both frames (SQL: full outer join)
-    * inner: use intersection of keys from both frames (SQL: inner join)
+    * inner: use intersection of keys from both frames (SQL: inner join),
+        preserving the order of the left keys
 on : label or list
     Field names to join on. Must be found in both DataFrames. If on is
     None and not merging on indexes, then it merges on the intersection of
@@ -149,7 +150,8 @@
     Use the index from the right DataFrame as the join key. Same caveats as
     left_index
 sort : boolean, default False
-    Sort the join keys lexicographically in the result DataFrame
+    Sort the join keys lexicographically in the result DataFrame. If False,
+    the order of the join keys depends on the join type (how keyword)
 suffixes : 2-length sequence (tuple, list, ...)
     Suffix to apply to overlapping column names in the left and right
     side, respectively
@@ -4486,6 +4488,7 @@ def join(self, other, on=None, how='left', lsuffix='', rsuffix='',
             * right: use other frame's index
             * outer: form union of calling frame's index (or column if on is
                 specified) with other frame's index, and sort it
+                lexicographically
             * inner: form intersection of calling frame's index (or column if
                 on is specified) with other frame's index, preserving the
                 order of the calling's one

pandas/indexes/base.py

@@ -2047,7 +2047,7 @@ def intersection(self, other):
         Form the intersection of two Index objects.
 
         This returns a new Index with elements common to the index and `other`,
-        preserving the calling index order.
+        preserving the order of the calling index.
 
         Parameters
         ----------
@@ -2788,9 +2788,7 @@ def _reindex_non_unique(self, target):
         new_index = self._shallow_copy_with_infer(new_labels, freq=None)
         return new_index, indexer, new_indexer
 
-    def join(self, other, how='left', level=None, return_indexers=False,
-             sort=False):
-        """
+    _index_shared_docs['join'] = """
         *this is an internal non-public method*
 
         Compute join_index and indexers to conform data
@@ -2804,10 +2802,16 @@ def join(self, other, how='left', level=None, return_indexers=False,
         return_indexers : boolean, default False
         sort : boolean, default False
 
+            .. versionadded:: 0.20.0
+
         Returns
         -------
         join_index, (left_indexer, right_indexer)
         """
+
+    @Appender(_index_shared_docs['join'])
+    def join(self, other, how='left', level=None, return_indexers=False,
+             sort=False):
         from .multi import MultiIndex
         self_is_mi = isinstance(self, MultiIndex)
         other_is_mi = isinstance(other, MultiIndex)

pandas/indexes/range.py

@@ -431,26 +431,9 @@ def union(self, other):
 
         return self._int64index.union(other)
 
+    @Appender(_index_shared_docs['join'])
     def join(self, other, how='left', level=None, return_indexers=False,
              sort=False):
-        """
-        *this is an internal non-public method*
-
-        Compute join_index and indexers to conform data
-        structures to the new index.
-
-        Parameters
-        ----------
-        other : Index
-        how : {'left', 'right', 'inner', 'outer'}
-        level : int or level name, default None
-        return_indexers : boolean, default False
-        sort : boolean, default False
-
-        Returns
-        -------
-        join_index, (left_indexer, right_indexer)
-        """
         if how == 'outer' and self is not other:
             # note: could return RangeIndex in more circumstances
             return self._int64index.join(other, how, level, return_indexers,

pandas/tests/frame/test_join.py

@@ -2,6 +2,8 @@
 
 from __future__ import print_function
 
+import numpy as np
+
 import pandas as pd
 
 from pandas.tests.frame.common import TestData
@@ -15,64 +17,75 @@ def test_join(self):
         df1 = pd.DataFrame({'a': [20, 10, 0]}, index=[2, 1, 0])
         df2 = pd.DataFrame({'b': [100, 200, 300]}, index=[1, 2, 3])
 
+        # default how='left'
         result = df1.join(df2)
-        expected = pd.DataFrame({'a': [20, 10, 0], 'b': [200, 100, None]},
+        expected = pd.DataFrame({'a': [20, 10, 0], 'b': [200, 100, np.nan]},
                                 index=[2, 1, 0])
         tm.assert_frame_equal(result, expected)
 
+        # how='left'
         result = df1.join(df2, how='left')
-        expected = pd.DataFrame({'a': [20, 10, 0], 'b': [200, 100, None]},
+        expected = pd.DataFrame({'a': [20, 10, 0], 'b': [200, 100, np.nan]},
                                 index=[2, 1, 0])
         tm.assert_frame_equal(result, expected)
 
+        # how='right'
         result = df1.join(df2, how='right')
-        expected = pd.DataFrame({'a': [10, 20, None], 'b': [100, 200, 300]},
+        expected = pd.DataFrame({'a': [10, 20, np.nan], 'b': [100, 200, 300]},
                                 index=[1, 2, 3])
         tm.assert_frame_equal(result, expected)
 
+        # how='inner'
         result = df1.join(df2, how='inner')
         expected = pd.DataFrame({'a': [20, 10], 'b': [200, 100]},
                                 index=[2, 1])
         tm.assert_frame_equal(result, expected)
 
+        # how='outer'
         result = df1.join(df2, how='outer')
-        expected = pd.DataFrame({'a': [0, 10, 20, None],
-                                 'b': [None, 100, 200, 300]},
+        expected = pd.DataFrame({'a': [0, 10, 20, np.nan],
+                                 'b': [np.nan, 100, 200, 300]},
                                 index=[0, 1, 2, 3])
         tm.assert_frame_equal(result, expected)
 
     def test_join_sort(self):
         df1 = pd.DataFrame({'a': [20, 10, 0]}, index=[2, 1, 0])
         df2 = pd.DataFrame({'b': [100, 200, 300]}, index=[1, 2, 3])
 
+        # default how='left'
         result = df1.join(df2, sort=True)
-        expected = pd.DataFrame({'a': [0, 10, 20], 'b': [None, 100, 200]},
+        expected = pd.DataFrame({'a': [0, 10, 20], 'b': [np.nan, 100, 200]},
                                 index=[0, 1, 2])
         tm.assert_frame_equal(result, expected)
 
+        # how='left'
         result = df1.join(df2, how='left', sort=True)
-        expected = pd.DataFrame({'a': [0, 10, 20], 'b': [None, 100, 200]},
+        expected = pd.DataFrame({'a': [0, 10, 20], 'b': [np.nan, 100, 200]},
                                 index=[0, 1, 2])
         tm.assert_frame_equal(result, expected)
 
+        # how='right' (already sorted)
         result = df1.join(df2, how='right', sort=True)
-        expected = pd.DataFrame({'a': [10, 20, None], 'b': [100, 200, 300]},
+        expected = pd.DataFrame({'a': [10, 20, np.nan], 'b': [100, 200, 300]},
                                 index=[1, 2, 3])
         tm.assert_frame_equal(result, expected)
 
+        # how='right'
         result = df2.join(df1, how='right', sort=True)
-        expected = pd.DataFrame([[None, 0], [100, 10], [200, 20]],
+        expected = pd.DataFrame([[np.nan, 0], [100, 10], [200, 20]],
                                 columns=['b', 'a'], index=[0, 1, 2])
         tm.assert_frame_equal(result, expected)
 
+        # how='inner'
         result = df1.join(df2, how='inner', sort=True)
         expected = pd.DataFrame({'a': [10, 20], 'b': [100, 200]},
                                 index=[1, 2])
         tm.assert_frame_equal(result, expected)
 
+        # how='outer'
         result = df1.join(df2, how='outer', sort=True)
-        expected = pd.DataFrame({'a': [0, 10, 20, None],
-                                 'b': [None, 100, 200, 300]},
+        expected = pd.DataFrame({'a': [0, 10, 20, np.nan],
+                                 'b': [np.nan, 100, 200, 300]},
                                 index=[0, 1, 2, 3])
         tm.assert_frame_equal(result, expected)