Merge remote-tracking branch 'upstream/master' into multi-index-join

Author: harisbal

ci/code_checks.sh

@@ -49,7 +49,7 @@ if [[ -z "$CHECK" || "$CHECK" == "lint" ]]; then
     # Note: this grep pattern is (intended to be) equivalent to the python
     # regex r'(?<![ ->])> '
     MSG='Linting .pyx code for spacing conventions in casting' ; echo $MSG
-    ! grep -r -E --include '*.pyx' --include '*.pxi.in' '> ' pandas/_libs | grep -v '[ ->]> '
+    ! grep -r -E --include '*.pyx' --include '*.pxi.in' '[a-zA-Z0-9*]> ' pandas/_libs
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     # readability/casting: Warnings about C casting instead of C++ casting
@@ -131,7 +131,7 @@ if [[ -z "$CHECK" || "$CHECK" == "doctests" ]]; then
 
     MSG='Doctests frame.py' ; echo $MSG
     pytest -q --doctest-modules pandas/core/frame.py \
-        -k"-axes -combine -itertuples -join -nlargest -nsmallest -nunique -pivot_table -quantile -query -reindex -reindex_axis -replace -round -set_index -stack -to_stata"
+        -k"-axes -combine -itertuples -join -nunique -pivot_table -quantile -query -reindex -reindex_axis -replace -round -set_index -stack -to_stata"
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Doctests series.py' ; echo $MSG

ci/requirements-optional-conda.txt

@@ -1,7 +1,7 @@
 beautifulsoup4>=4.2.1
 blosc
 bottleneck>=1.2.0
-fastparquet
+fastparquet>=0.1.2
 gcsfs
 html5lib
 ipython>=5.6.0
@@ -12,7 +12,7 @@ matplotlib>=2.0.0
 nbsphinx
 numexpr>=2.6.1
 openpyxl
-pyarrow>=0.4.1
+pyarrow>=0.7.0
 pymysql
 pytables>=3.4.2
 pytest-cov

ci/requirements-optional-pip.txt

@@ -3,7 +3,7 @@
 beautifulsoup4>=4.2.1
 blosc
 bottleneck>=1.2.0
-fastparquet
+fastparquet>=0.1.2
 gcsfs
 html5lib
 ipython>=5.6.0
@@ -14,9 +14,9 @@ matplotlib>=2.0.0
 nbsphinx
 numexpr>=2.6.1
 openpyxl
-pyarrow>=0.4.1
+pyarrow>=0.7.0
 pymysql
-tables
+pytables>=3.4.2
 pytest-cov
 pytest-xdist
 s3fs
@@ -27,4 +27,4 @@ statsmodels
 xarray
 xlrd
 xlsxwriter
-xlwt
+xlwt

ci/travis-27.yaml

@@ -22,7 +22,7 @@ dependencies:
   - patsy
   - psycopg2
   - py
-  - pyarrow=0.4.1
+  - pyarrow=0.7.0
   - PyCrypto
   - pymysql=0.6.3
   - pytables

ci/travis-37-numpydev.yaml

@@ -13,5 +13,5 @@ dependencies:
     - "git+git://github.com/dateutil/dateutil.git"
     - "-f https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com"
     - "--pre"
-    - "numpy<=1.16.0.dev0+20181015190246"
+    - "numpy"
     - "scipy"

doc/source/basics.rst

@@ -767,7 +767,7 @@ We encourage you to view the source code of :meth:`~DataFrame.pipe`.
 
 .. _dplyr: https://github.com/hadley/dplyr
 .. _magrittr: https://github.com/smbache/magrittr
-.. _R: http://www.r-project.org
+.. _R: https://www.r-project.org
 
 
 Row or Column-wise Function Application
@@ -2296,7 +2296,7 @@ For example, to select ``bool`` columns:
    df.select_dtypes(include=[bool])
 
 You can also pass the name of a dtype in the `NumPy dtype hierarchy
-<http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html>`__:
+<https://docs.scipy.org/doc/numpy/reference/arrays.scalars.html>`__:
 
 .. ipython:: python
 

doc/source/comparison_with_sas.rst

@@ -365,8 +365,8 @@ Length
 ~~~~~~
 
 SAS determines the length of a character string with the 
-`LENGTHN <http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002284668.htm>`__  
-and `LENGTHC <http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002283942.htm>`__ 
+`LENGTHN <https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002284668.htm>`__
+and `LENGTHC <https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002283942.htm>`__
 functions. ``LENGTHN`` excludes trailing blanks and ``LENGTHC`` includes trailing blanks.
 
 .. code-block:: sas
@@ -391,7 +391,7 @@ Find
 ~~~~
 
 SAS determines the position of a character in a string with the 
-`FINDW <http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002978282.htm>`__ function. 
+`FINDW <https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002978282.htm>`__ function.
 ``FINDW`` takes the string defined by the first argument and searches for the first position of the substring 
 you supply as the second argument.
 
@@ -417,7 +417,7 @@ Substring
 ~~~~~~~~~
 
 SAS extracts a substring from a string based on its position with the 
-`SUBSTR <http://www2.sas.com/proceedings/sugi25/25/cc/25p088.pdf>`__ function. 
+`SUBSTR <https://www2.sas.com/proceedings/sugi25/25/cc/25p088.pdf>`__ function.
 
 .. code-block:: sas
 
@@ -438,7 +438,7 @@ indexes are zero-based.
 Scan
 ~~~~
 
-The SAS `SCAN <http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000214639.htm>`__ 
+The SAS `SCAN <https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000214639.htm>`__
 function returns the nth word from a string. The first argument is the string you want to parse and the 
 second argument specifies which word you want to extract.
 
@@ -469,9 +469,9 @@ approaches, but this just shows a simple approach.
 Upcase, Lowcase, and Propcase
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The SAS `UPCASE <http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000245965.htm>`__ 
-`LOWCASE <http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000245912.htm>`__ and 
-`PROPCASE <http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/a002598106.htm>`__ 
+The SAS `UPCASE <https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000245965.htm>`__
+`LOWCASE <https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000245912.htm>`__ and
+`PROPCASE <https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/a002598106.htm>`__
 functions change the case of the argument.
 
 .. code-block:: sas
@@ -709,7 +709,7 @@ This means that the size of data able to be loaded in pandas is limited by your
 machine's memory, but also that the operations on that data may be faster.
 
 If out of core processing is needed, one possibility is the
-`dask.dataframe <http://dask.pydata.org/en/latest/dataframe.html>`_
+`dask.dataframe <https://dask.pydata.org/en/latest/dataframe.html>`_
 library (currently in development) which
 provides a subset of pandas functionality for an on-disk ``DataFrame``
 

doc/source/comparison_with_sql.rst

@@ -4,7 +4,7 @@
 Comparison with SQL
 ********************
 Since many potential pandas users have some familiarity with
-`SQL <http://en.wikipedia.org/wiki/SQL>`_, this page is meant to provide some examples of how
+`SQL <https://en.wikipedia.org/wiki/SQL>`_, this page is meant to provide some examples of how
 various SQL operations would be performed using pandas.
 
 If you're new to pandas, you might want to first read through :ref:`10 Minutes to pandas<10min>`
@@ -59,7 +59,7 @@ Filtering in SQL is done via a WHERE clause.
     LIMIT 5;
 
 DataFrames can be filtered in multiple ways; the most intuitive of which is using
-`boolean indexing <http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing>`_.
+`boolean indexing <https://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing>`_.
 
 .. ipython:: python
 

doc/source/contributing_docstring.rst

@@ -16,7 +16,7 @@ function or method, so programmers can understand what it does without having
 to read the details of the implementation.
 
 Also, it is a common practice to generate online (html) documentation
-automatically from docstrings. `Sphinx <http://www.sphinx-doc.org>`_ serves
+automatically from docstrings. `Sphinx <https://www.sphinx-doc.org>`_ serves
 this purpose.
 
 Next example gives an idea on how a docstring looks like:
@@ -68,7 +68,7 @@ As PEP-257 is quite open, and some other standards exist on top of it. In the
 case of pandas, the numpy docstring convention is followed. The conventions is
 explained in this document:
 
-* `numpydoc docstring guide <http://numpydoc.readthedocs.io/en/latest/format.html>`_
+* `numpydoc docstring guide <https://numpydoc.readthedocs.io/en/latest/format.html>`_
   (which is based in the original `Guide to NumPy/SciPy documentation
   <https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_)
 
@@ -78,7 +78,7 @@ The standard uses reStructuredText (reST). reStructuredText is a markup
 language that allows encoding styles in plain text files. Documentation
 about reStructuredText can be found in:
 
-* `Sphinx reStructuredText primer <http://www.sphinx-doc.org/en/stable/rest.html>`_
+* `Sphinx reStructuredText primer <https://www.sphinx-doc.org/en/stable/rest.html>`_
 * `Quick reStructuredText reference <http://docutils.sourceforge.net/docs/user/rst/quickref.html>`_
 * `Full reStructuredText specification <http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html>`_
 
@@ -119,7 +119,7 @@ backticks. It is considered inline code:
     function, prefix it with ``~``. For example, ``:class:`~pandas.Series```
     will link to ``pandas.Series`` but only display the last part, ``Series``
     as the link text. See `Sphinx cross-referencing syntax
-    <http://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-syntax>`_
+    <https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-syntax>`_
     for details.
 
 **Good:**

doc/source/cookbook.rst

@@ -52,7 +52,7 @@ Idioms
 These are some neat pandas ``idioms``
 
 `if-then/if-then-else on one column, and assignment to another one or more columns:
-<http://stackoverflow.com/questions/17128302/python-pandas-idiom-for-if-then-else>`__
+<https://stackoverflow.com/questions/17128302/python-pandas-idiom-for-if-then-else>`__
 
 .. ipython:: python
 
@@ -88,7 +88,7 @@ Or use pandas where after you've set up a mask
    df.where(df_mask,-1000)
 
 `if-then-else using numpy's where()
-<http://stackoverflow.com/questions/19913659/pandas-conditional-creation-of-a-series-dataframe-column>`__
+<https://stackoverflow.com/questions/19913659/pandas-conditional-creation-of-a-series-dataframe-column>`__
 
 .. ipython:: python
 
@@ -101,7 +101,7 @@ Splitting
 *********
 
 `Split a frame with a boolean criterion
-<http://stackoverflow.com/questions/14957116/how-to-split-a-dataframe-according-to-a-boolean-criterion>`__
+<https://stackoverflow.com/questions/14957116/how-to-split-a-dataframe-according-to-a-boolean-criterion>`__
 
 .. ipython:: python
 
@@ -115,7 +115,7 @@ Building Criteria
 *****************
 
 `Select with multi-column criteria
-<http://stackoverflow.com/questions/15315452/selecting-with-complex-criteria-from-pandas-dataframe>`__
+<https://stackoverflow.com/questions/15315452/selecting-with-complex-criteria-from-pandas-dataframe>`__
 
 .. ipython:: python
 
@@ -141,7 +141,7 @@ Building Criteria
    df.loc[(df['BBB'] > 25) | (df['CCC'] >= 75), 'AAA'] = 0.1; df
 
 `Select rows with data closest to certain value using argsort
-<http://stackoverflow.com/questions/17758023/return-rows-in-a-dataframe-closest-to-a-user-defined-number>`__
+<https://stackoverflow.com/questions/17758023/return-rows-in-a-dataframe-closest-to-a-user-defined-number>`__
 
 .. ipython:: python
 
@@ -152,7 +152,7 @@ Building Criteria
    df.loc[(df.CCC-aValue).abs().argsort()]
 
 `Dynamically reduce a list of criteria using a binary operators
-<http://stackoverflow.com/questions/21058254/pandas-boolean-operation-in-a-python-list/21058331>`__
+<https://stackoverflow.com/questions/21058254/pandas-boolean-operation-in-a-python-list/21058331>`__
 
 .. ipython:: python
 
@@ -189,7 +189,7 @@ DataFrames
 The :ref:`indexing <indexing>` docs.
 
 `Using both row labels and value conditionals
-<http://stackoverflow.com/questions/14725068/pandas-using-row-labels-in-boolean-indexing>`__
+<https://stackoverflow.com/questions/14725068/pandas-using-row-labels-in-boolean-indexing>`__
 
 .. ipython:: python
 
@@ -232,7 +232,7 @@ Ambiguity arises when an index consists of integers with a non-zero start or non
    df2.loc[1:3] #Label-oriented
 
 `Using inverse operator (~) to take the complement of a mask
-<http://stackoverflow.com/questions/14986510/picking-out-elements-based-on-complement-of-indices-in-python-pandas>`__
+<https://stackoverflow.com/questions/14986510/picking-out-elements-based-on-complement-of-indices-in-python-pandas>`__
 
 .. ipython:: python
 
@@ -259,13 +259,13 @@ Panels
    pf.loc[:,:,'F'] = pd.DataFrame(data, rng, cols);pf
 
 `Mask a panel by using np.where and then reconstructing the panel with the new masked values
-<http://stackoverflow.com/questions/14650341/boolean-mask-in-pandas-panel>`__
+<https://stackoverflow.com/questions/14650341/boolean-mask-in-pandas-panel>`__
 
 New Columns
 ***********
 
 `Efficiently and dynamically creating new columns using applymap
-<http://stackoverflow.com/questions/16575868/efficiently-creating-additional-columns-in-a-pandas-dataframe-using-map>`__
+<https://stackoverflow.com/questions/16575868/efficiently-creating-additional-columns-in-a-pandas-dataframe-using-map>`__
 
 .. ipython:: python
 
@@ -279,7 +279,7 @@ New Columns
    df[new_cols] = df[source_cols].applymap(categories.get);df
 
 `Keep other columns when using min() with groupby
-<http://stackoverflow.com/questions/23394476/keep-other-columns-when-using-min-with-groupby>`__
+<https://stackoverflow.com/questions/23394476/keep-other-columns-when-using-min-with-groupby>`__
 
 .. ipython:: python
 
@@ -308,7 +308,7 @@ MultiIndexing
 The :ref:`multindexing <advanced.hierarchical>` docs.
 
 `Creating a MultiIndex from a labeled frame
-<http://stackoverflow.com/questions/14916358/reshaping-dataframes-in-pandas-based-on-column-labels>`__
+<https://stackoverflow.com/questions/14916358/reshaping-dataframes-in-pandas-based-on-column-labels>`__
 
 .. ipython:: python
 
@@ -331,7 +331,7 @@ Arithmetic
 **********
 
 `Performing arithmetic with a MultiIndex that needs broadcasting
-<http://stackoverflow.com/questions/19501510/divide-entire-pandas-multiindex-dataframe-by-dataframe-variable/19502176#19502176>`__
+<https://stackoverflow.com/questions/19501510/divide-entire-pandas-multiindex-dataframe-by-dataframe-variable/19502176#19502176>`__
 
 .. ipython:: python
 
@@ -343,7 +343,7 @@ Slicing
 *******
 
 `Slicing a MultiIndex with xs
-<http://stackoverflow.com/questions/12590131/how-to-slice-multindex-columns-in-pandas-dataframes>`__
+<https://stackoverflow.com/questions/12590131/how-to-slice-multindex-columns-in-pandas-dataframes>`__
 
 .. ipython:: python
 
@@ -364,7 +364,7 @@ To take the cross section of the 1st level and 1st axis the index:
    df.xs('six',level=1,axis=0)
 
 `Slicing a MultiIndex with xs, method #2
-<http://stackoverflow.com/questions/14964493/multiindex-based-indexing-in-pandas>`__
+<https://stackoverflow.com/questions/14964493/multiindex-based-indexing-in-pandas>`__
 
 .. ipython:: python
 
@@ -387,13 +387,13 @@ To take the cross section of the 1st level and 1st axis the index:
    df.loc[(All,'Math'),(All,'II')]
 
 `Setting portions of a MultiIndex with xs
-<http://stackoverflow.com/questions/19319432/pandas-selecting-a-lower-level-in-a-dataframe-to-do-a-ffill>`__
+<https://stackoverflow.com/questions/19319432/pandas-selecting-a-lower-level-in-a-dataframe-to-do-a-ffill>`__
 
 Sorting
 *******
 
 `Sort by specific column or an ordered list of columns, with a MultiIndex
-<http://stackoverflow.com/questions/14733871/mutli-index-sorting-in-pandas>`__
+<https://stackoverflow.com/questions/14733871/mutli-index-sorting-in-pandas>`__
 
 .. ipython:: python