Merge remote-tracking branch 'upstream/master' into fu1+sort

Author: TomAugspurger

.gitignore

@@ -61,6 +61,7 @@ dist
 .coverage
 coverage.xml
 coverage_html_report
+*.pytest_cache
 
 # OS generated files #
 ######################
@@ -90,7 +91,6 @@ scikits
 
 # Unit / Performance Testing #
 ##############################
-.pytest_cache/
 asv_bench/env/
 asv_bench/html/
 asv_bench/results/

ci/build_docs.sh

@@ -10,11 +10,11 @@ echo "inside $0"
 
 git show --pretty="format:" --name-only HEAD~5.. --first-parent | grep -P "rst|txt|doc"
 
-if [ "$?" != "0" ]; then
-    echo "Skipping doc build, none were modified"
-    # nope, skip docs build
-    exit 0
-fi
+# if [ "$?" != "0" ]; then
+#     echo "Skipping doc build, none were modified"
+#     # nope, skip docs build
+#     exit 0
+# fi
 
 
 if [ "$DOC" ]; then

ci/requirements-2.7_COMPAT.pip

@@ -1,4 +1,4 @@
 html5lib==1.0b2
-beautifulsoup4==4.2.0
+beautifulsoup4==4.2.1
 openpyxl
 argparse

ci/requirements-optional-conda.txt

@@ -1,4 +1,4 @@
-beautifulsoup4
+beautifulsoup4>=4.2.1
 blosc
 bottleneck
 fastparquet

ci/requirements-optional-pip.txt

@@ -1,6 +1,6 @@
 # This file was autogenerated by scripts/convert_deps.py
 # Do not modify directly
-beautifulsoup4
+beautifulsoup4>=4.2.1
 blosc
 bottleneck
 fastparquet

doc/cheatsheet/Pandas_Cheat_Sheet.pdf

@@ -246,16 +246,16 @@ changes in this branch specific to one bug or feature so it is clear
 what the branch brings to *pandas*. You can have many shiny-new-features
 and switch in between them using the git checkout command.
 
-To update this branch, you need to retrieve the changes from the master branch::
+When creating this branch, make sure your master branch is up to date with
+the latest upstream master version. To update your local master branch, you
+can do::
 
-    git fetch upstream
-    git rebase upstream/master
+    git checkout master
+    git pull upstream master --ff-only
 
-This will replay your commits on top of the latest pandas git master.  If this
-leads to merge conflicts, you must resolve these before submitting your pull
-request.  If you have uncommitted changes, you will need to ``stash`` them prior
-to updating.  This will effectively store your changes and they can be reapplied
-after updating.
+When you want to update the feature branch with changes in master after
+you created the branch, check the section on
+:ref:`updating a PR <contributing.update-pr>`.
 
 .. _contributing.documentation:
 
@@ -964,32 +964,6 @@ Now you can commit your changes in your local repository::
 
     git commit -m
 
-Combining commits
------------------
-
-If you have multiple commits, you may want to combine them into one commit, often
-referred to as "squashing" or "rebasing".  This is a common request by package maintainers
-when submitting a pull request as it maintains a more compact commit history.  To rebase
-your commits::
-
-    git rebase -i HEAD~#
-
-Where # is the number of commits you want to combine.  Then you can pick the relevant
-commit message and discard others.
-
-To squash to the master branch do::
-
-    git rebase -i master
-
-Use the ``s`` option on a commit to ``squash``, meaning to keep the commit messages,
-or ``f`` to ``fixup``, meaning to merge the commit messages.
-
-Then you will need to push the branch (see below) forcefully to replace the current
-commits with the new ones::
-
-    git push origin shiny-new-feature -f
-
-
 Pushing your changes
 --------------------
 
@@ -1045,15 +1019,51 @@ release.  To submit a pull request:
 #. Click ``Send Pull Request``.
 
 This request then goes to the repository maintainers, and they will review
-the code. If you need to make more changes, you can make them in
-your branch, push them to GitHub, and the pull request will be automatically
-updated.  Pushing them to GitHub again is done by::
+the code. 
+
+.. _contributing.update-pr:
 
-    git push -f origin shiny-new-feature
+Updating your pull request
+--------------------------
+
+Based on the review you get on your pull request, you will probably need to make
+some changes to the code. In that case, you can make them in your branch, 
+add a new commit to that branch, push it to GitHub, and the pull request will be
+automatically updated.  Pushing them to GitHub again is done by::
+
+    git push origin shiny-new-feature
 
 This will automatically update your pull request with the latest code and restart the
 :ref:`Continuous Integration <contributing.ci>` tests.
 
+Another reason you might need to update your pull request is to solve conflicts
+with changes that have been merged into the master branch since you opened your
+pull request. 
+
+To do this, you need to "merge upstream master" in your branch::
+
+    git checkout shiny-new-feature
+    git fetch upstream
+    git merge upstream/master
+
+If there are no conflicts (or they could be fixed automatically), a file with a
+default commit message will open, and you can simply save and quit this file.
+
+If there are merge conflicts, you need to solve those conflicts. See for
+example at https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/
+for an explanation on how to do this.
+Once the conflicts are merged and the files where the conflicts were solved are
+added, you can run ``git commit`` to save those fixes.
+
+If you have uncommitted changes at the moment you want to update the branch with
+master, you will need to ``stash`` them prior to updating (see the
+`stash docs <https://git-scm.com/book/en/v2/Git-Tools-Stashing-and-Cleaning>`__).
+This will effectively store your changes and they can be reapplied after updating.
+
+After the feature branch has been update locally, you can now update your pull
+request by pushing to the branch on GitHub::
+
+    git push origin shiny-new-feature
 
 Delete your merged branch (optional)
 ------------------------------------

doc/cheatsheet/Pandas_Cheat_Sheet.pptx

@@ -266,6 +266,12 @@ Optional Dependencies
 * One of the following combinations of libraries is needed to use the
   top-level :func:`~pandas.read_html` function:
 
+  .. versionchanged:: 0.23.0
+
+  .. note::
+
+     If using BeautifulSoup4 a minimum version of 4.2.1 is required
+
   * `BeautifulSoup4`_ and `html5lib`_ (Any recent version of `html5lib`_ is
     okay.)
   * `BeautifulSoup4`_ and `lxml`_
@@ -282,9 +288,6 @@ Optional Dependencies
      * You are highly encouraged to read :ref:`HTML Table Parsing gotchas <io.html.gotchas>`.
        It explains issues surrounding the installation and
        usage of the above three libraries.
-     * You may need to install an older version of `BeautifulSoup4`_:
-       Versions 4.2.1, 4.1.3 and 4.0.2 have been confirmed for 64 and 32-bit
-       Ubuntu/Debian
 
   .. note::
 

doc/source/contributing.rst

@@ -13,6 +13,8 @@ pandas' own :ref:`10 Minutes to pandas<10min>`.
 
 More complex recipes are in the :ref:`Cookbook<cookbook>`.
 
+A handy pandas `cheat sheet <http://pandas.pydata.org/Pandas_Cheat_Sheet.pdf>`_.
+
 pandas Cookbook
 ---------------
 

doc/source/install.rst

@@ -358,13 +358,15 @@ Dependencies have increased minimum versions
 We have updated our minimum supported versions of dependencies (:issue:`15184`).
 If installed, we now require:
 
-+-----------------+-----------------+----------+
-| Package         | Minimum Version | Required |
-+=================+=================+==========+
-| python-dateutil | 2.5.0           |    X     |
-+-----------------+-----------------+----------+
-| openpyxl        | 2.4.0           |          |
-+-----------------+-----------------+----------+
++-----------------+-----------------+----------+---------------+
+| Package         | Minimum Version | Required |     Issue     |
++=================+=================+==========+===============+
+| python-dateutil | 2.5.0           |    X     | :issue:`15184`|
++-----------------+-----------------+----------+---------------+
+| openpyxl        | 2.4.0           |          | :issue:`15184`|
++-----------------+-----------------+----------+---------------+
+| beautifulsoup4  | 4.2.1           |          | :issue:`20082`|
++-----------------+-----------------+----------+---------------+
 
 .. _whatsnew_0230.api_breaking.dict_insertion_order:
 
@@ -711,6 +713,7 @@ Other API Changes
 - ``Categorical.fillna`` now validates its ``value`` and ``method`` keyword arguments. It now raises when both or none are specified, matching the behavior of :meth:`Series.fillna` (:issue:`19682`)
 - ``pd.to_datetime('today')`` now returns a datetime, consistent with ``pd.Timestamp('today')``; previously ``pd.to_datetime('today')`` returned a ``.normalized()`` datetime (:issue:`19935`)
 - :func:`Series.str.replace` now takes an optional `regex` keyword which, when set to ``False``, uses literal string replacement rather than regex replacement (:issue:`16808`)
+- :func:`DatetimeIndex.strftime` and :func:`PeriodIndex.strftime` now return an ``Index`` instead of a numpy array to be consistent with similar accessors (:issue:`20127`)
 
 .. _whatsnew_0230.deprecations:
 
@@ -736,6 +739,7 @@ Deprecations
 - :attr:`Timestamp.weekday_name`, :attr:`DatetimeIndex.weekday_name`, and :attr:`Series.dt.weekday_name` are deprecated in favor of :meth:`Timestamp.day_name`, :meth:`DatetimeIndex.day_name`, and :meth:`Series.dt.day_name` (:issue:`12806`)
 
 - ``pandas.tseries.plotting.tsplot`` is deprecated. Use :func:`Series.plot` instead (:issue:`18627`)
+- ``Index.summary()`` is deprecated and will be removed in a future version (:issue:`18217`)
 
 .. _whatsnew_0230.prior_deprecations:
 
@@ -841,6 +845,9 @@ Categorical
 - Bug in :meth:`Series.astype` and ``Categorical.astype()`` where an existing categorical data does not get updated (:issue:`10696`, :issue:`18593`)
 - Bug in :class:`Index` constructor with ``dtype=CategoricalDtype(...)`` where ``categories`` and ``ordered`` are not maintained (issue:`19032`)
 - Bug in :class:`Series` constructor with scalar and ``dtype=CategoricalDtype(...)`` where ``categories`` and ``ordered`` are not maintained (issue:`19565`)
+- Bug in ``Categorical.__iter__`` not converting to Python types (:issue:`19909`)
+- Bug in :func:`pandas.factorize` returning the unique codes for the ``uniques``. This now returns a ``Categorical`` with the same dtype as the input (:issue:`19721`)
+- Bug in :func:`pandas.factorize` including an item for missing values in the ``uniques`` return value (:issue:`19721`)
 
 Datetimelike
 ^^^^^^^^^^^^
@@ -937,6 +944,7 @@ Indexing
 - Bug in :class:`IntervalIndex` where set operations that returned an empty ``IntervalIndex`` had the wrong dtype (:issue:`19101`)
 - Bug in :meth:`DataFrame.drop_duplicates` where no ``KeyError`` is raised when passing in columns that don't exist on the ``DataFrame`` (issue:`19726`)
 - Bug in ``Index`` subclasses constructors that ignore unexpected keyword arguments (:issue:`19348`)
+- Bug in :meth:`Index.difference` when taking difference of an ``Index`` with itself (:issue:`20040`)
 
 
 MultiIndex