CI/TST: Disallow usage of pytest.xfail (#39624)

Author: rhshadrach

.pre-commit-config.yaml

@@ -127,6 +127,12 @@ repos:
         types: [python]
         files: ^pandas/tests/
         exclude: ^pandas/tests/extension/
+    -   id: unwanted-patters-pytest-xfail
+        name: Check for use of pytest.xfail
+        entry: pytest\.xfail
+        language: pygrep
+        types: [python]
+        files: ^pandas/tests/
     -   id: inconsistent-namespace-usage
         name: 'Check for inconsistent use of pandas namespace in tests'
         entry: python scripts/check_for_inconsistent_pandas_namespace.py

doc/source/development/code_style.rst

@@ -161,6 +161,46 @@ For example:
     # wrong
     from common import test_base
 
+Testing
+=======
+
+Failing tests
+--------------
+
+See https://docs.pytest.org/en/latest/skipping.html for background.
+
+Do not use ``pytest.xfail``
+---------------------------
+
+Do not use this method. It has the same behavior as ``pytest.skip``, namely
+it immediately stops the test and does not check if the test will fail. If
+this is the behavior you desire, use ``pytest.skip`` instead.
+
+Using ``pytest.mark.xfail``
+---------------------------
+
+Use this method if a test is known to fail but the manner in which it fails
+is not meant to be captured. It is common to use this method for a test that
+exhibits buggy behavior or a non-implemented feature. If
+the failing test has flaky behavior, use the argument ``strict=False``. This
+will make it so pytest does not fail if the test happens to pass.
+
+Prefer the decorator ``@pytest.mark.xfail`` and the argument ``pytest.param``
+over usage within a test so that the test is appropriately marked during the
+collection phase of pytest. For xfailing a test that involves multiple
+parameters, a fixture, or a combination of these, it is only possible to
+xfail during the testing phase. To do so, use the ``request`` fixture:
+
+.. code-block:: python
+
+    import pytest
+
+    def test_xfail(request):
+        request.node.add_marker(pytest.mark.xfail(reason="Indicate why here"))
+
+xfail is not to be used for tests involving failure due to invalid user arguments.
+For these tests, we need to verify the correct exception type and error message
+is being raised, using ``pytest.raises`` instead.
 
 Miscellaneous
 =============