DOC: Disallowing search engines in the development docs

[datapythonista]

• closes DOC: Disallow crawlers in development docs #26924
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[codecov]

Codecov Report

Merging #26931 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26931      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50716    50716              
==========================================
- Hits        46593    46589       -4     
- Misses       4123     4127       +4

Flag	Coverage Δ
#multiple	90.45% <ø> (ø)	⬆️
#single	41.1% <ø> (-0.09%)	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	88.88% <0%> (-11.12%)	⬇️
pandas/core/frame.py	96.88% <0%> (-0.12%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d39a6de...dbd3764. Read the comment docs.

[jreback]

sure. note sure how we can tell if this works?

[datapythonista]

The file has been created successfully, that worked: https://dev.pandas.io/robots.txt

I think Google has tools where they give you information on how they crawl your site, but not much idea how that works, and not really looking forward to start working in SEO now. :)

I think we can simply wait some days until Google updates its data, and then we shouldn't see all the results here: https://www.google.com/search?q=site%3Adev.pandas.io