bpo-30647: Check nl_langinfo(CODESET) in locale coercion

[ncoghlan]

• On some versions of FreeBSD, setting the "UTF-8" locale
  succeeds, but a subsequent "nl_langinfo(CODESET)" fails
• adding a check for this in the coercion logic means that
  coercion will happen on systems where this check succeeds,
  and will be skipped otherwise
• that way CPython should automatically adapt to changes in
  platform behaviour, rather than needing a new release to
  enable coercion at build time
• this also allows UTF-8 to be re-enabled as a coercion
  target, restoring the locale coercion behaviour on Mac OS X

[mention-bot]

@ncoghlan, thanks for your PR! By analyzing the history of the files in this pull request, we identified @ericsnowcurrently, @vadmium and @zooba to be potential reviewers.

[ncoghlan]

Custom builds of most relevance to this change:

FreeBSD: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%20custom/builds/15
Mac OS X: http://buildbot.python.org/all/builders/x86%20Tiger%20custom/builds/14

[ncoghlan]

The test condition aimed at determining whether or not nl_langinfo worked in the candidate locales wasn't right, so I've tweaked it and kicked off a new custom build.

FreeBSD: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%20custom/builds/16
Mac OS X: http://buildbot.python.org/all/builders/x86%20Tiger%20custom/builds/15

[ncoghlan]

OK, this is looking reasonably promising now: test_c_locale_coercion passed on both Mac OS X and FreeBSD, and the other custom buildbots remain green. test_readline did fail on FreeBSD, and that may be a real bug and/or change in behaviour.

However, the adaptive nature of the tests mean they're also quite vulnerable to nominally passing without actually testing anything, so I'm going to adjust the test case to skip the nl_langinfo check on Linux and Mac OS X.

[ncoghlan]

This looks good on Mac OS X now, but I'm not going to merge it until the FreeBSD 10 custom builder is back online: https://mail.python.org/pipermail/python-buildbots/2017-June/000121.html

[ncoghlan]

The FreeBSD 10 bot is back, and still showing the test_readline failure, so this isn't mergeable in its current form: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%20custom/builds/18/steps/test/logs/stdio

[ncoghlan]

I'm finally setting up a local FreeBSD 10 box in Vagrant, so hopefully I'll be able to work out a sensible way to handle FreeBSD's current behaviour (and hopefully whatever I come up with generalise reasonably to other BSD variants)

Current status: the unexpectedly failing tests turned out to be due to the VM having neither a hostname nor a local timezone set. The symptoms were a bit cryptic - the former resulted in an empty string from socket.gethostname(), which triggers a "Unknown server name" exception if you pass it to socker.gethostbyname_ex(), while the latter leads to an overflow error in time.mktime() if you pass it a time tuple where the is_dst flag is set to 1.

[ncoghlan]

Experimenting locally, I was able to confirm that the following sequence of commands triggers the test_readline failure:

const char *initial_locale = setlocale(LC_CTYPE, NULL);
setlocale(LC_CTYPE, "UTF-8");
setlocale(LC_CTYPE, initial_locale);

The same failure also occurs if the second call uses a normal locale like en_US.UTF-8.

However, I'm also seeing a behaviour where if I print initial_locale before and after the second call to set_locale, the content of the string changes to match the newly set locale...

[ncoghlan]

While the save-and-restore approach proved fruitless, it occurred to me that we haven't modified the environment at this point, so we can use that to restore the original locale settings.

Custom FreeBSD buildbot run for the latest variant: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%20custom/builds/19/steps/test/logs/stdio