Skip to content

closes gh-96734: Update to Unicode 15.0.0. #96809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Doc/library/stdtypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ Notes:
The numeric literals accepted include the digits ``0`` to ``9`` or any
Unicode equivalent (code points with the ``Nd`` property).

See https://www.unicode.org/Public/14.0.0/ucd/extracted/DerivedNumericType.txt
See https://www.unicode.org/Public/15.0.0/ucd/extracted/DerivedNumericType.txt
for a complete list of code points with the ``Nd`` property.


Expand Down
8 changes: 4 additions & 4 deletions Doc/library/unicodedata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@

This module provides access to the Unicode Character Database (UCD) which
defines character properties for all Unicode characters. The data contained in
this database is compiled from the `UCD version 14.0.0
<https://www.unicode.org/Public/14.0.0/ucd>`_.
this database is compiled from the `UCD version 15.0.0
<https://www.unicode.org/Public/15.0.0/ucd>`_.

The module uses the same names and symbols as defined by Unicode
Standard Annex #44, `"Unicode Character Database"
Expand Down Expand Up @@ -175,6 +175,6 @@ Examples:

.. rubric:: Footnotes

.. [#] https://www.unicode.org/Public/14.0.0/ucd/NameAliases.txt
.. [#] https://www.unicode.org/Public/15.0.0/ucd/NameAliases.txt

.. [#] https://www.unicode.org/Public/14.0.0/ucd/NamedSequences.txt
.. [#] https://www.unicode.org/Public/15.0.0/ucd/NamedSequences.txt
8 changes: 4 additions & 4 deletions Doc/reference/lexical_analysis.rst
Original file line number Diff line number Diff line change
Expand Up @@ -315,16 +315,16 @@ The Unicode category codes mentioned above stand for:
* *Nd* - decimal numbers
* *Pc* - connector punctuations
* *Other_ID_Start* - explicit list of characters in `PropList.txt
<https://www.unicode.org/Public/14.0.0/ucd/PropList.txt>`_ to support backwards
<https://www.unicode.org/Public/15.0.0/ucd/PropList.txt>`_ to support backwards
compatibility
* *Other_ID_Continue* - likewise

All identifiers are converted into the normal form NFKC while parsing; comparison
of identifiers is based on NFKC.

A non-normative HTML file listing all valid identifier characters for Unicode
14.0.0 can be found at
https://www.unicode.org/Public/14.0.0/ucd/DerivedCoreProperties.txt
15.0.0 can be found at
https://www.unicode.org/Public/15.0.0/ucd/DerivedCoreProperties.txt


.. _keywords:
Expand Down Expand Up @@ -1013,4 +1013,4 @@ occurrence outside string literals and comments is an unconditional error:

.. rubric:: Footnotes

.. [#] https://www.unicode.org/Public/11.0.0/ucd/NameAliases.txt
.. [#] https://www.unicode.org/Public/15.0.0/ucd/NameAliases.txt
76 changes: 6 additions & 70 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
@@ -1,74 +1,4 @@

****************************
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were all these removals intentional?

What's New In Python 3.12
****************************

:Release: |release|
:Date: |today|

.. Rules for maintenance:

* Anyone can add text to this document. Do not spend very much time
on the wording of your changes, because your text will probably
get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add
changes; it's therefore more important to add your changes to
Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness
is the purpose of Misc/NEWS. Some changes I consider too small
or esoteric to include. If such a change is added to the text,
I'll just remove it. (This is another reason you shouldn't spend
too much time on writing your addition.)

* If you want to draw your new text to the attention of the
maintainer, add 'XXX' to the beginning of the paragraph or
section.

* It's OK to just add a fragmentary note about a change. For
example: "XXX Describe the transmogrify() function added to the
socket module." The maintainer will research the change and
write the necessary text.

* You can comment out your additions if you like, but it's not
necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is
sufficient; the e-mail address isn't necessary.

* It's helpful to add the bug/patch number as a comment:

XXX Describe the transmogrify() function added to the socket
module.
(Contributed by P.Y. Developer in :issue:`12345`.)

This saves the maintainer the effort of going through the Mercurial log
when researching a change.

This article explains the new features in Python 3.12, compared to 3.11.

For full details, see the :ref:`changelog <changelog>`.

.. note::

Prerelease users should be aware that this document is currently in draft
form. It will be updated substantially as Python 3.12 moves towards release,
so it's worth checking back even after reading earlier versions.


Summary -- Release highlights
=============================

.. This section singles out the most important changes in Python 3.12.
Brevity is key.


.. PEP-sized items next.

Important deprecations, removals or restrictions:

* :pep:`623`, Remove wstr from Unicode


New Features
Expand Down Expand Up @@ -147,6 +77,12 @@ threading
profiling functions in all running threads in addition to the calling one.
(Contributed by Pablo Galindo in :gh:`93503`.)

unicodedata
-----------

* The Unicode database has been updated to version 15.0.0. (Contributed by
Benjamin Peterson in :gh:`96734`).


Optimizations
=============
Expand Down
6 changes: 3 additions & 3 deletions Lib/test/test_unicodedata.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
class UnicodeMethodsTest(unittest.TestCase):

# update this, if the database changes
expectedchecksum = '4739770dd4d0e5f1b1677accfc3552ed3c8ef326'
expectedchecksum = 'e708c31c0d51f758adf475cb7201cf80917362be'

@requires_resource('cpu')
def test_method_checksum(self):
Expand Down Expand Up @@ -71,7 +71,7 @@ class UnicodeFunctionsTest(UnicodeDatabaseTest):

# Update this if the database changes. Make sure to do a full rebuild
# (e.g. 'make distclean && make') to get the correct checksum.
expectedchecksum = '4975f3ec0acd4a62465d18c9bf8519b1964181f6'
expectedchecksum = '84b88a89f40aeae96852732f9dc0ee08be49780f'

@requires_resource('cpu')
def test_function_checksum(self):
Expand Down Expand Up @@ -224,7 +224,7 @@ def test_east_asian_width(self):
def test_east_asian_width_unassigned(self):
eaw = self.db.east_asian_width
# unassigned
for char in '\u0530\u0ece\u10c6\u20fc\uaaca\U000107bd\U000115f2':
for char in '\u0530\u0ecf\u10c6\u20fc\uaaca\U000107bd\U000115f2':
self.assertEqual(eaw(char), 'N')
self.assertIs(self.db.name(char, None), None)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update :mod:`unicodedata` database to Unicode 15.0.0.
5 changes: 3 additions & 2 deletions Modules/unicodedata.c
Original file line number Diff line number Diff line change
Expand Up @@ -1046,11 +1046,12 @@ is_unified_ideograph(Py_UCS4 code)
(0x3400 <= code && code <= 0x4DBF) || /* CJK Ideograph Extension A */
(0x4E00 <= code && code <= 0x9FFF) || /* CJK Ideograph */
(0x20000 <= code && code <= 0x2A6DF) || /* CJK Ideograph Extension B */
(0x2A700 <= code && code <= 0x2B738) || /* CJK Ideograph Extension C */
(0x2A700 <= code && code <= 0x2B739) || /* CJK Ideograph Extension C */
(0x2B740 <= code && code <= 0x2B81D) || /* CJK Ideograph Extension D */
(0x2B820 <= code && code <= 0x2CEA1) || /* CJK Ideograph Extension E */
(0x2CEB0 <= code && code <= 0x2EBE0) || /* CJK Ideograph Extension F */
(0x30000 <= code && code <= 0x3134A); /* CJK Ideograph Extension G */
(0x30000 <= code && code <= 0x3134A) || /* CJK Ideograph Extension G */
(0x31350 <= code && code <= 0x323AF); /* CJK Ideograph Extension H */
}

/* macros used to determine if the given code point is in the PUA range that
Expand Down
Loading