Add a last resort font for missing glyphs

Author: QuLogic

LICENSE/LICENSE_LAST_RESORT_FONT

@@ -0,0 +1,97 @@
+Last Resort High-Efficiency Font License
+========================================
+
+This Font Software is licensed under the SIL Open Font License,
+Version 1.1.
+
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font
+creation efforts of academic and linguistic communities, and to
+provide a free and open framework in which fonts may be shared and
+improved in partnership with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply to
+any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software
+components as distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to,
+deleting, or substituting -- in part or in whole -- any of the
+components of the Original Version, by changing formats or by porting
+the Font Software to a new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed,
+modify, redistribute, and sell modified and unmodified copies of the
+Font Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components, in
+Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the
+corresponding Copyright Holder. This restriction only applies to the
+primary font name as presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created using
+the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.
+
+SPDX-License-Identifier: OFL-1.1

doc/users/next_whats_new/last_resort_font.rst

@@ -0,0 +1,37 @@
+Missing glyphs use Last Resort font
+-----------------------------------
+
+Most fonts do not have 100% character coverage, and will fall back to a "not
+found" glyph for character that are not provided. Often, this glyph will be minimal
+(e.g., the default DejaVu Sans "not found" glyph is just a rectangle.) Such minimal
+glyphs provide no context as to the characters that are missing.
+
+Now, missing glyphs will fall back to the `Last Resort font
+<https://github.com/unicode-org/last-resort-font>`__ produced by the Unicode Consortium.
+This special-purpose font provides glyphs that represent types of Unicode characters.
+These glyphs show a representative character from the missing Unicode block,
+and at larger sizes, more context to help determine which character and font
+are needed.
+
+To disable this fallback behaviour, set :rc:`font.enable_last_resort` to ``False``.
+
+.. plot::
+    :alt: An example of missing glyph behaviour, with increasing font size from top to bottom.
+
+    text = '\N{Bengali Digit Zero}\N{Hiragana Letter A}\ufdd0'
+    sizes = [
+        (0.90, 6),
+        (0.85, 8),
+        (0.80, 10),
+        (0.75, 12),
+        (0.70, 16),
+        (0.63, 20),
+        (0.55, 24),
+        (0.45, 32),
+        (0.30, 48),
+        (0.10, 64),
+    ]
+
+    fig = plt.figure()
+    for y, size in sizes:
+        fig.text(0.01, y, f'{size}pt: {text}', fontsize=size)

lib/matplotlib/font_manager.py

@@ -1546,17 +1546,27 @@ def is_opentype_cff_font(filename):
 
 
 @lru_cache(64)
-def _get_font(font_filepaths, hinting_factor, *, _kerning_factor, thread_id):
+def _get_font(font_filepaths, hinting_factor, *, _kerning_factor, thread_id,
+              enable_last_resort):
     first_fontpath, *rest = font_filepaths
+    fallback_list = [
+        ft2font.FT2Font(fpath, hinting_factor, _kerning_factor=_kerning_factor)
+        for fpath in rest
+    ]
+    # Add Last Resort font so we always have glyphs regardless of font.
+    if enable_last_resort:
+        path = _cached_realpath(
+            cbook._get_data_path('fonts', 'ttf', 'LastResortHE-Regular.ttf'))
+        last_resort = ft2font.FT2Font(path, hinting_factor,
+                                      _kerning_factor=_kerning_factor,
+                                      _warn_if_used=True)
+        # Ensure we are using the right charmap; FreeType picks the Unicode one
+        # by default, but this exists only for Windows, and is empty.
+        last_resort.set_charmap(0)
+        fallback_list.append(last_resort)
     return ft2font.FT2Font(
         first_fontpath, hinting_factor,
-        _fallback_list=[
-            ft2font.FT2Font(
-                fpath, hinting_factor,
-                _kerning_factor=_kerning_factor
-            )
-            for fpath in rest
-        ],
+        _fallback_list=fallback_list,
         _kerning_factor=_kerning_factor
     )
 
@@ -1611,7 +1621,8 @@ def get_font(font_filepaths, hinting_factor=None):
         hinting_factor,
         _kerning_factor=mpl.rcParams['text.kerning_factor'],
         # also key on the thread ID to prevent segfaults with multi-threading
-        thread_id=threading.get_ident()
+        thread_id=threading.get_ident(),
+        enable_last_resort=mpl.rcParams['font.enable_last_resort'],
     )
 
 

lib/matplotlib/mpl-data/fonts/ttf/LastResortHE-Regular.ttf

@@ -278,6 +278,11 @@
 #font.fantasy:    Chicago, Charcoal, Impact, Western, xkcd script, fantasy
 #font.monospace:  DejaVu Sans Mono, Bitstream Vera Sans Mono, Computer Modern Typewriter, Andale Mono, Nimbus Mono L, Courier New, Courier, Fixed, Terminal, monospace
 
+## If font.enable_last_resort is True, then Unicode Consortium's Last Resort
+## font will be appended to all font selections. This ensures that there will
+## always be a glyph displayed.
+#font.enable_last_resort: true
+
 
 ## ***************************************************************************
 ## * TEXT                                                                    *

lib/matplotlib/mpl-data/matplotlibrc

@@ -1016,6 +1016,7 @@ def _convert_validator_spec(key, conv):
     "boxplot.meanprops.linewidth":       validate_float,
 
     ## font props
+    "font.enable_last_resort":     validate_bool,
     "font.family":     validate_stringlist,  # used by text object
     "font.style":      validate_string,
     "font.variant":    validate_string,

lib/matplotlib/rcsetup.py

@@ -889,7 +889,9 @@ def test__get_fontmap():
     )
     fontmap = ft._get_fontmap(test_str)
     for char, font in fontmap.items():
-        if ord(char) > 127:
+        if char == '\n':  # Not actually a rendered glyph.
+            assert Path(font.fname).name == 'LastResortHE-Regular.ttf'
+        elif ord(char) > 127:
             assert Path(font.fname).name == 'DejaVuSans.ttf'
         else:
             assert Path(font.fname).name == 'cmr10.ttf'

lib/matplotlib/tests/test_ft2font.py

@@ -823,6 +823,7 @@ def test_pdf_kerning():
 
 
 def test_unsupported_script(recwarn):
+    # plt.rcParams['font.enable_last_resort'] = False
     fig = plt.figure()
     t = fig.text(.5, .5, "\N{BENGALI DIGIT ZERO}")
     fig.canvas.draw()

lib/matplotlib/tests/test_text.py

@@ -258,8 +258,11 @@ FT2Font::get_path(std::vector<double> &vertices, std::vector<unsigned char> &cod
 FT2Font::FT2Font(FT_Open_Args &open_args,
                  long hinting_factor_,
                  std::vector<FT2Font *> &fallback_list,
-                 FT2Font::WarnFunc warn)
-    : ft_glyph_warn(warn), image(), face(nullptr)
+                 FT2Font::WarnFunc warn, bool warn_if_used)
+    : ft_glyph_warn(warn), warn_if_used(warn_if_used), image(), face(nullptr),
+      hinting_factor(hinting_factor_),
+      // set default kerning factor to 0, i.e., no kerning manipulation
+      kerning_factor(0)
 {
     clear();
 
@@ -268,12 +271,7 @@ FT2Font::FT2Font(FT_Open_Args &open_args,
         throw_ft_error("Can not load face", error);
     }
 
-    // set default kerning factor to 0, i.e., no kerning manipulation
-    kerning_factor = 0;
-
     // set a default fontsize 12 pt at 72dpi
-    hinting_factor = hinting_factor_;
-
     error = FT_Set_Char_Size(face, 12 * 64, 0, 72 * (unsigned int)hinting_factor, 72);
     if (error) {
         FT_Done_Face(face);
@@ -441,6 +439,8 @@ void FT2Font::set_text(
             char_to_font[codepoint] = ft_object_with_glyph;
             glyph_to_font[glyph_index] = ft_object_with_glyph;
             ft_object_with_glyph->load_glyph(glyph_index, flags, ft_object_with_glyph, false);
+        } else if (ft_object_with_glyph->warn_if_used) {
+            ft_glyph_warn((FT_ULong)codepoint, glyph_seen_fonts);
         }
 
         // retrieve kerning distance and move pen position
@@ -510,6 +510,8 @@ void FT2Font::load_char(long charcode, FT_Int32 flags, FT2Font *&ft_object, bool
             else if (glyph_error) {
                 throw_ft_error("Could not load charcode", glyph_error);
             }
+        } else if (ft_object_with_glyph->warn_if_used) {
+            ft_glyph_warn(charcode, glyph_seen_fonts);
         }
         ft_object = ft_object_with_glyph;
     } else {
@@ -569,7 +571,9 @@ bool FT2Font::load_char_with_fallback(FT2Font *&ft_object_with_glyph,
                                       bool override = false)
 {
     FT_UInt glyph_index = FT_Get_Char_Index(face, charcode);
-    glyph_seen_fonts.insert(face->family_name);
+    if (!warn_if_used) {
+        glyph_seen_fonts.insert(face->family_name);
+    }
 
     if (glyph_index || override) {
         charcode_error = FT_Load_Glyph(face, glyph_index, flags);

src/ft2font.cpp

@@ -71,7 +71,8 @@ class FT2Font
 
   public:
     FT2Font(FT_Open_Args &open_args, long hinting_factor,
-            std::vector<FT2Font *> &fallback_list, WarnFunc warn);
+            std::vector<FT2Font *> &fallback_list,
+            WarnFunc warn, bool warn_if_used);
     virtual ~FT2Font();
     void clear();
     void set_size(double ptsize, double dpi);
@@ -139,6 +140,7 @@ class FT2Font
 
   private:
     WarnFunc ft_glyph_warn;
+    bool warn_if_used;
     FT2Image image;
     FT_Face face;
     FT_Vector pen;    /* untransformed origin  */

src/ft2font.h

@@ -435,14 +435,20 @@ const char *PyFT2Font_init__doc__ = R"""(
     _kerning_factor : int, optional
         Used to adjust the degree of kerning.
 
+        .. warning::
+            This API is private: do not use it directly.
+
+    _warn_if_used : bool, optional
+        Used to trigger missing glyph warnings.
+
         .. warning::
             This API is private: do not use it directly.
 )""";
 
 static PyFT2Font *
 PyFT2Font_init(py::object filename, long hinting_factor = 8,
                std::optional<std::vector<PyFT2Font *>> fallback_list = std::nullopt,
-               int kerning_factor = 0)
+               int kerning_factor = 0, bool warn_if_used = false)
 {
     if (hinting_factor <= 0) {
         throw py::value_error("hinting_factor must be greater than 0");
@@ -491,7 +497,8 @@ PyFT2Font_init(py::object filename, long hinting_factor = 8,
         self->stream.close = nullptr;
     }
 
-    self->x = new FT2Font(open_args, hinting_factor, fallback_fonts, ft_glyph_warn);
+    self->x = new FT2Font(open_args, hinting_factor, fallback_fonts, ft_glyph_warn,
+                          warn_if_used);
 
     self->x->set_kerning_factor(kerning_factor);
 
@@ -1714,6 +1721,7 @@ PYBIND11_MODULE(ft2font, m, py::mod_gil_not_used())
         .def(py::init(&PyFT2Font_init),
              "filename"_a, "hinting_factor"_a=8, py::kw_only(),
              "_fallback_list"_a=py::none(), "_kerning_factor"_a=0,
+             "_warn_if_used"_a=false,
              PyFT2Font_init__doc__)
         .def("clear", &PyFT2Font_clear, PyFT2Font_clear__doc__)
         .def("set_size", &PyFT2Font_set_size, "ptsize"_a, "dpi"_a,