Don't allow restricted characters in identifiers (UTS 39, C1) (#11580)

Author: mrluc

lib/elixir/pages/unicode-security.md

@@ -0,0 +1,25 @@
+# Unicode Security
+
+(See [Unicode Syntax](unicode-syntax.html) for information on Unicode usage in Elixir).
+
+Elixir will prevent, or warn on, confusing or suspicious uses of Unicode in identifiers since Elixir v1.15, as defined in the [Unicode Technical Standard #39](https://unicode.org/reports/tr39/) on Security.
+
+The focus of this document is to describe how Elixir implements the conformance clauses from that standard, referred to as C1, C2, and so on. All quotes are from the spec unless otherwise noted.
+
+## C1. General Security Profile for Identifiers
+
+Elixir will not allow tokenization of identifiers with codepoints in `\p{Identifier_Status=Restricted}`.
+
+> An implementation following the General Security Profile does not permit any characters in \p{Identifier_Status=Restricted}, ...
+
+For instance, the 'HANGUL FILLER' (`ㅤ`) character, which is often invisible, is an uncommon codepoint and will trigger this warning.
+
+## C2, C3 (planned)
+
+Elixir may implement Confusable Detection, and Mixed-Script Confusable detection, in the future, and will likely emit warnings in those cases; there is a reference implementation.
+
+## C4, C5 (inapplicable)
+
+'C4 - Restriction Level detection' conformance is not claimed and is inapplicable. (It applies to classifying the level of safety of a given arbitrary string into one of 5 restriction levels).
+
+'C5 - Mixed number detection' conformance is inapplicable as Elixir does not support Unicode numbers.

lib/elixir/pages/unicode-syntax.md

@@ -6,6 +6,8 @@ Quoted identifiers, such as strings (`"olá"`) and charlists (`'olá'`), support
 
 Elixir also supports Unicode in identifiers since Elixir v1.5, as defined in the [Unicode Annex #31](https://unicode.org/reports/tr31/). The focus of this document is to describe how Elixir implements the requirements outlined in the Unicode Annex. These requirements are referred to as R1, R6 and so on.
 
+Elixir provides identifier security as defined in the [Unicode Technical Standard #39](https://unicode.org/reports/tr39/) on Security. The [Unicode Security](unicode-security.html) document describes how Elixir implements the clauses of that Standard.
+
 To check the Unicode version of your current Elixir installation, run `String.Unicode.version()`.
 
 ## R1. Default Identifiers

lib/elixir/test/elixir/kernel/parser_test.exs

@@ -499,6 +499,11 @@ defmodule Kernel.ParserTest do
     end
 
     test "invalid token" do
+      assert_syntax_error(
+        ~r/nofile:1:1: unexpected token: "#{"\u3164"}" \(column 1, code point U\+3164\)/,
+        'ㅤ = 1'
+      )
+
       assert_syntax_error(
         ~r/nofile:1:7: unexpected token: "#{"\u200B"}" \(column 7, code point U\+200B\)/,
         '[foo: \u200B]\noops'