Skip to content

Commit 13996e6

Browse files
committed
Define the Token rule
This defines a Token in the grammar so that we can easily refer to it (and to make it easier to see what all the tokens are).
1 parent a8e1afb commit 13996e6

File tree

1 file changed

+19
-0
lines changed

1 file changed

+19
-0
lines changed

src/tokens.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,25 @@
11
r[lex.token]
22
# Tokens
33

4+
r[lex.token.syntax]
5+
> **<sup>Lexer</sup>**\
6+
> Token :\
7+
> &nbsp;&nbsp; &nbsp;&nbsp; IDENTIFIER_OR_KEYWORD\
8+
> &nbsp;&nbsp; | RAW_IDENTIFIER\
9+
> &nbsp;&nbsp; | CHAR_LITERAL\
10+
> &nbsp;&nbsp; | STRING_LITERAL\
11+
> &nbsp;&nbsp; | RAW_STRING_LITERAL\
12+
> &nbsp;&nbsp; | BYTE_LITERAL\
13+
> &nbsp;&nbsp; | BYTE_STRING_LITERAL\
14+
> &nbsp;&nbsp; | RAW_BYTE_STRING_LITERAL\
15+
> &nbsp;&nbsp; | C_STRING_LITERAL\
16+
> &nbsp;&nbsp; | RAW_C_STRING_LITERAL\
17+
> &nbsp;&nbsp; | INTEGER_LITERAL\
18+
> &nbsp;&nbsp; | FLOAT_LITERAL\
19+
> &nbsp;&nbsp; | LIFETIME_TOKEN\
20+
> &nbsp;&nbsp; | PUNCTUATION\
21+
> &nbsp;&nbsp; | RESERVED_TOKEN
22+
423
r[lex.token.intro]
524
Tokens are primitive productions in the grammar defined by regular
625
(non-recursive) languages. Rust source input can be broken down

0 commit comments

Comments
 (0)