File tree Expand file tree Collapse file tree 1 file changed +19
-0
lines changed Expand file tree Collapse file tree 1 file changed +19
-0
lines changed Original file line number Diff line number Diff line change 1
1
r[ lex.token]
2
2
# Tokens
3
3
4
+ r[ lex.token.syntax]
5
+ > ** <sup >Lexer</sup >** \
6
+ > Token :\
7
+ >   ;  ;   ;  ; IDENTIFIER_OR_KEYWORD\
8
+ >   ;  ; | RAW_IDENTIFIER\
9
+ >   ;  ; | CHAR_LITERAL\
10
+ >   ;  ; | STRING_LITERAL\
11
+ >   ;  ; | RAW_STRING_LITERAL\
12
+ >   ;  ; | BYTE_LITERAL\
13
+ >   ;  ; | BYTE_STRING_LITERAL\
14
+ >   ;  ; | RAW_BYTE_STRING_LITERAL\
15
+ >   ;  ; | C_STRING_LITERAL\
16
+ >   ;  ; | RAW_C_STRING_LITERAL\
17
+ >   ;  ; | INTEGER_LITERAL\
18
+ >   ;  ; | FLOAT_LITERAL\
19
+ >   ;  ; | LIFETIME_TOKEN\
20
+ >   ;  ; | PUNCTUATION\
21
+ >   ;  ; | RESERVED_TOKEN
22
+
4
23
r[ lex.token.intro]
5
24
Tokens are primitive productions in the grammar defined by regular
6
25
(non-recursive) languages. Rust source input can be broken down
You can’t perform that action at this time.
0 commit comments