Skip to content

[Parse] Tokens.def cleanup and documentation #6618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[Parse] Tokens.def cleanup and documentation #6618

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7bee1a2
[Parse] Refactored internal structure of Tokens.def and documented us…
micahhainline Jan 6, 2017
09b3a06
[Parse] Made use of new guarantees and abstractions in Tokens.def
micahhainline Jan 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 75 additions & 31 deletions include/swift/Parse/Tokens.def
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,59 +9,100 @@
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
//
//===----------------------------------------------------------------------===//
//
// This file defines macros used for macro-metaprogramming lexer tokens.
//
///
/// This file defines macros used for macro-metaprogramming lexer tokens.
///
/// To use, simply define one or more of the macro functions listed below and
/// then import this file. The macro functions follow a hierarchical pattern,
/// and defining a higher-level macro will expend (by default) all the patterns
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/expend/expand/

/// beneath it. All of the macro functions will be undefined automatically at
/// the end, there is no need to do this manually.
///
/// KEYWORD(kw)
/// SWIFT_KEYWORD(kw)
/// DECL_KEYWORD(kw)
/// STMT_KEYWORD(kw)
/// EXPR_KEYWORD(kw)
/// PAT_KEYWORD(kw)
/// SIL_KEYWORD(kw)
/// POUND_KEYWORD(kw)
/// POUND_OBJECT_LITERAL(kw, desc, proto)
/// POUND_OLD_OBJECT_LITERAL(kw, new_kw, old_arg, new_arg)
/// POUND_CONFIG(kw)
/// PUNCTUATOR(name, str)
///
//===----------------------------------------------------------------------===//

/// KEYWORD(kw)
/// Expands for every Swift keyword, such as 'if', 'else', etc.
/// Expands by default for every Swift keyword and every SIL keyword, such as
/// 'if', 'else', 'sil_global', etc. If you only want to use Swift keywords
/// see SWIFT_KEYWORD.
#ifndef KEYWORD
#define KEYWORD(kw)
#endif

/// SWIFT_KEYWORD(kw)
/// Expands for every Swift keyword.
#ifndef SWIFT_KEYWORD
#define SWIFT_KEYWORD(kw) KEYWORD(kw)
#endif

/// DECL_KEYWORD(kw)
/// Expands for every Swift keyword that can be used in a declaration.
#ifndef DECL_KEYWORD
#define DECL_KEYWORD(kw) KEYWORD(kw)
#define DECL_KEYWORD(kw) SWIFT_KEYWORD(kw)
#endif

/// STMT_KEYWORD(kw)
/// Expands for every Swift keyword used in statement grammar.
#ifndef STMT_KEYWORD
#define STMT_KEYWORD(kw) KEYWORD(kw)
#define STMT_KEYWORD(kw) SWIFT_KEYWORD(kw)
#endif

/// EXPR_KEYWORD(kw)
/// Expands for every Swift keyword used in an expression, such as 'true',
/// 'false', and 'as'
#ifndef EXPR_KEYWORD
#define EXPR_KEYWORD(kw) SWIFT_KEYWORD(kw)
#endif

/// PAT_KEYWORD(kw)
/// Expands for every Swift keyword used in a pattern, which is currently
/// limited to '_'
#ifndef PAT_KEYWORD
#define PAT_KEYWORD(kw) SWIFT_KEYWORD(kw)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question about the taxonomy here: isn't every pattern keyword an expression keyword, and every expression keyword a statement keyword?

#endif

/// SIL_KEYWORD(kw)
/// Expands for every SIL keyword. These are only keywords when parsing SIL.
#ifndef SIL_KEYWORD
#define SIL_KEYWORD(kw) KEYWORD(kw)
#endif

/// POUND_KEYWORD(kw)
/// Every keyword in the #foo namespace.
/// Every keyword in the # namespace.
#ifndef POUND_KEYWORD
#define POUND_KEYWORD(kw)
#endif

/// POUND_OBJECT_LITERAL(kw, desc, proto)
/// Every keyword in the #foo namespace representing an object literal.
/// Every keyword in the # namespace representing a new-form object literal.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the comment below, I'd say that this is the syntax for an object literal. "New-form" is not a term used elsewhere to refer to the current syntax, and it implies that there is another form in the language, which there isn't.

#ifndef POUND_OBJECT_LITERAL
#define POUND_OBJECT_LITERAL(kw, desc, proto) POUND_KEYWORD(kw)
#endif

/// POUND_OLD_OBJECT_LITERAL(kw, new_kw, old_arg, new_arg)
/// Every keyword in the #foo namespace representing an object literal.
/// Every keyword in the # namespace representing an old-form object literal.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good addition, but perhaps it'd be better to explicitly say it's obsolete; it's not one of two valid ways of spelling an object literal, it's just here for migrator purposes.

#ifndef POUND_OLD_OBJECT_LITERAL
#define POUND_OLD_OBJECT_LITERAL(kw, new_kw, old_arg, new_arg) POUND_KEYWORD(kw)
#endif

/// POUND_CONFIG(kw)
/// Every keyword in the #foo namespace representing a configuration.
/// Every keyword in the # namespace representing a configuration.
#ifndef POUND_CONFIG
#define POUND_CONFIG(kw) POUND_KEYWORD(kw)
#endif

/// SIL_KEYWORD(kw)
/// Expands for every SIL keyword. These are only keywords when parsing SIL.
#ifndef SIL_KEYWORD
#define SIL_KEYWORD(kw) KEYWORD(kw)
#endif

/// PUNCTUATOR(name, str)
/// Expands for every Swift punctuator.
/// \param name The symbolic name of the punctuator, such as
Expand Down Expand Up @@ -128,27 +169,27 @@ STMT_KEYWORD(where)
STMT_KEYWORD(catch)

// Expression keywords.
KEYWORD(as)
KEYWORD(Any)
KEYWORD(false)
KEYWORD(is)
KEYWORD(nil)
KEYWORD(rethrows)
KEYWORD(super)
KEYWORD(self)
KEYWORD(Self)
KEYWORD(throw)
KEYWORD(true)
KEYWORD(try)
KEYWORD(throws)
EXPR_KEYWORD(as)
EXPR_KEYWORD(Any)
EXPR_KEYWORD(false)
EXPR_KEYWORD(is)
EXPR_KEYWORD(nil)
EXPR_KEYWORD(rethrows)
EXPR_KEYWORD(super)
EXPR_KEYWORD(self)
EXPR_KEYWORD(Self)
EXPR_KEYWORD(throw)
EXPR_KEYWORD(true)
EXPR_KEYWORD(try)
EXPR_KEYWORD(throws)
KEYWORD(__FILE__)
KEYWORD(__LINE__)
KEYWORD(__COLUMN__)
KEYWORD(__FUNCTION__)
KEYWORD(__DSO_HANDLE__)

// Pattern keywords.
KEYWORD(_)
PAT_KEYWORD(_)

// Punctuators.
PUNCTUATOR(l_paren, "(")
Expand Down Expand Up @@ -216,11 +257,14 @@ POUND_KEYWORD(dsohandle)


#undef KEYWORD
#undef SWIFT_KEYWORD
#undef DECL_KEYWORD
#undef STMT_KEYWORD
#undef EXPR_KEYWORD
#undef PAT_KEYWORD
#undef SIL_KEYWORD
#undef PUNCTUATOR
#undef POUND_KEYWORD
#undef POUND_OBJECT_LITERAL
#undef POUND_OLD_OBJECT_LITERAL
#undef POUND_CONFIG
#undef PUNCTUATOR
4 changes: 2 additions & 2 deletions lib/IDE/SyntaxModel.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@ SyntaxModelContext::SyntaxModelContext(SourceFile &SrcFile)
switch(Tok.getKind()) {
#define KEYWORD(X) case tok::kw_##X:
#include "swift/Parse/Tokens.def"
#undef KEYWORD
Kind = SyntaxNodeKind::Keyword;
// Some keywords can be used as an argument labels. If this one can and
// is being used as one, treat it as an identifier instead.
Expand All @@ -119,7 +118,8 @@ SyntaxModelContext::SyntaxModelContext(SourceFile &SrcFile)

#define POUND_OLD_OBJECT_LITERAL(Name, NewName, OldArg, NewArg) \
case tok::pound_##Name:
#define POUND_OBJECT_LITERAL(Name, Desc, Proto) case tok::pound_##Name:
#define POUND_OBJECT_LITERAL(Name, Desc, Proto) \
case tok::pound_##Name:
#include "swift/Parse/Tokens.def"
LiteralStartLoc = Loc;
continue;
Expand Down
1 change: 0 additions & 1 deletion lib/Parse/Lexer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,7 +599,6 @@ void Lexer::lexHash() {

// Map the character sequence onto
tok Kind = llvm::StringSwitch<tok>(StringRef(CurPtr, tmpPtr-CurPtr))
#define KEYWORD(kw)
#define POUND_KEYWORD(id) \
.Case(#id, tok::pound_##id)
#include "swift/Parse/Tokens.def"
Expand Down
9 changes: 4 additions & 5 deletions tools/SourceKit/lib/SwiftLang/SwiftCompletion.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -589,11 +589,10 @@ getCodeCompletionLiteralKindForUID(UIdent uid) {

static CodeCompletionKeywordKind
getCodeCompletionKeywordKindForUID(UIdent uid) {
#define SIL_KEYWORD(kw)
#define KEYWORD(kw) \
static UIdent Keyword##kw##UID("source.lang.swift.keyword." #kw); \
if (uid == Keyword##kw##UID) { \
return CodeCompletionKeywordKind::kw_##kw; \
#define SWIFT_KEYWORD(kw) \
static UIdent Keyword##kw##UID("source.lang.swift.keyword." #kw); \
if (uid == Keyword##kw##UID) { \
return CodeCompletionKeywordKind::kw_##kw; \
}
#include "swift/Parse/Tokens.def"

Expand Down