Skip to content

[syntax-coloring] Rework the syntax map to use offset + length and simplify the delta logic #10289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 19, 2017
Merged

[syntax-coloring] Rework the syntax map to use offset + length and simplify the delta logic #10289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
09cb55e
[syntax-coloring] Rework the syntax map to use offset+length rather t…
Jun 13, 2017
fa3873e
[syntax-coloring] Highlight unterminated regular and multi-line strin…
Jun 14, 2017
1158137
[syntax-coloring] Fix incorrectly reporting a no-op when a token is r…
Jun 14, 2017
04674ff
[syntax-coloring] Add test with nested tokens
Jun 14, 2017
3791e12
[syntax-coloring] Remove early exit from delta logic when there are n…
Jun 16, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions lib/IDE/SyntaxModel.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,15 @@ SyntaxModelContext::SyntaxModelContext(SourceFile &SrcFile)
break;
}

case tok::unknown: {
if (Tok.getRawText().startswith("\"")) {
// This is an invalid string literal
Kind = SyntaxNodeKind::String;
break;
}
continue;
}

default:
continue;
}
Expand Down
19 changes: 19 additions & 0 deletions test/IDE/coloring.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,15 @@ func f(x: Int) -> Int {
// CHECK: <str>"This is string </str>\<anchor>(</anchor>genFn({(a:<type>Int</type> -> <type>Int</type>) <kw>in</kw> a})<anchor>)</anchor><str> interpolation"</str>
"This is string \(genFn({(a:Int -> Int) in a})) interpolation"

// CHECK: <str>"This is unterminated</str>
"This is unterminated

// CHECK: <str>"This is unterminated with ignored \(interpolation) in it</str>
"This is unterminated with ignored \(interpolation) in it

// CHECK: <str>"This is terminated with invalid \(interpolation" + "in it"</str>
"This is terminated with invalid \(interpolation" + "in it"

// CHECK: <str>"""
// CHECK-NEXT: This is a multiline string.
// CHECK-NEXT: """</str>
Expand All @@ -236,9 +245,19 @@ func f(x: Int) -> Int {

// CHECK: <str>"""
// CHECK-NEXT: This is a multiline</str>\<anchor>(</anchor> <str>"interpolated"</str> <anchor>)</anchor><str>string
// CHECK-NEXT: </str>\<anchor>(</anchor>
// CHECK-NEXT: <str>"""
// CHECK-NEXT: inner
// CHECK-NEXT: """</str>
// CHECK-NEXT: <anchor>)</anchor><str>
// CHECK-NEXT: """</str>
"""
This is a multiline\( "interpolated" )string
\(
"""
inner
"""
)
"""
}

Expand Down
16 changes: 16 additions & 0 deletions test/IDE/unterminated_multiline.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
// RUN: %target-swift-ide-test -syntax-coloring -source-filename %s | %FileCheck %s
// RUN: %target-swift-ide-test -syntax-coloring -typecheck -source-filename %s | %FileCheck %s

// CHECK: <kw>let</kw> x = <str>"""
// CHECK-NEXT: This is an unterminated
// CHECK-NEXT: \( "multiline" )
// CHECK-NEXT: string followed by code
// CHECK-NEXT: ""
// CHECK-NEXT: func foo() {}
// CHECK-NEXT: </str>
let x = """
This is an unterminated
\( "multiline" )
string followed by code
""
func foo() {}
2 changes: 0 additions & 2 deletions test/SourceKit/DocumentStructure/mark_edit.swift.response
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@
]
}
{
key.offset: 0,
key.length: 3,
key.diagnostic_stage: source.diagnostic.stage.swift.parse,
key.substructure: [
{
Expand Down
5 changes: 5 additions & 0 deletions test/SourceKit/SyntaxMapData/Inputs/syntaxmap-edit-block-comment.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

let x = /*

*/ 2
func foo() {}
5 changes: 5 additions & 0 deletions test/SourceKit/SyntaxMapData/Inputs/syntaxmap-edit-chained-comment.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
let y = /*

*// /*

*/ 2
12 changes: 12 additions & 0 deletions test/SourceKit/SyntaxMapData/Inputs/syntaxmap-edit-multiline-string.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
let a = "value"
let x = """

\(
a
)



func foo () -> String {
return "foo"
}
19 changes: 19 additions & 0 deletions test/SourceKit/SyntaxMapData/Inputs/syntaxmap-edit-nested-token.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
/// This function does stuff
///
/// - parameter first: The first parameter
///
/// - returns: The return value
func foo(first: Int) -> String {
return ""
}

let x = "Changing this string should only affect this line"

/// This function does other stuff
///
/// - parameter first: The first parameter
///
/// - returns: The return value
func bar(first: Int) -> String {
return ""
}
4 changes: 4 additions & 0 deletions test/SourceKit/SyntaxMapData/Inputs/syntaxmap-edit-remove.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
let l = 2
l
\( 56
)
14 changes: 14 additions & 0 deletions test/SourceKit/SyntaxMapData/Inputs/syntaxmap-multiple-edits.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@

/// A comment
/// - Note: important things
let x = 42

struct Point {
let x: Int = x
let y: Int = x

func mag2() {
return x*x + y*y;
}
}

2 changes: 2 additions & 0 deletions test/SourceKit/SyntaxMapData/Inputs/syntaxmap-partial-delete.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@

let x = 421
118 changes: 118 additions & 0 deletions test/SourceKit/SyntaxMapData/syntaxmap-edit-block-comment.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
// RUN: %sourcekitd-test -req=open -print-raw-response %S/Inputs/syntaxmap-edit-block-comment.swift == -req=edit -print-raw-response %S/Inputs/syntaxmap-edit-block-comment.swift -pos=4:2 -replace=" " -length=1 == -req=edit -print-raw-response %S/Inputs/syntaxmap-edit-block-comment.swift -pos=4:2 -replace="/" -length=1 == -req=edit -print-raw-response %S/Inputs/syntaxmap-edit-block-comment.swift -pos=1:1 -replace="//" -length=2 | %sed_clean > %t.response
// RUN: %FileCheck -input-file=%t.response %s

// Initial state

// CHECK: {{^}}{
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 34,
// CHECK-NEXT: key.diagnostic_stage: source.diagnostic.stage.swift.parse,
// CHECK-NEXT: key.syntaxmap: [
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 3,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 7,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 11,
// CHECK-NEXT: key.length: 6
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.number,
// CHECK-NEXT: key.offset: 18,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 20,
// CHECK-NEXT: key.length: 4
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 25,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: }
// CHECK-NEXT: ],

// After removing the '/' from the comment close

// CHECK: {{^}}{
// CHECK-NEXT: key.offset: 3,
// CHECK-NEXT: key.length: 31,
// CHECK-NEXT: key.diagnostic_stage: source.diagnostic.stage.swift.parse,
// CHECK-NEXT: key.syntaxmap: [
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 3,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 7,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 11,
// CHECK-NEXT: key.length: 23
// CHECK-NEXT: }
// CHECK-NEXT: ],

// After adding the '/' back to the comment close

// CHECK: {{^}}{
// CHECK-NEXT: key.offset: 3,
// CHECK-NEXT: key.length: 31,
// CHECK-NEXT: key.diagnostic_stage: source.diagnostic.stage.swift.parse,
// CHECK-NEXT: key.syntaxmap: [
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 3,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 7,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 11,
// CHECK-NEXT: key.length: 6
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.number,
// CHECK-NEXT: key.offset: 18,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 20,
// CHECK-NEXT: key.length: 4
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 25,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: }
// CHECK-NEXT: ],

// after adding a single-line comment on the line above

// CHECK: {{^}}{
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 3,
// CHECK-NEXT: key.diagnostic_stage: source.diagnostic.stage.swift.parse,
// CHECK-NEXT: key.syntaxmap: [
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: }
// CHECK-NEXT: ],
100 changes: 100 additions & 0 deletions test/SourceKit/SyntaxMapData/syntaxmap-edit-chained-comment.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
// RUN: %sourcekitd-test -req=open -print-raw-response %S/Inputs/syntaxmap-edit-chained-comment.swift == -req=edit -print-raw-response %S/Inputs/syntaxmap-edit-chained-comment.swift -pos=1:9 -replace=" " -length=1 == -req=edit -print-raw-response %S/Inputs/syntaxmap-edit-chained-comment.swift -pos=1:9 -replace="/" -length=1 | %sed_clean > %t.response
// RUN: %FileCheck -input-file=%t.response %s

// Initial state

// CHECK: {{^}}{
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 25,
// CHECK-NEXT: key.diagnostic_stage: source.diagnostic.stage.swift.parse,
// CHECK-NEXT: key.syntaxmap: [
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 4,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 8,
// CHECK-NEXT: key.length: 6
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 16,
// CHECK-NEXT: key.length: 6
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.number,
// CHECK-NEXT: key.offset: 23,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: }
// CHECK-NEXT: ],


// After replacing the '/' from the comment open with ' '

// CHECK: {{^}}{
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 25,
// CHECK-NEXT: key.diagnostic_stage: source.diagnostic.stage.swift.parse,
// CHECK-NEXT: key.syntaxmap: [
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 4,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 13,
// CHECK-NEXT: key.length: 6
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.number,
// CHECK-NEXT: key.offset: 23,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: }
// CHECK-NEXT: ],

// After adding the '/' back to the comment open

// CHECK: {{^}}{
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 25,
// CHECK-NEXT: key.diagnostic_stage: source.diagnostic.stage.swift.parse,
// CHECK-NEXT: key.syntaxmap: [
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.keyword,
// CHECK-NEXT: key.offset: 0,
// CHECK-NEXT: key.length: 3
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.identifier,
// CHECK-NEXT: key.offset: 4,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 8,
// CHECK-NEXT: key.length: 6
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.comment,
// CHECK-NEXT: key.offset: 16,
// CHECK-NEXT: key.length: 6
// CHECK-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: key.kind: source.lang.swift.syntaxtype.number,
// CHECK-NEXT: key.offset: 23,
// CHECK-NEXT: key.length: 1
// CHECK-NEXT: }
// CHECK-NEXT: ],
2 changes: 1 addition & 1 deletion test/SourceKit/SyntaxMapData/syntaxmap-edit-del.swift.response
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
}
{
key.offset: 1,
key.length: 36,
key.length: 34,
key.diagnostic_stage: source.diagnostic.stage.swift.parse,
key.syntaxmap: [
{
Expand Down
2 changes: 0 additions & 2 deletions test/SourceKit/SyntaxMapData/syntaxmap-edit-del.swift.response2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,6 @@
]
}
{
key.offset: 36,
key.length: 0,
key.diagnostic_stage: source.diagnostic.stage.swift.parse,
key.syntaxmap: [
]
Expand Down
Loading