Skip to content

Improve JSON parser error recovery #42657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 23, 2021

Conversation

andrewbranch
Copy link
Member

Does two loosely related things:

  1. After we’ve finished parsing one of the top-level expression kinds that are valid in JSON, we check to see if we’re on the EndOfFileToken yet. If we’re not, we loop and parse another top-level expression, issuing an "unexpected token" error. At the end, if we have more than one top-level expression, we make them the elements of an ArrayLiteralExpression, and that becomes the single top-level expression of the file. This is to make sure that in a case like {} identifier, identifier actually belongs to a node, instead of being a token totally unaccounted for in the parse tree (which triggers a debug failure if the language service finds out about it). This fixes the remaining repro of Debug Failure. Did not expect SourceFile to have an Identifier in its trivia at addSyntheticNodes #39854.
  2. Fixes error reporting in tsconfig parsing when the top-level expression is not an object (Assert while trying to report error about a bad tsconfig file #37850).

Neither of these bugs are very important, and parser changes are inherently somewhat risky, so this should wait until 4.3.

Fixes #39854
Fixes #37850

@andrewbranch andrewbranch requested a review from sandersn February 5, 2021 00:31
@andrewbranch andrewbranch requested a review from rbuckton February 5, 2021 00:31
@typescript-bot typescript-bot added For Backlog Bug PRs that fix a backlog bug For Milestone Bug PRs that fix a bug with a specific milestone labels Feb 5, 2021
@@ -1790,7 +1790,26 @@ namespace ts {
return returnValue ? {} : undefined;
}

return convertPropertyValueToJson(sourceFile.statements[0].expression, knownRootOptions);
const rootExpression: Expression = sourceFile.statements[0].expression;
if (rootExpression.kind !== SyntaxKind.ObjectLiteralExpression) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is common functionality for json modules as well as tsconfig.
The way tsconfig ensures its object is by passing knownRootOptions? This should not break module which has array.

Copy link
Member Author

@andrewbranch andrewbranch Feb 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does break, because there’s no name for the root of the object itself. We try to create an error message that says Compiler option '{0}' requires a value of type 'object' but there’s nothing to substitute for {0}, and the error doesn’t make much sense calling the root object itself a “compiler option.” So I think there should be dedicated logic for the root somewhere, because there should be a dedicated error message.

This is common functionality for json modules as well as tsconfig.

Hah, ok, the failing test shows this. Very surprising since this is in commandLineParser.ts, but I’ll find another place for the tsconfig root error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hah, ok, the failing test shows this. Very surprising since this is in commandLineParser.ts

that’s just from fact that we wrote this for tsconfig first and then added support for json modules

@DanielRosenwasser
Copy link
Member

The approach seems reasonable - the only thing I'm a little concerned about is whether there are any places in the compiler/LS where we try to ask for brackets on a node (since they won't exist here)

@andrewbranch
Copy link
Member Author

the only thing I'm a little concerned about is whether there are any places in the compiler/LS where we try to ask for brackets on a node (since they won't exist here)

We generally don’t do any interactive stuff with JSON files in the language service since editors (at least VS Code) use another set of providers for JSON. The only way we discovered this was through API use and the playground. But if you have a suggestion of something else to test, I’ll add it.

@@ -1782,15 +1806,16 @@ namespace ts {
/*@internal*/
export function convertToObjectWorker(
sourceFile: JsonSourceFile,
rootExpression: Expression | undefined,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this potentially be a polymorphism deopt?

@andrewbranch
Copy link
Member Author

@typescript-bot perf test this faster

I don’t know if any of our perf tests resolve JSON modules though

@typescript-bot
Copy link
Collaborator

typescript-bot commented Feb 5, 2021

Heya @andrewbranch, I've started to run the abridged perf test suite on this PR at b6a2f39. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@andrewbranch
The results of the perf run you requested are in!

Here they are:

Comparison Report - master..42657

Metric master 42657 Delta Best Worst
Angular - node (v14.15.1, x64)
Memory used 321,871k (± 0.01%) 321,890k (± 0.01%) +18k (+ 0.01%) 321,840k 321,952k
Parse Time 1.92s (± 0.94%) 1.91s (± 0.89%) -0.01s (- 0.47%) 1.89s 1.96s
Bind Time 0.85s (± 0.68%) 0.84s (± 0.59%) -0.00s (- 0.59%) 0.84s 0.86s
Check Time 4.85s (± 0.39%) 4.84s (± 0.53%) -0.01s (- 0.21%) 4.78s 4.92s
Emit Time 5.51s (± 0.77%) 5.47s (± 0.54%) -0.04s (- 0.71%) 5.41s 5.55s
Total Time 13.13s (± 0.50%) 13.07s (± 0.38%) -0.06s (- 0.46%) 13.00s 13.22s
Compiler-Unions - node (v14.15.1, x64)
Memory used 200,800k (± 0.65%) 200,310k (± 0.54%) -490k (- 0.24%) 199,013k 202,914k
Parse Time 0.80s (± 0.81%) 0.80s (± 0.46%) -0.00s (- 0.25%) 0.79s 0.80s
Bind Time 0.53s (± 0.00%) 0.53s (± 0.56%) -0.00s (- 0.38%) 0.52s 0.53s
Check Time 9.72s (± 0.66%) 9.70s (± 0.38%) -0.02s (- 0.15%) 9.63s 9.79s
Emit Time 2.35s (± 1.32%) 2.37s (± 1.72%) +0.02s (+ 0.81%) 2.30s 2.45s
Total Time 13.39s (± 0.62%) 13.40s (± 0.47%) +0.01s (+ 0.04%) 13.25s 13.55s
Monaco - node (v14.15.1, x64)
Memory used 336,838k (± 0.01%) 336,829k (± 0.01%) -9k (- 0.00%) 336,786k 336,900k
Parse Time 1.57s (± 1.04%) 1.56s (± 0.57%) -0.02s (- 1.02%) 1.54s 1.58s
Bind Time 0.73s (± 0.64%) 0.73s (± 0.45%) 0.00s ( 0.00%) 0.72s 0.74s
Check Time 4.85s (± 0.38%) 4.84s (± 0.41%) -0.01s (- 0.12%) 4.79s 4.88s
Emit Time 2.91s (± 0.46%) 2.90s (± 0.66%) -0.01s (- 0.45%) 2.86s 2.95s
Total Time 10.07s (± 0.25%) 10.03s (± 0.33%) -0.04s (- 0.37%) 9.96s 10.12s
TFS - node (v14.15.1, x64)
Memory used 291,528k (± 0.01%) 291,562k (± 0.01%) +34k (+ 0.01%) 291,510k 291,612k
Parse Time 1.25s (± 0.92%) 1.26s (± 0.78%) +0.01s (+ 0.64%) 1.24s 1.29s
Bind Time 0.69s (± 0.68%) 0.69s (± 0.49%) +0.00s (+ 0.43%) 0.69s 0.70s
Check Time 4.50s (± 0.77%) 4.51s (± 0.27%) +0.01s (+ 0.18%) 4.48s 4.54s
Emit Time 3.05s (± 0.79%) 3.04s (± 0.58%) -0.01s (- 0.26%) 2.99s 3.08s
Total Time 9.49s (± 0.60%) 9.50s (± 0.24%) +0.01s (+ 0.12%) 9.45s 9.56s
material-ui - node (v14.15.1, x64)
Memory used 471,549k (± 0.00%) 471,569k (± 0.01%) +20k (+ 0.00%) 471,492k 471,635k
Parse Time 2.06s (± 0.24%) 2.05s (± 0.61%) -0.02s (- 0.87%) 2.01s 2.06s
Bind Time 0.70s (± 0.52%) 0.70s (± 0.71%) -0.01s (- 1.14%) 0.69s 0.71s
Check Time 12.55s (± 0.59%) 12.54s (± 0.51%) -0.01s (- 0.09%) 12.41s 12.68s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 15.32s (± 0.51%) 15.28s (± 0.45%) -0.04s (- 0.24%) 15.16s 15.44s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-198-generic
Architecturex64
Available Memory16 GB
Available Memory8 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v14.15.1, x64)
Benchmark Name Iterations
Current 42657 10
Baseline master 10

@@ -1767,11 +1767,35 @@ namespace ts {
onSetUnknownOptionKeyValueInRoot(key: string, keyNode: PropertyName, value: CompilerOptionsValue, valueNode: Expression): void;
}

function convertConfigFileToObject(sourceFile: JsonSourceFile, errors: Push<Diagnostic>, optionsIterator: JsonConversionNotifier): any {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You want to use this in parseConfigFileTextToJson as well since thats tsconfig ?

@armanio123 armanio123 mentioned this pull request Feb 5, 2021
@andrewbranch andrewbranch merged commit 2a01f92 into microsoft:master Feb 23, 2021
@andrewbranch andrewbranch deleted the bug/39854-2 branch February 23, 2021 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Backlog Bug PRs that fix a backlog bug For Milestone Bug PRs that fix a bug with a specific milestone
Projects
Archived in project
5 participants