Skip to content

Commit fc769ee

Browse files
bitjammerbitjammer
committed
[Migrator] Add a README for lib/Migrator
This describes the purpose and internals of lib/Migrator. NFC.
1 parent 6915ed5 commit fc769ee

File tree

1 file changed

+251
-0
lines changed

1 file changed

+251
-0
lines changed

‎lib/Migrator/README.md

Lines changed: 251 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,251 @@
1+
# Swift Migrator
2+
3+
This library implements functionality for the Swift 4 Migrator.
4+
5+
## Overview
6+
7+
The Migrator was rewritten from the ground up for Swift 4 with a few major differences:
8+
9+
- It's not a separate tool but integrated directly into the compiler binary
10+
- It understands Swift 3 and Swift 4 code equally
11+
- Can migrate individual targets in Xcode
12+
- A pipeline architecture with explicit immutable state changes
13+
14+
The Migrator runs during a normal frontend invocation, with the *primary file* being the target for migration, resulting in the following additional outputs:
15+
16+
1. The *replacement map* file (a.k.a. *remap* file)
17+
`-emit-remap-file-path <path>`
18+
2. The migrated file - optional, primarily for testing.
19+
`-emit-migrated-file-path <path>`
20+
3. The migration states - optional, primarily for testing.
21+
`-dump-migration-states-dir <dir>`
22+
23+
The majority of changes suggested by the Migrator are driven by:
24+
25+
- API changes from the Xcode 8.3\* SDKs and the Xcode 9 SDKs
26+
- Fix-its suggested by the compiler
27+
28+
There are a few passes that walk the AST and perform textual edits
29+
for some cases, discussed below.
30+
31+
## The Migrator Pipeline
32+
33+
The migrator has the following *passes*, each of which takes an input source text and produces and output source text, collecting a sequence of *states*, which includes the input and output text from the pass.
34+
35+
At the start of the normal frontend invocation, the compiler parses and type-checks the primary input, resulting in a type-checked AST, which is handed to the Migrator if one of the above flags were passed. For this initial step, the Migrator uses whatever Swift language version that was passed to the frontend.
36+
37+
Here are the passes:
38+
39+
1. Pre-fix-it Pass
40+
41+
If the compiler wasn't able to successfully type-check the primary input source file,
42+
the Migrator makes a best effort at applying any fix-its the compiler suggests and trying
43+
again, *up to two times*. If it still can't successfully type-check and get an AST, the
44+
pipeline stops.
45+
46+
> See lib/Migrator/Migrator.cpp: `Migrator::repeatFixitMigrations`
47+
48+
2. AST Passes
49+
50+
If the Pre-fix-it Pass was successful, or skipped because it was unnecessary, the
51+
*AST Passes* run. These include:
52+
53+
- API Diff Pass
54+
55+
This pass injests an *API Diff Data File*, a JSON file describing API changes
56+
from the previous SDK, and looks for API references, performing textual edits
57+
to update code for things like referenced type and argument names or optionality
58+
changes. This is where the majority of changes come from in the Migrator.
59+
60+
For a list of the different kinds of entries, see `include/swift/IDE/DigesterEnums.def`
61+
and the actual JSON data files in `lib/Migrator`.
62+
63+
There are also a few "special case" migrations implemented here, which
64+
were different from the typical API diff changes but also rare enough. These
65+
are mainly declared in `lib/Migrator/overlay.json`, implemented in `handleSpecialCases`.
66+
Some examples of these include:
67+
68+
- Migrating `Double.abs(0.0)` to `Swift.abs(0.0)`
69+
- A few NSOpenGL APIs
70+
- Standard Library `UIntMax` and `IntMax` APIs moving to reference `UInt64` and `Int64`
71+
72+
> See:
73+
> - lib/Migrator/APIDiffMigratorPass.cpp
74+
> - include/swift/IDE/DigesterEnums.def
75+
> - lib/IDE/APIDigesterData.cpp
76+
> - lib/Migrator/ios.json
77+
> - lib/Migrator/macos.json
78+
> - lib/Migrator/tvos.json
79+
> - lib/Migrator/watchos.json
80+
81+
- Tuple Splat Migrator Pass
82+
83+
This implements a few convenience transformations to ease the transition
84+
for [SE-0110: Distinguish between single-tuple and multiple-argument function types](https://github.com/apple/swift-evolution/blob/master/proposals/0110-distingish-single-tuple-arg.md).
85+
86+
In particular, this pass adds new variable bindings in closure expressions
87+
to destructure what are now are a single argument, a tuple. Prior to SE-0110,
88+
a closure's argument list may have been automatically matched up in Swift 3.
89+
90+
> See lib/Migrator/RewriteBufferEditsReceiver.cpp
91+
92+
- type(of:) Migrator Pass
93+
94+
This is a small convenience pass to account for `Swift.type(of:)` now being
95+
resolved by overload resolution. It was handled specially in Swift 3.
96+
97+
> See lib/Migrator/Migrator.cpp: `Migrator::performSyntacticPasses`
98+
99+
3. Post-fix-it Pass
100+
101+
Finally, the post-fix-it pass, like the pre-fix-it pass, ingests fix-its suggested
102+
by the compiler, explicitly set to Swift 4 Mode. This essentially handles automating
103+
acceptance of fix-its to convert the source file to Swift 4 in terms of the language
104+
itself instead of the APIs.
105+
106+
This pass is run up to seven times, a number tweaked based on historical observations.
107+
The reason the pass is run multiple times is that applying fix-its may reveal more
108+
problems to the type-checker, which can then suggest more fix-its, and so on.
109+
110+
Of note, this includes migration to *Swift 4 @objc Inference*. This is a nuanced topic
111+
with implications for your binary size and Objective-C interoperability. There is a
112+
link to discussion on this topic at the bottom of this README.
113+
114+
> See lib/Migrator/Migrator.cpp: `Migrator::repeatFixitMigrations`
115+
116+
As each of these passes run, a `MigrationState` is pushed onto the Migrator, describing
117+
the input and output text explicitly, and which pass produced the transformation.
118+
119+
Finally, at the end of the pipeline, the outputs are emitted. If `-emit-migrated-file-path` was given, the `OutputText` of the final `MigrationState` is written to that file path. If `-dump-migration-states-dir` was specified, the input and output text of each state is dumped into that directory. Finally, if `-emit-remap-file-path` was specified, a file describing the differences between the first and last `MigrationState`'s `OutputText` is emitted.
120+
121+
> See lib/Migrator/Migrator.cpp: `swift::migrator::updateCodeAndEmitRemap`
122+
123+
Other controls for the frontend:
124+
125+
- `-disable-migrator-fixits` - skips the fix-it passes during the migration pipeline.
126+
- `-migrate-keep-objc-visibility` - add `@objc` to declarations that were implicitly visible
127+
to Objective-C in Swift 3.
128+
- `-api-diff-data-file` - override the API diff JSON file.
129+
130+
> See include/swift/Migrator/MigratorOptions.h
131+
132+
### Fix-it Filter
133+
134+
For the pre- and post-fix-it passes, there are two basic rules for which fix-its the Migrator will take:
135+
136+
1. Fix-its attached to *error* diagnosics are taken by default and are opt-out.
137+
2. Fix-its attached to *warning* or *note* diagnostics are not taken by default and so are opt-in.
138+
139+
For the opt-out and opt-in cases, these are filtered in the `FixitFilter`, essentially just a small collection of Swift's compiler diagnostic IDs.
140+
141+
> See include/swift/Migrator/FixitFilter.h
142+
143+
### Applying Fix-its
144+
145+
The `FixitApplyDiagnosticConsumer` delegate class subscribes to fix-its emitted by the type-checker and decides whether to take the fix-it based on the following:
146+
147+
- The fix-it should be for the current primary file of the frontend invocation to prevent
148+
multiple fix-its from being emitted for the same file.
149+
- The fix-it should be permitted by the `FixitFilter`.
150+
- The fix-it doesn't suggest a duplicate change to the source file.
151+
152+
In order to produce a `MigrationState` for this pass, fix-its are applied immediately to a running *RewriteBuffer*, which is supplied by Clang. At the end of a fix-it pass, the resulting text is extracted from the `RewriteBufferEditsReceiver`.
153+
154+
> See:
155+
> - include/swift/Migrator/FixitApplyDiagnosticConsumer.h
156+
> - lib/Migrator/FixitApplyDiagnosticConsumer.cpp
157+
> - include/swift/Migrator/RewriteBufferEditsReceiver.h
158+
> - lib/Migrator/RewriteBufferEditsReceiver.cpp
159+
160+
## Remap File Format
161+
162+
This is a file describing textual replacements the input file, a JSON array-of-objects. Xcode ingests these files to generate the diff preview you see in the Migration Assistant.
163+
164+
- `file`: String
165+
166+
The absolute path to the input file.
167+
- `offset`: Number
168+
169+
The absolute offset into the input file.
170+
- `remove`: Number (Assumed 0 if missing)
171+
172+
Remove this many bytes at the given `offset`.
173+
- `text`: String (Assumed empty if missing)
174+
175+
Insert this text at the given `offset`.
176+
177+
These entries can describe *removals*, *insertions*, or *replacements*.
178+
179+
### Removals
180+
181+
For removals, you specify `file`, `offset`, and `remove`.
182+
183+
```json
184+
{
185+
"file": "/path/to/my/file.swift",
186+
"offset": 503,
187+
"remove": 10
188+
}
189+
```
190+
191+
This says to remove 10 bytes at offset 503 in the original source file. You can specify an empty string for the text.
192+
193+
### Insertions
194+
195+
For insertions, you specify `file`, `offset`, and `text`. You can specify `remove` to be 0.
196+
197+
```json
198+
{
199+
"file": "/path/to/my/file.swift",
200+
"offset": 61,
201+
"text": ".foo()"
202+
}
203+
```
204+
205+
This says to insert `.foo()` at offset 61 in the source file.
206+
207+
### Replacements
208+
209+
For replacements, you specify all four keys.
210+
211+
```json
212+
{
213+
"file": "/path/to/my/file.swift",
214+
"offset": 61,
215+
"remove": 3,
216+
"text": "bar"
217+
}
218+
```
219+
220+
This says to replace the 3 bytes starting at offset 61 with `foo` in the source file.
221+
222+
## Other Internals
223+
224+
There are two main other pieces of the Migrator's implementation, *diffing* and *editing*.
225+
226+
### Diffing
227+
228+
For diffing, we pulled in an STL port of Google's *diff-match-patch* library to perform the final diff of the start and end `MigrationState`'s text. This is a fairly standard implementation of the Myers Difference Algorithm (see *An O(ND) Difference Algorithm and Its Variations* by Eugene W. Myers).
229+
230+
> See include/swift/Basic/Diff.h
231+
232+
### Editing
233+
234+
For textual edits during the AST passes, we adapted libEdit's `Commit` and `EditedSource` functionality that was used in the ARC/Objective-C modernizer. Essentially a wrapper that converts Swift's `SourceLoc`, `SourceRange`, and `CharSourceRange` structures into Clang's,
235+
this implements basic operations like insertions, removals, and replacements. Originally, we planned on using lib/Syntax for these transformations but there wasn't enough time and we found that the edits we needed for the Migrator were straightforward enough.
236+
237+
> See:
238+
> - include/swift/Migrator/EditorAdaptor.h
239+
> - lib/Migrator/EditorAdaptor.h
240+
241+
### Migration States
242+
243+
This is an immutable container explicitly describing changes in state as the Migrator runs, which is not only for safety but also for debuggability. This clarifies which pass was responsible for a set of changes in the pipeline because there can be a lot of changes, sometimes conflicting or redundant, between the API diffs and compiler fix-its.
244+
245+
> See include/swift/Migrator/MigrationState.h
246+
247+
## More Information
248+
249+
[Migrating to Swift 4 on swift.org](https://swift.org/migration-guide/)
250+
251+
[Migrate to Swift 4 @objc inference on help.apple.com](https://help.apple.com/xcode/mac/current/#/deve838b19a1)

0 commit comments

Comments
 (0)