Skip to content

Fixup Bidirectional Map #662

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 14, 2021
Merged

Fixup Bidirectional Map #662

merged 1 commit into from
May 14, 2021

Conversation

CodaFi
Copy link
Contributor

@CodaFi CodaFi commented May 14, 2021

The setters here were both implemented in such a way that the map would
lose the invariant that it was truly bidirectional. In particular, one
must ensure that previous mappings from keys to values were struck from
the bidirectional map before emplacing a new mapping, else the mirror
image of the mapping would retain extra entries.

In addition, document the misuse of this data structure in the driver.
The mapping from swiftdeps to swift files is a not a bijection. There
are three sources of data that can all give conflicting mappings from
swiftdeps files to swift files in particular:

  1. The output file map
  2. The priors
  3. Swiftdeps files after the frontends have run

This means we can have at most three conflicting maps from some
swiftdeps files to some swift file. Currently, the driver overwrites
this mapping and the last writer wins. This is the root cause of
a family of crashers where the output file map's contents disagreed from
run to run.

When a project workspace is set up with multiple files with the same
base name but differing extensions, XCBuild will disambiguate any
products by appending a GUID derived from the structure of the project.
That means, if foo.swift and foo.c live in the same target's inputs,
then the output file map will have an entry

foo-<GUID_1>.swiftdeps

This GUID is derived from the structure of the project itself, and
changes with that structure. So, for example, one can delete and re-add
a reference to a swift file and the output file map entry for that
swiftdeps file will regenerate

foo-<GUID_2>.swiftdeps

This means that the priors' swiftdeps files now disagree in content with
the output file map, and the output file map's entries currently win.
But, only partially. Because the bidirectional map was broken, it was
able to successfully answer queries about the extra injective mapping
between swift files and swiftdeps files. It was, however, not equipped
to handle the other direction, and so it used to crash. We have since
changed this to redo the incremental build to flush this state.

There is a remaining lurking bug here that involves an extra step.
Suppose you build in a normal way with foo.swift/foo.c setup. Then you
rebuild and cancel it after a frontend has emitted a new swiftdeps file,
finally, you crash the driver by exploiting this bug. The driver is now
in a state where swiftdeps disagree with the priors which disagree with
the output file map.

All this to say, we should take the earliest opportunity to remove
BidirectionalMap and its uses. But this commit is not the one to make
such a structural change to the driver.

rdar://77804636

The setters here were both implemented in such a way that the map would
lose the invariant that it was truly bidirectional. In particular, one
must ensure that previous mappings from keys to values were struck from
the bidirectional map before emplacing a new mapping, else the mirror
image of the mapping would retain extra entries.

In addition, document the misuse of this data structure in the driver.
The mapping from swiftdeps to swift files is a not a bijection. There
are three sources of data that can all give conflicting mappings from
swiftdeps files to swift files in particular:

1) The output file map
2) The priors
3) Swiftdeps files after the frontends have run

This means we can have at most three conflicting maps from some
swiftdeps files to some swift file. Currently, the driver overwrites
this mapping and the last writer wins. This is the root cause of
a family of crashers where the output file map's contents disagreed from
run to run.

When a project workspace is set up with multiple files with the same
base name but differing extensions, XCBuild will disambiguate any
products by appending a GUID derived from the structure of the project.
That means, if foo.swift and foo.c live in the same target's inputs,
then the output file map will have an entry

foo-<GUID_1>.swiftdeps

This GUID is derived from the structure of the project itself, and
changes with that structure. So, for example, one can delete and re-add
a reference to a swift file and the output file map entry for that
swiftdeps file will regenerate

foo-<GUID_2>.swiftdeps

This means that the priors' swiftdeps files now disagree in content with
the output file map, and the output file map's entries currently win.
But, only partially. Because the bidirectional map was broken, it was
able to successfully answer queries about the extra injective mapping
between swift files and swiftdeps files. It was, however, not equipped
to handle the other direction, and so it used to crash. We have since
changed this to redo the incremental build to flush this state.

There is a remaining lurking bug here that involves an extra step.
Suppose you build in a normal way with foo.swift/foo.c setup. Then you
rebuild and cancel it after a frontend has emitted a new swiftdeps file,
finally, you crash the driver by exploiting this bug. The driver is now
in a state where swiftdeps disagree with the priors which disagree with
the output file map.

All this to say, we should take the earliest opportunity to remove
BidirectionalMap and its uses. But this commit is not the one to make
such a structural change to the driver.

rdar://77804636
@CodaFi CodaFi requested a review from artemcm May 14, 2021 02:17
@CodaFi
Copy link
Contributor Author

CodaFi commented May 14, 2021

@swift-ci test

// turn over when e.g. entries in the output file map change. This should be
// replaced by a multi-map from swift files to dependency sources,
// and a regular map from dependency sources to swift files -
// since that direction really is one-to-one.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn’t that technically many to one? A normal dictionary still works fine in that case, so not sure the distinction matters.

@CodaFi CodaFi merged commit bcc9a3b into swiftlang:main May 14, 2021
@CodaFi CodaFi deleted the mapping-out branch May 14, 2021 06:05
CodaFi added a commit to CodaFi/swift-driver that referenced this pull request May 21, 2021
This reverts commit 5e1f8ec.

We no longer need the simulated failure path since we can reproduce this
as of swiftlang#662
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants