Support hermetic index store data via path remappings #4207

DavidGoldman · 2022-04-11T19:58:48Z

When indexing, -fdebug-prefix-map will be used to remap
paths so the computed index data will be hermetic. This
includes the unit files' names as well as all paths embedded
in the unit files.
When reading, a reverse mapping can be used e.g. via
indexstore_store_create_with_prefix_mapping to automatically
remap the hermetic index data to the correct local paths.

DavidGoldman · 2022-04-12T15:47:59Z

keith · 2022-04-12T16:57:59Z

swiftlang/llvm-project#4207 adds path remappings to the index store to allow using hermetic index data but remapping for local use without the need of any external tools like index-import. This adds a new parameter to IndexStoreDB.init to allow giving the prefix mapping to remap from hermetic --> local paths.

- When indexing, `-fdebug-prefix-map` will be used to remap paths so the computed index data will be hermetic. This includes the unit files' names as well as all paths embedded in the unit files. - When reading, a reverse mapping can be used e.g. via `indexstore_store_create_with_prefix_mapping` to automatically remap the hermetic index data to the correct local paths.

DavidGoldman · 2022-04-18T15:50:31Z

Updated the tests to confirm that the unit file names are deterministic by including the hash in the expected output

bnbarham · 2022-04-19T17:31:52Z

clang/include/clang/Index/IndexDataStore.h

+      std::map<std::string, std::string, std::greater<std::string>>
+          PrefixMap,


I think it'd be nice to extract this out into a separate data structure. IIUC there's no need for this to be a map other than that it's convenient (though not really) to pass around. So if we had a separate class for this it could internally just store a SmallVector of <struct with original and remap path>.

I agree, and as an input to the API I think conceptually it should be an array of string pairs. Whether then the underlying implementation uses a std::map or a llvm::StringMap for the lookups, it would be an implementation detail.

FWIW I don't think a "lookup" is ever actually performed, we only ever iterate over them all and check whether there's a prefix match (hence my suggestion of a SmallVector internally).

I was mirroring how Clang stores its prefix maps: https://cs.github.com/llvm/llvm-project/blob/0f304ef0170231b860a249f34e07f50686392253/clang/lib/CodeGen/CGDebugInfo.h#L87 and https://cs.github.com/llvm/llvm-project/blob/0f304ef0170231b860a249f34e07f50686392253/clang/include/clang/Basic/LangOptions.h#L421. I can change this to be more like how Swift does it (they're different, like you mentioned)

Ideally those wouldn't be maps either 😅. So these are checked in alphabetical order rather than the order they were given on the CLI?

Yes, that's my understanding. I think it would make sense to clean up, but not sure it's in scope for here. Thoughts?

I'd say it's worth making whatever you add usable in those cases, but we'd want to fix that upstream anyway (so no, not in scope).

clang/include/indexstore/indexstore.h

clang/lib/Index/IndexUnitWriter.cpp

clang/tools/IndexStore/IndexStore.cpp

Also increment INDEXSTORE_VERSION_MINOR

clang/include/indexstore/indexstore.h

Also avoid storing relative paths in the index data - make them absolute first, which means they can be stored using UNIT_PATH_PREFIX_WORKDIR.

DavidGoldman · 2022-04-20T16:45:22Z

Also went ahead and made sure paths are absolutized in IndexUnitWriter --> meaning they will be stored relative to UNIT_PATH_PREFIX_WORKDIR.

…create(void)

DavidGoldman · 2022-04-25T15:02:54Z

@swift-ci please test

- The PathRemapper preserves insertion order of the remappings instead of sorting

Swiftc port of swiftlang/llvm-project#4207.

Swiftc port of swiftlang/llvm-project#4207. This introduces a new flag, `-file-prefix-map` which can be used instead of the existing `-debug-prefix-map` and `-coverage-prefix-map` flags, and also remaps paths in index information currently.

Support hermetic index store data via path remappings #4207

DavidGoldman · 2022-05-16T22:38:13Z

Closing, will re-open with cherry-picks from stable.

Swiftc port of swiftlang/llvm-project#4207. This introduces a new flag, `-file-prefix-map` which can be used instead of the existing `-debug-prefix-map` and `-coverage-prefix-map` flags, and also remaps paths in index information currently.

swiftlang/llvm-project#4207 adds path remappings to the index store to allow using hermetic index data but remapping for local use without the need of any external tools like index-import. This adds a new prefixMappings parameter to IndexStoreDB.init to allow giving the prefix mapping to remap from hermetic/canonical --> local paths. If prefix mappings are given while the new IndexStore API is unavailable, indexstoredb creation will fail noting that the new API is only available in newer versions of libIndexStore. In addition, I've added a new `indexstoredb_creation_options_*` API to allow configuring options when creating indexstoredb from Swift. To support testing the remappings, TibsBuilder now supports replacing arguments containing `$SRC_DIR` and `$BUILD_DIR` with the respective directories.

swiftlang/llvm-project#4207 adds path remappings to the index store to allow using hermetic index data but remapping for local use without the need of any external tools like index-import. This adds a new prefixMappings parameter to IndexStoreDB.init to allow giving the prefix mapping to remap from hermetic/canonical --> local paths. If prefix mappings are given while the new IndexStore API is unavailable, indexstoredb creation will fail noting that the new API is only available in newer versions of libIndexStore. In addition, I've added a new `indexstoredb_creation_options_*` API to allow configuring options when creating indexstoredb from Swift. To support testing the remappings, TibsBuilder now supports replacing arguments containing `$SRC_DIR` and `$BUILD_DIR` with the respective directories. (cherry picked from commit aff5369)

DavidGoldman mentioned this pull request Apr 15, 2022

Support hermetic indexing information + introduce -file-prefix-map swiftlang/swift#42394

Merged

DavidGoldman force-pushed the hermeticindexstore branch from 0a4dd82 to 836e3f0 Compare April 18, 2022 15:49

bnbarham requested review from bnbarham and akyrtzi April 19, 2022 17:17

bnbarham reviewed Apr 19, 2022

View reviewed changes

Swap to indexstore_store_create_with_options

c50d6c2

Also increment INDEXSTORE_VERSION_MINOR

akyrtzi reviewed Apr 20, 2022

View reviewed changes

clang/include/indexstore/indexstore.h Show resolved Hide resolved

Add documentation comments to indexstore.h

3b53e89

Also avoid storing relative paths in the index data - make them absolute first, which means they can be stored using UNIT_PATH_PREFIX_WORKDIR.

indexstore_creation_options_create() --> indexstore_creation_options_…

c394fb8

…create(void)

DavidGoldman mentioned this pull request Apr 28, 2022

Support hermetic index store data via path remappings #4207 #4267

Merged

Don't make stdout absolute

a0248b7

DavidGoldman force-pushed the hermeticindexstore branch from 4581e58 to a0248b7 Compare April 28, 2022 21:22

Add PathRemapper utility class

dd5dc70

- The PathRemapper preserves insertion order of the remappings instead of sorting

DavidGoldman added a commit to DavidGoldman/swift that referenced this pull request May 11, 2022

Support hermetic indexing information

6576454

Swiftc port of swiftlang/llvm-project#4207.

bnbarham added a commit that referenced this pull request May 16, 2022

Merge pull request #4267 from DavidGoldman/stable/20211026

fcfcf6e

Support hermetic index store data via path remappings #4207

DavidGoldman closed this May 16, 2022

DavidGoldman mentioned this pull request May 17, 2022

[5.7] Support hermetic indexing information swiftlang/swift#58946

Merged

DavidGoldman mentioned this pull request Jun 3, 2022

[5.7] Support for hermetic indexing + path remappings swiftlang/indexstore-db#152

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support hermetic index store data via path remappings #4207

Support hermetic index store data via path remappings #4207

Uh oh!

DavidGoldman commented Apr 11, 2022

Uh oh!

DavidGoldman commented Apr 12, 2022

Uh oh!

keith commented Apr 12, 2022

Uh oh!

DavidGoldman commented Apr 18, 2022

Uh oh!

bnbarham Apr 19, 2022 •

edited

Loading

Uh oh!

akyrtzi Apr 19, 2022

Uh oh!

bnbarham Apr 19, 2022

Uh oh!

DavidGoldman Apr 19, 2022

Uh oh!

bnbarham Apr 19, 2022

Uh oh!

DavidGoldman Apr 19, 2022

Uh oh!

bnbarham Apr 19, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DavidGoldman commented Apr 20, 2022

Uh oh!

DavidGoldman commented Apr 25, 2022

Uh oh!

DavidGoldman commented May 16, 2022

Uh oh!

Uh oh!

		std::map<std::string, std::string, std::greater<std::string>>
		PrefixMap,

Support hermetic index store data via path remappings #4207

Support hermetic index store data via path remappings #4207

Uh oh!

Conversation

DavidGoldman commented Apr 11, 2022

Uh oh!

DavidGoldman commented Apr 12, 2022

Uh oh!

keith commented Apr 12, 2022

Uh oh!

DavidGoldman commented Apr 18, 2022

Uh oh!

bnbarham Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akyrtzi Apr 19, 2022

Choose a reason for hiding this comment

Uh oh!

bnbarham Apr 19, 2022

Choose a reason for hiding this comment

Uh oh!

DavidGoldman Apr 19, 2022

Choose a reason for hiding this comment

Uh oh!

bnbarham Apr 19, 2022

Choose a reason for hiding this comment

Uh oh!

DavidGoldman Apr 19, 2022

Choose a reason for hiding this comment

Uh oh!

bnbarham Apr 19, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DavidGoldman commented Apr 20, 2022

Uh oh!

DavidGoldman commented Apr 25, 2022

Uh oh!

DavidGoldman commented May 16, 2022

Uh oh!

Uh oh!

bnbarham Apr 19, 2022 •

edited

Loading