Frontend: set up output file .swiftsourceinfo #27277

nkcsgexi · 2019-09-20T21:00:30Z

This patch will focus on teaching driver and frontend to emit this file.
The actual content and de-serialization parts will come later.

More details: https://forums.swift.org/t/proposal-emitting-source-information-file-during-compilation/28794

nkcsgexi · 2019-09-20T23:18:40Z

@swift-ci Please Build Toolchain macOS Platform

nkcsgexi · 2019-09-20T23:18:48Z

@swift-ci Please Build Toolchain Linux Platform

swift-ci · 2019-09-21T00:08:02Z

Linux Toolchain (Ubuntu 16.04)
Download Toolchain
Git Sha - 7878e8657efe96c9831d7546f15cac4d0081b4ef

Install command
tar zxf swift-PR-27277-277-ubuntu16.04.tar.gz
More info

swift-ci · 2019-09-21T00:44:41Z

macOS Toolchain
Download Toolchain
Git Sha - 7878e8657efe96c9831d7546f15cac4d0081b4ef

Install command
tar -zxf swift-PR-27277-373-osx.tar.gz --directory ~/

nkcsgexi · 2019-09-23T19:37:31Z

@swift-ci please smoke test

nkcsgexi · 2019-09-23T19:38:02Z

@swift-ci please smoke test

nkcsgexi · 2019-09-23T21:15:47Z

@swift-ci Please Build Toolchain macOS Platform

nkcsgexi · 2019-09-23T21:21:45Z

@swift-ci Please Build Toolchain macOS Platform

nkcsgexi · 2019-09-23T22:21:52Z

@swift-ci please smoke test

nkcsgexi · 2019-09-23T22:23:22Z

@swift-ci please smoke test

jrose-apple

Looks pretty good, also not happy how much it took. cc @davidungar for later thought on how to streamline some of this.

jrose-apple · 2019-09-24T16:35:38Z

lib/Driver/Driver.cpp

@@ -2786,6 +2789,41 @@ void Driver::chooseSwiftModuleOutputPath(Compilation &C,
  }
 }

+void Driver::choosePrivateOutputFilePath(Compilation &C,


Nitpick: can you put "module" in the name of this function somewhere? I don't know if all private outputs belong in the module folder.

Alternately, is it possible to combine this with the swiftdoc helper and just have a flag to say "Private" or not? Duplicated code is code that gets out of sync.

Sounds good! Will do.

jrose-apple · 2019-09-24T16:36:14Z

lib/Driver/Driver.cpp

+    llvm::sys::path::append(Path, "Private");
+    // If the build system has created a Private dir for us to include the file, use it.
+    if (!llvm::sys::fs::exists(Path)) {
+      llvm::sys::path::remove_filename(Path);


Ah, clever. And I guess we can look in both places?

yeah, the source info loader should look into both places and prefer the Private one.

jrose-apple · 2019-09-24T16:37:06Z

lib/Driver/ToolChains.cpp

@@ -907,6 +912,8 @@ ToolChain::constructInvocation(const MergeModuleJobAction &job,

  addOutputsOfType(Arguments, context.Output, context.Args,
                   file_types::TY_SwiftModuleDocFile, "-emit-module-doc-path");
+  addOutputsOfType(Arguments, context.Output, context.Args,
+                   file_types::TY_SwiftSourceInfoFile, "-emit-module-source-info-path");


Nitpick: 80 columns.

jrose-apple · 2019-09-24T16:37:46Z

lib/Frontend/FrontendInputsAndOutputs.cpp

@@ -435,6 +435,12 @@ bool FrontendInputsAndOutputs::hasModuleDocOutputPath() const {
        return outs.ModuleDocOutputPath;
      });
 }
+bool FrontendInputsAndOutputs::hasModuleSourceInfoOutputPath() const {
+  return hasSupplementaryOutputPath(
+  [](const SupplementaryOutputPaths &outs) -> const std::string & {


Nitpick: funny indentation here.

jrose-apple · 2019-09-24T16:41:57Z

lib/Serialization/Serialization.cpp

+    SharedTimer timer("Serialization, swiftsourceinfo, to buffer");
+    llvm::SmallString<1024> buf;
+    llvm::raw_svector_ostream stream(buf);
+    writeSourceInfoToStream(stream, DC);


@harlanhaskins This always using a buffer is unfortunate; ideally we'd only use it when we're going to keep it in memory.

jrose-apple · 2019-09-24T16:44:36Z

lib/Serialization/SerializeDoc.cpp

@@ -502,3 +502,7 @@ void serialization::writeDocToStream(raw_ostream &os, ModuleOrSourceFile DC,

  S.writeToStream(os);
 }
+
+void serialization::writeSourceInfoToStream(raw_ostream &os, ModuleOrSourceFile DC) {
+  os << "Some stuff";


Excellent source info. :-)

jrose-apple · 2019-09-24T16:45:21Z

test/Driver/sourceinfo_file.swift

+// RUN: mkdir -p %t/build/Private
+// RUN: %swiftc_driver -driver-print-jobs -emit-module %s -emit-module-path %t/build/sourceinfo_file.swiftmodule -module-name sourceinfo_file | %FileCheck %s -check-prefix CHECK-PRIVATE
+
+// CHECK-PRIVATE: Private/sourceinfo_file.swiftsourceinfo


Can you check that this is build/Private/sourceinfo_file.swiftsourceinfo?

nkcsgexi · 2019-09-24T18:43:14Z

@swift-ci please smoke test

nkcsgexi · 2019-09-24T18:43:39Z

@swift-ci Please Build Toolchain macOS Platform

nkcsgexi · 2019-09-24T18:43:48Z

@swift-ci Please Build Toolchain Linux Platform

This patch will focus on teaching driver and frontend to emit this file. The actual content and de-serialization parts will come later. More details: https://forums.swift.org/t/proposal-emitting-source-information-file-during-compilation/28794

nkcsgexi · 2019-09-24T20:53:05Z

@swift-ci Please Build Toolchain macOS Platform

nkcsgexi · 2019-09-24T21:48:59Z

@swift-ci please test

nkcsgexi · 2019-09-24T21:51:15Z

@swift-ci please test

nkcsgexi · 2019-09-24T22:01:36Z

@swift-ci Please test OS X platform

swift-ci · 2019-09-24T22:16:06Z

Build failed
Swift Test OS X Platform
Git Sha - 7878e8657efe96c9831d7546f15cac4d0081b4ef

davidungar · 2019-09-24T23:56:39Z

LGTM

nkcsgexi · 2019-09-25T00:21:19Z

@swift-ci please smoke test

nkcsgexi · 2019-09-25T01:42:36Z

@swift-ci Please smoke test

compnerd · 2019-09-26T23:09:21Z

@nkcsgexi I think that this may have regressed the windows build: https://dev.azure.com/compnerd/windows-swift/_build/results?buildId=9686&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=2fdae087-0aee-589c-2ab0-2b8fc539478d&l=184

Account for different path separators.

tests: fix test failure on Windows after #27277

Account for different path separators.

nkcsgexi force-pushed the add-source-info-to-driver branch 2 times, most recently from 7ec2641 to 7878e86 Compare September 20, 2019 23:10

nkcsgexi force-pushed the add-source-info-to-driver branch from 8d24ddf to 8a237ef Compare September 23, 2019 21:21

nkcsgexi changed the title ~~[WIP] Frontend: set up output file .swiftsourceinfo~~ Frontend: set up output file .swiftsourceinfo Sep 23, 2019

nkcsgexi requested review from harlanhaskins and jrose-apple September 23, 2019 23:32

jrose-apple requested a review from beccadax September 24, 2019 16:28

jrose-apple reviewed Sep 24, 2019

View reviewed changes

nkcsgexi added 6 commits September 24, 2019 13:52

test: update driver tests

6ec4fc8

cmake: exclude Private dir from installation

4cd46d9

test: add a test about .swiftsourceinfo file using Private directory

5759fe6

Driver: expose -emit-module-source-info-path as a driver option

ec9cf33

Address Jordan's code review comments

7013725

nkcsgexi force-pushed the add-source-info-to-driver branch from a1d0f60 to 7013725 Compare September 24, 2019 20:52

jrose-apple approved these changes Sep 24, 2019

View reviewed changes

nkcsgexi merged commit adb0e6e into swiftlang:master Sep 25, 2019

compnerd added a commit to compnerd/apple-swift that referenced this pull request Sep 26, 2019

tests: fix test failure on Windows after swiftlang#27277

ed958a7

Account for different path separators.

compnerd added a commit that referenced this pull request Sep 27, 2019

Merge pull request #27394 from compnerd/path-separator

40016b5

tests: fix test failure on Windows after #27277

roop pushed a commit to roop/swift that referenced this pull request Nov 5, 2019

tests: fix test failure on Windows after swiftlang#27277

33242da

Account for different path separators.

Frontend: set up output file .swiftsourceinfo #27277

Frontend: set up output file .swiftsourceinfo #27277

Uh oh!

Conversation

nkcsgexi commented Sep 20, 2019

Uh oh!

nkcsgexi commented Sep 20, 2019

Uh oh!

nkcsgexi commented Sep 20, 2019

Uh oh!

swift-ci commented Sep 21, 2019

Uh oh!

swift-ci commented Sep 21, 2019

Uh oh!

nkcsgexi commented Sep 23, 2019

Uh oh!

nkcsgexi commented Sep 23, 2019

Uh oh!

nkcsgexi commented Sep 23, 2019

Uh oh!

nkcsgexi commented Sep 23, 2019

Uh oh!

nkcsgexi commented Sep 23, 2019

Uh oh!

nkcsgexi commented Sep 23, 2019

Uh oh!

jrose-apple left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nkcsgexi commented Sep 24, 2019

Uh oh!

nkcsgexi commented Sep 24, 2019

Uh oh!

nkcsgexi commented Sep 24, 2019

Uh oh!

nkcsgexi commented Sep 24, 2019

Uh oh!

nkcsgexi commented Sep 24, 2019

Uh oh!

nkcsgexi commented Sep 24, 2019

Uh oh!

nkcsgexi commented Sep 24, 2019

Uh oh!

swift-ci commented Sep 24, 2019

Uh oh!

davidungar commented Sep 24, 2019

Uh oh!

nkcsgexi commented Sep 25, 2019

Uh oh!

nkcsgexi commented Sep 25, 2019

Uh oh!

compnerd commented Sep 26, 2019

Uh oh!

Uh oh!