RFC: Remove lib_InternalSwiftSyntaxParser as a link-time dependency

[owenv]

I'd appreciate any thoughts & feedback anyone has on this!

This PR removes the link-time dependency on a compatible lib_InternalSwiftSyntaxParser. Instead, it uses dlopen & dlsym at runtime, following the approach SwiftDriver uses to load the dependency scanning library. The motivation here is to be able to add a SwiftSyntax dependency to SwiftPM to implement SE-301's package manifest editing commands. Per the discussion here, the SwiftPM team wants to be able to build the package syntax support library (which uses SwiftSyntax) when there's no compatible parser library available, so that breaking changes can be detected before testing on CI. This change should make that possible, and a SwiftPM built in a local dev environment without a compatible library will report a library mismatch error.

The main downside of this approach is that it requires keeping the parser library's header in-sync with the one in the main swift repository, and I had to copy over a bit of TSCUtility code for cross-platform dlopen.

[owenv]

@swift-ci please test

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 0d8f1e1

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 0d8f1e1

[owenv]

@swift-ci please test

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 0d8f1e1

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 0d8f1e1

[owenv]

Hmm, I'm unable tor reproduce the source kit stress tester failure locally, I assume there's some kind of rpath problem though

[owenv]

@swift-ci please test

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 0653ca0

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 0653ca0

[ahoppen]

Hi @owenv,

To clarify what you are trying to achieve here: You want to remove the link-dependency from SwiftSyntax to lib_InternalSwiftSyntaxParser.dylib so that you can build SwiftSyntax without a lib_InternalSwiftSyntaxParser.dylib or with an incompatible lib_InternalSwiftSyntaxParser.dylib, correct? It is expected that any attempt to run SwiftSyntax without or with an incompatible library is expected to fail, right?

If this is what you are trying to achieve, I don’t think it’ll be of much benefit (unless I’m missing something). You will be able to verify that SwiftPM builds against the most recent SwiftSyntax library, but now we need to make sure that SwiftSyntax works with the current lib_InternalSwiftSyntaxParser.dylib API, so we’re just pushing the issue around, I think. And you won’t be able to run any tests, so I think the benefit on the SwiftPM side is marginal.

What I’d propose is that you pick up the lib_InternalSwiftSyntaxParser.dylib from the toolchain (if it’s missing in some toolchain, we can discuss adding it) and require the developers of SwiftPM to either (a) build SwiftPM with a recent enough Swift development snapshot that contains an lib_InternalSwiftSyntaxParser.dylib compatible with the main SwiftSyntax version or (b) modify the Package.swift in SwiftPM to point to the version of SwiftSyntax that matches the toolchain they’re using to build SwiftPM as described here. Would that work or is there a show-stopper with that approach? I didn’t follow the entire referenced thread.

[owenv]

As I understand it, the main issue is that folks developing SwiftPM want to be able to do so with an incompatible parser lib and without editing the manifest, but also want to know about API breakage on the main branch before running tests on CI. The original approach I had (which is better IMO) to adding the SwiftSyntax dependency conditionally compiled it out of the package when a compatible parser lib isn't available, but that's pretty much stalled so I was experimenting with this instead. That said, I think the approach here just creates more problems than it solves.

[akyrtzi]

want to be able to do so with an incompatible parser lib and without editing the manifest,

To clarify, the incompatibility of SwiftSyntax with parser lib is primarily manifested at runtime (with a runtime check when you try to use it to parse some source), it's not manifested at build time (the C API has been stable).