Add support for Netrc for Downloader (cherry-pick #2833)

[sstadelman]

Adds support for authenticated download for binary artifacts. Cherry-picking PR #2833 from main.
Depends on swift-tools-support-core PR #133.

[sstadelman]

@neonichu can this be considered for merge to the release/5.3 branch?

[MaxDesiatov]

@swift-ci please smoke test

[sstadelman]

@swift-ci please smoke test

@MaxDesiatov Depends on the pending tools PR referenced

[MaxDesiatov]

swiftlang/swift-tools-support-core#133
@swift-ci please smoke test

[sstadelman]

@MaxDesiatov I'm assuming self-hosted had a package caching issue?

[MaxDesiatov]

It could be because self-hosted build points to the release/5.3 branch of TSC, I'm not sure. I hope the reviewers can clarify this.

[neonichu]

Yep, self-hosted uses what's declared in the manifest and doesn't support cross-repo testing at the moment.

As far as inclusion into 5.3 goes, I kicked off a thread with the relevant folks to decide.

[sstadelman]

@neonichu any update on this?

[neonichu]

Hi @sstadelman, sorry for the delay. We discussed this now, but first would like to see two tweaks here:

• we should add a --netrc option that uses the default configuration file, similar to how curl works
• we should emit a warning if at least one artifact is being downloaded and one of the netrc options was specified, but the file isn't present

[sstadelman]

• we should add a --netrc option that uses the default configuration file, similar to how curl works
• we should emit a warning if at least one artifact is being downloaded and one of the netrc options was specified, but the file isn't present

Hi @neonichu ,

Right now, the behavior is closer to --netrc-optional or --netrc-file <file> --netrc-optional, where we always check the HOME or specified path for a netrc file.

I infer from the quote that we should not use the default configuration file, if --netrc is not supplied?

If the intention is to precisely imitate curl behavior, then --netrc requires the file to be present. Only --netrc-optional should gracefully warn on failure to find a file at the specified location.

Perhaps we should instead add --netrc-optional, such that absence indicates that the default location should not be searched? We could also update the behavior for --netrc-file that it respects --netrc-optional, as documented in the man page?

See https://curl.haxx.se/docs/manpage.html#--netrc-file

[neonichu]

Perhaps we should instead add --netrc-optional, such that absence indicates that the default location should not be searched? We could also update the behavior for --netrc-file that it respects --netrc-optional, as documented in the man page?

I think this sounds like a good course of action.

[MaxDesiatov]

@swift-ci please test

[sstadelman]

@MaxDesiatov I updated Package.swift to reference release/5.3 for llbuild. This was necessary to avoid compile error.

[sstadelman]

@MaxDesiatov had a local path in the test, should be clear now.

[MaxDesiatov]

@swift-ci please test

[neonichu]

We decided to not take this for 5.3, but as the changes are on main they will be part of 5.4.