Fix --libdir installs

[wthrowe]

Rather than modifying the installer to disable directory rewriting,
this patch modifies the directory structure passed to the installer so
that the rewriting gives the correct results. This means that if a
non-standard --libdir is passed to configure then the same --libdir
option (relative to the --prefix) must be passed to the install
script. In the make install case this is handled automatically.
Binary distributions are generally generated using the default
--libdir and then have paths optionally rewritten by the installer,
which should continue to work.

This has the advantage of not complicating the installer interface
intended for end-user use.

Fixes #29561

[rust-highfive]

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

r? @brson

Seems fine to me though. Just a standard "oh my god I can't read anything here and this is madness" it seems!

[wthrowe]

Although I have not tested, I believe making HLIB_RELATIVE_* fixed to lib (on Linux) would mean that you could no longer run the compiler from the build tree with a non-default --libdir option (because CFG_LIBDIR_RELATIVE is used to search for libraries). This would mean that distros could not run the test suite.

Edit: Actually, that presumably applies to building the stage2 compiler, not just running the test suite.

[brson]

Hm, you may be right about that. I still might be inclined to complexify the find_libdir method to fallback to CFG_LIBDIR_RELATIVE if lib doesn't exist in order to keep the in-tree structure consistent (assuming we did the same on windows, where we currently put some libs in bin).

Although, I think I recall my preferred option for fixing this problem previously was actually to kind of do the opposite of where this patch is going (and what I'm currently suggesting): remove the --libdir option and the libdir rewriting from rust-installer (since it can't work without help from the buildsystem and CFG_LIBDIR_RELATIVE), and continue what we're doing now in the buildsystem, customizing the in-tree layout and that in the installer tarball. That would solve the case for ditros using make install, without implying that the main installers we distribute can usefully customize --libdir.

Can you remind me the gentoo file system layout, why --libdir is customized?

[wthrowe]

Adding another fallback in find_libdir should work, yeah, and would be pretty easy. (In fact it already basically exists, it would just require checking whether the CFG_LIBDIR_RELATIVE path was valid and ignoring it if it wasn't.)

The only downsides I see to removing the libdir rewriting in the installer are (1) rewriting lib to lib64/lib32 as appropriate should currently work (although I wouldn't be surprised if no one used that featue) and (2) the installer would still have to figure out where to install the manifest and similar files somehow, so either these would have to move out of $libdir, $libdir would have to be encoded in the installer (or tarballs) somehow, or we'd still be relying on configure and install.sh getting the same value passed.

Gentoo wants to allow multiple versions of Rust to be installed simultaneously. In theory you could have stable, beta, nightly, and git versions of rust installed simultaneously (and also binary distributions of the first three). I was not involved in deciding on the structure, so I can only report what is going on, not give reasons.

The executables are installed to, for example, /usr/bin/rustc-1.4.0 (although the versions appear to be lies in the non-stable channels: things like rustc-99). There is a system to maintain a /usr/bin/rustc symlink.

The --libdir is set to /usr/lib{,32,64}/rust-${version}, presumably so that the different channels don't try to install files over one another. The libraries themselves have suffixes so they wouldn't conflict (although I'm not sure how well rustc handles finding multiple versions of the same library at the moment), but things like the gdb scripts don't. (Not that the gdb scripts work currently, since rust-gdb hardcodes lib/rustlib/etc as the path to the python scripts.)

The man pages are put in /usr/share/rust-${version}/man, and the miscellaneous documentation is put in /usr/share/doc/rust-${version} (manually, since Rust's configure script doesn't have a --docdir option). There's also an environment file installed to add the appropriate values to LDPATH and MANPATH.

There are also packages that don't compile from source, which appear to install to someplace in /opt and then symlink the binaries to /usr/bin. I've never tried using those.

[brson]

the installer would still have to figure out where to install the manifest and similar files somehow, so either these would have to move out of $libdir, $libdir would have to be encoded in the installer (or tarballs) somehow, or we'd still be relying on configure and install.sh getting the same value passed.

Oh, good point. Argh.

Thanks for the explanation of what Gentoo is doing.

OK, I'm going to accept this as-is to get it working - it's a small change that does improve the current state. I'd like to come up with a side-by-side scheme that we can feel confident about supporting and that is workable for all distros, but right now I don't have great ideas.

I'd suggest that gentoo consider setting up their custom filesystem layout differently though, in a way that doesn't tax our build / install system so much. Instead of using the --*dir options (which I understand are traditional and should work as expected), just do a typical --prefix install to /usr/lib/rust-$version, which is guaranteed to work and not break weirdly. Then use symlinks to set up the user-facing file system as described.

@bors r+

[bors]

📌 Commit ea798b9 has been approved by brson

[bors]

⌛ Testing commit ea798b9 with merge edcfeb7...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt