[rm nixpkgs] use path-from-hash-part command when updating lock file

[savil]

Summary

Motivation
With Remove Nixpkgs feature on, installing [email protected] was failing with:

[3/7] [email protected]
        don't know how to build these paths:
          /nix/store/0xyi7w3cm3g7gzwfpwqzx0n7xks2sdc6-curl-8.0.1
        asked 'https://cache.nixos.org/' for '/nix/store/0xyi7w3cm3g7gzwfpwqzx0n7xks2sdc6-curl-8.0.1' but got '/nix/store/0xyi7w3cm3g7gzwfpwqzx0n7xks2sdc6-curl-8.0.1-bin'
        error: path '/nix/store/0xyi7w3cm3g7gzwfpwqzx0n7xks2sdc6-curl-8.0.1' is required, but there is no substituter that can build it

Explanation
Thanks to @gcurtis.
curl’s first (and therefore default) output is bin, not out, so Nix is appending -bin to the path.

From man nix-build:

If a derivation has multiple outputs, nix-build will build the default (first) output. You can also build all outputs:
$ nix-build '' --attr openssl.all
This will create a symlink for each output named result-outputname. The suffix is omitted if the output name is out. So if openssl has outputs out, bin and man, nix-build will create symlinks result, result-bin and result-man.

Fix
Observe:

> nix store path-from-hash-part --store https://cache.nixos.org/ bbkz21vmx16wl0h3xk36g7gxpbymnm6d --extra-experimental-features "nix-command flakes"
/nix/store/bbkz21vmx16wl0h3xk36g7gxpbymnm6d-curl-8.0.1-bin

So:

• During devbox update, we derive the store-path from the search API and write to devbox.lock
• Previously, we would manually construct store-path from hash-name-version from search API response
• Now, we invoke nix store path-from-hash-part <hash> and write the response store-path in the devbox.lock.
  • if this fails (it shouldn't!), we skip writing the store-path to the devbox.lock, and print a warning. Maybe we should track it in sentry?

How was it tested?

> cat devbox.json
{
  "packages": [
    "[email protected]"
  ]
}

> devbox update
Info: Updated system information for [email protected]
Ensuring packages are installed.

Installing package: [email protected].

[1/1] [email protected]
[1/1] [email protected]: Success
Info: Running "nix flake update"

> cat devbox.lock
{
  "lockfile_version": "1",
  "packages": {
    "[email protected]": {
      "last_modified": "2023-05-31T02:09:55Z",
      "resolved": "github:NixOS/nixpkgs/9cfaa8a1a00830d17487cb60a19bb86f96f09b27#curl",
      "source": "devbox-search",
      "version": "8.0.1",
      "systems": {
        "aarch64-darwin": {
          "store_path": "/nix/store/7qwsslwdrjyg8hs654nsqpq7lly6dhq5-curl-8.0.1-bin"
        },
        "aarch64-linux": {
          "store_path": "/nix/store/g9g1p4l9146kclnx4lv0hfbzhr9hk7n7-curl-8.0.1-bin"
        }
      }
    }
  }
}

[savil]

Current dependencies on/for this PR:

• main
  • PR [rm nixpkgs] use path-from-hash-part command when updating lock file #1442 👈
    • PR [rm nixpkgs] enable feature flag (assuming tests pass) #1443

This comment was auto-generated by Graphite.

[savil]

@gcurtis instead of running this logic in the CLI, I'm now thinking it may make sense for the search indexer to run it. WDYT?

[gcurtis]

This is something we should add to the API, but feel free to keep it as a workaround until the API is updated.

I plan on adding a new outputs field to the JSON alongside the current store_hash/name/version fields. The new response will look like:

{
  "store_hash": "00263n102bi1r6qihvdghxb3jx6n3k1d",
  "store_name": "libcxxabi",
  "store_version": "16.0.6",
  "outputs": [
    {
      "path": "/nix/store/00263n102bi1r6qihvdghxb3jx6n3k1d-libcxxabi-16.0.6",
      "name": "out",
      "hash": "00263n102bi1r6qihvdghxb3jx6n3k1d",
      "default": true
    },
    {
      "path": "/nix/store/04r1bxcai0bzb2z01n5r47j8r2387s2r-libcxxabi-16.0.6-dev",
      "name": "dev",
      "hash": "04r1bxcai0bzb2z01n5r47j8r2387s2r",
      "default": false
    }
  ]
}

The store_hash field will be kept for compatibility, but new code will use the outputs array instead. Would this new format get you what you're looking for?

[savil]

@gcurtis That would be neat. A few questions:

1. I understand that default would refer to the first output, as per nix build docs. Does that always necessarily imply its also the output cached by cache.nixos.org? Will you confirm that server-side, or do I need to keep a check similar to the one in this PR in the CLI?

2. For packages with multiple outputs (ref. [Package Issue Report]: prometheus does not provide promtool #1431 where prometheus has cli output as well), your proposal would be really helpful. However, I'm guessing the cache.nixos.org will not have a prebuilt binary for the cli output, since its not the default output. Let me know if I'm incorrect. I'm thinking we may need the user to specify outputs they'd like. Something like:

devbox add prometheus --outputs default,cli

which are set as:

// devbox.json
{
    "prometheus: {
        "version": "latest",
        "outputs": [default, cli],
    },
}

And unfortunately then we'd need to install this cli using the nixpkgs-full path.
Does that sounds good, directionally at least?

[savil]

Lets land this PR then so we can start dogfooding internally!

[gcurtis]

1. I understand that default would refer to the first output, as per nix build docs. Does that always necessarily imply its also the output cached by cache.nixos.org? Will you confirm that server-side, or do I need to keep a check similar to the one in this PR in the CLI?

Default isn't always just the first output. Some packages define multiple outputs to install by default. That's a good point about the cache check. I think the best approach would be to add a cached field to each output in the response.

However, I'm guessing the cache.nixos.org will not have a prebuilt binary for the cli output, since its not the default output.

I had to look this one up to double check. It looks like all of the prometheus outputs are cached, including cli. I'm not sure what the best way is for installing non-default outputs. Off the top of my head, we could either:

• Do your suggestion - add an outputs field to devbox.json. I'm not a huge fan of adding yet another field though.
  • Alternatively, we just work it into the name like prometheus-cli@latest or curl-lib@latest.
• Install all outputs by default (nixpkgs#prometheus.all). Downside is this will probably really bloat installs.
• Special-case outputs to always install. In other words, always install out, bin, man, and cli.

Edit: it looks like nix profile install uses the syntax nixpkgs#prometheus^cli.