Skip to content

Commit 79a7049

Browse files
angermanangerman
authored
Adds documentation (#46)
* Adds documentation
1 parent 24b6d6a commit 79a7049

File tree

8 files changed

+536
-0
lines changed

8 files changed

+536
-0
lines changed

‎docs/architecture.md

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Architecture
2+
3+
There are multiple components that play a part in the haskell.nix
4+
infrastructure. These are `nix-tools`, `haskell.nix`, `hackage.nix`,
5+
and `stackage.nix`.
6+
7+
```no-highlight
8+
.-------------. .-------------.
9+
.- nix-tools ------. | haskell.nix | .- | hackage.nix |
10+
| .--------------. | .----------------. '-------------' | '-------------'
11+
| | stack-to-nix |---> | stack-pkgs.nix |-. | | |
12+
| '--------------' | '----------------' | v | v
13+
| .-------------. | .----------. '--> .----------. <-' .--------------.
14+
| | plan-to-nix |----> | plan.nix |------.---> | pkgs.nix | <--- | stackage.nix |
15+
| '-------------' | '----------' | '----------' '--------------'
16+
| .--------------. | .--------------. | |
17+
| | cabal-to-nix |---> | $package.nix |--' v
18+
| '--------------' | '--------------' .-------------.
19+
'------------------' | default.nix |
20+
'-------------'
21+
|
22+
v
23+
.-------------.
24+
| release.nix |
25+
'-------------'
26+
```
27+
haskell.nix diagram
28+
29+
## [nix-tools](https://github.com/input-output-hk/nix-tools)
30+
31+
nix-tools is a Haskell package that provides the following tools:
32+
33+
- `cabal-to-nix`: a `.cabal` to `.nix` transformer that retains
34+
conditional expressions.
35+
36+
- `stack-to-nix`: a `stack.yaml` to `.nix` transformer that will read
37+
in a `stack.yaml` expression an generate a `pkgs.nix` file suited for
38+
use with `haskell.nix`.
39+
40+
- `plan-to-nix`: a `plan.json` to `.nix` transformer that will read in
41+
a `plan.json` file and generate a `pkgs.nix` file suited for use
42+
with `haskell.nix`.
43+
44+
as well as a few other tools used to generate `hackage.nix` and `stackage.nix`.
45+
46+
## [haskell.nix](https://github.com/input-output-hk/haskell.nix)
47+
48+
haskell.nix is the runtime system for this Haskell infrastructure. It
49+
contains the component builder, as well as the system package and
50+
license mapping. Without haskell.nix the expressions generated by
51+
either of the `nix-tools` tools make little sense on their own.
52+
53+
## [hackage.nix](https://github.com/input-output-hk/hackage.nix)
54+
55+
hackage.nix provides all cabal expressions from hackage as nix
56+
expressions. It is periodically updated to keep in sync with the set
57+
of packages available on hackage.
58+
59+
## [stackage.nix](https://github.com/input-output-hk/stackage.nix)
60+
61+
stackage.nix is similar to hackage.nix but provides all stackage
62+
snapshots (lts, and nightly) as nix expressions. It naturally depends
63+
on hackage.nix to resolve package names, versions and revisions to the
64+
repsective packages from hackage.nix.

‎docs/index.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# haskell.nix
2+
3+
haskell.nix is an alternative Haskell infrastructure for nixpkgs. See
4+
[Nixpkgs current Users' Guide to Haskell Infrastructure](https://nixos.org/nixpkgs/manual/#users-guide-to-the-haskell-infrastructure) for comparison.
5+
6+
## Motivation
7+
8+
Why do we need another Haskell infrastructure for nix? Doesn't nixpkgs
9+
provide a sufficiently good Haskell infrastructure already? These are
10+
good questions. And it boils down to the following reasons for us to
11+
embark on a new infrastructure:
12+
13+
- first class support for cross compilation
14+
- first class support for package sets
15+
- component level control when building packages
16+
- reduction of `dontCheck` for cyclic dependencies
17+
- reducing build times by building libraries and tests in parallel
18+
- more logic encoded in nix expressions
19+
- decoupling of Haskell and nixpkgs
20+
21+
### cross compilation
22+
23+
`nixpkgs` has quite good support for cross compilation, however the
24+
Haskell infrastructure suffers from the fact that it heavily relies on
25+
the `cabal2nix` tool. `cabal2nix` (as well as tools that depend on it
26+
like `stack2nix`) flattens the `.cabal` file at conversion time to a
27+
given os/arch/flags configuration. Thus to make cross compilation
28+
work with `cabal2nix` you will have to generate a separate `nix`
29+
expression for each configuration. This becomes a major maintenance
30+
burden over time. Therefore the tooling that translates cabal files
31+
into nix-expressions for use with Haskell.nix retains the full
32+
conditional tree from the cabal file and exposes it to `nix`. In
33+
addition it will also expose the `build-type` value, which allows us
34+
to cache the `Setup.hs` for build-type simple and not have to rebuild
35+
it every time.
36+
37+
### package sets
38+
39+
We often rely on either package sets as provided by stackage or
40+
computed by cabal. `nixpkgs` provides it's own curated package set
41+
which might or might not work for the projects we work on.
42+
`stack2nix` tries to solve this issue, here we go one step further and
43+
provide the infrastructure to allow any form of package set.
44+
45+
### component level control
46+
47+
The Haskell builder in `nixpkgs` provides control over executables and
48+
libraries, to build a specific executable only however is rather
49+
tricky to do. This also leads to the cyclic dependencies issue.
50+
51+
### cyclic dependencies
52+
53+
Because the Haskell builder in `nixpkgs` exposes packages at the
54+
package level, if packages mutually depend on each other through tests
55+
and libraries, lead to cyclic dependencies that nix can't resolve. By
56+
exposing the components to nix as separate derivations this will only
57+
occur if you have mutually dependent components.
58+
59+
### build times
60+
61+
The Haskell builder in nixpkgs build package sequentially, first the
62+
library than the executables and finally the tests. It then executes
63+
the tests before the package is considered done. The upshot of this
64+
is that packages are only considered done if the test-suites
65+
passed. The downside is that if you have to compile multiple packages
66+
the likelihood of them failing is low, you have unnecessarily
67+
serialized you build. In a more aggressive setting libraries could
68+
start building as early as their dependent libraries are built. Of
69+
course they will have to be invalidated later should the test-suites
70+
of their dependencies fail, but this way we can make use of parallel
71+
building. In an ideal scenario this will reduce build times close to
72+
the optimum.
73+
74+
### more logic in nix
75+
76+
The `cabal2nix` tool has a resolver that resolved system dependencies
77+
and licenses to values in `nixpkgs`. This logic end up being a simple
78+
dictionary lookup and can be a simple nix expression. This also
79+
offloads some of the work the cabal to nix translation tool needs to
80+
do into nix, and as such if changes are necessary (or needed to be
81+
performed ad hoc) there is no need to rebuild the conversion tool and
82+
subsequently mark every derived expression as out of date.
83+
84+
### decoupleing
85+
86+
Finally by treating Haskell.nix and nixpkgs as separate entities we
87+
can decouple the Haskell packages and infrastructure from the nixpkgs
88+
package set, and rely on it to provide us with system packages while
89+
staying up to date with Haskell packages from hackage while retaining
90+
a stable (or known to be good) nixpkgs revision.

‎docs/iohk-nix.md

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# IOHK's nix tooling
2+
3+
## [iohk-nix](https://github.com/input-output-hk/iohk-nix)
4+
5+
iohk-nix is IOHK's shared nix library. It provides some templates to
6+
make working with haskell.nix trivial but is non-essential to use
7+
haskell.nix infrastructure.
8+
9+
lib.nix
10+
```nix
11+
let
12+
# iohk-nix can be overridden for debugging purposes by setting
13+
# NIX_PATH=iohk_nix=/path/to/iohk-nix
14+
iohkNix = import (
15+
let try = builtins.tryEval <iohk_nix>;
16+
in if try.success
17+
then builtins.trace "using host <iohk_nix>" try.value
18+
else
19+
let
20+
spec = builtins.fromJSON (builtins.readFile ./iohk-nix.json);
21+
in builtins.fetchTarball {
22+
url = "${spec.url}/archive/${spec.rev}.tar.gz";
23+
inherit (spec) sha256;
24+
}) {};
25+
26+
pkgs = iohkNix.pkgs;
27+
lib = pkgs.lib;
28+
in lib // { inherit iohkNix pkgs; inherit (iohkNix) nix-tools; }
29+
```
30+
31+
iohk-nix.json
32+
```json
33+
{
34+
"url": "https://github.com/input-output-hk/iohk-nix",
35+
"rev": "c92f0119ef5814b0ed1f445c2fdcf8894e326294",
36+
"sha256": "05r90x6x3yp1nb66rkc4n0i8q15c634rrdsr2zvb118s3sdcmmrm",
37+
"fetchSubmodules": false
38+
}
39+
```
40+
41+
nix/pkgs.nix
42+
```
43+
{ pkgs ? import <nixpkgs> {}
44+
, iohk-overlay ? {}
45+
, iohk-module ? {}
46+
, haskell
47+
, hackage
48+
, stackage
49+
, ...
50+
}:
51+
let
52+
# our packages
53+
stack-pkgs = import ./.stack-pkgs.nix;
54+
55+
# packages which will require TH and thus
56+
# will need -fexternal-interpreter treatment
57+
# when cross compiling.
58+
th-packages = [
59+
"hedgehog" "cardano-crypto-wrapper"
60+
"cardano-crypto-test" "cardano-chain"
61+
"small-steps" "cs-ledger" ];
62+
63+
# Build the packageset with module support.
64+
# We can essentially override anything in the modules
65+
# section.
66+
#
67+
# packages.cbors.patches = [ ./one.patch ];
68+
# packages.cbors.flags.optimize-gmp = false;
69+
#
70+
compiler = (stack-pkgs.overlay hackage).compiler.nix-name;
71+
pkgSet = haskell.mkNewPkgSet {
72+
inherit pkgs;
73+
pkg-def = stackage.${stack-pkgs.resolver};
74+
# The overlay allows extension or restriction of the set of
75+
# packages we are interested in. By using the stack-pkgs.overlay
76+
# we restrict our package set to the ones provided in stack.yaml.
77+
pkg-def-overlays = [
78+
stack-pkgs.overlay
79+
iohk-overlay.${compiler}
80+
];
81+
# package customizations
82+
modules = [
83+
# This module will ensure that we get the necessary
84+
# patches ontop of GHC packages that for which the
85+
# ones that GHC ships are not identical to the ones
86+
# we find on hackage. These patches will make sure
87+
# they are identical by augmenting the packages on
88+
# hackage to match those that ship with ghc.
89+
haskell.ghcHackagePatches.${compiler}
90+
91+
# the iohk-module will supply us with the necessary
92+
# cross compilation plumbing to make Template Haskell
93+
# work when cross compiling. For now we need to
94+
# list the packages that require template haskell
95+
# explicity here.
96+
(iohk-module { nixpkgs = pkgs;
97+
inherit th-packages; })
98+
];
99+
};
100+
in
101+
pkgSet.config.hsPkgs // { _config = pkgSet.config; }
102+
```
103+
104+
105+
default.nix
106+
```
107+
let
108+
localLib = import ./lib.nix;
109+
in
110+
# This file needs to export a function that takes
111+
# the arguments it is passed and forwards them to
112+
# the default-nix template from iohk-nix. This is
113+
# important so that the release.nix file can properly
114+
# parameterize this file when targetting different
115+
# hosts.
116+
{ ... }@args:
117+
# We will instantiate the defaul-nix template with the
118+
# nix/pkgs.nix file...
119+
localLib.nix-tools.default-nix ./nix/pkgs.nix args
120+
# ... and add a few custom packages as well.
121+
// { }
122+
```

‎doc/removing-with-package-wrapper.md renamed to ‎docs/removing-with-package-wrapper.md

File renamed without changes.

‎docs/user-guide-cabal.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# User Guide (cabal project)
2+
3+
Here we will look into how to generate the `pkgs.nix` file for a
4+
`cabal.project` project. For the full integration please see the [User
5+
Guide](/user-guide)
6+
7+
## Using `plan-to-nix`
8+
9+
*We currently don't have a `project-to-nix` tool yet, as such creating
10+
the relevant `pkgs.nix` file for a `cabal.project` is slightly more
11+
involved than for a corresponding stack project*.
12+
13+
With [nix-tools](https://github.com/input-output-hk/nix-tools) in
14+
`PATH`, we can simply run the following command on a stack project:
15+
16+
```bash
17+
# make sure the cabal project is configured (the plan.json file is generated)
18+
cabal new-configure
19+
# convert the plan.json file into a pkgs.nix file
20+
plan-to-nix dist-newstyle/cache/plan.json > nix/plan.nix
21+
```
22+
23+
This will produce a `nix/plan.nix` file that looks like the following:
24+
```nix
25+
hackage:
26+
{
27+
packages = {
28+
"o-clock" = hackage.o-clock."0.1.1".revisions.default;
29+
...
30+
};
31+
compiler = { ... };
32+
}
33+
```
34+
35+
it specifically does not include any of our local packages yet. We
36+
will need to run
37+
38+
```bash
39+
cabal-to-nix $path > nix/$pkg.nix
40+
```
41+
or
42+
```bash
43+
cabal-to-nix $url $rev > nix/$pkg.nix
44+
```
45+
for each local (or source) package.
46+
47+
With this in place we can then proceed to build the `nix/pkgs.nix`
48+
file as follows:
49+
50+
```nix
51+
let plan = import ./plan.nix; in
52+
{ ... }:
53+
{ pkg-def = plan;
54+
overlay =
55+
{ local-package-a = ./local-package-a.nix;
56+
local-package-b = ./local-package-b.nix;
57+
source-import-a = ./source-import-a.nix;
58+
source-import-b = ./source-import-b.nix;
59+
...
60+
};
61+
}
62+
```
63+
64+
*If you came here from the [User Guide](/user-guide), go back and
65+
complete the setup.*

‎docs/user-guide-stack.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# User Guide (stack project)
2+
3+
Here we will look into how to generate the `pkgs.nix` file for a
4+
`stack.yaml` project. For the full integration please see the [User
5+
Guide](/user-guide)
6+
7+
## Using `stack-to-nix`
8+
9+
With [nix-tools](https://github.com/input-output-hk/nix-tools) in
10+
`PATH`, we can simply run the following command on a stack project:
11+
12+
```bash
13+
stack-to-nix -o nix stack.yaml > nix/.stack-pkgs.nix
14+
```
15+
16+
This will produce a `nix/.stack-pkgs.nix` file that looks like the following:
17+
```nix
18+
{
19+
resolver = "lts-12.17";
20+
overlay = hackage:
21+
{
22+
packages = {
23+
"o-clock" = hackage.o-clock."0.1.1".revisions.default;
24+
...
25+
} // {
26+
my-package = ./.stack.nix/my-package.nix;
27+
...
28+
};
29+
};
30+
}
31+
```
32+
33+
This file contains the stackage resolver, as well as an overlay of
34+
packages. The overlay specifies which `extra-deps` (here: clock-0.1.1)
35+
we wanted to overlay over the stackage snapshot, and what local
36+
packages we want (here: my-package).
37+
38+
We will then create the following `nix/pkgs.nix` file:
39+
40+
```nix
41+
let stack-pkgs = import ./.stack-pkgs.nix; in
42+
{ stackage, ... }:
43+
{ pkg-def = stackage.${stack-pkgs.resolver};
44+
inherit (stack-pkgs) overlay;
45+
}
46+
```
47+
48+
*If you came here from the [User Guide](/user-guide), go back and
49+
complete the setup.*

0 commit comments

Comments
 (0)