Adds documentation

docs/architecture.md

@@ -0,0 +1,64 @@
+# Architecture
+
+There are multiple components that play a part in the haskell.nix
+infrastructure.  These are `nix-tools`, `haskell.nix`, `hackage.nix`,
+and `stackage.nix`.
+
+```no-highlight
+                                             .-------------.     .-------------.
+.- nix-tools ------.                         | haskell.nix |  .- | hackage.nix |
+| .--------------. |   .----------------.    '-------------'  |  '-------------'
+| | stack-to-nix |---> | stack-pkgs.nix |-.         |         |         |
+| '--------------' |   '----------------' |         v         |         v
+| .-------------.  |   .----------.       '--> .----------. <-'  .--------------.
+| | plan-to-nix |----> | plan.nix |------.---> | pkgs.nix | <--- | stackage.nix |
+| '-------------'  |   '----------'      |     '----------'      '--------------'
+| .--------------. |   .--------------.  |          |
+| | cabal-to-nix |---> | $package.nix |--'          v
+| '--------------' |   '--------------'       .-------------.
+'------------------'                          | default.nix |
+                                              '-------------'
+                                                    |
+                                                    v
+                                              .-------------.
+                                              | release.nix |
+                                              '-------------'
+```
+haskell.nix diagram
+
+## [nix-tools](https://github.com/input-output-hk/nix-tools)
+
+nix-tools is a Haskell package that provides the following tools:
+
+- `cabal-to-nix`: a `.cabal` to `.nix` transformer that retains
+  conditional expressions.
+
+- `stack-to-nix`: a `stack.yaml` to `.nix` transformer that will read
+  in a `stack.yaml` expression an generate a `pkgs.nix` file suited for
+  use with `haskell.nix`.
+
+- `plan-to-nix`: a `plan.json` to `.nix` transformer that will read in
+  a `plan.json` file and generate a `pkgs.nix` file suited for use
+  with `haskell.nix`.
+
+as well as a few other tools used to generate `hackage.nix` and `stackage.nix`.
+
+## [haskell.nix](https://github.com/input-output-hk/haskell.nix)
+
+haskell.nix is the runtime system for this Haskell infrastructure.  It
+contains the component builder, as well as the system package and
+license mapping.  Without haskell.nix the expressions generated by
+either of the `nix-tools` tools make little sense on their own.
+
+## [hackage.nix](https://github.com/input-output-hk/hackage.nix)
+
+hackage.nix provides all cabal expressions from hackage as nix
+expressions.  It is periodically updated to keep in sync with the set
+of packages available on hackage.
+
+## [stackage.nix](https://github.com/input-output-hk/stackage.nix)
+
+stackage.nix is similar to hackage.nix but provides all stackage
+snapshots (lts, and nightly) as nix expressions.  It naturally depends
+on hackage.nix to resolve package names, versions and revisions to the
+repsective packages from hackage.nix.

docs/index.md

@@ -0,0 +1,90 @@
+# haskell.nix
+
+haskell.nix is an alternative Haskell infrastructure for nixpkgs. See
+[Nixpkgs current Users' Guide to Haskell Infrastructure](https://nixos.org/nixpkgs/manual/#users-guide-to-the-haskell-infrastructure) for comparison.
+
+## Motivation
+
+Why do we need another Haskell infrastructure for nix?  Doesn't nixpkgs
+provide a sufficiently good Haskell infrastructure already?  These are
+good questions.  And it boils down to the following reasons for us to
+embark on a new infrastructure:
+
+- first class support for cross compilation
+- first class support for package sets
+- component level control when building packages
+- reduction of `dontCheck` for cyclic dependencies
+- reducing build times by building libraries and tests in parallel
+- more logic encoded in nix expressions
+- decoupling of Haskell and nixpkgs
+
+### cross compilation
+
+`nixpkgs` has quite good support for cross compilation, however the
+Haskell infrastructure suffers from the fact that it heavily relies on
+the `cabal2nix` tool.  `cabal2nix` (as well as tools that depend on it
+like `stack2nix`) flattens the `.cabal` file at conversion time to a
+given os/arch/flags configuration.  Thus to make cross compilation
+work with `cabal2nix` you will have to generate a separate `nix`
+expression for each configuration.  This becomes a major maintenance
+burden over time.  Therefore the tooling that translates cabal files
+into nix-expressions for use with Haskell.nix retains the full
+conditional tree from the cabal file and exposes it to `nix`.  In
+addition it will also expose the `build-type` value, which allows us
+to cache the `Setup.hs` for build-type simple and not have to rebuild
+it every time.
+
+### package sets
+
+We often rely on either package sets as provided by stackage or
+computed by cabal.  `nixpkgs` provides it's own curated package set
+which might or might not work for the projects we work on.
+`stack2nix` tries to solve this issue, here we go one step further and
+provide the infrastructure to allow any form of package set.
+
+### component level control
+
+The Haskell builder in `nixpkgs` provides control over executables and
+libraries, to build a specific executable only however is rather
+tricky to do.  This also leads to the cyclic dependencies issue.
+
+### cyclic dependencies
+
+Because the Haskell builder in `nixpkgs` exposes packages at the
+package level, if packages mutually depend on each other through tests
+and libraries, lead to cyclic dependencies that nix can't resolve.  By
+exposing the components to nix as separate derivations this will only
+occur if you have mutually dependent components.
+
+### build times
+
+The Haskell builder in nixpkgs build package sequentially, first the
+library than the executables and finally the tests.  It then executes
+the tests before the package is considered done.  The upshot of this
+is that packages are only considered done if the test-suites
+passed.  The downside is that if you have to compile multiple packages
+the likelihood of them failing is low, you have unnecessarily
+serialized you build.  In a more aggressive setting libraries could
+start building as early as their dependent libraries are built.  Of
+course they will have to be invalidated later should the test-suites
+of their dependencies fail, but this way we can make use of parallel
+building.  In an ideal scenario this will reduce build times close to
+the optimum.
+
+### more logic in nix
+
+The `cabal2nix` tool has a resolver that resolved system dependencies
+and licenses to values in `nixpkgs`.  This logic end up being a simple
+dictionary lookup and can be a simple nix expression.  This also
+offloads some of the work the cabal to nix translation tool needs to
+do into nix, and as such if changes are necessary (or needed to be
+performed ad hoc) there is no need to rebuild the conversion tool and
+subsequently mark every derived expression as out of date.
+
+### decoupleing
+
+Finally by treating Haskell.nix and nixpkgs as separate entities we
+can decouple the Haskell packages and infrastructure from the nixpkgs
+package set, and rely on it to provide us with system packages while
+staying up to date with Haskell packages from hackage while retaining
+a stable (or known to be good) nixpkgs revision.

docs/iohk-nix.md

@@ -0,0 +1,122 @@
+# IOHK's nix tooling
+
+## [iohk-nix](https://github.com/input-output-hk/iohk-nix)
+
+iohk-nix is IOHK's shared nix library.  It provides some templates to
+make working with haskell.nix trivial but is non-essential to use
+haskell.nix infrastructure.
+
+lib.nix
+```nix
+let
+  # iohk-nix can be overridden for debugging purposes by setting
+  # NIX_PATH=iohk_nix=/path/to/iohk-nix
+  iohkNix = import (
+    let try = builtins.tryEval <iohk_nix>;
+    in if try.success
+    then builtins.trace "using host <iohk_nix>" try.value
+    else
+      let
+        spec = builtins.fromJSON (builtins.readFile ./iohk-nix.json);
+      in builtins.fetchTarball {
+        url = "${spec.url}/archive/${spec.rev}.tar.gz";
+        inherit (spec) sha256;
+      }) {};
+
+  pkgs = iohkNix.pkgs;
+  lib = pkgs.lib;
+in lib // { inherit iohkNix pkgs; inherit (iohkNix) nix-tools; }
+```
+
+iohk-nix.json
+```json
+{
+  "url": "https://github.com/input-output-hk/iohk-nix",
+  "rev": "c92f0119ef5814b0ed1f445c2fdcf8894e326294",
+  "sha256": "05r90x6x3yp1nb66rkc4n0i8q15c634rrdsr2zvb118s3sdcmmrm",
+  "fetchSubmodules": false
+}
+```
+
+nix/pkgs.nix
+```
+{ pkgs ? import <nixpkgs> {}
+, iohk-overlay ? {}
+, iohk-module ? {}
+, haskell
+, hackage
+, stackage
+, ...
+}:
+let
+  # our packages
+  stack-pkgs = import ./.stack-pkgs.nix;
+
+  # packages which will require TH and thus
+  # will need -fexternal-interpreter treatment
+  # when cross compiling.
+  th-packages = [
+    "hedgehog" "cardano-crypto-wrapper"
+    "cardano-crypto-test" "cardano-chain"
+    "small-steps" "cs-ledger" ];
+
+  # Build the packageset with module support.
+  # We can essentially override anything in the modules
+  # section.
+  #
+  #  packages.cbors.patches = [ ./one.patch ];
+  #  packages.cbors.flags.optimize-gmp = false;
+  #
+  compiler = (stack-pkgs.overlay hackage).compiler.nix-name;
+  pkgSet = haskell.mkNewPkgSet {
+    inherit pkgs;
+    pkg-def = stackage.${stack-pkgs.resolver};
+    # The overlay allows extension or restriction of the set of
+    # packages we are interested in. By using the stack-pkgs.overlay
+    # we restrict our package set to the ones provided in stack.yaml.
+    pkg-def-overlays = [
+      stack-pkgs.overlay
+      iohk-overlay.${compiler}
+    ];
+    # package customizations
+    modules = [
+      # This module will ensure that we get the necessary
+      # patches ontop of GHC packages that for which the
+      # ones that GHC ships are not identical to the ones
+      # we find on hackage. These patches will make sure
+      # they are identical by augmenting the packages on
+      # hackage to match those that ship with ghc.
+      haskell.ghcHackagePatches.${compiler}
+
+      # the iohk-module will supply us with the necessary
+      # cross compilation plumbing to make Template Haskell
+      # work when cross compiling.  For now we need to
+      # list the packages that require template haskell
+      # explicity here.
+      (iohk-module { nixpkgs = pkgs;
+                     inherit th-packages; })
+    ];
+  };
+in
+  pkgSet.config.hsPkgs // { _config = pkgSet.config; }
+```
+
+
+default.nix
+```
+let
+  localLib = import ./lib.nix;
+in
+# This file needs to export a function that takes
+# the arguments it is passed and forwards them to
+# the default-nix template from iohk-nix. This is
+# important so that the release.nix file can properly
+# parameterize this file when targetting different
+# hosts.
+{ ... }@args:
+# We will instantiate the defaul-nix template with the
+# nix/pkgs.nix file...
+localLib.nix-tools.default-nix ./nix/pkgs.nix args
+# ... and add a few custom packages as well.
+// { }
+```

docs/user-guide-cabal.md

@@ -0,0 +1,65 @@
+# User Guide (cabal project)
+
+Here we will look into how to generate the `pkgs.nix` file for a
+`cabal.project` project.  For the full integration please see the [User
+Guide](/user-guide)
+
+## Using `plan-to-nix`
+
+*We currently don't have a `project-to-nix` tool yet, as such creating
+ the relevant `pkgs.nix` file for a `cabal.project` is slightly more
+ involved than for a corresponding stack project*.
+
+With [nix-tools](https://github.com/input-output-hk/nix-tools) in
+`PATH`, we can simply run the following command on a stack project:
+
+```bash
+# make sure the cabal project is configured (the plan.json file is generated)
+cabal new-configure
+# convert the plan.json file into a pkgs.nix file
+plan-to-nix dist-newstyle/cache/plan.json > nix/plan.nix
+```
+
+This will produce a `nix/plan.nix` file that looks like the following:
+```nix
+hackage:
+  {
+    packages = {
+      "o-clock" = hackage.o-clock."0.1.1".revisions.default;
+      ...
+    };
+    compiler = { ... };
+  }
+```
+
+it specifically does not include any of our local packages yet. We
+will need to run
+
+```bash
+cabal-to-nix $path > nix/$pkg.nix
+```
+or
+```bash
+cabal-to-nix $url $rev > nix/$pkg.nix
+```
+for each local (or source) package.
+
+With this in place we can then proceed to build the `nix/pkgs.nix`
+file as follows:
+
+```nix
+let plan = import ./plan.nix; in
+{ ... }:
+{ pkg-def = plan;
+  overlay =
+    { local-package-a = ./local-package-a.nix;
+      local-package-b = ./local-package-b.nix;
+      source-import-a = ./source-import-a.nix;
+      source-import-b = ./source-import-b.nix;
+      ...
+    };
+}
+```
+
+*If you came here from the [User Guide](/user-guide), go back and
+ complete the setup.*

docs/user-guide-stack.md

@@ -0,0 +1,49 @@
+# User Guide (stack project)
+
+Here we will look into how to generate the `pkgs.nix` file for a
+`stack.yaml` project.  For the full integration please see the [User
+Guide](/user-guide)
+
+## Using `stack-to-nix`
+
+With [nix-tools](https://github.com/input-output-hk/nix-tools) in
+`PATH`, we can simply run the following command on a stack project:
+
+```bash
+stack-to-nix -o nix stack.yaml > nix/.stack-pkgs.nix
+```
+
+This will produce a `nix/.stack-pkgs.nix` file that looks like the following:
+```nix
+{
+  resolver = "lts-12.17";
+  overlay = hackage:
+    {
+      packages = {
+        "o-clock" = hackage.o-clock."0.1.1".revisions.default;
+        ...
+      } // {
+        my-package = ./.stack.nix/my-package.nix;
+        ...
+      };
+    };
+}
+```
+
+This file contains the stackage resolver, as well as an overlay of
+packages.  The overlay specifies which `extra-deps` (here: clock-0.1.1)
+we wanted to overlay over the stackage snapshot, and what local
+packages we want (here: my-package).
+
+We will then create the following `nix/pkgs.nix` file:
+
+```nix
+let stack-pkgs = import ./.stack-pkgs.nix; in
+{ stackage, ... }:
+{ pkg-def = stackage.${stack-pkgs.resolver};
+  inherit (stack-pkgs) overlay;
+}
+```
+
+*If you came here from the [User Guide](/user-guide), go back and
+ complete the setup.*