Implement system and user provisioning scripts

[jandubois]

Fixes #33

In addition this PR implements a bare-vm setting that disables installation of optional requirements (containerd and nerdctl), and also skips installation of sshfs if there are no directories to mount.

Example use to provision kubernetes via k3s (although not in rootless mode):

$ cat k3s.yaml
images:
- location: "~/Downloads/hirsute-server-cloudimg-amd64.img"
  arch: "x86_64"

mounts: []

ssh:
  localPort: 60022

bare-vm: true

provision:
  system: |
    #!/bin/sh
    curl -sfL https://get.k3s.io | sh -
$ limactl start --tty=false k3s.yaml
[...]
INFO[0032] READY. Run `lima bash` to open the shell.
INFO[0032] Forwarding "/run/user/501/lima-guestagent.sock" (guest) to "/Users/jan/.lima/k3s/ga.sock" (host)
INFO[0051] Forwarding TCP port 6444
INFO[0051] Forwarding TCP port 6443
INFO[0054] Forwarding TCP port 10010
[...]

And in a second shell:

$ export KUBECONFIG=$PWD/kubeconfig.yaml
$ limactl shell k3s sudo cat /etc/rancher/k3s/k3s.yaml >$KUBECONFIG
$ kubectl get no
NAME       STATUS   ROLES                  AGE   VERSION
lima-k3s   Ready    control-plane,master   69s   v1.21.1+k3s1
$ kubectl get po -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-7448499f4d-gsv6h                  1/1     Running     0          67s
kube-system   local-path-provisioner-5ff76fc89d-kbmtj   1/1     Running     0          67s
kube-system   metrics-server-86cbb8457f-g7bjg           1/1     Running     0          67s
kube-system   helm-install-traefik-crd-sr9m8            0/1     Completed   0          67s
kube-system   helm-install-traefik-f587z                0/1     Completed   1          67s
kube-system   svclb-traefik-sqj88                       2/2     Running     0          47s
kube-system   traefik-97b44b794-j8t8s                   1/1     Running     0          47s

[jandubois]

I think I made the requirements logic overly complicated, and would prefer to simplify it:

1. Generate the requirements dynamically in a method instead of having a static list. That way the whole logic for skipping requirements during checking becomes unnecessary, and the user will not see the potentially confusing "Skipping requirement..." message either.

2. With the availability of bare-vm it makes less sense to treat nerdctl/containerd as "optional". A single "essential" requirements list of all configured functionality should be sufficient.

3. There should be a way to add a check for the provision scripts. I would use a top-level readiness block for that:

   provision:
     system: |
       #!/bin/bash
       curl -sfL https://get.k3s.io | sh -
   readiness:
     description: k3s kubeconfig to be created
     script: |
       #!/bin/bash
       set -eux -o pipefail
       if ! timeout 30s bash -c "until test -f /etc/rancher/k3s/k3s.yaml; do sleep 3; done"; then
               echo >&2 "k3s is not running yet"
               exit 1
       fi
     debug-hint: |
       The k3s kubeconfig file has not yet been created.
       Run "lima bash sudo journalctl -u k3s" to check the log.
       If that is still empty, check the bottom of the log at "/var/log/cloud-init-output.log".
   

Please let me know if I should implement any/all of these suggestions, either by amending this PR or by creating another one on top of it!

[AkihiroSuda]

Thanks a lot for working on this.

I think I made the requirements logic overly complicated, and would prefer to simplify it:

1. Generate the requirements dynamically in a method instead of having a static list. That way the whole logic for skipping requirements during checking becomes unnecessary, and the user will not see the potentially confusing "Skipping requirement..." message either.

SGTM

2. With the availability of bare-vm it makes less sense to treat nerdctl/containerd as "optional". A single "essential" requirements list of all configured functionality should be sufficient.

I'd rather prefer to have containerd structure as I suggested above.

3. There should be a way to add a check for the provision scripts. I would use a top-level readiness block for that:

   provision:
     system: |
       #!/bin/bash
       curl -sfL https://get.k3s.io | sh -
   readiness:
     description: k3s kubeconfig to be created
     script: |
       #!/bin/bash
       set -eux -o pipefail
       if ! timeout 30s bash -c "until test -f /etc/rancher/k3s/k3s.yaml; do sleep 3; done"; then
               echo >&2 "k3s is not running yet"
               exit 1
       fi
     debug-hint: |
       The k3s kubeconfig file has not yet been created.
       Run "lima bash sudo journalctl -u k3s" to check the log.
       If that is still empty, check the bottom of the log at "/var/log/cloud-init-output.log".
   

SGTM, but I guess we should make sure that we can have multiple "readiness" scripts. Probably we want to support "liveness" probes as well.

e.g.

probes:
- mode: readiness
  script: ...
- mode: readiness
  script: ...
- mode: liveness
  script: ...

Please let me know if I should implement any/all of these suggestions, either by amending this PR or by creating another one on top of it!

Either is fine to amend this PR or create another PR.

[jandubois]

@AkihiroSuda I believe I have implemented everything you suggested.

However, I was not sure how you would like to deploy the system-level containerd; should it also install buildkit and stargz?

I've not implemented liveness probes, mainly because I'm not sure what they are supposed to do when they fail.

Oh, and I kept essential and optional requirements separate, contrary to what I wrote earlier: I think it is better to mount directories after sshfs has been installed, so that the provisioning scripts can wait until the mounts are available, if they so desire.

[AkihiroSuda]

I believe I have implemented everything you suggested.

Thanks! 👍

However, I was not sure how you would like to deploy the system-level containerd; should it also install buildkit and stargz?

Yes, but that can be separate PR.

[AkihiroSuda]

Thanks, I'll merge after some testing

[AkihiroSuda]

=== RUN   TestTemplate
    template_test.go:23: assertion failed: error is not nil: template: :87: unexpected "/" in command
--- FAIL: TestTemplate (0.00s)
FAIL
FAIL	github.com/AkihiroSuda/lima/pkg/cidata	0.011s

[jandubois]

BTW, when you have multiple user-provisioning scripts, then each one will be preceded by the wait logic:

until [ -e "/run/user/{{.UID}}/systemd/private" ]; do sleep 3; done

This is redundant, but should be very fast. I found that extending the template to only generate it for the first script just makes the template less readable for very little gain.

[jandubois]

Just for the record, here is my k3s.yaml test file adapted to the settings now supported by this PR:

images:
- location: "~/Downloads/hirsute-server-cloudimg-amd64.img"
  arch: "x86_64"

mounts: []

ssh:
  localPort: 60022

containerd:
  system: false
  user: false

provision:
- mode: system
  script: |
    #!/bin/sh
    curl -sfL https://get.k3s.io | sh -

probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until test -f /etc/rancher/k3s/k3s.yaml; do sleep 3; done"; then
            echo >&2 "k3s is not running yet"
            exit 1
    fi
  hint: |
    The k3s kubeconfig file has not yet been created.
    Run "lima bash sudo journalctl -u k3s" to check the log.
    If that is still empty, check the bottom of the log at "/var/log/cloud-init-output.log".

[AkihiroSuda]

Thanks, can we have k3s.yaml in the examples directory?

[jandubois]

Thanks, can we have k3s.yaml in the examples directory?

Sure, do you want me to add it to this PR? Just the plain file, or does it need additional documentation beyond just a comment at the top?

[AkihiroSuda]

Sure, do you want me to add it to this PR?

Either is fine to me.

Just the plain file, or does it need additional documentation beyond just a comment at the top?

An additional documentation would be appreciated, but can be just a single or a few lines.