arduino-cli should provide "environments"

[s-celles]

Hello,

I wonder if the concept of "environment" (comparable to conda environments or Julia Pkg environments) have been considered (it could help to have several versions of a library installed without odd effects).

Here is a possible workflow for dealing with environment with arduino-cli :

Create an environment named myenv

$ arduino-cli env create myenv

It should create an empty directory name myenv in ~/Documents/Arduino/envs

List all existing environments

$ arduino-cli env list

Add packages to this environment

Latest version of a package

$ arduino-cli lib install WiFi101 --env myenv

A given version

$ arduino-cli lib install "ArduinoJson==5.13.4" --env myenv

(this will need #105 to be fixed)

(An environment could also be created directly from a file but this could probably be implemented later)

I assume core is installed in the global environment (because I think it's important to keep ability to have libraries and core installed in a shared manner)

$ arduino-cli compile --fqbn arduino:samd:mkr1000 Arduino/MyFirstSketch --env myenv

This last command will use ArduinoJson==5.13.4 even if, for example, ArduinoJson==6.x.y is installed in the "global" environment.

What is your opinion of such an idea?

Kind regards

[s-celles]

I assume core is installed in the global environment (because I think it's important to keep ability to have libraries and core installed in a shared manner)

Just an additional idea: libraries installed in an environment should "override" libraries installed in the global (shared) environment

[skandragon]

I faked this up using a Docker image per "environment" I want to use. This lets me be very explicit about what each build platform sees. It's a WIP, and not currently public, but if there is interest I can share.

I know others have similar base images, but for some reason I chose to make one from scratch.

[binaryben]

I'm not personally keen on environments, but having a system like the Node Package Manager (NPM) for managing libraries would be awesome. It could just use Github Repos to start with (no need to set up an official library registry... yet):

arduino-cli install --save @FastLED/FastLED

I would love to have a package.json file (or similar) though which lists required libraries, and then have the cli download them to the sketch directory as needed. You can also allow the user to install them globally like npm does, and that would fulfill @scls19fr's request.

EDIT:

Hold up. Libraries can be saved to the the ./src directory in a sketch directory. I could write a simple package manager cli utility that makes use of that fact. It would be a simple case of checking a package file and downloading the relevant code to ./src before calling arduino-cli separately. Is there a use case that having environments provides additional benefits that this possible solution wouldn't @scls19fr?

[masci]

+1 for using Docker, we're working on official releases for images shipping the CLI that should make this even easier.

You can already have something really close to an env by setting up ARDUINO_DATA_DIR and ARDUINO_SKETCHBOOK_DIR env vars to different values for different envs. You could even use Conda itself to manage such environments but any other dot-env system would work as well.

At this point we've no plan to add a sophisticated env manager with specific commands and stuff like global fallback but let's keep this issue open to collect feedback and alternative approaches.

[matthijskooijman]

#32 was closed, pointing to this issue for further discussion. The core of that issue was to be able to include libraries in the sketch directory, in order to make a sketch directory more self-contained and easier to distribute to others.

In this issue, a broader solution using environments is proposed, which serves a similar goal, but as @masci already says is quite a lot more sophisticated and thus less likely to be implemented.

Also, it seems that such an environment-bases approach needs more setup (e.g. it needs the user to explicitly create an environment and install libraries into it), while the simplier "libraries subdir in the sketch directory" approach is more implicit. The way I envison that, it only requires distributing library files along with the sketch, which will be used automatically, without any specific action required on the part of the user that compiles the sketch. Something similar could be achieved with an environment-based approach, but that does require some additional thought.

[federicobond]

I have been looking into this topic for a while in order to improve the tooling we use and get a nice reproducible environment for an Arduino project I am working on, and I think there is a key insight here:

You can already have something really close to an env by setting up ARDUINO_DATA_DIR and ARDUINO_SKETCHBOOK_DIR env vars to different values for different envs.

arduino-cli is 90% of the way there with regards to having usable separate environments. As long as you prefix each command with ARDUINO_SKETCHBOOK_DIR=., libraries are installed in ./libraries relative to the current directory and sketches are built using those libraries only. What would be really helpful is for the CLI to recognize that we are on a project laid out in such a way and set the sketchbook directory to the project root automatically.

This can be done by placing a special file in the project root directory. We could call this file project.json or similar. The most important thing this file does is to put the CLI into this new "project" mode so the sketchbook directory is configured appropriately and libraries are installed locally by default. We could also list those libraries and their versions in the file (so you can install them all easily by running a single command like arduino-cli install), set a default board fqbn (so you don't need to add the --fqbn arg to each compile program) and other default compile settings.

I think this solves a bunch of common problems previously reported by users and requires little change to the CLI. I would love to hear your opinions on it!