New feature: mbed cache #627

screamerbg · 2018-02-18T12:05:26Z

This feature aims to minimize traffic and reduce import times, by making Mbed CLI cache repositories as a default behavior. Caching is done via storing repository indexes under the Mbed CLI user config folder - typically ~/.mbed/mbed-cache/ on UNIX systems, or %userprofile%/.mbed/mbed-cache/ on Windows systems.

Compared to a fully checked out repository, indexes are significantly smaller in size and number of files, and contain the whole revision history of that repository. This allows Mbed CLI to quickly create copies of previously downloaded repository indexes and pull/fetch only the latest changes from the remote repositories, therefore dramatically reducing network traffic and download times, especially for big repositories like mbed-os.

Workflow
No impact to existing workflows.

This PR introduces caching as default behavior and also a new mbed cache sub-command for cache management:

mbed cache [on|off|dir <path>|ls|purge|-h|--help]

on - Turn repository caching on. Will use either the default or the user specified cache directory.
off - Turn repository caching off. Note that this doesn't purge cached repositories. See "purge".
dir - Set cache directory. Set to "default" to let mbed CLI determine the cache directory location. Typically this is ~/.mbed/mbed-cache/ on UNIX systems, or %%userprofile%%/.mbed/mbed-cache/ on Windows systems.
ls - List cached repositories and their size.
purge - Purge cached repositories. Note that this doesn't turn caching off.
-h or --help - Print cache command options.

If no sub-command is specified to mbed cache, then mbed CLI would print the current cache setting (ENABLED or DISABLED) and the path to the local cache directory.

For safety reasons, Mbed CLI will always use mbed-cache subfolder to a user specified location. This ensure that no user files will deleted during purge even if the user has specified root/system folder as a cache location (e.g. mbed cache dir / or mbed cache dir C:\).

Security notice: It's generally recommended to user cache location inside your own home directory. If you use cache location outside your user home/profile, then other system users might be able to access the repository cache and therefore the data of the cached repositories

How this works

Behind the scenes during mbed import or mbed add, Mbed CLI will check whether repository caching is enabled and whether a cache folder exists for that repository in the correct location. Location is determined based on URL, e.g.
https://github.com/ARMmbed/mbed-os-example-client will be cached in `~/.mbed/mbed-cache/github.com/ARMmbed/mbed-os-example-client' (the actual files of the repositories will not be checked out, just the index).

If repo index exists, Mbed CLI would make a carbon copy to the destination folder and try fetch for Git, or pull for Mercurial.

If fetch / pull succeeds, then Mbed CLI checks-outs the repository normally.
If fetch/pull fails, then it's highly likely that the remote repository has been rewritten (bad bad repo admin!). In that case Mbed CLI ignores the cached repo, wipes the copy and does a normal clone of the repository, and lastly "feeds" the new/fresh index to the cache.

Listing of cached repositories is done via mbed cache ls, e.g.

$ mbed cache ls

[mbed] Listing cached repositories in "/Users/mihsto01/.mbed"
* https://github.com/ARMmbed/atmel-rf-driver                            240.4KB
* https://github.com/ARMmbed/easy-connect                               112.3KB
* https://github.com/ARMmbed/esp8266-driver                             161.6KB
* https://github.com/ARMmbed/mbed-client                                  4.3MB
* https://github.com/ARMmbed/mbed-client-c                                5.0MB
* https://github.com/ARMmbed/mbed-client-classic                        436.7KB
* https://github.com/ARMmbed/mbed-client-mbed-tls                       296.8KB
* https://github.com/ARMmbed/mbed-os                                    196.6MB
* https://github.com/ARMmbed/mbed-os-example-client                       8.1MB
* https://github.com/ARMmbed/mcr20a-rf-driver                            93.3KB
* https://github.com/ARMmbed/pal                                          5.6MB
* https://github.com/ARMmbed/stm-spirit1-rf-driver                      384.8KB
* https://github.com/ARMmbed/wifi-x-nucleo-idw01m1                      285.1KB
* https://github.com/ARMmbed/wizfi310-driver                            119.9KB
-------------------------------------------------------------------------------
Total size:                                                             221.8MB

Documentation
Documentation is included with this PR. There's a new section called "Repository caching".

@AnotherButler please review

Tests
Not yet

CC @sg- @theotherjimmy @janjongboom
Note that due to the additionally stored files, there might be impact to CI systems @JanneKiiskila @studavekar @0xc0170 @adbridge.

…tion

Enable repo caching as default behavior

…stency Also improve help (a lot)

screamerbg · 2018-02-18T12:25:02Z

Comparison between first import and subsequent imports with caching enabled. The first import is heavily dependent on internet speed, and in the case below mbed-os was downloaded with 1.7 Megabytes per second.

Prep

$ mbed cache purge

First import

$ time mbed import mbed-os-example-client
[mbed] Importing program "mbed-os-example-client" from "https://github.com/ARMmbed/mbed-os-example-client" at latest revision in the current branch
[mbed] Adding library "easy-connect" from "https://github.com/ARMmbed/easy-connect" at rev #06594ba91bc1
[mbed] Adding library "easy-connect/atmel-rf-driver" from "https://github.com/ARMmbed/atmel-rf-driver" at rev #ca9782e68f5f
[mbed] Adding library "easy-connect/esp8266-driver" from "https://github.com/ARMmbed/esp8266-driver" at rev #b0d79dad507d
[mbed] Adding library "easy-connect/mcr20a-rf-driver" from "https://github.com/ARMmbed/mcr20a-rf-driver" at rev #93661a696735
[mbed] Adding library "easy-connect/stm-spirit1-rf-driver" from "https://github.com/ARMmbed/stm-spirit1-rf-driver" at rev #ce9e2f81f95f
[mbed] Adding library "easy-connect/wifi-x-nucleo-idw01m1" from "https://github.com/ARMmbed/wifi-x-nucleo-idw01m1" at rev #257d0878561b
[mbed] Adding library "easy-connect/wizfi310-driver" from "https://github.com/ARMmbed/wizfi310-driver" at rev #e78ec79cf496
[mbed] Adding library "mbed-client" from "https://github.com/ARMmbed/mbed-client" at rev #ea04c5de7822
[mbed] Adding library "mbed-client/mbed-client-c" from "https://github.com/ARMmbed/mbed-client-c" at rev #ecfa619e42b2
[mbed] Adding library "mbed-client/mbed-client-classic" from "https://github.com/ARMmbed/mbed-client-classic" at rev #4e66929607c3
[mbed] Adding library "mbed-client/mbed-client-mbed-tls" from "https://github.com/ARMmbed/mbed-client-mbed-tls" at rev #7e1b6d815038
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #569159b784f7
[mbed] Adding library "pal" from "https://github.com/ARMmbed/pal" at rev #60ce64d5ec35
                                                                                
real	2m15.357s
user	0m18.448s
sys	0m10.882s

Downloaded ~221MB with average speed 1.7MB/s

Second import

$ rm -fr mbed-os-example-client

$ time mbed import mbed-os-example-client
[mbed] Importing program "mbed-os-example-client" from "https://github.com/ARMmbed/mbed-os-example-client" at latest revision in the current branch
[mbed] Adding library "easy-connect" from "https://github.com/ARMmbed/easy-connect" at rev #06594ba91bc1
[mbed] Adding library "easy-connect/atmel-rf-driver" from "https://github.com/ARMmbed/atmel-rf-driver" at rev #ca9782e68f5f
[mbed] Adding library "easy-connect/esp8266-driver" from "https://github.com/ARMmbed/esp8266-driver" at rev #b0d79dad507d
[mbed] Adding library "easy-connect/mcr20a-rf-driver" from "https://github.com/ARMmbed/mcr20a-rf-driver" at rev #93661a696735
[mbed] Adding library "easy-connect/stm-spirit1-rf-driver" from "https://github.com/ARMmbed/stm-spirit1-rf-driver" at rev #ce9e2f81f95f
[mbed] Adding library "easy-connect/wifi-x-nucleo-idw01m1" from "https://github.com/ARMmbed/wifi-x-nucleo-idw01m1" at rev #257d0878561b
[mbed] Adding library "easy-connect/wizfi310-driver" from "https://github.com/ARMmbed/wizfi310-driver" at rev #e78ec79cf496
[mbed] Adding library "mbed-client" from "https://github.com/ARMmbed/mbed-client" at rev #ea04c5de7822
[mbed] Adding library "mbed-client/mbed-client-c" from "https://github.com/ARMmbed/mbed-client-c" at rev #ecfa619e42b2
[mbed] Adding library "mbed-client/mbed-client-classic" from "https://github.com/ARMmbed/mbed-client-classic" at rev #4e66929607c3
[mbed] Adding library "mbed-client/mbed-client-mbed-tls" from "https://github.com/ARMmbed/mbed-client-mbed-tls" at rev #7e1b6d815038
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #569159b784f7
[mbed] Adding library "pal" from "https://github.com/ARMmbed/pal" at rev #60ce64d5ec35

real	0m20.927s
user	0m4.017s
sys	0m4.932s

Downloaded ~1.2MB

janjongboom

I can't comment on the code, but I applaud the idea.

sg-

👍

screamerbg · 2018-02-22T16:29:12Z

Bump @theotherjimmy @JanneKiiskila @studavekar @0xc0170 @adbridge.

JanneKiiskila · 2018-02-22T16:35:00Z

Haven't had time to review the code, but the idea is superb.

JanneKiiskila · 2018-02-22T16:37:19Z

mbed/mbed.py

+@subcommand('cache',
+    dict(name='on', nargs='?', help='Turn repository caching on. Will use either the default or the user specified cache directory.'),
+    dict(name='off', nargs='?', help='Turn repository caching off. Note that this doesn\'t purge cached repositories. See "purge".'),
+    dict(name='dir', nargs='?', help='Set cache directory. Set to "default" to let mbed CLI determine the cache directory location. Typically this is "~/.mbed/mbed-cache/" on UNIX, or "%%userprofile%%/.mbed/mbed-cache/" on Windows.'),


Is it possible to actually print out the folder it WOULD really use rather than guess? Do we know at this point or can we use just %s and substitute it in?

Is this what you mean?

$ mbed cache [mbed] Repository cache is ENABLED. [mbed] Cache location "/Users/mihsto01/.mbed"

Note that the actual folder is /Users/mihsto01/.mbed/mbed-cache

I think Janne is asking about the help. so mbed cache -h prints something OS + user specific.

Pushed minor patch to support this.

studavekar · 2018-02-22T17:02:00Z

Looks good 👍

theotherjimmy · 2018-02-22T17:19:56Z

README.md

@@ -851,6 +851,30 @@ You can combine the options of the Mbed update command for the following scenari

 Use these with caution because your uncommitted changes and unpublished libraries cannot be restored.

+## Repository caching
+
+To minimize traffic and reduce import times, by default Mbed CLI would cache repositories by storing their indexes under the Mbed CLI user config folder - typically `~/.mbed/mbed-cache/` on UNIX systems, or `%userprofile%/.mbed/mbed-cache/` on Windows systems. Compared to a fully checked out repository, indexes are significantly smaller in size and number of files, and contain the whole revision history of that repository. This allows Mbed CLI to quickly create copies of previously downloaded repository indexes and pull/fetch only the latest changes from the remote repositories, therefore dramatically reducing network traffic and download times, especially for big repositories like `mbed-os`.


Tense is weird here. Could you make it all active?

would cache -> caches

theotherjimmy · 2018-02-22T17:21:59Z

README.md

+mbed cache [on|off|dir <path>|ls|purge|-h|--help]
+```
+
+ * `on` - Turn repository caching on. Will use either the default or the user specified cache directory.


The second sentence needs a subject.

"The cache will either be in the user specified location or the default location if the user has not specified a location"

theotherjimmy · 2018-02-22T17:24:44Z

README.md

+ * `purge` - Purge cached repositories. Note that this doesn't turn caching off.
+ * `-h` or `--help` - Print cache command options.
+
+If no sub-command is specified to `mbed cache`, then mbed CLI would print the current cache setting (ENABLED or DISABLED) and the path to the local cache directory.


tense: would print -> prints

How is that different compared to https://github.com/ARMmbed/mbed-cli/blob/master/README.md#compiler-detection-through-the-path. E.g. "If none of the above are configured, the mbed compile command will fall back to checking your PATH for an...". Are you suggesting that the tense should be corrected in the that part of the documentation as well? Note that you can find other occurrences of this all over the mbed CLI docs.

I think that we should probably correct the tense throughout the document. That should come as another PR then.

theotherjimmy · 2018-02-22T17:26:39Z

README.md

+
+If no sub-command is specified to `mbed cache`, then mbed CLI would print the current cache setting (ENABLED or DISABLED) and the path to the local cache directory.
+
+For safety reasons, Mbed CLI will always use `mbed-cache` subfolder to a user specified location. This ensure that no user files will deleted during `purge` even if the user has specified root/system folder as a cache location (e.g. `mbed cache dir /` or `mbed cache dir C:\`).


I think that the first sentence is hard to understand.
Proposal:
"For safty reasons, Mbed CLI always uses a sub-directory, mbed-cache, within the user specified cache location.

theotherjimmy · 2018-02-22T17:27:28Z

README.md

+
+For safety reasons, Mbed CLI will always use `mbed-cache` subfolder to a user specified location. This ensure that no user files will deleted during `purge` even if the user has specified root/system folder as a cache location (e.g. `mbed cache dir /` or `mbed cache dir C:\`).
+
+**Security notice**: It's generally recommended to use cache location inside your profile home directory. If you use cache location outside your user home/profile, then other system users might be able to access the repository cache and therefore the data of the cached repositories


There's no subject on the first sentence. Who is recommending the cache location? Probably the Mbed CLI developers or something like that.

theotherjimmy

Looks good. The cache command function could be broken up into sub commands as functions, and I would find that more readable.

The reported cache location will now contain "mbed-cache" additional sub-folder.

adbridge

LGTM

Copy edit for minor grammar nits.

screamerbg · 2018-03-13T14:27:26Z

All stakeholders approved this PR. Will be merged soon (just fixing CI issues).

screamerbg · 2018-03-14T10:08:15Z

@JanneKiiskila @studavekar Notice that this feature will turn caching on by default. On a CI system you might want to turn caching off as it will cache every PR repository. It's done by simply running mbed cache off while logged with the same user as the CI process.

screamerbg added 8 commits February 13, 2018 11:54

Introduce 'mbed cache' command to allow cache management and manipula…

835debb

…tion

Add 'mbed cache ls' implementation and related functions

11645c0

Polish 'mbed cache ls' output

7e6d7cc

Move default cache dir to user home/.mbed/

c476875

Enable repo caching as default behavior

Change mbed cache --dir <path> to mbed cache dir <path> for consi…

a314b33

…stency Also improve help (a lot)

Add Repository caching section to documentation

49cceeb

Minor fix to README.md

2a1f07d

Minor fix to README.md

f4b9270

screamerbg requested review from sg-, theotherjimmy, janjongboom, JanneKiiskila, AnotherButler and studavekar February 18, 2018 12:05

Remove cache from configuration help in README.md

172b5ca

janjongboom approved these changes Feb 19, 2018

View reviewed changes

sg- approved these changes Feb 19, 2018

View reviewed changes

JanneKiiskila reviewed Feb 22, 2018

View reviewed changes

JanneKiiskila approved these changes Feb 22, 2018

View reviewed changes

studavekar approved these changes Feb 22, 2018

View reviewed changes

theotherjimmy reviewed Feb 22, 2018

View reviewed changes

theotherjimmy approved these changes Feb 22, 2018

View reviewed changes

screamerbg added 2 commits February 23, 2018 18:21

Fix documentation and leftover print

c5a0d01

mbed cache -h will now print the default location as well.

b728558

The reported cache location will now contain "mbed-cache" additional sub-folder.

adbridge approved these changes Feb 27, 2018

View reviewed changes

Copy edit README.md

3966ab4

Copy edit for minor grammar nits.

AnotherButler approved these changes Mar 1, 2018

View reviewed changes

AnotherButler mentioned this pull request Mar 1, 2018

Add new content to cli-create.md ARMmbed/mbed-os-5-docs#426

Merged

screamerbg merged commit 3a979d9 into ARMmbed:master Mar 13, 2018


		If no sub-command is specified to `mbed cache`, then mbed CLI would print the current cache setting (ENABLED or DISABLED) and the path to the local cache directory.

		For safety reasons, Mbed CLI will always use `mbed-cache` subfolder to a user specified location. This ensure that no user files will deleted during `purge` even if the user has specified root/system folder as a cache location (e.g. `mbed cache dir /` or `mbed cache dir C:\`).


		For safety reasons, Mbed CLI will always use `mbed-cache` subfolder to a user specified location. This ensure that no user files will deleted during `purge` even if the user has specified root/system folder as a cache location (e.g. `mbed cache dir /` or `mbed cache dir C:\`).

		Security notice: It's generally recommended to use cache location inside your profile home directory. If you use cache location outside your user home/profile, then other system users might be able to access the repository cache and therefore the data of the cached repositories

New feature: mbed cache #627

New feature: mbed cache #627

Uh oh!

Conversation

screamerbg commented Feb 18, 2018

Uh oh!

screamerbg commented Feb 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janjongboom left a comment

Choose a reason for hiding this comment

Uh oh!

sg- left a comment

Choose a reason for hiding this comment

Uh oh!

screamerbg commented Feb 22, 2018

Uh oh!

JanneKiiskila commented Feb 22, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

studavekar commented Feb 22, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

adbridge left a comment

Choose a reason for hiding this comment

Uh oh!

screamerbg commented Mar 13, 2018

Uh oh!

screamerbg commented Mar 14, 2018

Uh oh!

Uh oh!

screamerbg commented Feb 18, 2018 •

edited

Loading