feat(hub): adding downloadFileToCacheDir
#1034
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
axel7083
changed the title
downloadFileToCacheDir
coyotte508
reviewed
Nov 15, 2024
coyotte508
reviewed
Nov 15, 2024
|
Thanks a lot, probably a |
Should be good now :) ! |
|
Can you find a way to skip the test in browser environment? (I can look into it otherwise) |
Yes, I excluded it from |
coyotte508
reviewed
Nov 18, 2024
Co-authored-by: Eliott C. <[email protected]>
|
Thanks!! |
1 task
coyotte508
added a commit
that referenced
this pull request
Nov 19, 2024
## Description We can now create a `snapshotDownload` method similator to the `snapshot_download` of the PY lib[^1], clone to the cache (only cache supported for now) a repository (either model, space or dataset) [^1]: https://huggingface.co/docs/huggingface_hub/en/guides/download#download-an-entire-repository ## Related issues/PR With the amazing help of @coyotte508 we were able to merge the following changes - #1034 - #1031 - #999 Which allow this PR to provide a python compliant clone of a hugging face repository to the cache directory. ## Testing - [x] unit tests are covering the new feature **Manually** ```ts await snapshotDownload({ repo: { name: 'OuteAI/OuteTTS-0.1-350M', type: 'model', }, }); ``` assert using the `huggingface-cli` tool (python) ``` $: huggingface-cli scan-cache REPO ID REPO TYPE SIZE ON DISK NB FILES LAST_ACCESSED LAST_MODIFIED REFS LOCAL PATH ----------------------------------- --------- ------------ -------- ----------------- ----------------- ---- ---------------------------------------------------------------------------------- OuteAI/OuteTTS-0.1-350M model 731.6M 14 5 minutes ago 5 minutes ago main /home/axel7083/.cache/huggingface/hub/models--OuteAI--OuteTTS-0.1-350M ``` --------- Co-authored-by: Eliott C. <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Following #1031 which added a
pathsInfomethod which can return the etag/commitHash for a given file. Allowing to be compliant with the_hf_hub_download_to_cache_dir1 method from the python library.Potential issue
The JS implementation do not handle the .lock files as the python library does.. This could be a problem if using the JS and PY function.. ?
The JS could make a basic implementation of the lock file that the PY lib is doing if this is a hard requirement.
Testing
I wrote tests for the existing
downloadFilefunction (no change to the implementation) and the new one addeddownloadFileToCacheDir.Footnotes
huggingface_hub/file_download.py#L882 ↩