[safetensors parser] RE_SAFETENSORS_SHARD_FILE

[mishig25]

const RE_SAFETENSORS_SHARD_FILE = /\d{5}-of-\d{5}\.safetensors$/;

is this regex good to detect shard file based on safetensors filename ?

[julien-c]

looks good (cc @Wauplin @LysandreJik on the filename convention)

Actually we could spec HF-wide that sharded files always have this pattern, given it's the one also used by datasets library for sharded files as was just confirmed to me by @lhoestq

[lhoestq]

FYI sharded files from datasets have a similar pattern, e.g.

train-00000-of-00004.parquet
train-00001-of-00004.parquet
train-00002-of-00004.parquet
train-00003-of-00004.parquet

BUT this is not the case for many datasets uploaded manually that can use arbitrary patterns, e.g.

0001.tar
0002.tar
0003.tar

part0.json
part1.json
part2.json

[Wauplin]

is this regex good to detect shard file based on safetensors filename ?

Correct. Though it's not used yet, harmonization for model sharding is happening here.

[Wauplin]

FYI sharded files from datasets have a similar pattern, e.g.

train-0000-of-0004.parquet
train-0001-of-0004.parquet
train-0002-of-0004.parquet
train-0003-of-0004.parquet

In your example, @lhoestq the numbers are on 4 digits, @mishig25's regex has 5.

[julien-c]

anyways not always the case for models either: https://huggingface.co/hpcai-tech/grok-1/tree/main

[lhoestq]

In your example, @lhoestq the numbers are on 4 digits, @mishig25's regex has 5.

oops it's actually 5 digits as well, sorry (edited my message)