Skip to content

Commit e0ce144

Browse files
mikekgfbJack-Khuu
andauthored
Update run-docs to enable run-docs evaluation (#1383)
* Update run-docs Enable evaluation tests from docs/evaluation.md * Update evaluation.md Install executorch for running test * Update evaluation.md wording * Update evaluation.md Avoid bleedthru of markup for test --------- Co-authored-by: Jack-Khuu <[email protected]>
1 parent dacabcd commit e0ce144

File tree

2 files changed

+6
-4
lines changed

2 files changed

+6
-4
lines changed

.ci/scripts/run-docs

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,9 +75,6 @@ if [ "$1" == "advanced" ]; then
7575
fi
7676

7777
if [ "$1" == "evaluation" ]; then
78-
79-
exit 0
80-
8178
echo "::group::Create script to run evaluation"
8279
python3 torchchat/utils/scripts/updown.py --file torchchat/utils/docs/evaluation.md --replace 'llama3:stories15M,-l 3:-l 2' --suppress huggingface-cli,HF_TOKEN > ./run-evaluation.sh
8380
# for good measure, if something happened to updown processor,

torchchat/utils/docs/evaluation.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,13 @@
44

55
# Evaluation Features
66

7+
<!--
8+
79
[shell default]: ./install/install_requirements.sh
810
11+
[shell default]: TORCHCHAT_ROOT=${PWD} ./torchchat/utils/scripts/install_et.sh
12+
13+
-->
914

1015
Torchchat provides evaluation functionality for your language model on
1116
a variety of tasks using the
@@ -14,7 +19,7 @@ library.
1419

1520
## Usage
1621

17-
The evaluation mode of `torchchat.py` script can be used to evaluate your language model on various tasks available in the `lm_eval` library such as "wikitext". You can specify the task(s) you want to evaluate using the `--tasks` option, and limit the evaluation using the `--limit` option. If no task is specified, it will default to evaluating on "wikitext".
22+
The evaluation mode of `torchchat.py` script can be used to evaluate your language model on various tasks available in the `lm_eval` library such as "wikitext". You can specify the task(s) you want to evaluate using the `--tasks` option, and limit the evaluation using the `--limit` option. If no task is specified, the task will default to evaluating on "wikitext".
1823

1924
**Examples**
2025

0 commit comments

Comments
 (0)