Skip to content

Commit d5c9151

Browse files
authored
Update evaluation.md to include AOTI
Update evaluation.md to include running tests with AOTI
1 parent e0ce144 commit d5c9151

File tree

1 file changed

+28
-3
lines changed

1 file changed

+28
-3
lines changed

torchchat/utils/docs/evaluation.md

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,24 +21,49 @@ library.
2121

2222
The evaluation mode of `torchchat.py` script can be used to evaluate your language model on various tasks available in the `lm_eval` library such as "wikitext". You can specify the task(s) you want to evaluate using the `--tasks` option, and limit the evaluation using the `--limit` option. If no task is specified, the task will default to evaluating on "wikitext".
2323

24-
**Examples**
24+
## Examples
25+
26+
**Python**
2527

2628
Running wikitext for 10 iterations
2729
```
2830
python3 torchchat.py eval stories15M --tasks wikitext --limit 10
2931
```
3032

31-
Running an exported model
33+
Running wikitext with torch.compile for 10 iterations
34+
```
35+
python3 torchchat.py eval stories15M --compile --tasks wikitext --limit 10
36+
```
37+
38+
Running multiple tasks and calling eval.py directly (with torch.compile):
39+
```
40+
python3 torchchat.py eval stories15M --compile --tasks wikitext hellaswag
41+
```
42+
43+
**exported PTE**
44+
Running an exported model as PTE
3245
```
3346
python3 torchchat.py export stories15M --output-pte-path stories15M.pte
3447
python3 torchchat.py eval stories15M --pte-path stories15M.pte
3548
```
3649

37-
Running multiple tasks and calling eval.py directly:
50+
Running multiple tasks and calling eval.py directly (with PTE):
3851
```
3952
python3 torchchat.py eval stories15M --pte-path stories15M.pte --tasks wikitext hellaswag
4053
```
4154

55+
**exported AOTI**
56+
Running an exported model as PTE
57+
```
58+
python3 torchchat.py export stories15M --dtype fast16 --output-dso-path stories15M.so
59+
python3 torchchat.py eval stories15M --dtype fast16 --pte-path stories15M.so
60+
```
61+
62+
Running multiple tasks and calling eval.py directly (with AOTI):
63+
```
64+
python3 torchchat.py eval stories15M --dso-path stories15M.so --tasks wikitext hellaswag
65+
```
66+
4267
For more information and a list of tasks/metrics see [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).
4368

4469
[end default]: end

0 commit comments

Comments
 (0)