|
1 | 1 | # Chat with LLMs Everywhere
|
2 |
| -Torchchat is a small codebase to showcase running large language models (LLMs) within Python OR within your own (C/C++) application on mobile (iOS/Android), desktop and servers. |
| 2 | +Torchchat is a compact codebase to showcase the capability of running large language models (LLMs) seamlessly across diverse platforms. With Torchchat, you could run LLMs from with Python, your own (C/C++) application on mobile (iOS/Android), desktop or servers. |
3 | 3 |
|
4 | 4 | ## Highlights
|
5 | 5 | - Command line interaction with popular LLMs such as Llama 3, Llama 2, Stories, Mistral and more
|
6 |
| - - Supporting [some GGUF files](docs/GGUF.md) and the Hugging Face checkpoint format |
| 6 | + - Supports [common GGUF formats](docs/GGUF.md) and the Hugging Face checkpoint format |
7 | 7 | - PyTorch-native execution with performance
|
8 | 8 | - Supports popular hardware and OS
|
9 | 9 | - Linux (x86)
|
@@ -59,16 +59,16 @@ with `python3 torchchat.py remove llama3`.
|
59 | 59 | * [Chat](#chat)
|
60 | 60 | * [Generate](#generate)
|
61 | 61 | * [Run via Browser](#browser)
|
62 |
| -* [Quantizing your model (suggested for mobile)](#quantization) |
| 62 | +* [Quantize your models (suggested for mobile)](#quantization) |
63 | 63 | * Export and run models in native environments (C++, your own app, mobile, etc.)
|
64 |
| - * [Exporting for desktop/servers via AOTInductor](#export-server) |
65 |
| - * [Running exported .so file via your own C++ application](#run-server) |
| 64 | + * [Export for desktop/servers via AOTInductor](#export-server) |
| 65 | + * [Run exported .so file via your own C++ application](#run-server) |
66 | 66 | * in Chat mode
|
67 | 67 | * in Generate mode
|
68 |
| - * [Exporting for mobile via ExecuTorch](#export-executorch) |
| 68 | + * [Export for mobile via ExecuTorch](#export-executorch) |
69 | 69 | * in Chat mode
|
70 | 70 | * in Generate mode
|
71 |
| - * [Running exported executorch file on iOS or Android](#run-mobile) |
| 71 | + * [Run exported ExecuTorch file on iOS or Android](#run-mobile) |
72 | 72 |
|
73 | 73 | ## Models
|
74 | 74 | These are the supported models
|
@@ -242,7 +242,7 @@ python3 torchchat.py export stories15M --output-pte-path stories15M.pte
|
242 | 242 | python3 torchchat.py generate --device cpu --pte-path stories15M.pte --prompt "Hello my name is"
|
243 | 243 | ```
|
244 | 244 |
|
245 |
| -See below under Mobile Execution if you want to deploy and execute a model in your iOS or Android app. |
| 245 | +See below under [Mobile Execution](#run-mobile) if you want to deploy and execute a model in your iOS or Android app. |
246 | 246 |
|
247 | 247 |
|
248 | 248 | ## Quantization
|
|
0 commit comments