Skip to content

Commit 5b687ce

Browse files
committed
Update on "[llava][21/N] Add llava runner test binary and build script"
Add a `main.cpp` and CMakeLists.txt for llava runner. This runner takes in an image in the format of `.pt` (a serialized pytorch model) along with text prompt. It will generate text tokens in a way similar to llama runner. Run `build.sh` to build the runner. To serialize the image into a `.pt` file, run the following script: ```python import torch from torch import nn copy = torch.tensor(resized) m = nn.Module() par = nn.Parameter(copy, requires_grad=False) m.register_parameter("0",par) tensors = torch.jit.script(m) tensors.save("image.pt") ``` To run the runner, use the following command: ``` cmake-out/examples/models/llava/llava_main \ --tokenizer_path tokenizer.bin \ --model_path llava_kv_768.pte \ --prompt "\nWhat are the things I should be cautious about when I visit here?" \ --image_path image.pt \ --temperature 0 ``` Differential Revision: [D61146432](https://www.internalfb.com/diff/D61146432) [ghstack-poisoned]
2 parents 3809535 + 7462a20 commit 5b687ce

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

.ci/scripts/test_llava.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ run_and_verify() {
7979
RESULT=$(cat result.txt)
8080
# set the expected prefix to be the same as prompt because there's a bug in sdpa_with_kv_cache that causes <unk> tokens.
8181
EXPECTED_PREFIX="ASSISTANT:"
82-
if [[ "${RESULT}" == "${EXPECTED_PREFIX}"* ]]; then
82+
if [[ "${RESULT}" == *"${EXPECTED_PREFIX}"* ]]; then
8383
echo "Expected result prefix: ${EXPECTED_PREFIX}"
8484
echo "Actual result: ${RESULT}"
8585
echo "Success"

0 commit comments

Comments
 (0)