Sending an image to server that serves LLava

[rezacopol]

Hi,

I tried to follow the instruction on the server readme to host LLava, it says it support openAI format but there is slight difference after I read the readme. My goal is access the model similar to openAI sever so later I can replace my gpt4 calls with a local model I host with minimal code changes (changing base_url).

I thought to post this to learn how to do it and also I think it's useful for others with same question. Here is the code I have to encode and image and sending to server

client = OpenAI(base_url = "http://localhost:8082", api_key="no key", timeout=60)
# Function to encode a local image into data URL found on MSFT website and works with OAI GPT
def local_image_to_data_url(image_path):
    # Guess the MIME type of the image based on the file extension
    mime_type, _ = guess_type(image_path)
    if mime_type is None:
        mime_type = 'application/octet-stream'  # Default MIME type if none is found

    # Read and encode the image file
    with open(image_path, "rb") as image_file:
        base64_encoded_data = base64.b64encode(image_file.read()).decode('utf-8')

    # Construct the data URL

    #return f"data:{mime_type};base64,{base64_encoded_data}" ## this works for open AI. tried it and doesn't work
    return f"{base64_encoded_data}" ## i thought sending this might be better

# Example usage
image_path = <local_image_path>
data_url = local_image_to_data_url(image_path)
system_prompt = "You are a helpful AI chatbot"
user_prompt = "[img-12] Describe the image in detail." ## I also tried with <image> token
response = client.chat.completions.create(
        model="junk",
        messages=[
            {"role": "system",
             "content": system_prompt},
            {"role": "user", "content": user_prompt, "image_data": [{"data": data_url, "id": 12}]}
        ]
    )

I don't get any error but response has no clue about the image and it's pure hallucination by the LLM.

Couldn't find much info on the readme except formatting the code as above, anyone made it work with LLava can help me out on what am I doing wrong?

Here is what I see on server logs after sending the request in case it is helpful. I don't see any sign of 576 image tokens for the image by llava 1.5

INFO [                    init] initializing slots | tid="0x1eeece080" timestamp=1718215254 n_slots=1
INFO [                    init] new slot | tid="0x1eeece080" timestamp=1718215254 id_slot=0 n_ctx_slot=4096
INFO [                    main] model loaded | tid="0x1eeece080" timestamp=1718215254
INFO [                    main] chat template | tid="0x1eeece080" timestamp=1718215254 chat_example="<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHi there<|im_end|>\n<|im_start|>user\nHow are you?<|im_end|>\n<|im_start|>assistant\n" built_in=true
INFO [                    main] HTTP server listening | tid="0x1eeece080" timestamp=1718215254 port="8082" n_threads_http="23" hostname="127.0.0.1"
INFO [            update_slots] all slots are idle | tid="0x1eeece080" timestamp=1718215254
INFO [   launch_slot_with_task] slot is processing task | tid="0x1eeece080" timestamp=1718215287 id_slot=0 id_task=0
INFO [            update_slots] kv cache rm [p0, end) | tid="0x1eeece080" timestamp=1718215287 id_slot=0 id_task=0 p0=0
INFO [           print_timings] prompt eval time     =     506.42 ms /    63 tokens (    8.04 ms per token,   124.40 tokens per second) | tid="0x1eeece080" timestamp=1718215287 id_slot=0 id_task=0 t_prompt_processing=506.418 n_prompt_tokens_processed=63 t_token=8.038380952380953 n_tokens_second=124.40316102508204
INFO [           print_timings] generation eval time =     390.81 ms /    30 runs   (   13.03 ms per token,    76.76 tokens per second) | tid="0x1eeece080" timestamp=1718215287 id_slot=0 id_task=0 t_token_generation=390.815 n_decoded=30 t_token=13.027166666666666 n_tokens_second=76.76266264089148
INFO [           print_timings]           total time =     897.23 ms | tid="0x1eeece080" timestamp=1718215287 id_slot=0 id_task=0 t_prompt_processing=506.418 t_token_generation=390.815 t_total=897.233
INFO [            update_slots] slot released | tid="0x1eeece080" timestamp=1718215287 id_slot=0 id_task=0 n_ctx=4096 n_past=92 n_system_tokens=0 n_cache_tokens=0 truncated=false
INFO [            update_slots] all slots are idle | tid="0x1eeece080" timestamp=1718215287
INFO [      log_server_request] request | tid="0x16e30b000" timestamp=1718215287 remote_addr="127.0.0.1" remote_port=60701 status=200 method="POST" path="/chat/completions" params={}

[rezacopol]

Here is my code on how to run the server in case it is helpful
/server --port 8082 -m ../models/ggml_llava-v1.5-7b/ggml-model-q5_k.gguf --mmproj ../models/ggml_llava-v1.5-7b/mmproj-model-f16.gguf -c 4096 -ngl 100

[ElhamAhmedian]

Hello,
But can I use it like so:

 curl -X POST http://0.0.0.0:7860/predict \
>     -F "image=@path_to_your_image.jpg" \
>     -F "prompt=Your text prompt here"

I mean how can I send an image to a VLM?

[pnschumacher]

Any updates on this? @rezacopol