[Bug] Ollama requests fail when including an Image

[kmeehl]

What happened?

Hi! Very interesting project.

I'm trying to use dspy to do image classification. As a first step, I'd like to just generate a description of an image, using minicpm-v model on ollama.

The following example results in an error:

import dspy

class Describe(dspy.Signature):
    """Describe the image in detail. Respond only in English."""

    image: dspy.Image = dspy.InputField(desc="A photo")
    description: str = dspy.OutputField(desc="Detailed description of the image.")

image_path="/tmp/9221487.jpg"
minicpm = dspy.LM('ollama/minicpm-v:latest', api_base='http://localhost:11434', api_key='')

p = dspy.Predict(Describe)
p.set_lm(minicpm)
result = p(image=dspy.Image.from_url(image_path))
print(result.description)

Output:

2025/04/13 13:22:40 WARNING dspy.adapters.json_adapter: Failed to use structured output format. Falling back to JSON mode. Error: litellm.BadRequestError: Invalid Message passed in {'role': 'system', 'content': 'Your input fields are:\n1. `image` (Image): A photo\nYour output fields are:\n1. `description` (str): Detailed description of the image.\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\nInputs will have the following structure:\n\n[[ ## image ## ]]\n{image}\n\nOutputs will be a JSON object with the following fields.\n\n[[ ## description ## ]]\n{description}\nIn adhering to this structure, your objective is: \n        Describe the image in detail. Respond only in English.'}
Traceback (most recent call last):
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/chat_adapter.py", line 41, in __call__
    return super().__call__(lm, lm_kwargs, signature, demos, inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/base.py", line 33, in __call__
    outputs = lm(messages=inputs, **lm_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/utils/callback.py", line 266, in wrapper
    return fn(instance, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/base_lm.py", line 52, in __call__
    response = self.forward(prompt=prompt, messages=messages, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/utils/callback.py", line 266, in wrapper
    return fn(instance, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 112, in forward
    results = completion(
              ^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 268, in wrapper
    output = func_cached(key, request, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/cachetools/_decorators.py", line 94, in wrapper
    v = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 257, in func_cached
    return func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 282, in cached_litellm_completion
    return litellm_completion(
           ^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 301, in litellm_completion
    return litellm.completion(
           ^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/utils.py", line 1213, in wrapper
    raise e
  File "~/.local/lib/python3.12/site-packages/litellm/utils.py", line 1091, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/main.py", line 3093, in completion
    raise exception_type(
  File "~/.local/lib/python3.12/site-packages/litellm/main.py", line 2815, in completion
    response = base_llm_http_handler.completion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 239, in completion
    data = provider_config.transform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/llms/ollama/completion/transformation.py", line 315, in transform_request
    modified_prompt = ollama_pt(model=model, messages=messages)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/litellm_core_utils/prompt_templates/factory.py", line 265, in ollama_pt
    raise litellm.BadRequestError(
litellm.exceptions.BadRequestError: litellm.BadRequestError: Invalid Message passed in {'role': 'system', 'content': 'Your input fields are:\n1. `image` (Image): A photo\nYour output fields are:\n1. `description` (str): Detailed description of the image.\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## image ## ]]\n{image}\n\n[[ ## description ## ]]\n{description}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n        Describe the image in detail. Respond only in English.'}


.
.
.

Traceback (most recent call last):
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/json_adapter.py", line 67, in __call__
    return super().__call__(lm, lm_kwargs, signature, demos, inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/chat_adapter.py", line 49, in __call__
    raise e
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/chat_adapter.py", line 41, in __call__
    return super().__call__(lm, lm_kwargs, signature, demos, inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/base.py", line 33, in __call__
    outputs = lm(messages=inputs, **lm_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/utils/callback.py", line 266, in wrapper
    return fn(instance, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/base_lm.py", line 52, in __call__
    response = self.forward(prompt=prompt, messages=messages, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/utils/callback.py", line 266, in wrapper
    return fn(instance, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 112, in forward
    results = completion(
              ^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 268, in wrapper
    output = func_cached(key, request, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/cachetools/_decorators.py", line 94, in wrapper
    v = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 257, in func_cached
    return func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 282, in cached_litellm_completion
    return litellm_completion(
           ^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/clients/lm.py", line 301, in litellm_completion
    return litellm.completion(
           ^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/utils.py", line 1213, in wrapper
    raise e
  File "~/.local/lib/python3.12/site-packages/litellm/utils.py", line 1091, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/main.py", line 3093, in completion
    raise exception_type(
  File "~/.local/lib/python3.12/site-packages/litellm/main.py", line 2815, in completion
    response = base_llm_http_handler.completion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 239, in completion
    data = provider_config.transform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/llms/ollama/completion/transformation.py", line 315, in transform_request
    modified_prompt = ollama_pt(model=model, messages=messages)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/litellm/litellm_core_utils/prompt_templates/factory.py", line 265, in ollama_pt
    raise litellm.BadRequestError(
litellm.exceptions.BadRequestError: litellm.BadRequestError: Invalid Message passed in {'role': 'system', 'content': 'Your input fields are:\n1. `image` (Image): A photo\nYour output fields are:\n1. `description` (str): Detailed description of the image.\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\nInputs will have the following structure:\n\n[[ ## image ## ]]\n{image}\n\nOutputs will be a JSON object with the following fields.\n\n[[ ## description ## ]]\n{description}\nIn adhering to this structure, your objective is: \n        Describe the image in detail. Respond only in English.'}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/projects/1central/image_classifier/image.py", line 32, in <module>
    result = p(image=dspy.Image.from_url(image_path))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/utils/callback.py", line 266, in wrapper
    return fn(instance, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/predict/predict.py", line 77, in __call__
    return self.forward(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/predict/predict.py", line 107, in forward
    completions = adapter(
                  ^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/chat_adapter.py", line 50, in __call__
    return JSONAdapter()(lm, lm_kwargs, signature, demos, inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/lib/python3.12/site-packages/dspy/adapters/json_adapter.py", line 69, in __call__
    raise RuntimeError(
RuntimeError: Both structured output format and JSON mode failed. Please choose a model that supports `response_format` argument. Original error: litellm.BadRequestError: Invalid Message passed in {'role': 'system', 'content': 'Your input fields are:\n1. `image` (Image): A photo\nYour output fields are:\n1. `description` (str): Detailed description of the image.\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\nInputs will have the following structure:\n\n[[ ## image ## ]]\n{image}\n\nOutputs will be a JSON object with the following fields.\n\n[[ ## description ## ]]\n{description}\nIn adhering to this structure, your objective is: \n        Describe the image in detail. Respond only in English.'}

I've also tried the same code, replacing the model provider with ollama_chat:
minicpm = dspy.LM('ollama_chat/minicpm-v:latest', api_base='http://localhost:11434', api_key='')

This results in the same error.

Questions

• Are there any known limitations to dspy when using local LLMs with Ollama?
• Are there additional configurations, or alternate strategies I should try?
• Any tips or directions you can point me in for debugging this?

Additional Info
ollama version: 0.6.5
dspy version: 2.6.17

Steps to reproduce

Copy and run the example code above, changing image_path to point to a real image on your hard drive.

DSPy version

2.6.17

[okhat]

Try dspy.LM('ollama_chat/...') ?

[kmeehl]

Hi @okhat , I tried that. That's the second error output that I added above.

I actually thought they were different errors, but upon running it again, they appear to be the same. I'll edit my post to reflect that.

[arnavsinghvi11]

hey @kmeehl , do you get this error for all your datapoints, or does it only happen on some?

I believe most multimodal LLMs are not well adapted for structured JSON outputting (from our newly-updated JsonAdapter) all the time, so that's what's triggering the error. (I've been noticing this on meta-llama/Llama-3.2-11B-Vision-Instruct as well).

You can bypass this to ensure runs aren't halted by setting max_errors in dspy.Evaluate or any of the optimizer initialization configs to a high value. Ideally, applying optimizers will take care of this or at least minimize how many examples fail.

lmk if this helps!

[carvalho28]

Hey @kmeehl
I also had similar issues, for me a working solution is the following:

import dspy

class Describe(dspy.Signature):
    """Describe the image in detail. Respond only in English."""

    image: dspy.Image = dspy.InputField(desc="A photo")
    description: str = dspy.OutputField(desc="Detailed description of the image.")

image_path="image.png"
minicpm = dspy.LM('openai/gemma3:12b-it-qat', base_url='http://localhost:11454/v1', api_key='ollama', cache=False)

p = dspy.Predict(Describe)
p.set_lm(minicpm)
result = p(image=dspy.Image.from_url(image_path))
print(result.description)

Hope this works—don’t forget to swap in your own image path, model name, and port!

[kmeehl]

Thanks for the responses!

Hey @okhat , 'ollama_chat/...' results in the following error:
Client error '400 Bad Request' for url 'http://localhost:11434/api/chat'

Hey @carvalho28, I haven't tried a ton of different LLMs, but I have yet to see it work on any of the ones that I have tried.

Hey @carvalho28, I gave your solution a try, but it results in the same error:
litellm.BadRequestError: Invalid Message passed in {'role': 'system', 'content': 'Your input fields are:\n1. `image`...'

I have been able to get dspy talking to my LLM via ollama by bypassing the "standard" dspy way of doing it:

 minicpm = dspy.LM('ollama_chat/minicpm-v:latest', api_base='http://localhost:11434', api_key='')
img = Util.image_base64_uri(image_base_path)
image_detail_prompt = "Describe the image in detail. Respond only in English."
messages = [{"role": "user", "content": [{"type": "text", "text": image_detail_prompt}, {"type": "image_url", "image_url": {"url": img}} ]}]
detail = minicpm(messages=messages)
print(detail)

I believe this works because messages is formatted differently. Specifically, content is not a string, but an array of json objects. The error I'm getting from dspy shows that dspy is constructing content as just a string.