Skip to content

[WIP] support for OpenAI Responses API and image gen #8331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

arnavsinghvi11
Copy link
Collaborator

Supports OpenAI Responses API
This also supports image generation in DSPy through native calls through OpenAI Responses (Note that you can do this through gpt-image-1 or through models like gpt-4.1-nano, gpt-4.1-mini, gpt-4.1, etc. which call gpt-image-1` under the hood)

Example usage of DSPy image gen:

import base64

lm = dspy.LM(model = 'gpt-4.1-nano', model_type = 'responses', tools=[{
        "type": "image_generation",
        "quality": "low", 
        "size": "1024x1024", 
        "background": "opaque"
    }])
   
dspy.settings.configure(lm=lm)

class DogImageSignature(dspy.Signature):
    """Generate an image of a dog based on the given breed."""
    breed: str = dspy.InputField(desc="The breed of the dog")
    image: dspy.Image = dspy.OutputField(desc="A small image of the dog")

generate_dog = dspy.Predict(DogImageSignature)

response = generate_dog(breed="corgi")

img_obj = response.image

if hasattr(img_obj, "url") and img_obj.url.startswith("data:image/"):
    header, b64data = img_obj.url.split(",", 1)
    with open("corgi.png", "wb") as f:
        f.write(base64.b64decode(b64data))
    print("Image saved to corgi.png")

(For further reference on image gen parameters for OpenAI LMs)

try:
parsed = annotation.parse(output)
value = {output_field_name: parsed}
except Exception as e:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is related to the implicit warning that certain custom fields must be the only OutputField (aka can't have 2 Image OutputFields or 1 Image, 1 ToolCall OutputField) but maybe we can express this better? @chenmoneygithub @TomeHirata

@@ -41,10 +41,26 @@ def pretty_print_history(history, n: int = 1):
else:
image_str = f"<image_url: {c['image_url']['url']}>"
print(_blue(image_str.strip()))
elif c["type"] == "input_audio":
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this got lost in an earlier PR, adding back in

@@ -219,7 +219,7 @@ class MySignature(dspy.Signature):
assert user_message_content[2]["type"] == "text"

# Assert that the image is formatted correctly
expected_image_content = {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
expected_image_content = {"type": "input_image", "image_url": "https://example.com/image.jpg"}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all test related changes are for the refactoring of dspy.Image to handle the Responses Image format (the API strictly requires input_image)

@@ -374,3 +375,64 @@ async def test_async_lm_call_with_cache(tmp_path):
assert mock_alitellm_completion.call_count == 2

dspy.cache = original_cache

def make_response(output_blocks, *, model="openai/dspy-test-model", tools=None):
return ResponsesAPIResponse(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not setting all these params triggers pydantic_core._pydantic_core.ValidationError bc LiteLLM doesn't set defaults for them :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant