-
Notifications
You must be signed in to change notification settings - Fork 1.9k
[WIP] support for OpenAI Responses API and image gen #8331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
try: | ||
parsed = annotation.parse(output) | ||
value = {output_field_name: parsed} | ||
except Exception as e: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is related to the implicit warning that certain custom fields must be the only OutputField (aka can't have 2 Image OutputFields or 1 Image, 1 ToolCall OutputField) but maybe we can express this better? @chenmoneygithub @TomeHirata
@@ -41,10 +41,26 @@ def pretty_print_history(history, n: int = 1): | |||
else: | |||
image_str = f"<image_url: {c['image_url']['url']}>" | |||
print(_blue(image_str.strip())) | |||
elif c["type"] == "input_audio": |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this got lost in an earlier PR, adding back in
@@ -219,7 +219,7 @@ class MySignature(dspy.Signature): | |||
assert user_message_content[2]["type"] == "text" | |||
|
|||
# Assert that the image is formatted correctly | |||
expected_image_content = {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} | |||
expected_image_content = {"type": "input_image", "image_url": "https://example.com/image.jpg"} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all test related changes are for the refactoring of dspy.Image to handle the Responses Image format (the API strictly requires input_image
)
@@ -374,3 +375,64 @@ async def test_async_lm_call_with_cache(tmp_path): | |||
assert mock_alitellm_completion.call_count == 2 | |||
|
|||
dspy.cache = original_cache | |||
|
|||
def make_response(output_blocks, *, model="openai/dspy-test-model", tools=None): | |||
return ResponsesAPIResponse( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not setting all these params triggers pydantic_core._pydantic_core.ValidationError
bc LiteLLM doesn't set defaults for them :(
Supports OpenAI Responses API
This also supports image generation in DSPy through native calls through OpenAI Responses (Note that you can do this through
gpt-image-1
or through models likegpt-4.1-nano
,gpt-4.1-mini,
gpt-4.1, etc. which call
gpt-image-1` under the hood)Example usage of DSPy image gen:
(For further reference on image gen parameters for OpenAI LMs)