Make it easier to do sync streaming #8183
Conversation
| By default calling a streamified DSPy program produces an async generator. In order to get back | ||
| a sync generator, you can set the flag `async_streaming=False`: | ||
|
|
||
| ``` |
There was a problem hiding this comment.
| ``` | |
| ```python |
There was a problem hiding this comment.
Actually is this block necessary? It seems to be duplicated with the following example.
There was a problem hiding this comment.
ah thank you for catching it
| @@ -249,11 +249,11 @@ def __call__(self, x: str, **kwargs): | |||
| dspy.streaming.StreamListener(signature_field_name="judgement"), | |||
| ], | |||
| include_final_prediction_in_output_stream=False, | |||
| async_streaming=False, | |||
There was a problem hiding this comment.
Is this test run in the CI pipeline? It seems to require OPENAI_API_KEY
There was a problem hiding this comment.
it's not running in CI, but written for local checking. streaming behavior is a bit unreliable to verify with mock, and we should find a way to set up API key for github actions.
There was a problem hiding this comment.
I see, is it difficult to mock the LiteLLM call with a dummy generator? I agree testing with a real API can increase safety, but having a unit test without relying on external infra is also valuable.
There was a problem hiding this comment.
Mocking streaming is messy but feasible, the more concerning part is the assumption on stream tokens can go wrong, which could lead to false positive test. But I agree with you that we can add another test based on mock. Merging this one now, will follow up with the test shortly.
Introduce a bool flag
async_streamingtodspy.streamifyso that users can switch easily between sync and async streaming.