feat: allow tool calling to respond without a tool #2614
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
drbh
changed the title
drbh
force-pushed
the
identify-function-name-while-streaming-and-unroll
branch
from
7bd57e6 to
fcc822f
Compare
Narsil
reviewed
Oct 8, 2024
router/src/server.rs
Outdated
Comment on lines
469
to
470
| let event = Event::default(); | ||
| yield Ok(event); |
There was a problem hiding this comment.
Why ? You're changing functionality here.
There was a problem hiding this comment.
great catch, Thanks. I've updated the logic to emit an event in both cases of a token or an error (to match the existing functionality).
drbh
force-pushed
the
identify-function-name-while-streaming-and-unroll
branch
from
d8ef7ac to
7d2aa27
Compare
Narsil
previously approved these changes
Oct 9, 2024
Narsil
reviewed
Oct 9, 2024
| print(content_generated) | ||
| assert ( | ||
| content_generated | ||
| == "There is no weather related function available to answer your prompt" |
There was a problem hiding this comment.
yea I agree, I think its because the model is technically responding to the error field in the notify_error tool, so the text is more of an error text than an actual response. Going to make a couple changes to see if I can minimize the issue
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is an improved version of #2597 and handles processing streaming responses as well as non streaming generations. This PR unrolls tool responses that do not select one of the user provided tools.
This implementation parses the first few tokens of generations requests with user provided tools to determine if a tool was selected. If no tool was selected the response is parsed and returned as a normal
contentvalue.This PR also supersedes draft PR #2606 as this approach has more benefits and less downsides