improve docs for attaching evaluator to dataset #837
+10
−32
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| @@ -2,7 +2,7 @@ | |||
| sidebar_position: 2 | |||
| --- | |||
|
|
|||
| # How to bind an evaluator to a dataset in the UI | |||
| # Automatically run evaluators on experiments | |||
|
|
|||
| While you can specify evaluators to grade the results of your experiments programmatically (see [this guide](./evaluate_llm_application) for more information), you can also bind evaluators to a dataset in the UI. | |||
There was a problem hiding this comment.
Suggested change
| While you can specify evaluators to grade the results of your experiments programmatically (see [this guide](./evaluate_llm_application) for more information), you can also bind evaluators to a dataset in the UI. | |
| You can grade your experiment results in two ways: **programmatically**, by specifying evaluators in your code (see [this guide](./evaluate_llm_application) for details) or you can **automatically run evaluators** defined in the UI. By binding evaluators to a dataset in the UI, they'll automatically run on your experiments. These UI-configured evaluators complement any you've set up via the SDK. We support both LLM-based and custom Python code evaluators. |
|
|
||
|  | ||
| Once you have saved your new evaluator, **subsequent** experiment run from the dataset will now be evaluated by the evaluator you configured. Note that in the below image, each run in the experiment has a "correctness" score. |
There was a problem hiding this comment.
Suggested change
| Once you have saved your new evaluator, **subsequent** experiment run from the dataset will now be evaluated by the evaluator you configured. Note that in the below image, each run in the experiment has a "correctness" score. | |
| Once you have created an evaluator, **subsequent** experiments on that dataset will be automatically graded by the evaluator(s) you configured. |
|
Would be good to add a "Next Steps" section at the end of the doc. Can link out to: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.