Updating MIPROv2 Auto Settings #8165
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
klopsahlong
commented
May 3, 2025
| @@ -349,7 +349,7 @@ def propose_instructions_for_program( | |||
|
|
|||
| # Create an instruction for each predictor | |||
| for pred_i, predictor in enumerate(program.predictors()): | |||
| for demo_set_i in range(num_demos): | |||
| for demo_set_i in range(num_demos)[:min(N, num_demos)]: | |||
There was a problem hiding this comment.
only create up to N new instructions, not up to num_demos instructions (since now num_demos != N)
klopsahlong
commented
May 3, 2025
| "heavy": {"num_trials": 50, "val_size": 1000}, | ||
| "light": {"n": 6, "val_size": 100}, | ||
| "medium": {"n": 12, "val_size": 300}, | ||
| "heavy": {"n": 18, "val_size": 1000}, |
There was a problem hiding this comment.
we now pin auto run settings to the number of candidates we want to evaluate in each setting, not the number of trials we're willing to spend
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the MIPROv2 auto settings based on intuition we've formed on ideal run settings from past experiments.
The new auto settings are calculated by pinning the number of candidates we want to explore for each model in
light,medium,heavy, modes ton=6,n=12,n=18respectively. To calculate the number of trials needed for proper exploration of each candidate, we use the following formula:trials = max(c*M*log(n), c=2, 3/2*n), where M is the number of modules in a program. In non zero-shot settings, when both few-shot examples and instructions are being optimized, we set the number of candidate few-shot examples ton, and the number of candidate instructions ton/2.The impacts of this change will generally be to ensure that
lightmode explores enough candidates to make an impact on performance, and to generally decrease the # of candidates explored inheavymode, which we found was often unnecessarily high before.