Skip to content

Commit 8a7252d

Browse files
xaedesolexiyb
authored andcommitted
ggml : fix backward rope after YaRN (ggml-org#3974)
* fix backward process of rope rope backward process was broken after YaRN RoPE (ggml-org#2268) implementation, due to missing changes in backward functions. the code for the backward process is nearly identically to the forward process: the only difference is the sign of the sin-values. to avoid future regressions remove the near-duplicate backward functions and reuse the forward code: for this a new function argument `bool forward` was added to `ggml_compute_forward_rope_f32` and `ggml_compute_forward_rope_f16`. the sin-values will be negated when forward is false. * fix finetune rope call to use correct default attn_factor of 1.0f * remove unused `ggml_rope_xpos_back` it is better to have only one `ggml_rope_back` function that accepts all rope parameters, so that `ggml_compute_backward` can propagate all parameters without having to switch between different rope_back variants. * fix comments explaining the sinus sign in ggml_forward_rope * add missing function arguments in declaration * fix function argument type in declaration
1 parent d3e96f8 commit 8a7252d

File tree

3 files changed

+84
-253
lines changed

3 files changed

+84
-253
lines changed

examples/finetune/finetune.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -643,7 +643,7 @@ static struct ggml_tensor * llama_build_lora_finetune_graphs(
643643

644644
return ggml_rope_custom(ctx,
645645
t, KQ_pos, n_rot, rope_mode, n_ctx, 0,
646-
rope_freq_base, rope_freq_scale, 0.0f, 0.0f, 0.0f, 0.0f
646+
rope_freq_base, rope_freq_scale, 0.0f, 1.0f, 0.0f, 0.0f
647647
);
648648
};
649649

0 commit comments

Comments
 (0)