Skip to content

add alias for lora adaptors #8636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

add alias for lora adaptors #8636

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions include/llama.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -443,6 +443,10 @@ extern "C" {
struct llama_model * model,
struct llama_context_params params);

LLAMA_API void llama_ctx_switch_adaptor(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the correct usage of lora adapter in llama.cpp

The adapter is loaded independently from inference context (llama_context). To control which adapter is enabled or not, use llama_lora_adapter_set and llama_lora_adapter_remove

See: #8332

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the correct usage of lora adapter in llama.cpp

The adapter is loaded independently from inference context (llama_context). To control which adapter is enabled or not, use llama_lora_adapter_set and llama_lora_adapter_remove

See: #8332

Thanks for replying.
If the adaptors are expected to set/remove outside of the library. It should not be loaded while initialized the model.

IMO, the adaptors can be handled outsize the library, but the exist adaptors are unknown and invisiable. Need explicity way to clean current adaptors. It would be better to have llama_lora_adaptors_clean to clean exist adaptors.
In real scenarios, apply multiple adaptors on the foundation model is not common(let me know if there are cases). Apply one adaptor on foundation model is a common partten. And when setting a adaptor, usually we need clean exist adaptors outside of the library, but the exist adaptors are invisiable.

Copy link
Collaborator

@ngxson ngxson Jul 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the adaptors are expected to set/remove outside of the library. It should not be loaded while initialized the model.

I think there's small misconception here. llama.cpp comes with 2 main interfaces: llama_context and llama_model. When you load the model, tensors are stored in llama_model as read-only. If you want to use these tensors for inference, you need to pass the model to llama_context. This makes it possible to re-use one llama_model in multiple llama_context

The same idea applies to llama_lora_adapter. When adapter is loaded, you can reuse it in multiple different llama_context. Imagine one context can use adapter for task A and the other can use adapter for task B.

The list of active adapters is held by each llama_context (with the corresponding scales)

When you want to remove adapter from llama_context, use llama_lora_adapter_remove. To free an adapter (i.e. remove it completely from memory), use llama_lora_adapter_free. The adapters are also free if llama_model is freed, because adapter is linked to specific model.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the adaptors are expected to set/remove outside of the library. It should not be loaded while initialized the model.

I think there's small misconception here. llama.cpp comes with 2 main interfaces: llama_context and llama_model. When you load the model, tensors are stored in llama_model as read-only. If you want to use these tensors for inference, you need to pass the model to llama_context. This makes it possible to re-use one llama_model in multiple llama_context

The same idea applies to llama_lora_adapter. When adapter is loaded, you can reuse it in multiple different llama_context. Imagine one context can use adapter for task A and the other can use adapter for task B.

The list of active adapters is held by each llama_context (with the corresponding scales)

When you want to remove adapter from llama_context, use llama_lora_adapter_remove. To free an adapter (i.e. remove it completely from memory), use llama_lora_adapter_free. The adapters are also free if llama_model is freed, because adapter is linked to specific model.

After lctx, model loaded outside of the library:
std::tie(model, ctx) = llama_init_from_gpt_params(params);
The ctx handles the adaptors and invisible outside library scope. The inference context allocate the memory for computation graph calculation. The efficient way is reused the same context.
Or the adaptors have to be handled outside the library and prevent load any adaptors within the library scope.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, the adaptors can be handled outsize the library, but the exist adaptors are unknown and invisiable. Need explicity way to clean current adaptors. It would be better to have llama_lora_adaptors_clean to clean exist adaptors.

I agree that common use case is one adapter per context, and llama_lora_adaptors_clean maybe useful to detach all adapters from a given context. But IMO llama.cpp should only provides most low-level API and user can build their own wrapper logic on top of it. Much like how operating systems only provide most basic syscalls, and libraries like glibc build on top of these syscalls.

In your case, user can simply write a llama_lora_adapter_switch(llama_lora_adapter * ptr) that automatically detach old adapter using llama_lora_adapter_remove, then attach the new one with llama_lora_adapter_set. It could be implement as a utility (see common folder).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, the adaptors can be handled outsize the library, but the exist adaptors are unknown and invisiable. Need explicity way to clean current adaptors. It would be better to have llama_lora_adaptors_clean to clean exist adaptors.

I agree that common use case is one adapter per context, and llama_lora_adaptors_clean maybe useful to detach all adapters from a given context. But IMO llama.cpp should only provides most low-level API and user can build their own wrapper logic on top of it. Much like how operating systems only provide most basic syscalls, and libraries like glibc build on top of these syscalls.

In your case, user can simply write a llama_lora_adapter_switch(llama_lora_adapter * ptr) that automatically detach old adapter using llama_lora_adapter_remove, then attach the new one with llama_lora_adapter_set. It could be implement as a utility (see common folder).

Agree adding a llama_lora_adaptors_clear to clean exist adaptors. If this cleaning functionality is visible outside of the library’s scope, users can manipulate the adaptors outside of the library to customize them.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please have a look on #8653

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After lctx, model loaded outside of the library:
std::tie(model, ctx) = llama_init_from_gpt_params(params);
The ctx handles the adaptors and invisible outside library scope.

llama_init_from_gpt_params is not part of the library, it's only part of the examples. 3rd party applications should not rely on it.

struct llama_context* ctx,
const char* adaptor_alias);

// Frees all allocated memory
LLAMA_API void llama_free(struct llama_context * ctx);

Expand Down
33 changes: 31 additions & 2 deletions src/llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2851,6 +2851,8 @@ struct llama_lora_weight {

struct llama_lora_adapter {
struct llama_model * base_model;
std::string alias;
bool enabled = true;
// map tensor name to lora_a_b
std::unordered_map<std::string, struct llama_lora_weight> ab_map;
std::vector<struct ggml_context *> ctxs;
Expand Down Expand Up @@ -7892,6 +7894,9 @@ static struct ggml_tensor * llm_build_lora_mm(
struct ggml_tensor * cur) {
struct ggml_tensor * res = ggml_mul_mat(ctx0, w, cur);
for (auto & it : lctx.lora_adapters) {
if(!it.first->enabled) {
continue;
}
struct llama_lora_weight * lora = it.first->get_weight(w);
if (lora == nullptr) {
continue;
Expand All @@ -7918,6 +7923,9 @@ static struct ggml_tensor * llm_build_lora_mm_id(
struct ggml_tensor * ids) {
struct ggml_tensor * res = ggml_mul_mat_id(ctx0, w, cur, ids);
for (auto & it : lctx.lora_adapters) {
if (!it.first->enabled) {
continue;
}
struct llama_lora_weight * lora = it.first->get_weight(w);
if (lora == nullptr) {
continue;
Expand Down Expand Up @@ -18604,7 +18612,7 @@ static void llama_model_quantize_internal(const std::string & fname_inp, const s
}

static void llama_lora_adapter_init_internal(struct llama_model * model, const char * path_lora, struct llama_lora_adapter & adapter) {
LLAMA_LOG_INFO("%s: loading lora adapter from '%s' ...\n", __func__, path_lora);
LLAMA_LOG_INFO("%s: loading lora adapter from '%s' as '%s' ...\n", __func__, path_lora, adapter.alias.c_str());

ggml_context * ctx = nullptr;
struct gguf_init_params meta_gguf_params = {
Expand Down Expand Up @@ -19404,6 +19412,18 @@ struct llama_context * llama_new_context_with_model(
return ctx;
}

void llama_ctx_switch_adaptor(struct llama_context* ctx, const char* adaptor_alias) {
llama_synchronize(ctx);
for (auto& adaptor : ctx->lora_adapters) {
if (adaptor.first->alias == adaptor_alias) {
adaptor.first->enabled = true;
}
else {
adaptor.first->enabled = false;
}
}
}

void llama_free(struct llama_context * ctx) {
delete ctx;
}
Expand Down Expand Up @@ -19615,8 +19635,17 @@ uint32_t llama_model_quantize(

struct llama_lora_adapter * llama_lora_adapter_init(struct llama_model * model, const char * path_lora) {
try {
std::string alias;
std::string path = path_lora;
struct llama_lora_adapter * adapter = new llama_lora_adapter(model);
llama_lora_adapter_init_internal(model, path_lora, *adapter);
size_t pos = path.find("=");
if (pos != std::string::npos) {
alias = path.substr(0, pos);
path = path.substr(pos + 1);
adapter->alias = alias;
}

llama_lora_adapter_init_internal(model, path.c_str(), *adapter);
return adapter;
} catch (const std::exception & err) {
LLAMA_LOG_ERROR("%s: failed to apply lora adapter: %s\n", __func__, err.what());
Expand Down