Webui: Enable communication with parent html (if webui is in iframe): #11940

igardev · 2025-02-18T08:35:41Z

Listens for "setText" command from parent with "text" and "context" fields. "text" is set in inputMsg, "context" is used as hidden context on the following requests to llama.cpp server
On pressing an Escape button sends command "escapePressed" to the parent

Example handling from the parent html side:

Send command "setText" from parent html to webui in iframe:
const iframe = document.getElementById('askAiIframe');
if (iframe) {
iframe.contentWindow.postMessage({ command: 'setText', text: text, context: context }, '*');
}
Listen for Escape key from webui on parent html:
// Listen for escape key event from iframe
if (command === 'escapePressed') {
// Process case when escape is pressed in webui
};

Make sure to read the contributing guidelines before submitting a PR

I need this change for llama.vscode extension (code completion). The goal is to show inside VS Code a window with Webui and send from VS Code the project context to Webui (and also set the selected text, if any, in inputMsg).

- Listens for "setText" command from parent with "text" and "context" fields. "text" is set in inputMsg, "context" is used as hidden context on the following requests to the llama.cpp server - On pressing na Escape button sends command "escapePressed" to the parent Example handling from the parent html side: - Send command "setText" from parent html to webui in iframe: const iframe = document.getElementById('askAiIframe'); if (iframe) { iframe.contentWindow.postMessage({ command: 'setText', text: text, context: context }, '*'); } - Listen for Escape key from webui on parent html: // Listen for escape key event in the iframe window.addEventListener('keydown', (event) => { if (event.key === 'Escape') { // Process case when Escape is pressed inside webui } });

ngxson · 2025-02-18T09:15:11Z

examples/server/webui/src/utils/storage.ts

@@ -21,6 +21,7 @@ const dispatchConversationChange = (convId: string) => {
 const db = new Dexie('LlamacppWebui') as Dexie & {
  conversations: Table<Conversation>;
  messages: Table<Message>;
+  extraContext: string;


I think this should be stored in app.context.tsx instead of here. The DB class is meant to store permanent data.

Also, you only define the type here, not the initial value. extraContext can de undefined.

extraContext is in some sense permanent as it is reused in each request to llama.cpp server. However, you are right - it is better to move it in app.context.

ngxson · 2025-02-18T09:15:58Z

examples/server/webui/src/utils/storage.ts

+  async setExtraContext(extraContext: string): Promise<void> {
+    db.extraContext = extraContext
+  },
+  async getExtraContext(): Promise<string> {


Either define a default value, or explicitly tell that it is undefine-able:

Suggested change

async getExtraContext(): Promise<string> {

async getExtraContext(): Promise<string | undefined> {

As extraContext will be moved in app.context, this function will be removed.

ngxson · 2025-02-18T09:17:47Z

examples/server/webui/src/utils/app.context.tsx

+      let extraContext:string = await StorageUtils.getExtraContext()
+      if (extraContext && extraContext != ""){
+        // insert extra context just after the systemMessage
+        messages.splice(config.systemMessage.length === 0 ? 0 : 1, 0, { role: 'user', content:extraContext } as APIMessage)


This may not be a good idea. Some models ignore system prompt. It should be injected into user message instead.

I think this puts the extra context message as a user message just before the other user messages. Is there something wrong?

ngxson · 2025-02-18T09:29:35Z

Btw, I'm having a plan to introduce a set of extraElements into the Message object. This will allow storing text, file, thought process, etc into the message. I think that would fit better into what you're trying to do here, as you can just put the extraContent as file content.

igardev · 2025-02-18T12:16:35Z

Message

OK, if it is not a problem, let's introduce this change now and when you update the Message class we could change it. It is not a big update.

ngxson · 2025-02-18T13:47:54Z

I'll push the new Message object to this PR, I'm working on it. This can also allow you to specify the file name for the input content.

ngxson · 2025-02-18T14:20:51Z

@igardev Here is how it looks now, can you test it please?

window.postMessage({ command: 'setText', text: 'Spot the syntax error', context: 'def test()\n  return 123' }, '*');

examples/server/webui/src/utils/llama-vscode.ts

igardev

Thank you for the improvements!
One remark - the extraContext could be huge and changes slowly. If we send the complete extraContext with every message the conversation size will increase quickly.
Let's merge the current implementation and continue thinking for a better one.

examples/server/webui/src/utils/llama-vscode.ts

igardev · 2025-02-18T18:56:23Z

examples/server/webui/src/utils/misc.ts

+    for (const extra of msg.extra ?? []) {
+      if (extra.type === 'textFile') {
+        // TODO: allow user to customize this via Settings
+        newContent += `\n===\nExtra file: ${extra.name}\n${extra.content}\n\n===\n`;


The extraContext is not one file, but a combination of many:

"Extra file: " could confuse the model. I would suggest to remove it.

Removed in 4bbd48d

examples/server/webui/src/components/ChatMessage.tsx

ngxson · 2025-02-18T19:38:20Z

One remark - the extraContext could be huge and changes slowly. If we send the complete extraContext with every message the conversation size will increase quickly.

extraContext is only attached to the first message of the conversation and will stay there. It won't be added to the second message sent from user. I imagine this should be the same with other platforms like chatgpt / GH copilot / etc

If we completely remove the extra from the conversation, it will make a bad UX because user may ask follow up questions about the same code.

…n iframe) (ggml-org#11940) * Webui: Enable communication with parent html (if webui is in iframe): - Listens for "setText" command from parent with "text" and "context" fields. "text" is set in inputMsg, "context" is used as hidden context on the following requests to the llama.cpp server - On pressing na Escape button sends command "escapePressed" to the parent Example handling from the parent html side: - Send command "setText" from parent html to webui in iframe: const iframe = document.getElementById('askAiIframe'); if (iframe) { iframe.contentWindow.postMessage({ command: 'setText', text: text, context: context }, '*'); } - Listen for Escape key from webui on parent html: // Listen for escape key event in the iframe window.addEventListener('keydown', (event) => { if (event.key === 'Escape') { // Process case when Escape is pressed inside webui } }); * Move the extraContext from storage to app.context. * Fix formatting. * add Message.extra * format + build * MessageExtraContext * build * fix display * rm console.log --------- Co-authored-by: igardev <[email protected]> Co-authored-by: Xuan Son Nguyen <[email protected]>

igardev requested a review from ngxson as a code owner February 18, 2025 08:35

github-actions bot added examples server labels Feb 18, 2025

ngxson reviewed Feb 18, 2025

View reviewed changes

Move the extraContext from storage to app.context.

42eb6c4

Fix formatting.

cd32872

add Message.extra

1920cc9

ngxson reviewed Feb 18, 2025

View reviewed changes

examples/server/webui/src/utils/llama-vscode.ts Outdated Show resolved Hide resolved

format + build

e7ec817

igardev commented Feb 18, 2025

View reviewed changes

ngxson added 2 commits February 18, 2025 20:35

MessageExtraContext

4bbd48d

build

8ddb353

ngxson added 2 commits February 18, 2025 20:42

fix display

96e2d46

rm console.log

ff378aa

ngxson approved these changes Feb 18, 2025

View reviewed changes

ngxson merged commit b58934c into ggml-org:master Feb 18, 2025
6 checks passed

	async getExtraContext(): Promise<string> {
	async getExtraContext(): Promise<string \| undefined> {

Webui: Enable communication with parent html (if webui is in iframe): #11940

Webui: Enable communication with parent html (if webui is in iframe): #11940

Uh oh!

Conversation

igardev commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson commented Feb 18, 2025

Uh oh!

igardev commented Feb 18, 2025

Uh oh!

ngxson commented Feb 18, 2025

Uh oh!

ngxson commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

igardev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngxson commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igardev commented Feb 18, 2025 •

edited

Loading

ngxson commented Feb 18, 2025 •

edited

Loading

ngxson commented Feb 18, 2025 •

edited

Loading