Skip to content

[Executorch][llm] Enable leveraging ring kv cache via module swap #10835

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 13, 2025
Merged

[Executorch][llm] Enable leveraging ring kv cache via module swap #10835

merged 1 commit into from
May 13, 2025

Conversation

pytorchbot
Copy link
Collaborator

This PR was created by the merge bot to help merge the original PR into the main branch.
ghstack PR number: #10611 by @kimishpatel
^ Please use this as the source of truth for the PR details, comments, and reviews
ghstack PR base: https://github.com/pytorch/executorch/tree/gh/kimishpatel/188/base
ghstack PR head: https://github.com/pytorch/executorch/tree/gh/kimishpatel/188/head
Merge bot PR base: https://github.com/pytorch/executorch/tree/gh/kimishpatel/187/orig
Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/kimishpatel/188/orig
@diff-train-skip-merge

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented May 13, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10835

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 1 New Failure, 8 Pending

As of commit 084bfe2 with merge base 0b231c4 (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 13, 2025
@kimishpatel kimishpatel force-pushed the gh/kimishpatel/187/orig branch from bfcaf89 to b994a1a Compare May 13, 2025 14:31
Base automatically changed from gh/kimishpatel/187/orig to main May 13, 2025 16:15
@kimishpatel
[Executorch][llm] Enable leveraging ring kv cache via module swap
084bfe2 
Pull Request resolved: #10611

This allows us to make some of the attention modules to use sliding window kv cache. Will help enable models like gemma3.
ghstack-source-id: 283404677

Differential Revision: [D73891426](https://our.internmc.facebook.com/intern/diff/D73891426/)
@kimishpatel kimishpatel force-pushed the gh/kimishpatel/188/orig branch from ed32169 to 084bfe2 Compare May 13, 2025 16:17
@github-actions GitHub Actions
Copy link

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:.

If not, please add the release notes: none label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@kimishpatel kimishpatel merged commit ef30b25 into main May 13, 2025
91 of 92 checks passed
@kimishpatel kimishpatel deleted the gh/kimishpatel/188/orig branch May 13, 2025 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kimishpatel kimishpatel kimishpatel approved these changes

@lucylq lucylq Awaiting requested review from lucylq lucylq is a code owner

@jackzhxng jackzhxng Awaiting requested review from jackzhxng jackzhxng is a code owner

@iseeyuan iseeyuan Awaiting requested review from iseeyuan

@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820

@swolchok swolchok Awaiting requested review from swolchok

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pytorchbot @kimishpatel @facebook-github-bot