feat: Hybrid unified/recurrent cache #13276

gabe-l-hart · 2025-05-02T23:08:10Z

Description

This implementation covers both llama_memory_i and llama_kv_cache interfaces, but they could very well not be correct.

Discussion

I'm putting this up for discussion even though it doesn't have much value as standalone. My ultimate goal is support for the just-released granite 4 which is a combination of mamba2 and granitemoeshared layers. I opened #13275 to track the full scope of model architecture changes.

compilade

Awesome to see this progress!

compilade · 2025-05-02T23:14:35Z

src/llama-kv-cache.cpp

+    // TODO: Will it cause problems if some caches are able to remove the seq
+    //  but others aren't?


Yes it will cause problems if this breaks the coherency between caches. (e.g. part of a sequence is removed in one cache but not the other).

This is what I was referring to in #12799 (comment) when I wrote:

The hardest part will be handling errors and properly keeping coherency between the different types of caches (because they don't necessarily roll-back states in the same way).

I think the seq_rm API might fundamentally be too specific to self-attention KV cache. Recurrent models can't rollback their state, because intermediate states are not kept since keeping them for all tokens would take too much space. (when seq_rm returns false, it means the states have to be re-calculated from scratch for the affected sequence (at least that was the intention in #5328))

Ideally, if there was some API to create snapshots and rollback to them, the implementation would be simpler for recurrent models (and for hybrid models by extension). (technically, sequences (with seq_id) already kind of do this (and are copy-on-write), but snapshots within sequences might be more convenient to manage in user code, since managing which state is the latest per sequence could be done transparently)

But that would also mean having to manage the lifetime of explicit state snapshots (in examples/server/server.cpp among others) instead of directly dealing with ranges of token positions (and might make things like largest-common-prefix context caching harder to handle). I've previously shared some ideas about state snapshots/checkpoints in #7531 (comment) (although the first half of the comment is about session restore as in state_read).

Ok, interesting. I'm definitely still learning on-the-fly here, but based on this description and the logic here in server.cpp, it seems like the most correct implementation would be to leak implementation details of the child caches or introduce a new member API for can_seq_rm that is const but returns the same logic. I think I'll give that a shot and see how far I can get.

Ok, I've pushed an attempt at doing this safely. One thing I noticed is that these mutating methods don't seem to have any sort of locking mechanism, so the way I have it implemented could certainly be prone to thread safety problems if concurrent threads tried to call seq_rm. I don't think this is any different than for the current cache implementations since those would also be sensitive to the same races where the validated condition changes after validation but before the members get mutated, but I wanted to double check if this kind of thread safety is guarded against elsewhere (or just assumed to be handled in the client layer).

compilade · 2025-05-02T23:44:16Z

src/llama-kv-cache.cpp

+    // If any of the caches are recurrent, require simple split
+    return llama_sbatch(batch, m_hparams.n_embd, m_has_recurrent, logits_all);


Simple split should not be used with recurrent models, they expect equal split.

See #7531 (comment) which illustrates the splits

Suggested change

// If any of the caches are recurrent, require simple split

return llama_sbatch(batch, m_hparams.n_embd, m_has_recurrent, logits_all);

// If any of the caches are recurrent, require non-simple split

return llama_sbatch(batch, m_hparams.n_embd, !m_has_recurrent, logits_all);

Thanks for the comment pointer, this is super helpful for understanding what the consequences of these actually are!

compilade · 2025-05-02T23:45:24Z

src/llama-kv-cache.cpp

+    if (m_has_recurrent) {
+        return sbatch.split_simple(n_ubatch);
+    }


This will not work, recurrent models expect split_equal to be used.

Ok, I'm following now. I had them backwards in my head

compilade · 2025-05-02T23:47:49Z

src/llama-kv-cache.cpp

+    // TODO: Is this correct?
+    // If any children can shift, return true
+    for (const auto & cache : m_children) {
+        if (cache->get_can_shift()) {
+            return true;
+        }
+    }


Maybe this should be if all children can shift, then return true.

But as you've noticed elsewhere, can_shift should technically always be true for all currently-implemented cache types, so I don't know if that part of the API will stay anyway.

gabe-l-hart · 2025-05-14T20:16:56Z

@compilade I now have a proof-point (#13550) that this works to some extend, though I haven't tested it robustly for edge cases. There are a few additional changes I needed to make on that branch that should maybe come over to this branch, but it gets a little hairy because they interact with adding the actual model architectures. Some possible paths forward:

Review this one as-is and take the updates from the Granite 4 branch when they come
Try to move the changes to hparams to determine per-layer recurrence over to this branch, and then just have unreachable code when initializing the hybrid cache in llama-model.cpp
Scrap this PR and just review all hybrid cache changes with the incoming models in the Granite 4 branch

Thoughts / Preferences?

gabe-l-hart · 2025-05-16T17:32:34Z

I thought about it further and decided that the cleanest separation between this and the Granite 4 branch is to pull over the key parts of hparam parsing and kv cache instantiation and isolate the mismatch to the llm_arch_is_hybrid function which will always return false for now until there are real hybrid models.

gabe-l-hart · 2025-05-16T20:51:21Z

src/llama-model.h

@@ -402,7 +402,10 @@ struct llama_model {

    // note: can mutate `cparams`
    // TODO: move this to new llm_arch_model_i interface
-    llama_memory_i * create_memory(const llama_memory_params & params, llama_cparams & cparams) const;
+    llama_memory_i * create_memory(


I just realized that this is entirely unnecessary since llama_model already has hparams as a member variable 🤦

gabe-l-hart · 2025-05-19T13:45:29Z

@compilade @ggerganov I'm going to keep working on some unit tests for the different cache types and try to sort through the CI failure. In the mean time, would it be possible to get an initial review to make sure I'm on the right track?

gabe-l-hart · 2025-05-19T21:13:32Z

It looks like the failing tests were all on Windows and therefore fall into this note about not running tests using internal APIs on Windows due to the lack of LLAMA_API exports.

ggerganov · 2025-05-20T05:24:52Z

I think the hybrid implementation can become similar to the new iSWA implementation - mainly for consistency.

I would like to now simplify the llama_kv_cache interface, namely:

Try to eliminate set_full() completely. We can instead clear() + init batches with appropriate sizes
Rework the sbatch_init(), ubatch_next() and find_slot(). It seems that this should be possible to be replaced with a single init(llama_batch) call that would prepare all the ubatches beforehand

Will open a draft PR and we can discuss details there.

Adding tests would be really useful - we can merge these from a separate PR without too much delays.

gabe-l-hart · 2025-05-20T14:43:58Z

@ggerganov Thank you for pointing that out! I'll work to rebase this change on the latest iSWA changes. I'll also try to pull out the unit tests into their own PR.

gabe-l-hart · 2025-05-20T19:03:12Z

@ggerganov Ported PR with minimal unit tests: #13669

I didn't try to expand them to incorporate the iSWA cache yet, but will look to do that as I go through the changes there and refactor this hybrid cache branch.

gabe-l-hart · 2025-05-20T21:04:57Z

@ggerganov This branch is now rebuilt based on the iSWA cache. I kept most of the implementation of llama_kv_cache_hybrid the way it was where the sub-types of the child caches are not specified to hopefully make it easier to extend this to other per-layer hybrid styles in the future. Like before, I've plumbed through the instantiation of the hybrid cache to the point where any model that answers true to llm_arch_is_hybrid_recurrent will instantiate a hybrid cache in create_memory, but currently there are no models that fit that, so it will not be exercised until the next PR with the concrete hybrid architectures.

gabe-l-hart · 2025-05-20T22:26:41Z

src/llama-kv-cache.cpp

@@ -1898,6 +1916,11 @@ llama_kv_cache_recurrent::llama_kv_cache_recurrent(
    v_l.reserve(n_layer);

    for (int i = 0; i < n_layer; i++) {
+        if (filter && !filter(i)) {


This is buggy because push_back is used below, so later when we index directly into the given per-layer tensor vectors, the final layers will be out-of-bounds reads.

gabe-l-hart · 2025-05-20T22:41:24Z

I've updated the Granite 4 / Bamba branch (#13550) to test this out and found a couple of bugs that should now be fixed and functional.

… signatures Branch: HybridCache Signed-off-by: Gabe Goodhart <[email protected]>

gabe-l-hart · 2025-05-27T23:16:12Z

Converting to draft until #13746 is resolved

This follows the pattern in iswa where the two child caches are held explicitly to support the case where a model requires a single attention cache and a single recurrent cache where each layer uses exactly one of the caches. This is a rewrite of the more generic approach in the original hybrid cache PR: ggml-org#13276 Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]>

gabe-l-hart · 2025-05-29T22:17:54Z

I'm closing this PR for now in favor of #13904 which is a simpler approach that handles the hybrid attention/recurrent case explicitly instead of trying to handle all possible hybrid cache combinations in the abstract.

This follows the pattern in iswa where the two child caches are held explicitly to support the case where a model requires a single attention cache and a single recurrent cache where each layer uses exactly one of the caches. This is a rewrite of the more generic approach in the original hybrid cache PR: ggml-org#13276 Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]>

@younesbelkada

* feat: Add llama_model_is_hybrid API call Also, split llama_model_is_recurrent into llm_arch_is_recurrent in llama-arch with llama_model_is_recurrent delegating to llm_arch_is_recurrent. The same split is done for hybird. This is needed because there are places where the llama_model has not yet been initialized but we need to check if the model is recurrent (specifically for the per-layer recurrent check array in hparams). Branch: GraniteFour Signed-off-by: Gabe Goodhart <[email protected]> * feat: Add c++ side constants for attention layer indices hparam Branch: GraniteFour * feat: Add support for distinguishing recurrent vs non-recurrent layers in hparams Branch: GraniteFour Signed-off-by: Gabe Goodhart <[email protected]> * feat: Auto-fill hparams.recurrent_layer_arr based on whether the model is recurrent Branch: GraniteFour Signed-off-by: Gabe Goodhart <[email protected]> * refactor: rename *_is_hybrid -> *_is_hybrid_recurrent The implementation of the hybrid cache intentionally does not specify the types of the child caches, so there was a naming mismatch with these predicate functions that used "hybrid" to imply "hybrid recurrent." Branch: HybridCache Signed-off-by: Gabe Goodhart <[email protected]> * feat: Add layer filter to recurrent cache Branch: HybridCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Use per-layer sizing everywhere in kv caches Branch: GraniteFour Signed-off-by: Gabe Goodhart <[email protected]> * feat: First pass at llama_kv_cache_hybrid_recurrent This follows the pattern in iswa where the two child caches are held explicitly to support the case where a model requires a single attention cache and a single recurrent cache where each layer uses exactly one of the caches. This is a rewrite of the more generic approach in the original hybrid cache PR: #13276 Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * feat: Construct hybrid recurrent cache for hybrid recurrent models This includes a refactor of the create_memory logic to avoid needing to use the arch enum explicitly unless a model needs explicit cache instantiation logic beyond the standard logic for recurrent, hybrid, unified, and iswa. Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Fix wrong bool condition for split equal in hybrid cache Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Fix shift logic to defer to unified cache Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * feat: Support hybrid recurrent in llama-graph NOTE: I intentionally did not add support for s_mask since it will be going away soon Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Fix logic for initializing inputs and attn layers for hybrid caches Branch: GraniteFour Signed-off-by: Gabe Goodhart <[email protected]> * fix: Update recurrent cache for changes to remove intermediate kv_cache interface Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Fix status for init_update sig for recurrent cache state Branch: GraniteFour Signed-off-by: Gabe Goodhart <[email protected]> * fix: Add missing padding to n_ctx for hybrid cache construction Branch: GraniteFour Signed-off-by: Gabe Goodhart <[email protected]> * fix: Update clear signature for data argument after rebase Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Remove errant virtual destructor leftover from previous impl attempt Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Use per-layer n_embd_k/v_s calls for mamba (1) layers Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refactor: Remove n_embd_k/v_s from unified cache No longer needed now that unified isn't also supporting recurrent #13979 (comment) Branch: HybridRecurrentCache * refactor: Remove layer index from n_embd_k/v_s Now that it's not used at all in the unified cache, we don't need to use the layer index to zero it out for attention layers. Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refactor: Remove n_embd_k/v_gqa from recurrent cache This is no longer needed now that there are separate implementations #13979 (comment) Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * feat: Allow custom layer filters for hybrid recurrent This should help support architectures like Falcon H1 where there is overlap between layers that need attention and recurrent caches. #13979 (comment) Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Remove logits_all after rebase Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Remove llama_model_is_hybrid_Recurrent public API #13979 (comment) Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refactor: Use llama_memory_state_ptr for child states in hybrid memory state Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * feat: Overhaul build_recurrent_state / build_inp_s_copy to match attention pattern https://github.com/ggml-org/llama.cpp/pull/13979/files#r2141701738 This is a big overhaul to bring consistency between how inputs and per- layer components are created for attention layers and recurrent layers. The main changes are: - Rename class llm_graph_input_s_copy -> llm_graph_input_rs - Add a corresponding llm_graph_input_rs_hybrid_recurrent - Rename build_inp_s_copy -> build_rs_inp_recurrent - Add a corresponding build_rs_inp_hybrid_recurrent - Rename build_recurrent_state -> build_rs to match build_attn w/ llm_graph_input_rs android-build AUTHORS bamba-9b-2.2T.gguf bamba-9b-2.2T.q4_k_m.gguf broken.log build build-rel build-xcframework.sh build.android build.android.bak ci cmake CMakeLists.txt CMakePresets.json CODEOWNERS common common.o CONTRIBUTING.md convert_hf_to_gguf_update.py convert_hf_to_gguf.py convert_llama_ggml_to_gguf.py convert_lora_to_gguf.py debug.log docs examples flake.lock flake.nix ggml ggml-alloc.o ggml-backend.o ggml-metal.o ggml-model-BF16.gguf ggml-model-Q4_K_M.gguf ggml-quants.o ggml.o gguf-py grammar-parser.o grammars include LICENSE licenses llama.log llama.o llamacpp_trace.log main.log Makefile media models mypy.ini pocs poetry.lock prompts pyproject.toml pyrightconfig.json q4_k_m_boot.log q8_0_boot.log quant.log quant2.log README.md requirements requirements.txt sampling.o scripts SECURITY.md src test-grammar-output.tmp test-json-schema-input.tmp tests tools vendor working.log as the first input - Add a corresponding overload of build_rs w/ llm_graph_input_rs_hybrid_recurrent android-build AUTHORS bamba-9b-2.2T.gguf bamba-9b-2.2T.q4_k_m.gguf broken.log build build-rel build-xcframework.sh build.android build.android.bak ci cmake CMakeLists.txt CMakePresets.json CODEOWNERS common common.o CONTRIBUTING.md convert_hf_to_gguf_update.py convert_hf_to_gguf.py convert_llama_ggml_to_gguf.py convert_lora_to_gguf.py debug.log docs examples flake.lock flake.nix ggml ggml-alloc.o ggml-backend.o ggml-metal.o ggml-model-BF16.gguf ggml-model-Q4_K_M.gguf ggml-quants.o ggml.o gguf-py grammar-parser.o grammars include LICENSE licenses llama.log llama.o llamacpp_trace.log main.log Makefile media models mypy.ini pocs poetry.lock prompts pyproject.toml pyrightconfig.json q4_k_m_boot.log q8_0_boot.log quant.log quant2.log README.md requirements requirements.txt sampling.o scripts SECURITY.md src test-grammar-output.tmp test-json-schema-input.tmp tests tools vendor working.log as the first input - Add a llm_graph_input_attn_kv_hybrid_recurrent analogous to llm_graph_input_attn_kv_unified - Add a build_attn override that takes llm_graph_input_attn_kv_hybrid_recurrent android-build AUTHORS bamba-9b-2.2T.gguf bamba-9b-2.2T.q4_k_m.gguf broken.log build build-rel build-xcframework.sh build.android build.android.bak ci cmake CMakeLists.txt CMakePresets.json CODEOWNERS common common.o CONTRIBUTING.md convert_hf_to_gguf_update.py convert_hf_to_gguf.py convert_llama_ggml_to_gguf.py convert_lora_to_gguf.py debug.log docs examples flake.lock flake.nix ggml ggml-alloc.o ggml-backend.o ggml-metal.o ggml-model-BF16.gguf ggml-model-Q4_K_M.gguf ggml-quants.o ggml.o gguf-py grammar-parser.o grammars include LICENSE licenses llama.log llama.o llamacpp_trace.log main.log Makefile media models mypy.ini pocs poetry.lock prompts pyproject.toml pyrightconfig.json q4_k_m_boot.log q8_0_boot.log quant.log quant2.log README.md requirements requirements.txt sampling.o scripts SECURITY.md src test-grammar-output.tmp test-json-schema-input.tmp tests tools vendor working.log as the first input This makes the two paradigms fully consistent. The main drawback is the code duplication in the build_attn and build_rs implementations where the only difference between implementations is how they cast the memory state. Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * fix: Fix resize vs reserve and skip null tensors in size computation https://github.com/ggml-org/llama.cpp/pull/13979/files#r2149469788 Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> Co-Authored-By: @younesbelkada * fix: Fix initialization of child states Since initially writing this PR, the logic in the child state types changed such that using the "init full" signature and keeping the ubatches on the parent struct no longer worked. Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refactor: Use a common build_recurrent_state method that is cache-agnostic This reduces the code duplication between the different build_rs impls and also retains a similar signature to the previous build_recurrent_state method while standardizing on the input-dispatched build_rs implementation. Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * recurrent : rework graph inputs + add TODOs ggml-ci * refactor: Make status and child states const in hybrid and iswa Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refactor: Rename llama_kv_cache_[recurrent|hybrid_recurrent] to remove kv cache This removes the notion of "kv" from the interface names for these memory types. There are still many references to kv in the implementation of the recurrent memory which will need further adjustment. Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refactor!: Rename all k/v related values for recurrent/hybrid to r/s Anywhere that "kv_<state|cell|size|etc>" is used, I've used the more generic "mem_" prefix. The specifics of "k" (key) translate to "r" (recurrent state) and "v" (value) translate to "s" (state-space embedding states). Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refacor: _recurrent -> _recr for brevity It just _happens_ to have the same number of letters as _attn! Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * style: Fix spacing for ref Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * refactor: recurrent_layer() -> is_recurrent() Branch: HybridRecurrentCache Signed-off-by: Gabe Goodhart <[email protected]> * style: Fix spacing for size_s_bytes declaration Co-authored-by: Georgi Gerganov <[email protected]> --------- Signed-off-by: Gabe Goodhart <[email protected]> Co-authored-by: Georgi Gerganov <[email protected]>

compilade reviewed May 3, 2025

View reviewed changes

gabe-l-hart force-pushed the HybridCache branch from e6dc215 to 5c410a5 Compare May 13, 2025 16:27

This was referenced May 13, 2025

llama : initial Mamba-2 support #9126

Open

Granite Four #13550

Draft

gabe-l-hart marked this pull request as ready for review May 14, 2025 20:13

gabe-l-hart force-pushed the HybridCache branch from 0b3853d to cbf6b10 Compare May 16, 2025 17:27

gabe-l-hart changed the title ~~feat: First pass at llama_kv_cache_hybrid~~ feat: Hybrid unified/recurrent cache May 16, 2025

gabe-l-hart commented May 16, 2025

View reviewed changes

github-actions bot added the testing Everything test related label May 16, 2025

gabe-l-hart mentioned this pull request May 20, 2025

Memory tests #13669

Closed

gabe-l-hart force-pushed the HybridCache branch from ec08571 to 403ba7a Compare May 20, 2025 20:58

gabe-l-hart commented May 20, 2025

View reviewed changes

gabe-l-hart requested review from JohannesGaessler, ngxson and ggerganov as code owners May 23, 2025 23:29

github-actions bot added documentation Improvements or additions to documentation Nvidia GPU Issues specific to Nvidia GPUs Vulkan Issues specific to the Vulkan backend examples python python script changes devops improvements to build systems and github actions labels May 23, 2025

tests(wip): Comment out broken test for now and fix other constructor…

58994f6

… signatures Branch: HybridCache Signed-off-by: Gabe Goodhart <[email protected]>

gabe-l-hart force-pushed the HybridCache branch from c8da32d to 58994f6 Compare May 27, 2025 23:15

gabe-l-hart marked this pull request as draft May 27, 2025 23:16

gabe-l-hart changed the base branch from master to gg/kv-cache-simplify-part3 May 28, 2025 12:43

ggerganov force-pushed the gg/kv-cache-simplify-part3 branch from 825efad to eed741e Compare May 29, 2025 13:22

gabe-l-hart mentioned this pull request May 29, 2025

Hybrid recurrent cache #13904

Closed

1 task

gabe-l-hart closed this May 29, 2025

gabe-l-hart mentioned this pull request Jun 2, 2025

Hybrid recurrent cache #13979

Merged

		// TODO: Will it cause problems if some caches are able to remove the seq
		// but others aren't?

		// If any of the caches are recurrent, require simple split
		return llama_sbatch(batch, m_hparams.n_embd, m_has_recurrent, logits_all);

feat: Hybrid unified/recurrent cache #13276

feat: Hybrid unified/recurrent cache #13276

Uh oh!

Conversation

gabe-l-hart commented May 2, 2025

Description

Discussion

Uh oh!

compilade left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

compilade May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

compilade May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabe-l-hart commented May 14, 2025

Uh oh!

gabe-l-hart commented May 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabe-l-hart commented May 19, 2025

Uh oh!

gabe-l-hart commented May 19, 2025

Uh oh!

ggerganov commented May 20, 2025

Uh oh!

gabe-l-hart commented May 20, 2025

Uh oh!

gabe-l-hart commented May 20, 2025

Uh oh!

gabe-l-hart commented May 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabe-l-hart commented May 20, 2025

Uh oh!

gabe-l-hart commented May 27, 2025

Uh oh!

gabe-l-hart commented May 29, 2025

Uh oh!

Uh oh!

compilade May 2, 2025 •

edited

Loading

compilade May 2, 2025 •

edited

Loading