This episode examines "Cartridges++," an Apple paper on learned, compressed KV-cache memories of a document (Cartridges) and a failure they cause on questions unrelated to that document. It explains how KV-cache compression works and compares three families: eviction methods like H2O and SnapKV, text summaries, and learned methods such as Cartridges, prefix-tuning and Attention Matching. The central argument is that a Cartridge's training objective only rewards matching the teacher on document-related queries and says nothing about unrelated ones. So the memory that best reproduces the document can derail the model elsewhere, for example by quoting a stored policy document when asked what Saturn is made of. The discussion separates context fidelity from capability preservation and flags interference, meaning fabricated quotes from the document. It argues the fair baseline is the full uncompressed document pasted as text, which already distracts models somewhat. Listeners get a clear account of why scoring a compressed memory on document questions alone hides a real cost, and why the problem looks like catastrophic forgetting applied to cache entries.
Sources:
1. Cartridges++: KV Cache Compression without Off-Context Derailment — Sonia Laguna, Joao Monteiro, Marco Cuturi, Pierre Ablin, Eleonora Gualdoni, 2026
http://arxiv.org/abs/2609.35621
2. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré, 2025
https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study
3. Overcoming catastrophic forgetting in neural networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell, 2017
https://scholar.google.com/scholar?q=Overcoming+catastrophic+forgetting+in+neural+networks
4. LoRA Learns Less and Forgets Less — Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, and colleagues (Databricks / Columbia), 2024
https://scholar.google.com/scholar?q=LoRA+Learns+Less+and+Forgets+Less
5. Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou, 2023
https://scholar.google.com/scholar?q=Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context
6. Fast KV Compaction via Attention Matching — Adam Zweiger et al., 2026
https://scholar.google.com/scholar?q=Fast+KV+Compaction+via+Attention+Matching
7. Cartridges at Scale — Hardalov et al., 2026
https://scholar.google.com/scholar?q=Cartridges+at+Scale
8. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021
https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation
9. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao et al., 2024
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks
10. Conformal Risk Control — Anastasios Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, Tal Schuster, 2024
https://scholar.google.com/scholar?q=Conformal+Risk+Control
11. Lost in the Middle / Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi et al. (distraction); Nelson Liu et al. (lost in the middle), 2023
https://scholar.google.com/scholar?q=Lost+in+the+Middle+%2F+Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context
12. Training plug-and-play knowledge modules with deep context distillation — Lucas Caccia, Alan Ansell, Edoardo Ponti, Ivan Vulić, Alessandro Sordoni, 2025
https://scholar.google.com/scholar?q=Training+plug-and-play+knowledge+modules+with+deep+context+distillation
13. Doc-to-LoRA: Learning to instantly internalize contexts — Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert Tjarko Lange, 2026
https://scholar.google.com/scholar?q=Doc-to-LoRA%3A+Learning+to+instantly+internalize+contexts
14. The pitfalls of KV cache compression — Alex Chen, Renato Geh, Aditya Grover, Guy Van den Broeck, Daniel Israel, 2026
https://scholar.google.com/scholar?q=The+pitfalls+of+KV+cache+compression
15. KV-Distill: Nearly lossless learnable context compression for LLMs — Vivek Chari, Guanghui Qin, Benjamin Van Durme, 2025
https://scholar.google.com/scholar?q=KV-Distill%3A+Nearly+lossless+learnable+context+compression+for+LLMs
Interactive Visualization: Cartridges++: Compressed KV Memory Without Off-Context Derailment
