Products & tools · first seen 21 Aug, updated 21 Aug
From KV Cache to Depth Attention: The Bottlenecks Kimi K3 Had to Break
Part 2 of Inside Kimi K3 — how Moonshot rebuilt long-context memory with Kimi Delta Attention, why it still kept global attention, and how the same idea was extended across 93 layers Previously in Part 1, we looked at the first problem crea…
Summary from Towards AI.