[RFC / Idea] Beyond Autoregressive Generation: Dynamic Backspace, In-Place Line Rewriting & Self-Healing Code Execution
Hi Kimi team and community,
Standard code LLMs are strictly forward-moving autoregressive systems ($P(x_t \mid x_{<t})$). When an LLM realizes it made a logical flaw 50 tokens earlier, it typically has to resort to conversational backtracking ("Sorry, let me rewrite that...") or append a correction at the very end. This wastes context, pollutes the code buffer, and breaks execution flow.
I would love to open a discussion on enabling a "Self-Healing / Non-Monotonic Generation" mechanism for code synthesis, where the model can emit dynamic editing primitives (like a virtual cursor/backspace) to mutate earlier lines after seeing subsequent logic or execution output.
The Core Problem
- Premature Commitments: A model defines an API signature or data structure on Line 2. By Line 25, it realizes a different structure or parameter is required. Current autoregressive generation cannot cleanly "backspace" Line 2.
- Execution Feedback: If an in-line REPL / test run fails midway, the model should heal the flawed block directly in-place rather than generating an entirely new code block.
Proposed Mechanics
1. Dynamic Cursor / Editing Primitives (Special Control Tokens)
Introduce dedicated editing tokens into the vocabulary:
<BACKSPACE_N>: Erase the previous $N$ characters/tokens.<EDIT_LINE_K> ... </EDIT_LINE_K>: Jump back to Line $K$, perform an in-place patch, and fast-forward back to the insertion point.<VERIFY_AND_REPAIR>: Trigger an execution checkpoint. If a runtime/syntax error occurs, roll back the generation pointer to the exact call site.
2. KV-Cache Truncation & Rollback
From an inference system perspective:
- Instead of running a fresh multi-turn prompt to rewrite code, the decoding engine simply pops the invalidated KV-cache entries back to token index $j$.
- The model regenerates from index $j$ conditioned on the new insight, saving massive memory and avoiding context bloat.
3. Diffusion / Non-Monotonic Fine-Tuning
- Train the model on human coding sessions (e.g., git commits, LSP undo/redo logs, terminal keystroke logs) where code development naturally includes deletion, cursor navigation, and refactoring.
- Formulate this as an interleaved Write $\rightarrow$ Execute $\rightarrow$ Backspace/Patch $\rightarrow$ Continue policy using Reinforcement Learning with Verifiable Rewards (RLVR).
Why this fits Kimi-K2.7-Code
Kimi's architecture already demonstrates exceptional capabilities in handling long context and deep technical reasoning. Equipping a coding model with the ability to self-heal in-place rather than just generating linearly would be a huge leap toward true agentic software engineering (closer to how humans actually write code in an IDE).
Would love to hear the team and community’s thoughts on:
- Feasibility of integrating control tokens for KV-cache splicing during inference.
- Training strategies using diff/patch datasets versus explicit cursor tokens.