For now, I ran a quick experiment. There may be something useful in teasing apart why 60% seems to hold up:
My short answer to the four questions would be:
- The current protocol looks useful as a pilot. In particular, evaluating the resulting adapter again on the full context is a good choice.
- Before scaling the experiment up, I think the highest-information next step is probably to separate “60% is enough” from “this selector chose a particularly good 60%.”
- For task-level evaluation, I would start with a very small controlled cross-block dependency test rather than a large benchmark suite.
- The closest literature I found spans several different design points—TokenTune, TokenSeek, LeMo, LongLoRA, LongQLoRA, and PoSE—but I would not treat any of them as identical to what you are doing.
If I were trying to get the most information from the fewest additional runs, my default path would be roughly:
base / step 0
full-context 100%
matched random-block 60%
Spiral-selected 60%
with the same dataset split, seed, block size, block-joining policy, position-ID policy, and loss masking.
That one control seems unusually valuable because both possible outcomes are informative:
random 60% ≈ Spiral 60%
→ a large part of the result may come from generic shortening /
redundancy in this corpus
Spiral 60% > random 60%
→ much stronger evidence that the selector itself is contributing
The first outcome would not make the experiment uninteresting. It could mean that this workload admits a much simpler compression rule than expected, which is useful information in its own right.
I would also record the step-0 full-context held-out loss if that is cheap. That separates:
"both methods learned, and ended up similarly good"
from
"after 48 steps, neither method has moved very far yet"
I would not jump straight from 48 steps to a large long-running sweep just to answer this. If intermediate checkpoints already exist, even something like 0 → 12 → 24 → 48 would be enough to see whether the curves are meaningfully separating.
Two other implementation details seem worth making explicit because they change what “drop 40% of the context” actually means:
- whether the reported timing includes the block scoring/selection work;
- what happens to positions and loss targets when non-adjacent blocks are joined.
Those are not objections to the approach; they are useful axes for separating the mechanism.
Why I would add a matched random-block control first
The main ambiguity I see in the current pilot is attribution.
There are at least two fairly different hypotheses consistent with:
full context and selected 60% have similar held-out full-context loss.
Hypothesis A — generic shortening is enough here
The local corpus may contain enough redundancy that many 60% subsets work.
In that case, the interesting result becomes something like:
for this data distribution and training horizon, a substantial amount of context can be removed before QLoRA quality visibly degrades.
That could be useful even if a sophisticated selector turns out not to be necessary.
Hypothesis B — the selector matters
The particular 60% chosen by the selector may retain substantially more useful training signal than an equally sized random subset.
Then the selector itself becomes the interesting part.
A matched random-block condition seems cleaner than random-token dropping here, because it preserves the same granularity as the method:
same original sequence
same block size
same number of retained blocks
same ordering rule
same position policy
same loss policy
only:
selector score → random block choice
There is some nearby precedent for making this kind of distinction.
TokenTune reduces the number of token positions participating in backpropagation while retaining full contextual information in the forward pass.
TokenSeek moves from data-agnostic token selection toward instance-aware selection. The exact mechanism is different from pre-forward block deletion, but the underlying experimental question is useful here:
is the gain caused by having a smaller token budget, or by choosing the right tokens within that budget?
So I would use those more as evaluation precedents / neighboring design points than as claims that they are the same method.
If a second cheap control is available after random blocks, one optional variant could be uniformly spaced blocks at the same retention rate.
That would help distinguish:
content-aware selection
from
simply preserving broad coverage across the original sequence
but I would consider that secondary. Random 60% seems like the first control.
What I would try next for task-level quality
I would probably not begin with a large LongBench/HELMET-style evaluation.
First I would build one deliberately small case that should fail if a necessary dependency is cut.
For example:
Block A:
Project Cedar uses key K17.
... several unrelated blocks ...
Block B:
Key K17 opens archive 739251.
... several unrelated blocks ...
Target:
Which archive can Project Cedar access?
Then vary only a few things:
A and B both retained
A retained / B removed
B retained / A removed
A and B close together
A and B far apart
low distractor density
high distractor density
The important quantity is no longer just:
retained token ratio = 60%
but something closer to:
did the selected context retain the dependency needed for this target?
This is closely related to why I think RULER is a useful source of test ideas here.
RULER deliberately goes beyond simple single-needle retrieval and includes configurable:
- multiple-needle retrieval,
- multi-hop variable tracing,
- aggregation.
Its own documentation is careful that these synthetic tests are not a replacement for realistic downstream tasks, which seems exactly the right role here: use a tiny synthetic test as an inexpensive mechanism probe, not as the final quality claim.
A practical progression could therefore be:
tiny controlled dependency probe
↓
small RULER-style subset
↓
realistic RAG / QA workload
↓
broader downstream validation
For the later stage, HELMET is useful because it found that simple needle-in-a-haystack results do not reliably predict performance across realistic long-context categories. It covers recall, RAG, long-document QA, reranking, citation, summarization, and ICL.
The HELMET authors specifically suggest RAG tasks as a relatively convenient development-time signal before doing a broad evaluation.
So I would see the synthetic dependency test and HELMET as complementary:
controlled synthetic test:
"what mechanism breaks?"
realistic task:
"does the break matter in an application?"
Why I would keep the full-context loss, but not make it the final metric
I think the existing full-context held-out loss is worth keeping.
It is stronger than evaluating the model only on the selected/truncated sequence, because it at least asks the trained adapter to operate again under the original context.
The caveat is that average language-model loss can hide a small subset of tokens that actually depend on distant context.
This is the main issue studied by LongPPL / “What is Wrong with Perplexity for Long-context Language Modeling?”.
Their argument is roughly:
most tokens may be predictable from local context
+
only a minority may genuinely benefit from distant context
+
ordinary PPL averages over everything
=
a model can lose some long-range behavior without a dramatic PPL change
They address that by comparing long-context and short-context predictions to identify “key tokens” whose prediction depends more on the distant context.
I do not think you need to implement LongPPL immediately.
A much cheaper version of the same diagnostic idea would simply be:
create a small set of target tokens for which you know distant blocks are necessary, and inspect those separately from aggregate loss.
That gives the current full-context held-out loss a useful partner rather than replacing it.
Position IDs may be a separate experimental variable
One thing I would make explicit in the method description is what happens to positions after block selection.
Suppose the original sequence contains:
block 1 @ positions 0–511
block 7 @ positions 3072–3583
block 12 @ positions 5632–6143
and those three blocks are selected.
There are at least two conceptually different ways to feed them back:
A. compact positions
block 1 → 0–511
block 7 → 512–1023
block 12 → 1024–1535
or something closer to:
B. preserve original positional distance
block 1 → 0–511
block 7 → 3072–3583
block 12 → 5632–6143
These are not necessarily “right” and “wrong”; they are different training signals.
PoSE is useful context here because it deliberately decouples physical training length from positional distance. It trains on short chunks but manipulates their position indices so that the model still experiences positions across a much larger target window.
So if selected blocks are being compacted, the method may effectively combine:
content selection
+
positional-distance compression
If original positions are preserved, it is closer to:
content selection
while preserving long positional gaps
Either can be a legitimate design choice. I just think documenting it makes the result easier to interpret, and later it becomes a cheap ablation if needed.
I tried a very small independent sanity check on this point with Qwen2.5-0.5B-Instruct: I kept exactly the same selected tokens and changed only whether their positions were compacted or kept at their original offsets. Even in that toy setup the loss changed measurably.
I would not read anything universal into the direction or magnitude of that change—the setup was synthetic and much smaller than your experiment—but it convinced me that position policy is cheap enough to treat as a first-class experimental variable rather than an invisible implementation detail.
What happens at a discontinuity between selected blocks?
There is a second, separate issue when physically non-adjacent blocks become neighbors.
For example:
original:
... end of block 3 ...
[blocks 4–9]
... beginning of block 10 ...
after selection may become:
... end of block 3 ...
... beginning of block 10 ...
In a causal-LM objective, unless something special is done, the first token of block 10 is now predicted from a prefix ending in block 3.
That transition did not exist in the original document.
Again, I do not think there is one universally mandatory policy. Two possibilities are:
1. keep the transition as a real training target
2. mask the first loss-bearing token after a discontinuity
The distinction matters because in case 1 the experiment measures:
context selection
+
learning on newly created block joins
while in case 2 it more narrowly measures the selected within-block language-model targets.
Hugging Face’s current TRL SFTTrainer documentation and its padding-free/packing implementation are useful examples of why these details are normally explicit: packed sequences carry position information, and non-loss-bearing locations can be represented with labels == -100.
That is not the same problem as this block-selection setup, but the general lesson transfers:
once sequence topology is changed, input_ids, position_ids, and labels become separate parts of the experimental contract.
So if non-adjacent blocks are directly concatenated, I would simply document the policy.
A one-example debug dump is probably enough to make it unambiguous:
selected block IDs
input_ids around each join
position_ids around each join
labels around each join
Where this seems to sit relative to nearby methods
The neighboring work became easier for me to understand when I stopped putting all of it under “token pruning.”
They reduce different things:
| Method |
What is reduced / changed? |
Useful comparison point |
| Your block-selection approach |
Input/context visible before the forward pass |
Physical sequence shortening |
| TokenTune |
Token positions participating in backward/activation storage |
Keeps forward context |
| TokenSeek |
Instance-aware token participation |
Selection quality vs data-agnostic selection |
| LeMo |
Token involvement dynamically across inputs/layers |
Long-context token sparsity inside the model |
| LongLoRA |
Attention interactions during training |
Keeps tokens, sparsifies attention |
| LongQLoRA |
QLoRA + positional/attention-side long-context adaptation |
Similar resource-constrained problem setting |
| PoSE |
Physical train length vs positional range |
Shows that token distance itself is an experimental variable |
I would not call these direct substitutes.
To me, the interesting part of your approach is that it acts earlier:
long sequence
↓
select blocks
↓
physically shorter sequence
↓
normal forward/backward
That makes large real wall-clock savings plausible in a way that some “selective backward” methods do not automatically get, because fewer tokens enter the expensive forward computation in the first place.
For that reason, I would also report timing in a way that makes the boundary clear:
selection/scoring time
+
model forward/backward time
=
end-to-end step time
If the reported ~1.675× already includes selection, that is especially useful to know.
If it currently excludes selection, reporting both numbers would make it easier to compare future selector variants whose scoring costs differ.
A later design axis: retain dependencies, not just tokens
This is probably not something I would implement before the cheap random control, but it may become useful if the basic result keeps holding.
There is a separate long-context literature arguing that:
a long document is not automatically a document with useful long-range dependencies.
For example, ProLong scores training documents using dependency strength, dependency distance, and dependency specificity, with the goal of identifying samples that genuinely exercise long-range modeling.
NExtLong approaches the same broad issue from the opposite direction: it inserts hard-negative distractors between related chunks so that the model must maintain useful relationships over larger distances.
Those papers are not doing your block selection.
But they suggest a useful future distinction:
retained-token ratio
is not necessarily the same as:
retained long-range dependency
A selector could retain 60% of the tokens while accidentally severing the one pair of distant blocks that supplies the important training signal.
Conversely, a selector that protects a relatively small set of dependency-carrying block pairs might be able to remove even more raw tokens.
So if the simple selector beats random selection, a possible later direction is not merely “find even more important individual blocks,” but:
identify groups of blocks whose value comes from their relationship to one another.
I would treat that as a second-generation design question, not as something needed to validate the current pilot.
What my quick sanity check did — and did not — show
I also tried a deliberately small independent probe, mainly to see whether the proposed diagnostic axes were observable at all.
It used:
Qwen2.5-0.5B-Instruct
small synthetic context
random block retention at roughly 60%
It was not a reproduction of your 7B / 8K / 48-step setup, and it did not implement your selector.
Two things were useful:
1. Same selected tokens, different position policy
Keeping the token IDs fixed but switching between compacted and original position IDs changed the measured NLL.
I would not generalize that number to your setup, but it supports treating position handling as an explicit variable.
2. Drop one actually necessary evidence block
In a tiny synthetic case where the answer depended on one distant evidence block, the candidate-answer NLL moved strongly when that block was removed.
Again, this is not evidence about your selector.
What it showed me is simply that a very cheap controlled dependency probe can have enough sensitivity to be useful before running a large task suite.
I did not get a clean independent training-speed comparison from that probe—the backward benchmark hit memory limits—so I would not use it to say anything about your reported speed/VRAM results.
A possible low-cost decision tree
I think this is the cleanest way to avoid turning the validation process into a long checklist:
1. Compare step 48 with step 0
|
+-- little/no learning signal yet
| |
| → extend only far enough to establish that both runs
| are actually learning
|
+-- clear learning signal
|
v
2. Add matched random-block 60%
|
+-- random ≈ Spiral
| |
| → generic shortening / corpus redundancy may explain
| much of the result
|
| → a simple selector may already be useful
|
+-- Spiral > random
|
→ selector-specific value becomes much more plausible
|
→ optionally try a small retention sweep
(for example 40 / 60 / 80)
3. Run one tiny cross-block dependency probe
|
+-- aggregate loss holds, dependency probe drops
| |
| → separate "context reduction" from
| "dependency-preserving selection"
|
+-- dependency probe also holds
|
→ stronger reason to move to a realistic task
or design-partner workload
And independently of those branches, I would keep these three things explicit:
timing boundary:
does end-to-end time include selection?
position policy:
compact selected blocks or preserve original offsets?
discontinuity policy:
what happens to the first loss-bearing token after a removed span?
That seems like enough information to interpret almost every branch without requiring a large evaluation campaign up front.
So overall, I think the pilot is already pointing at a useful question.
I would not try to prove “60% is universally safe” from this stage. I would instead ask:
what made 60% safe in this run?
If a matched random 60% baseline performs just as well, that tells you something important about redundancy and how simple the method might be.
If the selected 60% reliably beats random 60%, that makes the selector itself substantially more interesting.
And if both look fine on aggregate loss but differ on a tiny long-range dependency probe, that gives a fairly clean next design boundary without invalidating the efficiency idea.
That seems like a lot of information for only a few small additional checks.