I did a quick search around to start:
I think the pilot already has a useful core. The main thing I would separate is three slightly different questions:
- Does disconnecting a branch actually restore the intended clean model context?
- Does removing semantic contamination help beyond merely shortening the prompt?
- When does explicit user control beat, complement, or lose to automatic context selection?
That separation also makes the four questions in the post easier to answer.
My short answers to the four questions
1. Model / context length: I would start with one model, not several, and vary the actual serialized input length before doing cross-model comparisons. qwen3:8b seems like a sensible baseline because it is already in the ThoughtDAG Ollama setup examples, and the Qwen3-8B model card gives a native 32,768-token context.
One important distinction is:
allocated context window != actual serialized input length
With Ollama, I would record the actual loaded CONTEXT from ollama ps rather than infer it from the model card or UI. For the first experiment I would keep the allocated window comfortably large enough for every condition, then vary actual input length separately. That separates semantic interference from simple capacity pressure.
2. Missing contamination cases: the current three look like a good starting set. The two additions I would rank highest are:
- implicit staleness: later evidence makes an older belief invalid without explicitly saying “the old statement was wrong”;
- a previous model-generated mistake re-entering later context: a bad interpretation produced by the LLM itself becomes evidence for a later turn.
The first is close to the failure mode studied in STALE; the second connects to “hallucination snowballing”.
I would also add one control rather than another contamination family: clean context plus approximately the same number of neutral tokens as the polluted condition. There is evidence that context length alone can hurt performance even with perfect retrieval, so this helps distinguish semantic contamination from simply making the prompt longer.
3. Manual pruning vs retrieval / automatic memory: I would not treat this as a two-way choice. The interesting comparison looks more like:
manual selection
automatic selection
automatic suggestion + human confirmation/override
NoteEx is particularly close here. It is a different domain (LLM-assisted exploratory data analysis in notebooks), so I would not treat it as evidence for ThoughtDAG specifically, but its formative study found the same tension: pure manual curation becomes burdensome, while automatic context selection can miss the user’s mental-model dependencies. Their resulting design suggests context automatically, then lets the user add/remove items before sending.
That hybrid seems especially worth comparing with ThoughtDAG’s explicit wiring.
4. Inspection panel: ThoughtDAG already exposes a lot more than a typical chat UI: the current feature list includes a context tree, token/message/file preview, layer breakdown, staleness, node roles, etc.
So rather than adding more graph terminology, I would make the next layer answer four ordinary questions:
WHAT is going into this request?
WHY is this item included?
WHERE did it come from?
WHAT changes if I remove it?
Then keep an optional deeper “exact request/debug” view for benchmark/reproducibility work.
A recent HCI system called Contexty is useful background: it treats AI context as something that should be inspectable and correctable by the user, and exposes provenance such as source application, URL, timestamp, raw captured content, and the system’s interpretation.
A small first benchmark I would actually run
I would resist making the first pilot too large. Something like this already answers quite a lot:
one fixed model
one fixed runtime
one comfortably large allocated context window
C0 clean evidence only
C1 clean evidence + semantic contaminant
C1N clean evidence + approximately token-matched neutral context
C2 same task after explicitly disconnecting the contaminating branch
C3 polluted context summarized before answering (optional)
For every run I would keep or record:
exact model / quantization
runtime + version
actual allocated context window
actual input token count
thinking/non-thinking mode
sampling parameters
system / role state
tools/search state
attachments
ambient memory state
ordered outgoing messages
condition ID
output
The useful comparisons are:
| Comparison |
What it mainly tests |
| C0 vs C1 |
effect of adding the perturbation |
| C1 vs C1N |
semantic contamination vs extra context length |
| C0 vs C2 |
whether explicit pruning restores the clean condition |
| C2 vs C3 |
selection/removal vs representation/compression |
I would especially keep C0 vs C2. If the final effective request really returns to the C0 state after disconnecting the edge, then C0 ~= C2 is a useful mechanism sanity check that “wire removal” is genuinely “context removal.”
I just would not interpret that result alone as evidence that a DAG interaction model is superior to ordinary branching, retrieval, or automatic memory selection. That is a second experiment.
1. Model and context length: window size vs actual input size
There are at least three “context length” variables hiding inside the question:
model's advertised/native context capacity
runtime's actually allocated context window
actual serialized input length used by this benchmark case
For qwen3:8b, the official model card lists 32,768 tokens natively and describes YaRN for longer inputs. It also cautions that static YaRN can hurt shorter-text performance, so I probably would not enable YaRN in the first <=32K pilot.
I would also hold Qwen3’s thinking behavior constant. Whether the pilot uses thinking or non-thinking is less important than making it identical across C0/C1/C2/C3 and recording it.
The Ollama detail seems worth checking explicitly
Current Ollama documentation says its default context allocation depends on VRAM and recommends checking the allocation with:
ollama ps
where CONTEXT shows the actual loaded value.
There is also a small integration wrinkle. ThoughtDAG’s current model probe stores a context window when an OpenAI-compatible /models response supplies a numeric context_length.
Ollama’s OpenAI-compatible /v1/models does not normally provide that field yet. There is an Ollama issue about exposing it, and the corresponding PR #17422 is still open as of August 17, 2026.
So for this benchmark I would treat the Ollama-side value as authoritative rather than assuming the client can always infer it.
Vary actual occupancy before model family
Conceptually:
Qwen3-8B, same runtime, same allocated window
short fixture
medium fixture
long fixture
where clean and polluted conditions at each level still fit comfortably.
That asks:
As actual history grows, does semantic contamination become harder to ignore?
Then a separate capacity-pressure test could deliberately push the polluted history toward the limit:
polluted condition -> near context capacity
pruned condition -> comfortably inside capacity
That asks a different question:
Does pruning help because it removes interference, because it prevents truncation/capacity pressure, or both?
Benchmarks such as RULER have repeatedly found a gap between advertised maximum context and effective long-context behavior, while Context Length Alone Hurts LLM Performance Despite Perfect Retrieval suggests input length itself can be meaningful before the nominal limit is reached.
Only after that would I swap model families. Otherwise a cross-model result simultaneously changes architecture, tokenizer/chat template, capability, quantization, and context handling.
The Hugging Face chat-template docs are relevant here: the same abstract {role, content} messages can become different token sequences for different model families.
2. Contamination cases: two additions plus a length control
Your current three families already cover useful axes:
irrelevant-but-plausible
superseded/outdated
conflicting-but-valid-in-a-different-scope
I would keep those isolated at first rather than immediately mixing them.
A. Implicit staleness
A harder version of “later evidence corrects the old assumption” is when the new evidence never explicitly negates it.
For example:
Earlier:
"The participant commutes by bicycle every day."
Later:
"The participant broke their leg and will be on crutches
for the next two months."
Question:
"What is a practical commuting recommendation for tomorrow?"
The older fact is not textually contradicted, but it is no longer an appropriate current-state assumption.
STALE studies this kind of implicit conflict and separates recognizing the current state from resisting stale premises and adapting downstream behavior. That maps nicely onto ThoughtDAG because an old branch can remain visible without necessarily remaining active context.
B. Previous model output as contamination
This may be especially realistic for a research workspace:
primary source
|
v
old LLM interpretation <-- mistaken assumption
|
v
later synthesis
That differs from injecting external misinformation: the bad evidence is an earlier derived interpretation that the workspace itself helped produce.
The older hallucination snowballing work showed that models can over-commit to an early error and generate further false claims in support of it.
One ThoughtDAG-shaped fixture would therefore be:
source node remains wired
incorrect old interpretation remains wired
vs.
incorrect interpretation disconnected
That asks whether explicit context control can prevent a model’s own stale interpretation from becoming self-reinforcing history.
C. Optional later variant: duplicated derived evidence
ThoughtDAG can turn material digests and other derived artifacts into downstream context.
Eventually it may be interesting to test:
original source
+
passage extracted from source
+
LLM digest of the same source
These are several context objects but not several independent pieces of evidence. I would not add this to the first pilot unless it arises naturally.
D. Neutral-length control
C0 clean
C1 clean + meaningful contaminant
C1N clean + similarly sized neutral material
Ideally match not only token count but roughly message/role structure, because chat templates add per-turn control tokens.
If C1 degrades but C1N does not, semantic interference becomes more plausible.
If both degrade similarly, sheer context size / message structure may explain much of the effect.
A position control (early / middle / late contaminant) would also be useful eventually, but I would add that only after seeing a signal in the simpler matrix.
3. Manual vs automatic selection: the hybrid condition looks important
I would frame this less as:
manual pruning vs automatic retrieval: which is better?
and more as:
under what conditions should selection authority live with the user, the system, or both?
Manual control is most interesting when “relevant” is not simply “semantically similar”
For example:
- this study is on the same topic, but the population is wrong;
- this branch is factually correct, but I do not want it to be a premise of this synthesis;
- this answer is an old interpretation;
- I want to preserve a counterexample but exclude it from the current inference;
- two branches are equally related lexically, but only one belongs to the current hypothesis.
That is closer to an epistemic/task-scope judgment than ordinary retrieval relevance.
In a small enough research graph, the person doing the work may genuinely know something the retriever does not: “I know why this branch exists.”
Automatic selection gets more attractive as the graph grows
At 20 nodes, explicit control may feel like agency. At 2,000 nodes, checking every incoming branch could become context management as a second job.
There are already automatic approaches specifically targeting long dialogue. For example, DyCP dynamically selects query-relevant spans from prior dialogue rather than simply sending the full history.
That is not directly comparable to a user-edited DAG, but it is a stronger automatic baseline than a naive summary.
The hybrid seems especially plausible
NoteEx is the closest comparison I found for this design question.
Its domain is computational notebooks, not conversational research graphs, but the observed tension is similar:
- pure manual context curation was burdensome;
- automatic selection could miss the analyst’s intent/dependencies;
- participants wanted a useful default while retaining the ability to correct it;
- the resulting interface proposes context in an editable preview before the LLM call.
That suggests a future comparison such as:
M: manual
user explicitly controls relevant context
A: automatic
selector chooses without intervention
H: hybrid
selector proposes
user accepts/removes/adds before generation
The interesting metrics are not answer quality alone:
answer correctness / task quality
contaminant retained
useful evidence accidentally removed
time to select
number of interactions
user corrections to automatic selection
A manual approach that gains a little quality but triples selection effort is a very different result from one that improves both quality and clarity.
Retrieval and evidence use are separate problems
The recent MemTrace benchmark reports a useful diagnostic distinction: in its evaluated memory systems, many failures occurred even when relevant evidence was retrievable.
That is a different domain, so I would not transfer the numerical results directly. But it suggests:
retrieval problem:
the right evidence never reaches the model
evidence-use/interference problem:
the right evidence is present,
but stale / irrelevant / conflicting context competes with it
ThoughtDAG may be particularly interesting in the second regime.
So the strongest hypothesis to test may not be:
“manual wiring retrieves better”
but:
“visible context control helps decide which reachable evidence should actually participate in this inference.”
That seems very close to “wires are the context.”
ContextBranch is also useful nearby work: its reported branching benefits were concentrated especially in more complex cases involving conceptually distant exploration. That is another reason not to expect pruning to win uniformly; the more interesting result may be where explicit isolation becomes useful.
4. Inspection panel: expose effects rather than graph operations
Since ThoughtDAG already has a substantial inspection layer, I would avoid adding controls just for the sake of it.
The current feature list already describes context send preview, material/reference/conversation breakdown, context tree, token counts, roles, staleness, and reference depth.
For someone who should not need to understand graph mechanics, I would translate that into an effect-oriented context contract:
WHAT
The items that will influence the next response
WHY
"Inherited through Research Question"
"Explicit reference from Current Hypothesis"
SOURCE
User note
Model answer
PDF p. 12
Link snapshot
Generated digest
Ambient memory
STATE
Current
Stale
Edited since generation
Derived from another source
COST
1,340 tokens
CHANGE
Remove from next request
-> removes 2 messages / ~1,340 tokens
The UI verb can be:
Remove from next request
rather than:
Delete incoming structural edge X.
The graph remains the implementation / advanced mental model; the normal user sees the consequence.
Contexty seems relevant here because it explicitly explores user-inspectable and correctable AI context, including provenance such as source application, URL, timestamp, captured content, system interpretation, and user-authored memo.
Two inspection layers may be enough
Normal view: understandable context
included item
why included
source
current/stale/derived status
token cost
effect of removing it
Optional research/debug view: effective request
ordered messages
roles
system instructions
ambient memory present/absent
attachments
model identifier
runtime
allocated context window
estimated/actual tokens
generation settings
A normal user usually cares about:
“Why is the model seeing this?”
A benchmark author cares about:
“What exact request was sent?”
Keeping those layers separate may make the system understandable without forcing everyone to learn graph operations.
5. A few controls that could prevent ambiguous pilot results
These are mostly small controls rather than requests for a larger benchmark.
A. Keep non-graph context fixed
In current main, ordinary ThoughtDAG generations can include ambient memory in addition to the graph-built context; the memory block is assembled at generation time after buildContext().
So for C0/C1/C2 comparisons I would either:
disable ambient memory
or use a fixed empty/identical memory state.
Same idea for:
web/search tools
MCP tools
attachments
role/system prompt
model override
Those features are fine; they are simply extra causal inputs that are better held constant in a context-control benchmark.
B. Be careful with the built-in stale marker
ThoughtDAG’s current context builder can mark stale downstream answers with a label like:
[Stale: this answer was written against an earlier version of its upstream]
and the server’s base directive tells the model that bracketed provenance markers should affect how the information is trusted.
If the “outdated assumption” fixture is produced through that built-in staleness mechanism, the condition is therefore not simply:
old unsupported assumption remains in context
but closer to:
old assumption remains in context
+ the system explicitly tells the model that it is stale
Both are interesting, but they test different things.
So, if that path is being used, I would distinguish:
unlabelled stale evidence
labelled [Stale] evidence
disconnected stale evidence
If the fixture is just hand-authored text, this point disappears.
C. Save the linearized request for multi-parent cases
One small reproducibility detail in current main: walkUpAncestors() walks incoming edges in their stored order.
For a diamond:
A
/ \
B C
\ /
D
a multi-parent traversal can potentially produce:
A -> B -> C -> D
or:
A -> C -> B -> D
depending on parent ordering.
The context builder clearly intends deterministic assembly, so I would treat this as an implementation-level canonicalization detail rather than a blocker.
For the benchmark, any of these is enough:
- avoid ambiguous multi-parent fixtures initially;
- store the final ordered message list;
- canonicalize parent ordering for fixtures.
D. C3 is a different intervention
C2 is basically:
selection intervention
remove part of the evidence set
C3 is:
representation intervention
transform the polluted evidence set
A summary can change salience, uncertainty, wording, provenance, and apparent authority. If generated dynamically, it is another model call.
So I would either freeze/log the C3 summary or repeat the summarization step too.
Then C3 > C1 means “transforming/compressing polluted context helped,” not automatically “automatic memory selection beat manual pruning.”
E. One sampled answer may be noisy
I would not make the benchmark depend heavily on one stochastic generation per condition.
Either use a reproducible fixed generation setup if the runtime supports it, or run a small number of repetitions per condition.
The important part is not to mistake one sampling flip for a context-control effect.
6. How I would read the first results
C1 worse than C0
C2 returns near C0
C1N stays near C0
-> semantic contamination is a plausible driver,
and removing that branch reverses it in this fixture
C1 worse than C0
C1N also degrades about as much
-> extra length / structure may explain a large part
of what initially looked like contamination
C1 worse than C0
C1N somewhat worse
C1 worse still
-> length and semantic interference may both matter
C2 does not recover toward C0
-> before concluding that pruning failed, inspect:
final messages, ambient memory, system/role state,
stale markers, message order, and run variance
benefit exists only at small context budgets
-> capacity pressure / limited effective context
may be an important moderator
benefit persists while all conditions remain comfortably
inside the model's context window
-> the effect looks less like simple capacity
and more like relevance / interference / evidence use
manual pruning improves quality
but requires lots of interaction
-> hybrid suggestion + user confirmation
becomes especially interesting
An automatic selector and an explicit context graph could conceivably have similar average answer accuracy while producing very different debugging experiences.
For a research workspace, being able to explain:
“this answer changed because this branch entered the request”
may itself be part of the value worth measuring.
7. Nearby work I found most useful
These are not claiming the same thing, but they seem like useful neighbors.
Probably the closest HCI comparison for the manual/automatic question: task-relevant context is proposed automatically but remains editable before the LLM call.
A nearby branch-isolation approach. It is tree/version-control oriented rather than arbitrary editable DAG context, which makes it useful as a comparison rather than a duplicate.
Useful automatic-side comparison: dynamic query-conditioned pruning of prior dialogue rather than full-history prompting.
The main reason I would include a token-matched neutral control before attributing every C1 drop to semantic contamination.
Useful for a harder outdated-context condition where later evidence invalidates an older state without explicit negation.
Useful for the “model’s own earlier answer becomes contamination” case.
Different problem domain, but useful for separating evidence retrieval from evidence use.
Useful for the inspection-panel question: context as something users can inspect, understand provenance for, and correct.
So if I had to reduce all of this to one default path, I would not expand the pilot into a giant memory benchmark yet.
I would do:
1. qwen3:8b (or another single fixed local model)
2. verify the actual Ollama context allocation
3. keep every first-pass fixture comfortably inside that window
4. run:
C0 clean
C1 semantic contamination
C1N token/structure-matched neutral control
C2 explicit disconnect
C3 summary only if useful
5. freeze/log every non-graph input
6. interpret C0/C2 as mechanism recovery first
7. only after that:
vary actual context occupancy,
then model family,
then manual vs automatic vs hybrid interaction
That seems small enough to run, but already tells you much more than a plain clean-vs-polluted comparison.
The broader thing I find interesting is that existing work makes “context pruning can help” less surprising than it might initially sound. The less-settled question is more specific:
When does making context selection visible and user-governed provide enough additional control, debuggability, or resistance to evidence interference to justify the interaction cost?
ThoughtDAG seems unusually well positioned to test that because the context boundary is already an explicit object in the interface rather than a hidden retriever decision.
