This may sit at the intersection of several research lines
:
My short answer to the literature questions is: yes, there are fairly close precedents for almost every individual component you described, but I would not collapse the whole idea into “this is just MoE” or “this is just continual learning.” The interesting comparison seems to be how the pieces are connected.
The rough map I would use is:
| Your idea |
Nearby research line |
Important difference |
| A weight carries persistent history / importance |
Synaptic Intelligence, Memory Aware Synapses |
Mostly consolidation/continual learning, not input-conditioned routing |
| Stability / plasticity differs across connections |
Differentiable Plasticity |
Plasticity is learned in a different meta-learning/local-plasticity setting |
confidence controls how much a weight may change |
UCB, MESU in Bayesian continual learning and forgetting in neural networks |
Close only if your confidence means something like parameter uncertainty |
| Input selects only part of the network |
conditional computation / MoE / modular networks |
Usually expert/module granularity rather than individual scalar weights |
| Selection history identifies what should be protected |
Conditional Channel Gated Networks, Hash |
Filters/prompts rather than scalar weights |
| Parameters themselves act like experts |
ParaX |
Trainable parameter matrices are routed/aggregated; no persistent synaptic state like your proposal |
| Two routing stages |
CaRE |
Router → expert routing, rather than WClass → scalar-weight routing |
| Individual weights can be active/inactive |
Piggyback, Supermasks in Superposition |
Mostly task-specific masks over fixed weights, not per-input stateful routing |
| Only part of the backward/update is used |
meProp, SparseProp |
Different selection rule and objective; this does not by itself remove backprop |
So the part I would investigate most carefully is not whether a Weight can be richer than a scalar — that has quite a lot of precedent — but something closer to:
persistent per-parameter state
|
+------> routing decision
|
+------> update / plasticity decision
input
|
v
WClass selection
|
v
parameter-subset selection
|
v
forward / credit assignment / update
I did not find an exact match for that complete combination in this search, especially with persistent multi-dimensional state at individual-weight granularity plus input-dependent two-stage routing. That is not a novelty claim; it is only where the closest references I found stop matching.
If I were turning the idea into a first prototype, I would first separate five things that are currently bundled together:
- parameter value
- persistent state / memory
- routing
- update rule
- actual execution sparsity
That separation lets you change one part without changing the underlying idea. It also makes comparisons with the existing literature much easier.
A minimal first experiment could then be something like:
dense baseline
state only
routing only
routing + state
capacity-matched random routing
and measure both retention and new-learning performance. The random-routing control is cheap but useful: it helps distinguish “the router learned something useful” from “splitting parameters reduced interference anyway.”
I would probably start at block/channel/small-module granularity, even if scalar-weight routing is the eventual goal. It preserves the main idea while making routing behavior, optimizer behavior, and real compute effects much easier to inspect. Scalar routing can then be a second experiment rather than an assumption built into the first one.
Why I think these research lines are close, but not identical
1. Stateful synapses: Synaptic Intelligence / MAS / EWC-family methods
Synaptic Intelligence (SI) is unusually close to your initial intuition.
Its motivation explicitly contrasts the usual ANN synapse — essentially one scalar parameter — with biological synapses that have richer internal dynamics. SI maintains an online estimate of how important each parameter was to previous learning and then resists changing important parameters during later tasks.
So something like:
weight value
+
history-dependent importance
already has a very direct precedent.
Memory Aware Synapses (MAS) is useful for another reason: its importance measure is not simply “how often was this parameter used?” It estimates importance from the sensitivity of the learned function to changes in that parameter.
That suggests that your separation between:
usage
importance
may actually be worth keeping.
A frequently active parameter is not necessarily a uniquely important parameter, and an infrequently active parameter might be critical for a rare subset of inputs.
There is also a useful recent caution here. EWC-DR (CVPR 2026) revisits parameter-importance estimation and shows that the details of how parameters are identified/protected matter; “having an importance score” is not by itself a solved design problem.
So I would treat importance as an operational definition to choose and test, rather than a self-explanatory field.
2. Selection + protection: Conditional Channel Gating
Conditional Channel Gated Networks may be one of the closest older examples of the whole select → use → identify importance → protect cycle.
It adds task-specific gates to convolutional layers, uses the gate execution patterns to identify important filters, protects those filters, and promotes sparse filter selection so unused capacity remains for future tasks.
The difference matters, though:
their unit: filter/channel
their context: task-aware continual learning
your proposed unit: possibly individual weight
your router: potentially input-dependent WClass -> weight selection
So I would use it as a structural comparison, not as “the same architecture.”
A related example is the combination of context-dependent gating and synaptic stabilization studied by Masse et al.. That work combines sparse context-dependent gating with SI/EWC-like stabilization.
Again, the qualifier is important: this is context/task gating, not necessarily a learned sample-wise router of the form you sketched.
3. Usage/history feeding back into protection: Hash
A particularly relevant recent paper is Is Parameter Isolation Better for Prompt-Based Continual Learning? (Hash).
It maintains a global prompt pool, sparsely routes prompts, records cumulative prompt activation statistics, and uses that history to protect frequently used prompts from excessive updating.
That is conceptually close to:
usage history
|
v
future routing / protection / plasticity
The main difference is granularity: these are prompt parameters, not arbitrary scalar weights.
But I think it is a useful precedent for the feedback loop part of your proposal.
4. Routing parameters themselves: ParaX
ParaX: Parameters as Experts is another comparison I would definitely look at.
Its shared expert centers contain trainable parameter matrices. A module dynamically selects/aggregates matrices from those centers to produce input-dependent weight matrices.
So the basic viewpoint:
parameters themselves can be the routed experts
is already quite explicit there.
But ParaX is doing adapter-style PEFT with parameter matrices, and the routed parameters do not carry the persistent importance / stability / usage / confidence state you described.
That makes ParaX especially useful for isolating what may be different in your design: not parameter routing alone, but routing plus persistent parameter history plus state-dependent updates.
There is also a naming wrinkle in the current material: the official repository uses the name ParaX, while the current arXiv abstract still describes the method internally as AdaRoute. I would follow the official repository/paper title when searching for it.
5. Two-level routing: CaRE
CaRE uses a bi-level routing MoE:
router selection
|
v
expert routing
Structurally, that gives a recent reference point for:
WClass selection
|
v
within-class selection
But CaRE’s second level still selects experts/modules, so it does not establish the individual-weight version of your proposal.
6. Individual-weight masks: Piggyback / SupSup
If the question is specifically whether selection can go all the way down to individual connections, there is older work at that granularity.
Piggyback learns binary masks over individual weights of a fixed network.
Supermasks in Superposition likewise uses learned masks over fixed weights for many sequential tasks.
These differ from your proposal because the underlying weights can remain fixed and the masks are typically task-oriented rather than dynamically generated from every input.
Still, they are useful evidence that weight-level membership/selection is a real design axis, not something that has to stop at channels or experts.
A design split that might make the idea easier to prototype
One way to preserve your terminology while making the contracts explicit would be:
PARAMETER VALUE
W
PERSISTENT STATE
usage U
importance I
confidence C
...
DERIVED UPDATE STATE
stability S = f(U, I, C, history, ...)
plasticity P = g(U, I, C, history, ...)
ROUTING
r_class = class_router(x, state)
r_weight = weight_router(x, r_class, state)
FORWARD
y = apply(W, r_class, r_weight, x)
CREDIT / UPDATE
gradients or another credit signal
-> optimizer/update rule
-> parameter change modulated by P
STATE UPDATE
U, I, C, ... <- observations from this step
This immediately exposes several independent decisions.
What exactly does each state mean?
For example:
usage
- number of times selected?
- cumulative routing probability?
- activation magnitude?
- contribution to output?
importance
- loss sensitivity?
- output sensitivity?
- Fisher-like estimate?
- accumulated contribution to loss reduction?
confidence
- router confidence?
- certainty that a weight is useful?
- uncertainty of the parameter estimate?
stability
- an independently learned state?
- or a value derived from importance/history?
plasticity
- multiplier on raw gradient?
- multiplier on optimizer learning rate?
- multiplier on the realized parameter update?
- coefficient of a local plasticity rule?
Those choices are not merely naming details; they can produce different algorithms.
For example, if by confidence you mean something like parameter uncertainty, then there is a surprisingly direct research branch.
Uncertainty-guided Continual Learning (UCB) adapts learning rates according to uncertainty in Bayesian weight distributions.
More recently, the MESU method in Bayesian continual learning and forgetting in neural networks makes this relationship especially explicit: uncertainty controls metaplasticity, so uncertain parameters remain more adaptable while confident parameters become more stable.
That suggests one possible simplification:
confidence / uncertainty
|
v
plasticity
instead of assuming that confidence, stability, and plasticity all have to be independent learned variables.
That is only one design option, though; your confidence may mean something entirely different.
Where should the state live?
It also does not necessarily need to live inside a Python object representing each scalar.
Conceptually you can keep a “stateful weight” while physically representing the states as same-shaped tensors:
weight
usage
importance
plasticity
or keeping some of them in:
model state
optimizer state
router/controller state
external memory
This is partly why learned optimizer work is another useful neighboring area: the rule deciding how a parameter changes can itself consume parameter/gradient/history information, without requiring the model parameter object to own all of that logic.
The question I would use to choose the boundary is:
Who reads this state, and who is allowed to update it?
For example:
router reads usage
optimizer reads plasticity
importance estimator writes importance
state update writes usage
is a much easier contract to test than one object implicitly doing all four jobs.
Selection is not the only form of conditional parameterization
There is also a useful neighboring design choice:
select stored parameters
vs
compose stored parameter bases
vs
generate parameters from context
Your current sketch is mostly the first.
CondConv is a clean example of the second: it computes input-conditioned coefficients and combines expert kernels before the convolution.
ParaX is also closer to composition than to a pure one-hot hard selector.
This matters because hard scalar routing creates one set of optimization/system problems, while soft composition creates another. If the high-level goal is “different inputs should use different effective parameters,” it may be useful to keep all three implementations open initially.
Cheap controls I would use before scaling the idea up
I think a small controlled experiment can answer more than starting immediately with a large model.
A useful minimal matrix is:
| Condition |
State |
Learned routing |
| Dense baseline |
no |
no |
| State only |
yes |
no |
| Routing only |
no |
yes |
| Routing + state |
yes |
yes |
| Random-routing control |
same capacity |
no learned router |
The last condition is particularly useful.
If learned routing beats dense but not a capacity-matched random partition, the improvement may come from reducing parameter interference or increasing effective modularity rather than from the routing policy itself.
If learned routing also beats random routing, there is stronger evidence that input-conditioned selection is doing useful work.
An oracle router can also be useful in a synthetic experiment as an upper-bound control, but obviously not as a deployable method.
What I would log
At minimum:
old-task performance after new learning
new-task learning speed / final performance
routing frequency per WClass
routing entropy
overlap between parameter subsets
fraction of parameters never selected
fraction selected almost all the time
parameter-update norm
drift of "protected" parameters
usage distribution
importance distribution
correlation between the two
Those measurements would also catch an easy failure mode if usage feeds back into future routing:
selected often
-> usage increases
-> gets protected/preferred
-> selected even more
That may or may not happen in your design, but ordinary MoE systems already need mechanisms for routing imbalance. The Hugging Face Switch Transformers documentation exposes things such as expert capacity and router auxiliary losses for related reasons.
I would therefore log utilization before trying to design a complicated anti-collapse mechanism.
A small sanity check produced two implementation warnings
I tried this only on tiny synthetic sequential-learning problems, so I would not treat the result as validation of the architecture. It was useful mainly for exposing implementation semantics.
1. Learned routing should have a random-routing control
In the toy setup, a learned soft router reduced forgetting more than both a dense model and a fixed-random router.
However, the expert indices did not cleanly correspond to task identities, so I would not describe this as discovering hard task-specific subnetworks.
A safer interpretation was simply:
the learned conditional mixture reduced interference better than the controls in that toy problem.
That is exactly why the random-routing condition seems worth keeping.
2. plasticity depends on where it acts in the optimizer pipeline
I also tried using a per-parameter plasticity value as a gradient multiplier.
With SGD this behaved as expected.
With AdamW, a fixed raw-gradient scale could be almost cancelled by the adaptive normalization, while a time-varying scale changed the trajectory. Applying plasticity to the realized parameter update after the AdamW step also produced different behavior.
I would not generalize that toy result into “AdamW ignores plasticity.” The useful lesson is narrower:
plasticity needs an operational definition that includes where it acts relative to the optimizer.
These are different algorithms:
raw gradient
|
* plasticity
|
AdamW
versus:
raw gradient
|
AdamW
|
realized update
|
* plasticity
and there are other possibilities.
If the aim is for plasticity = 0.1 to literally mean “this parameter changes one tenth as much,” that contract should be tested on the actual parameter delta, not assumed from a gradient multiplier.
One PyTorch trap: inactive, zero-gradient, and frozen are different states
This became important when thinking about individual scalar selection.
These statements are not equivalent:
this parameter contributes zero to this forward pass
this parameter's gradient tensor contains zero
this parameter received no gradient (grad is None)
the optimizer does not update this parameter
this parameter value is guaranteed not to move
PyTorch documents this distinction explicitly in Optimizer.zero_grad: optimizers behave differently when a gradient is zero versus None; in one case the optimizer can perform a step with a zero gradient, while in the other it skips the step.
This matters with momentum/Adam state and weight decay.
In a small test, an inactive branch that was genuinely absent from the graph (grad=None) stayed unchanged, while a branch that was still in the graph but multiplied by a zero mask could drift from previously accumulated optimizer state. Clearing that state removed the moment-driven drift in the no-weight-decay condition.
So if stability or WClass selection is supposed to mean strictly frozen, I would test the contract directly:
before = parameter.detach().clone()
optimizer.step()
drift = (parameter.detach() - before).abs().max()
rather than assuming that a zero mask guarantees it.
There is also a practical granularity issue here.
If thousands of scalar weights are coordinates inside one Parameter tensor, you cannot normally give each coordinate an independent grad=None state; None exists at the Parameter level.
That does not make weight-level protection impossible, but it suggests that strict scalar-level freezing may require something like:
custom coordinate-wise optimizer masking
+
coordinate-wise optimizer-state masking/reset
rather than only masking the forward value or raw gradient.
That is one reason I would prototype the semantics at block/channel granularity first, then move downward once the update contract is clear.
Partial backprop is quite plausible; no-backprop is a separate question
I would split your backprop question into several levels.
Level 1: only selected parameters receive/update gradients
This is already normal in conditionally executed computation graphs if the inactive branch really is not used.
At the more explicit sparse-gradient end, meProp is directly relevant: it performs the normal forward pass but keeps only top-k gradient components in the backward pass, updating only a small subset of parameters.
So “do I have to update every parameter on every sample?” is clearly no.
Level 2: make sparse backward actually cheaper
Logical gradient sparsity does not automatically produce wall-clock speedup.
SparseProp is useful precisely because it addresses the systems side: it implements sparse backpropagation specialized for sparse weights and demonstrates CPU training speedups.
That distinction becomes especially important if you go all the way to arbitrary scalar sparsity.
Level 3: replace the ordinary global backward dependency
That becomes a different research branch.
Decoupled Neural Interfaces / Synthetic Gradients predicts gradients locally so modules can be updated without waiting for true downstream backpropagated gradients.
The Forward-Forward Algorithm goes further and investigates replacing forward+backward training with two forward passes using local objectives.
There are also local/plasticity-rule approaches, including Differentiable Plasticity.
I would therefore phrase the relationship as:
stateful selective routing
|
+--> selective parameter update [very direct connection]
|
+--> sparse / partial backprop [plausible extension]
|
+--> no backprop at all [separate credit-assignment problem]
In other words, your architecture may make partial backprop natural, but it does not automatically solve the problem that backprop normally solves: assigning credit to the router and the selected parameters.
For a hard discrete selector, that credit-assignment question becomes especially visible. A soft differentiable router, a straight-through estimator, reinforcement-style routing, evolutionary selection, etc. are different possible answers rather than implementation details.
Logical sparsity is not necessarily hardware-efficient sparsity
If compute reduction is one of the goals, I would keep a separate metric for it.
There are several different meanings of “only some weights are active”:
semantic/routing sparsity
graph sparsity
gradient sparsity
optimizer-update sparsity
hardware-efficient sparsity
They need not coincide.
For example, multiplying a dense tensor by a binary mask may give the desired mathematical behavior while still executing dense matrix multiplications.
The current Hugging Face Experts backends documentation is a nice concrete example of this distinction. At the high level, every backend performs the same MoE semantics:
router selects k experts
-> selected expert projections
-> aggregate outputs
but the actual execution can be an eager loop, batched matrix multiplication, grouped matrix multiplication, or specialized fused GPU kernels, with substantially different performance characteristics.
So if scalar selection is primarily about learning behavior, arbitrary scalar masks are a reasonable experiment.
If it is primarily about speed, I would seriously consider testing progressively coarser structure:
scalar
|
small block
|
channel / neuron
|
matrix / adapter
|
expert / module
and measure where the hardware benefit begins to dominate the routing overhead.
This is another place where I would avoid assuming that the most biologically synapse-like granularity is automatically the most useful computational granularity.
A compact way I would navigate the literature is:
If your main goal is preventing forgetting:
Synaptic Intelligence
MAS / EWC-family methods
Conditional Channel Gating
UCB / MESU
Hash
If your main goal is input-dependent specialization:
conditional computation / MoE
Modular Deep Learning
ParaX
CondConv
If your main goal is individual-weight selection:
Piggyback
Supermasks in Superposition
If your main goal is learned per-weight plasticity:
Differentiable Plasticity
UCB / MESU
learned optimizer literature
If your main goal is sparse training compute:
meProp
SparseProp
structured/block/expert routing implementations
If your main goal is eliminating backprop:
synthetic gradients / DNI
Forward-Forward
local-learning / plasticity-rule research
For a broader taxonomy, the Modular Deep Learning survey is also useful because it explicitly separates computation, routing, aggregation, and training instead of treating “modularity” as one algorithm.
If I were choosing one default route for a first implementation, I would probably do this:
1. Use a very small model.
2. Make WClass a block/channel/small-module grouping first.
3. Start with only two persistent quantities:
usage
importance
4. Derive stability/plasticity from them initially,
rather than making five independent state variables at once.
5. Compare:
dense
state only
routing only
routing + state
random routing
6. Record:
retention
new-task adaptation
routing entropy/utilization
overlap of selected parameters
actual parameter drift
7. Only after those semantics are clear:
try scalar-level routing,
more state dimensions,
hard routing,
sparse backward,
or non-backprop credit assignment.
That would keep the original idea intact while making each claim independently testable.
The part I would be most interested in seeing separated experimentally is:
Does persistent parameter history improve the routing policy, the update policy, or both?
If that distinction becomes clear, it should also become much easier to tell which existing research line is the closest comparison — and which part of the design is genuinely doing something different.