For now, after running a few experiments in Colab, I think I found something along these lines:
Short answer: yes — I think the q_proj / v_proj default is worth revisiting. But I would separate two questions:
- Is
q/v a universal optimum? Probably not.
- Does the current TIM scalar score already provide a universally calibrated replacement? My small tests do not support that stronger claim yet.
What I found instead is, to me, a more interesting middle ground: the TIM module-level diagnostic seems to contain useful signal, but placement quality can depend strongly on interactions between selected modules, enough that simply averaging/adding independent module scores can miss a very bad configuration.
That suggests a relatively cheap workflow:
TIM diagnostic
↓
small shortlist of candidate placements
↓
verify actual PEFT targets + trainable parameter budget
↓
tiny local / one-swap neighborhood check
↓
compare retention at matched new-task acquisition
↓
choose the final placement
In other words, I would not throw away the measurement idea. I would probably separate measurement, candidate generation, configuration validation, and final selection.
There is also an important distinction depending on what “better placement” is supposed to optimize:
Maximum downstream task performance
→ all-linear is an important baseline
Small adapter / fewer adapted modules / serving efficiency
→ selective placement is directly the optimization problem
Retention-aware or continual adaptation
→ compare old-task retention at comparable new-task learning
and placement/interference diagnostics become especially relevant
That distinction matters because the original LoRA q/v choice was not arbitrary. The original LoRA work, under its GPT-3 attention-only fixed-parameter-budget ablation, found q+v to be a strong configuration, and the official LoRA repository describes q/v as a “simple yet effective setup”. But the same repository explicitly says LoRA can be applied to other subsets and that the optimal configuration is likely to vary by architecture and task.
Later work makes that boundary even clearer. QLoRA found, for its LLaMA-7B Alpaca experiment, that standard q/v LoRA did not match the 16-bit baseline and that adapting all linear Transformer-block layers was important for matching it. Current PEFT LoRA consequently supports explicit targets as well as target_modules="all-linear". So I would treat q/v as a useful historical baseline, rather than as something that needs to be defended as a theorem.
For a recent independent example that target choice still matters materially, Amazon’s Nova 2.0 Lite target-module ablation found quite different behavior among qkv, o_proj, feed-forward targets and combinations across tasks. That does not validate TIM — it is a different model and objective — but it does support the broader premise that module placement is a real optimization variable.
A few cheap checks also seem worth doing before making the selector itself more complicated:
- compare actual LoRA trainable parameters, not only the number of targeted modules;
- record the modules that PEFT actually adapted;
- evaluate acquisition and retention together, rather than retention alone;
- keep generated baselines and published/stored baselines distinguishable if they are not literally identical.
For the first two, current PEFT guidance for custom models already exposes useful sanity checks:
peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)
I found these surprisingly useful, because a configuration that looks structurally matched can still differ in actual LoRA capacity or in the modules PEFT really attached to.
What I tested in Colab
Scope first
This is not a reproduction of the four-model headline experiment.
I used:
HuggingFaceTB/SmolLM2-135M- a fixed small synthetic code/prose task pair
- three seeds for the GPU comparisons
- a five-depth panel at layers
0, 7, 14, 22, 29
q_proj, k_proj, v_proj, o_proj- the same general TIM score construction used by the implementation I was testing
- explicit checks of PEFT targets and actual trainable parameters
So I would read the results as a controlled sanity experiment about the selector and placement behavior, not as evidence that the same numerical effect must occur on Llama-3, HumanEval, CodeSearchNet/OpenWebText, etc.
1. The module diagnostic itself was reasonably stable
Before training anything, I repeated the small TIM measurement on two independent sample blocks.
On the later calibrated run, module-score agreement was approximately:
- Spearman: 0.8075
- Kendall: 0.6316
So, at least in this toy setup, “the TIM score is just random sample noise” did not look like the best explanation.
I also did an earlier parameter-matched training comparison between:
- a stable TIM-selected placement,
- an
o_proj placement,
- a budget-matched q/v placement,
- and a simple Task-B-only gradient-salience placement.
Under a first-crossing control for Task-A learning, the selected TIM placement was encouraging: it beat the matched q/v arm on the main retention/final-A/final-B readouts across the three toy seeds, and its retention was better than the simple o_proj and Task-B-salience alternatives.
I would call that positive selector evidence, but narrow evidence. It does not by itself tell us what the numerical TIM score means globally.
2. So I tested the stronger score claim directly
The next question was:
If TIM score is being used as a placement-quality quantity, do progressively higher-scoring placements actually forget progressively less?
To make that harder to explain away with simple capacity differences, I enumerated 390 candidate placements and selected five score quantiles while holding the gross placement structure fixed.
Every selected placement had:
- 6 target modules
- exactly
2 o_proj + 1 q_proj + 1 k_proj + 2 v_proj
- coverage of all five panel layers
- exactly 46,080 actual PEFT trainable parameters
- the same data and optimizer setup
- the same three seeds
- Task-A checkpoints selected by a common earliest first-crossing criterion before Task B
The result was:
| score band |
mean TIM placement score |
mean Task-A forgetting ΔNLL |
| very low |
0.450847 |
-0.0385 |
| low-mid |
0.482286 |
-0.0809 |
| middle |
0.494613 |
+1.2799 |
| high-mid |
0.508072 |
-0.0878 |
| very high |
0.528385 |
-0.0004 |
The expected direction, if “higher score = safer retention”, would be a negative score/forgetting correlation.
Observed Spearman was +0.10.
For that expected negative direction, the exact one-sided permutation p-value was 0.6083.
With only five placements this is obviously not a high-powered statistical calibration study, so I would not interpret this as “TIM is disproven”. The narrower statement seems much safer:
The tested module-level diagnostic was reasonably stable, but its simple placement-level scalar aggregation was not a monotonic predictor of forgetting in this matched small placement space.
Those two observations are compatible. A signal can be useful for generating candidates without its numerical value being a calibrated global utility function.
3. The strange part was the middle placement
The middle placement was:
layer 0 q_proj
layer 7 o_proj
layer 7 v_proj
layer 14 o_proj
layer 22 v_proj
layer 29 k_proj
In the original calibration its Task-A forgetting ΔNLL was:
seed 11 +1.3542
seed 37 +1.2379
seed 83 +1.2476
mean +1.2799
The other score-quantile placements were near zero or slightly negative.
Task-B training loss still fell normally, so this did not look like a simple “training failed to run” case.
At that point, the obvious structural clue was the colocated o_proj + v_proj at layer 7. But that still leaves several explanations:
o7 itself might be problematic;v7 itself might be problematic;- the specific
o7 + v7 pair might be problematic;
- any duplicated placement at layer 7 might be problematic;
- the destination topology of the other modules might matter;
- or the six-module configuration might have a higher-order interaction that cannot be assigned to one module.
A full combinatorial sweep seemed unnecessary, so I tried the smallest local intervention I could think of.
4. Strict one-swap neighborhood
I reran the center plus six neighbors.
Each neighbor changes exactly one of the two layer-7 modules while preserving:
- 6 modules,
2o + 1q + 1k + 2v,- all five depth locations represented,
- LoRA rank/alpha,
- optimizer and data,
- 46,080 actual trainable parameters,
- the three seeds,
- and seed-local Task-A first-crossing matching.
The pathological center also reproduced qualitatively in this second experiment:
middle forgetting ΔNLL
seed 11 +1.2671
seed 37 +0.7483
seed 83 +1.0657
mean +1.0270
The exact magnitude moved because the seven-arm experiment recomputed the seed-local common reachable Task-A target, but the large center pathology remained in every seed.
Then the neighborhood did this:
| placement |
score |
Δscore vs middle |
mean Task-A forgetting ΔNLL |
mean Task-B acquisition ΔNLL |
middle |
0.494613 |
0 |
+1.0270 |
+0.0581 |
o7 → o0 |
0.459956 |
-0.034657 |
-0.0378 |
+0.1546 |
o7 → o22 |
0.464996 |
-0.029617 |
-0.0158 |
+0.1847 |
o7 → o29 |
0.465893 |
-0.028719 |
-0.0498 |
+0.1903 |
v7 → v0 |
0.487957 |
-0.006656 |
+0.0207 |
+0.1127 |
v7 → v14 |
0.492006 |
-0.002607 |
+0.1098 |
+0.0762 |
v7 → v29 |
0.507622 |
+0.013010 |
-0.0378 |
+0.1413 |
Every one of the six strict neighbors reduced forgetting substantially relative to the center for all three paired seeds.
I would not call that “18 independent successes” — those paired comparisons are strongly related — but the directional pattern is difficult to miss.
5. It was not simply “the neighbor learned Task B less”
This was important to check because LoRA retention can otherwise be deceptive.
The paper LoRA Learns Less and Forgets Less gives a good general warning here: less forgetting can accompany less learning of the target distribution.
That is why I would avoid interpreting retention without an acquisition measure.
In this one-swap experiment, however, the rescue does not look like that trivial trade-off.
The pathological middle arm had mean Task-B acquisition of only about 0.058 NLL, while the neighbors were around 0.076–0.190 NLL.
So the neighbors generally:
- learned Task B more, and
- forgot Task A dramatically less.
That looks more like escaping a bad placement/configuration region than merely reducing plasticity to preserve Task A.
I still would not infer a specific mechanism from that result alone, but it makes “they simply did not learn B” a poor explanation here.
6. What seems localized — and what does not
The result makes the exact layer-7 configuration a strong suspect, but I would keep the wording narrow.
It is not enough to say “putting o_proj and v_proj in the same layer is generally bad.”
For example:
o7 → o22 creates an o22 + v22 pair and rescues the pathology;v7 → v14 creates an o14 + v14 pair and also rescues it.
So generic o+v co-location is not sufficient to explain this toy result.
What the experiment really says is closer to:
Something about the particular layer-7 configuration, or its interaction with the rest of this placement, matters a lot.
A useful next localization experiment if this phenomenon itself becomes interesting would be a tiny layer-7 pair-identity panel:
o7 + v7 current pathological center
q7 + v7
q7 + o7
k7 + v7
k7 + o7
with the same type-count / layer-coverage / parameter-budget controls.
That would distinguish “specifically o7+v7” from “two adapters at layer 7 are problematic”.
But I would regard that as a research extension, not something needed before TIM is useful.
7. Where simple additivity becomes questionable
The cleanest counterexample to a simple additive interpretation was v7 → v29.
Its mean scalar placement score moved up:
middle 0.494613
v7 → v29 0.507622
yet its Task-A forgetting moved from approximately:
+1.027 → -0.038
So the local behavioral improvement was enormous while the additive score changed in the opposite direction from what a simple “higher score = less forgetting” interpretation would predict.
That is why I currently think the useful distinction is:
module-level diagnostic signal
≠
complete configuration-level utility function
The first can be useful even if the second needs an interaction term.
There is some broader theoretical precedent for being cautious about additivity here. The recent paper Understanding and Guiding Layer Placement in Parameter-Efficient Fine-Tuning of Large Language Models explicitly treats layer coupling as a separate quantity, with approximately additive layer contributions arising in a weak-coupling regime. I would not claim that paper explains this particular layer 7 observation — different setup, theory and objective — but it gives a useful language for the general issue: independent per-location measurements need not compose additively when adaptation sites interact.
How I would separate TIM's roles
The result above makes me think TIM may become easier to reason about if four roles are kept separate.
1. Measurement
Measure the Task-A / Task-B gradient geometry at individual candidate modules.
This is where the distinctive part of TIM lives.
The small repeated-block experiment gave me some confidence that there is a reproducible signal here.
2. Candidate generation
Use the measurement to shrink the placement search space.
This does not require claiming that the scalar score is perfectly calibrated.
For example, the output could be:
top candidate
+
a few near-top structurally different alternatives
rather than:
highest mean score = final answer
3. Configuration validation
Before doing the expensive run:
- verify actual adapted modules,
- verify the actual trainable parameter budget,
- perturb the proposed placement locally,
- and run a short matched-acquisition check.
The one-swap result makes this step look particularly high-information.
You do not need a huge search. Four to six deliberately chosen neighbors may already reveal whether the top candidate is sitting on a sharp interaction pathology.
4. Final decision
Pick the operating point using the quantity actually relevant to the use case.
For continual adaptation I would prefer something like:
new-task acquisition
vs.
old-task forgetting / retained old-task performance
rather than ranking placements by forgetting alone.
If the goal is ordinary single-task SFT performance instead, I would change the baseline set rather than forcing a continual-learning objective onto it.
The baseline depends on the goal
I think this distinction could prevent several comparisons from becoming unnecessarily confusing.
If the goal is maximum downstream performance
Include an all-linear configuration.
This is the natural lesson from QLoRA: in its LLaMA-7B Alpaca experiment, q/v-only LoRA did not match the tuned 16-bit baseline, while LoRA on all linear Transformer-block layers did.
That does not imply all-linear is always best. It means it is an important performance-oriented reference point.
If the goal is sparse adapters or inference efficiency
Then all-linear may be the wrong target entirely.
Now the real question becomes:
Which small subset buys most of the useful adaptation?
This is where a placement selector is directly valuable.
A recent example is the Amazon Nova 2.0 Lite target-module study, where o_proj, qkv and feed-forward combinations produced materially different accuracy/latency trade-offs across tasks.
Again, that particular result is model-specific. I mainly cite it because it makes the optimization problem concrete: target selection can matter precisely because adding adapters everywhere is not free at serving time.
If the goal is retention-aware adaptation
Then TIM has a particularly interesting niche because it explicitly compares Task-A and Task-B geometry rather than only asking “where is Task B salient?”
That is a different problem from:
- static weight geometry,
- ordinary new-task gradient salience,
- rank allocation,
- or simply adapting every linear layer.
Current PEFT even contains a different automatic target selector, KappaTune / find_kappa_target_modules, based on weight condition numbers. That might make a useful low-integration-cost comparator if you ever want one.
I would not add a zoo of selectors immediately. One contrastive selector is probably enough to answer the useful question:
Is Task-A/Task-B interference geometry adding information beyond a task-agnostic/static placement heuristic?
A few implementation/reproducibility notes that looked cheap to fix or record
These are secondary to the main result, but they were cheap checks and may help make future comparisons cleaner.
1. Same target-module count does not necessarily mean the same LoRA parameter budget
For the Llama-3 configuration I checked, the stored TIM and conventional placements both contained 31 target modules, but because Llama-3 uses GQA, the projection shapes are not all equal.
At rank 8, I calculated:
TIM placement 1,933,312 LoRA trainable parameters
conventional placement 1,662,976 LoRA trainable parameters
So the suggested placement has roughly 16% more LoRA parameters despite having the same number of targeted modules.
I would not interpret this as “therefore the reported result disappears”. It only means that a pure placement interpretation becomes cleaner if actual LoRA capacity is either:
- matched,
- or at least reported beside module count.
This is one reason I like leaving print_trainable_parameters() in the experiment log.
2. Parameter names and PEFT module targets are slightly different contracts
The diagnostic naturally produces names like:
model.layers.7.self_attn.q_proj.weight
but PEFT’s normal target_modules path targets modules such as:
model.layers.7.self_attn.q_proj
In the PEFT version I tested, passing the .weight form through the normal target_modules route did not behave like targeting the module name.
Current PEFT LoRA documentation also explicitly distinguishes:
target_modules for modules such as nn.Linear;target_parameters for cases where an nn.Parameter itself needs to be targeted.
So I would make that conversion an explicit boundary in the code and then assert the result:
peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)
That turns a potentially silent targeting mistake into a cheap runtime failure.
3. Published and regenerated baselines can be named separately
In the revision I tested, the fresh conventional q/v constructor did not reconstruct the stored/published conventional placement exactly, even though both were q/v-style placements.
That can be made unambiguous without changing the method at all by keeping artifacts named along the lines of:
published_conventional
generated_conventional
and recording the exact target list beside each result.
That also makes future comparisons easier if baseline generation logic changes.
4. A small amount of provenance buys a lot
For each reported arm I would keep:
model revision
PEFT version
seed
rank / alpha
exact requested targets
actual targeted_module_names
actual trainable parameter count
placement JSON / hash
Task-A matching criterion
Task-B training budget
Most of this costs almost nothing and makes placement experiments much easier for another person to reproduce.
If I were choosing follow-up work by information gain
I would roughly order it this way.
Tier 0: nearly free
Using existing runs:
- report actual LoRA parameter counts;
- record exact
targeted_module_names;
- put acquisition and retention in the same table;
- plot TIM placement score against measured forgetting;
- distinguish stored/published and freshly generated baselines.
Tier 1: small GPU cost — probably my default
For a TIM-selected candidate:
- keep the candidate;
- construct a handful of same-budget one-swap neighbors;
- match new-task acquisition approximately;
- compare old-task retention;
- only take survivors into the expensive evaluation.
This is the part that my Colab result changed my mind about most. A tiny local perturbation exposed much more information than another broad score sweep probably would have.
Tier 2: only if the selector itself is the research question
Add one comparator, not ten.
For example:
- PEFT’s KappaTune selector,
- a simple Task-B gradient/Fisher salience score,
- or another cheap static selector.
Then the interesting result is not merely “placement A beats q/v”, but:
Does the Task-A/Task-B relation measured by TIM contain useful information beyond simply selecting modules that look adaptable or important for Task B?
Tier 3: only if the interaction effect reproduces elsewhere
Then it could be worth extending the score itself:
placement utility
≈
module terms
+
pair / topology terms
+
possibly higher-order terms
But I would not start there.
The strict one-swap result is strong enough to justify checking whether this interaction behavior reproduces on another task/model, but not strong enough to justify building an elaborate interaction model immediately.
So my current interpretation would be:
q_proj/v_proj is still a perfectly reasonable baseline, and it has a real empirical history behind it. But I would not treat it as an architecture-independent optimum.
I also would not interpret my small experiment as a rejection of TIM. Quite the opposite: I saw a reasonably stable module-level signal and one encouraging selected-placement result. The main boundary I found was that the simple scalar/additive placement interpretation was much weaker than the underlying diagnostic signal, and a very small local intervention exposed a large configuration effect.
That makes me think a practical version of the idea could be:
TIM for interference-aware candidate generation, followed by a cheap configuration check, with the final choice made on an acquisition–retention operating point.
That seems to preserve the useful part of the idea while avoiding the need to assume that independently measured module scores compose perfectly.