Hmm… For now, based on what I found looking through the existing evidence:
I think the intuition here has quite a bit of support, but I would separate “is this a good sample?” from “is this the sample I should spend training budget on right now?”
My short version would be:
- Validity is a gate: the task/answer/verifier needs to be trustworthy enough to train on at all.
- Value is model-relative: a valid sample can still be redundant, irrelevant to the current weakness, or provide little learning signal at the current checkpoint.
- Difficulty is objective-relative: what makes useful SFT data is not necessarily what makes useful DPO or RLVR data.
- Targeting can be very data-efficient, but I do not think the evidence says it has replaced scale. Broad coverage and targeted correction look more complementary than competing.
- A useful default loop seems to be:
broad / reasonably balanced base mixture
↓
held-out evaluation
↓
identify failure / capability clusters
↓
check that the failure is really a capability/data problem
↓
targeted selection / generation / reweighting
↓
retrain
↓
target eval + broad regression + unseen eval
↓
update the mixture
↺
That is fairly close to what the open Tülu 3 post-training recipe looks like in practice: skill-specific mixtures, synthetic data to fill gaps, iterative add/remove decisions for lagging skills, and explicit development/final evaluation plus decontamination.
So for “what makes a sample worthy of inclusion?”, I would probably break “quality” into something like:
- correctness / verifier validity
- target relevance
- novelty / non-redundancy
- diversity / capability coverage
- useful signal for the specific training objective
- transfer to held-out or unseen examples
And for “scale vs. targeting?”, I would lean toward:
Targeting can be much more efficient under constrained budgets or for localized deficits, but broad data still seems important for coverage, and there are regimes where random selection or simply more useful data remains hard to beat.
The cheapest controls also seem unusually informative: same-size targeted vs. random, fine-tuned vs. current/zero-shot checkpoint, and target metric + broad/unseen regression.
Why I would separate SFT, preference tuning, and RL
The “golden middle” idea looks much stronger for some training objectives than others.
SFT
For supervised fine-tuning, I would not discard a sample simply because the current model cannot solve it.
If there is a correct expert demonstration, a currently-unsolved problem can be useful precisely because it demonstrates a strategy the model does not yet produce.
That also makes “harder = better” questionable. The ACL 2026 paper What Makes Good Instruction-Tuning Data? studies instruction-data selection under constrained budgets and reports that sample difficulty was negatively correlated with their estimated teaching influence.
That is not evidence that “easy examples are always best”; it is evidence that difficulty alone is not a reliable SFT value score.
For SFT I would therefore lean more toward:
Does this demonstration teach useful behavior
on related examples?
than:
Can the current model already solve this prompt?
Preference tuning / DPO
Preference data seems to have another definition of value again.
Things that matter include:
- whether the chosen answer is actually preferable;
- whether the chosen/rejected contrast is informative rather than trivial;
- annotation/judge reliability;
- prompt diversity;
- whether the responses are sufficiently related to the current policy;
- balance across capabilities.
Tülu 3’s DPO ablations are useful here. They report that increasing the number of unique prompts helped, introducing new prompts not reused from SFT helped, and mixing in on-policy completions improved aggregate downstream performance compared with entirely off-policy preference data.
So preference-data quality looks more like:
contrast quality
+ diversity
+ current-policy relevance
than simply task difficulty.
RL / RLVR
Here the “golden middle” intuition has a much more direct connection to the objective.
LILO: Learning to Reason at the Frontier of Learnability prioritizes questions on which multiple rollouts sometimes succeed and sometimes fail. Questions where the current policy always succeeds or always fails provide much less useful training signal in the group-relative RL settings they study.
They report improved final test accuracy across multiple models/algorithms/datasets and up to about a 3× reduction in training steps to reach a given performance level.
Current Hugging Face TRL tooling exposes essentially the same diagnostic idea through frac_reward_zero_std in the GRPO trainer: groups where all completions receive the same reward contribute no useful within-group reward variation.
That makes Parsewave’s published practice of difficulty calibration understandable. Their methodology says they use a pilot batch, automated and human QA, and target roughly 10–30% first-pass success on the target model.
I would treat that particular 10–30% range as their operational heuristic, though, rather than a universal optimum established by the literature.
The more general idea seems to be:
useful difficulty is relative to the current policy and the learning objective, and therefore can move during training.
Targeting a failure mode still leaves a measurement problem
One result I found particularly useful is that “targeted data” itself contains at least two separate design choices:
How do we represent relevance?
+
How do we select data using that representation?
The 2026 study A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn’t) explicitly separates those two pieces and compares them across models, tasks, candidate pools, and budgets.
Their result is more nuanced than “pick the examples most similar to your failures”:
- gradient-based representations were the only tested representations whose distance to the target/query set consistently predicted downstream usefulness;
- no single selection algorithm dominated everywhere;
- gradient representations with a simple round-robin-style selector were particularly strong at low budgets;
- the relative advantages changed as the data budget changed;
- random and zero-shot baselines were often surprisingly informative.
I also re-aggregated the result CSVs released with their public repository. This is not an independent retraining replication, just another view of the authors’ released experiments.
In those released query-loss results, the LESS gradient representation showed the expected relation — samples farther from the target set tended to produce worse target loss — extremely consistently across the model/task/pool cells I checked. Simpler representation families were substantially less consistent.
At the same time, individual targeted methods did not beat a same-budget random baseline everywhere. Depending on the method, the win rate over random across the released result cells was only roughly in the mid-50% to mid-60% range. There were also conditions where even the best fine-tuned result available in the comparison did not improve over the zero-shot/current model.
To me, that suggests a useful distinction:
“Does this example look similar to my failure cases?”
is not necessarily the same question as:
“Does this relevance signal predict a useful model update?”
That is probably one of the reasons I would keep a same-size random subset as a default control before investing in a complicated selector.
Why I would not frame this as quality replacing scale
There is real evidence that small targeted subsets can beat much larger datasets.
For example, LESS reports settings where a target-aware subset around 5% of the candidate training data outperforms training on the full candidate set.
But that is not universal even within that family of results.
If the full pool already contains a lot of useful target-relevant information, removing most of it may not help. And selection itself can distort the training distribution.
BIDS is useful here because it shows that influence-based selection can systematically over-select particular tasks when influence scores are not comparable across capabilities. The resulting imbalance can hurt other capabilities and can even hurt the supposedly favored high-influence task.
Their solution effectively separates:
target influence
from:
capability balance / coverage
That feels like a useful principle beyond their exact algorithm:
Use relevance as one signal, but keep coverage as a separate constraint.
There is also evidence for regimes where scale remains very important. Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning studies a specific mathematical-reasoning SFT→RL setting and reports that dataset scale was the main determinant of post-training potential, with trajectory difficulty behaving more like a multiplier.
So I would probably think in terms of regimes rather than one rule:
| Situation |
What I would expect to matter |
| Small training budget |
Selection can matter a lot |
| Clear localized capability deficit |
Targeted data can be very efficient |
| Highly redundant candidate pool |
Selection/deduplication can matter a lot |
| Broad generalist model |
Coverage/balance remains important |
| Candidate pool already rich in relevant data |
More relevant examples may still help |
| Current checkpoint already saturated |
Additional tuning may have little headroom |
| Different objective (SFT/DPO/RL) |
“valuable data” can mean something different |
So instead of:
scale → quality
I think a more useful picture may be:
broad coverage
+
targeted correction
+
adaptive weighting
+
evaluation feedback
A related distinction: selecting data vs. changing the mixture over time
There may be another axis beyond keep/drop selection:
A sample can be useful now without deserving permanently high sampling weight.
Once a model learns a capability, yesterday’s valuable failure examples may become today’s redundant examples.
That makes post-training look less like constructing one perfect static dataset and more like allocating training mass over time.
Tülu 3 already does a manual version of this by building skill-specific mixtures and then adding/removing data to improve lagging skills.
There are also methods explicitly formalizing dynamic data mixtures. For example, DynamixSFT updates dataset sampling probabilities based on estimates of their usefulness to the model in its current state, while regularizing toward the original mixture so that adaptation does not destroy diversity and coverage.
For preference optimization, AutoMixAlign similarly treats multi-task preference-data composition as an adaptive mixing problem.
This seems quite compatible with the direction in the OP:
identify current weak region
↓
increase useful training mass there
↓
model changes
↓
re-measure
rather than permanently declaring a particular class of examples “high quality”.
Synthetic vs. human-authored may not be the most useful dividing line
I agree with the concern that synthetic datasets can create enormous numbers of superficially different examples that exercise essentially the same capability.
I would just phrase the underlying problem more generally.
Instead of:
human
vs.
synthetic
I think the more informative axes are:
informative vs. redundant
grounded vs. ungrounded
verified vs. weakly judged
coverage-expanding vs. paraphrastic
Human-authored data can also be redundant, ambiguous, or badly distributed. Synthetic data can also deliberately explore missing failure regions.
Tülu 3 is one practical example: its published prompt pool combines public and in-house synthetic data, with synthetic generation explicitly used to fill gaps in target skills rather than simply scale the number of paraphrases.
For software engineering, SWE-smith takes another route: synthetic tasks are grounded in real repositories and filtered using executable tests.
There is also work explicitly generating synthetic examples from observed failures. For example, Learning from Reasoning Failures via Synthetic Data Generation analyzes a weaker model’s reasoning failures and generates/filter examples aimed at those weaknesses.
That particular result is from a multimodal setting, so I would not generalize its numerical results to all post-training. But it is a useful counterexample to the idea that synthetic data inherently means repeated surface variation.
The practical question seems closer to:
Does the generated data expand capability/failure coverage, or merely produce more surface forms of something already represented?
“Verifiable” is important, but the verifier is part of the data pipeline too
For real-world engineering tasks, I think the emphasis on clear ground truth and review is well motivated.
Parsewave’s public Harbor Bot also makes part of that methodology concrete: it includes task validation, Oracle/Nop checks, similarity checking, model runs, and debugging/QA steps.
One distinction I would still keep explicit is:
executable / verifiable
!=
automatically valid
The verifier itself can be incomplete, overly strict, overly broad, environment-sensitive, or misaligned with the written task.
SWE-bench provides a useful example of why this matters.
SWE-bench Verified was created by having expert software engineers review 1,699 candidate tasks, with each independently reviewed by three experts, resulting in a curated 500-task set.
A later OpenAI re-audit of SWE-bench Verified examined 138 tasks that a frontier model did not solve consistently. At least 59.4% of that audited subset contained material problems in the tests and/or problem descriptions, including tests that were too narrow, tests requiring functionality not specified by the task, and environment-sensitive failures.
That does not mean expert review failed or that executable benchmarks are a bad idea — Verified was designed to improve known problems in the original benchmark.
It does show why I would split an observed failure before using it to drive data generation:
observed failure
│
├─ genuine capability deficit
├─ missing training coverage
├─ ambiguous / underspecified task
├─ verifier / test mismatch
├─ environment / dependency failure
├─ rollout / search configuration issue
└─ evaluation contamination / leakage
If the failure is really a verifier artifact, generating hundreds of neighboring examples could reinforce the wrong target.
So failure validation looks like a cheap step worth placing before expensive targeted data production.
Cheap controls that seem unusually informative
Before building a sophisticated data-valuation system, I would probably start with three controls because they separate several explanations at very low cost.
1. Targeted vs. same-size random
1,000 targeted examples
vs.
1,000 random examples
Keep the token/update budget and training configuration fixed.
This answers a surprisingly important question:
Did the selection rule add value, or did the model simply benefit from another 1,000 training examples?
The recent targeted-selection comparison explicitly emphasizes random baselines because random sampling is often much more competitive than expected.
2. Fine-tuned vs. current checkpoint
Keep the zero-shot/current checkpoint in the table.
If:
targeted fine-tune
<=
current checkpoint
there are several possibilities besides “we need a better selector”:
- the candidate pool contains little new information;
- the current model already has the capability;
- the selected examples are redundant;
- the optimization setup damages an already-good capability.
This is an inexpensive way of checking whether there is actual training headroom.
3. Target + broad + unseen evaluation
For a targeted intervention, I would separate:
targeted failure metric
+
broad regression suite
+
unseen related examples
The last part is especially important if the training examples were created from benchmark failures.
It helps distinguish:
learning the underlying capability
from:
learning the neighborhood of the benchmark
For larger pipelines, tooling such as the Open-Instruct decontamination utilities can help detect train/evaluation overlap, but even a modest unseen sibling set already gives a lot of information.
Optional next controls
If those three look promising, then I would consider:
- same selector at several budgets;
- several selectors at the same budget;
- re-estimating sample value after checkpoints change;
- for RL, multiple rollouts and reward/success variance;
- repeated verifier runs for executable tasks.
I would see those as extensions, not prerequisites for testing the basic idea.
So overall, I think the direction in the OP is real, but I would describe the change a little differently than simply “quality instead of scale.”
Something like:
from static bulk data toward increasingly measured allocation of training signal
seems to cover more of the evidence.
That “measurement” can ask:
Is the sample valid?
Is there actual training headroom?
Is this information new?
Is it relevant to this model now?
Does this training objective learn from it?
Does the mixture remain broad enough?
Does the gain transfer beyond the failures that selected the data?
Under that framing, scale still has a role in providing coverage; targeted data has a role in repairing current deficits; adaptive mixtures can change emphasis as the model learns; and evaluation closes the loop.
So I would probably think of a strong post-training dataset less as a static pile of universally “best” samples, and more as a validated, controlled mixture whose marginal value changes with the model and the stage of training.