Hmm… from what I can tell from the public CSVs, maybe something like this:
I think there are several plausible ML uses here, and the one that looks especially natural to me is controlled evaluation / diagnostics, not only using the outputs as generic synthetic training data.
The interesting part may not be that the images are synthetic by itself, but that your generator can potentially turn design parameters into known, controllable interventions with exact metadata.
From the public CSVs, the three examples already seem to point toward somewhat different task families:
| Collection |
What the metadata looks useful for |
| Wavy Gradient |
Reading visual parameters such as colors, blur, twirl, and noise; controlled one-parameter changes |
| Pinwheel Shapes |
Recovering a procedural transformation recipe; reasoning over multi-step transformations |
| Flowy Image Blend |
Reading animation/compositing properties from video, or generating video from a structured composition specification |
That seems reasonably close to work that is already appearing in graphic-design evaluation. For example, GraphicDesignBench (GDB) has executable benchmarks for layout, typography, SVG, templates, and animation. Its property-extraction tasks include things such as font size, letter spacing, motion type, duration, and timing parameters.
So I would probably keep the current collections as they are, and—if you wanted a research/evaluation-oriented version as well—derive a small benchmark layer from them rather than redesigning the original data.
For example, a very small Wavy subset like this seems potentially high-leverage:
same palette
same twirl
same noise
blur = 40
vs.
blur = 70
Then a model can be asked:
What changed?
Which parameter changed?
What were the before/after values?
That gives a rather different kind of dataset from broad parameter-space sampling, even though it can come from exactly the same generator.
The older TRANCE / Transformation Driven Visual Reasoning benchmark is a useful conceptual precedent: it evaluates inference of a single-step transformation or a sequence of transformations from initial and final visual states. More recent counterfactual VLM evaluations use a similar general idea of changing one factor while trying to leave the others fixed.
So if I had to pick only one small experiment, a tiny one-factor-at-a-time paired subset would probably tell people a lot about what is special about this generator setup without requiring you to change the larger collections.
How I would map the three CSV examples to actual tasks
1. Wavy Gradient: parameter reading + controlled counterfactuals
The Wavy CSV is perhaps the easiest one to turn into a conventional benchmark because most of the important variables are scalar or categorical:
top/bottom color
middle color 1
middle color 2
Gaussian blur
twirl
noise
A first task does not require generating anything new at all:
rendered image
↓
predict:
colors
blur
twirl
noise
The evaluation can also be quite simple:
- categorical accuracy for discrete labels,
- a color-distance metric or exact palette classification for colors,
- MAE or another numerical error metric for blur/twirl/noise.
This is broadly the same shape as some GDB property-extraction tasks. For example, GDB extracts multiple typography parameters from a rendered design and evaluates categorical properties separately from numerical ones.
A second task: controlled change detection
The more distinctive version would be to generate matched pairs:
A:
blur = 40
twirl = 130
noise = 12
B:
blur = 70
twirl = 130
noise = 12
or:
A:
twirl = -120
B:
twirl = +120
with everything else held constant.
That lets you test not merely whether a model can associate an image with a parameter, but whether it is actually sensitive to the intended intervention.
The current CSV looks more like broad sampling over the parameter space—I did not find obvious one-variable-only pairs in it—so I would see this as a derived evaluation set, not as something missing from the current collection.
One useful caution: design rules are also structure
The Wavy metadata also appears to encode actual design constraints rather than treating every value as freely interchangeable. In particular, the outer and middle color roles appear to come from different color pools.
That seems completely reasonable for a design generator; I would not try to remove that from the production dataset.
It just means that, for evaluation, I would distinguish:
"randomly sampled factors"
from:
"controlled changes that remain inside the generator's design rules"
That second formulation may actually be more interesting.
It also makes deliberately constructed challenge splits possible later—for example, checking whether a model has learned the intended visual property or merely a correlated palette/position rule.
2. Pinwheel Shapes: inverse procedural reasoning
The Pinwheel CSV seems qualitatively different.
Rather than just describing a finished image, it contains something much closer to a procedural recipe, with fields corresponding to operations such as:
shrink
twirl
noise
rotate
twirl
radial blur
noise
and some rows contain multiple values/stages.
That suggests an unusual inverse task:
final rendered image
↓
recover the generation recipe
This could be evaluated at several levels rather than with one all-or-nothing score:
Did the model identify the correct operations?
Did it recover the numerical parameters?
Did it get their order correct?
Did it reconstruct the complete recipe exactly?
That decomposition seems useful because a model could understand most of a procedure while missing one operation or getting one numerical value wrong.
TRANCE is again a useful conceptual connection here. Its simpler setting predicts a single transformation, while the harder settings predict a sequence of transformations.
If you can cheaply retain or render an initial state—or perhaps a few intermediate states—the task could become even cleaner:
initial
↓ shrink
state 1
↓ twirl
state 2
↓ noise
...
↓
final
Then you can separately test single-step and multi-step reasoning.
But that is optional. The existing final image → recipe mapping is already an interesting inverse-procedural task.
There is also a potentially useful shortcut-control example already in the CSV
One thing I noticed is that the apparent recipe depth and palette size are not independent in the public Pinwheel CSV.
That does not make the collection “wrong”—it may simply reflect the intended design rules.
For a diagnostic benchmark, though, this gives you a nice opportunity:
normal split:
keep the natural correlation
challenge split:
balance palette size across recipe depths
Then you can see whether a model really learned something about transformation complexity, or whether it just counted colors.
That is the sort of distinction procedural generation makes unusually easy to test.
3. Flowy Image Blend: animation/composition tasks
Flowy looks different again because the metadata describes something closer to:
source assets
+
layer/composition settings
+
animation settings
↓
rendered video
The CSV records things such as source images, blend modes, scales, loop duration/cycles, animation mode, and some effect switches.
That maps quite naturally onto the temporal side of GDB. GDB includes tasks for things such as:
- motion type classification,
- animation property extraction,
- duration/start-time estimation,
- animation parameter generation,
- trajectory generation,
- short-form video generation.
So two directions seem possible.
Understanding
MP4
↓
predict:
animation mode
blend modes
layer scales
loop duration
effect switches
Generation
source images
+
structured composition/animation recipe
↓
target video
The second direction is especially interesting because the CSV is not merely a label table—it is close to a compact generation specification.
One thing I would separate for benchmark purposes is animation mode from the other conditions.
In the current public CSV, the Flowy and Random groups also differ in some other settings/batch characteristics, so I would not treat those existing rows as a clean experiment saying:
the only independent variable is animation mode
But again, that does not require changing the collection.
A tiny derivative set such as:
same 3 source images
same blend modes
same scales
same loop settings
Animation = Flowy
vs.
Animation = Random
would give you the clean comparison if you ever wanted it.
Why the paired subset is different from simply generating more samples
I think there are really two useful products hiding in the same generator.
Broad collection
Good for:
- coverage,
- training,
- retrieval,
- discovering the parameter space,
- generating large numbers of examples,
- general structured metadata.
Controlled evaluation slice
Good for:
- sensitivity tests,
- causal/counterfactual-style diagnostics,
- change detection,
- compositional generalization,
- shortcut detection,
- model regression testing.
The second does not have to be large.
For example:
20 pairs: blur only
20 pairs: twirl only
20 pairs: noise only
20 pairs: palette variable only
could already answer questions that thousands of independently sampled examples cannot answer as cleanly.
A useful pattern might therefore be:
existing collection
|
+-- normal train/dev sample
|
+-- one-factor paired evaluation
|
+-- optional challenge split
rather than trying to make one split serve every purpose.
Later: structured holdouts
If the use case becomes model evaluation rather than only data generation, there are some stronger splits you could add without changing the underlying generator:
template-held-out
parameter-combination-held-out
recipe-family-held-out
For example:
train:
blue + low noise
red + high noise
challenge:
blue + high noise
red + low noise
That tests whether a model has learned the factors compositionally rather than memorizing the combinations seen during training.
I would consider this a later step, though. A few controlled pairs probably have a better effort/information ratio initially.
A small Hugging Face-native sample could make this easier to experiment with
If the goal is to get feedback from ML people rather than to change the original delivery format, I think a small Hub sample could help a lot.
It does not need to contain the whole catalog.
Something like a few hundred Wavy examples with:
asset_id
file_name
colors
blur
twirl
noise
generator/version fields if available
would be enough for someone to load it and try an evaluation quickly.
The current Hugging Face ImageFolder documentation supports metadata.csv, metadata.jsonl, or metadata.parquet, linked to media using relative file_name / *_file_name fields. It also supports examples that refer to multiple images, which would be useful for before/after pairs.
So, for example, a paired subset could conceptually expose:
input_file_name
output_file_name
changed_parameter
input_value
output_value
without inventing a custom loader.
The existing CSVs do not need to disappear
I would probably treat the current CSVs as the human-readable/export form and, if useful, add a machine-oriented representation alongside them.
This is most noticeable in Pinwheel, where fields such as:
'-82;192
appear to represent lists.
For ML tooling, the same information could optionally be represented as actual typed lists/nested structures in JSONL or Parquet.
For example:
{
"twirl_1": [-82, 192],
"noise_1": [14, 28]
}
or, if the stages really form repeated operation records:
{
"stages": [
{
"twirl": -82,
"noise": 14
},
{
"twirl": 192,
"noise": 28
}
]
}
I would only choose the latter after confirming that this is really the semantic structure of the generator.
The main point is not “CSV is bad”; the current CSVs are quite useful for inspection.
It is just that a typed Hub representation can remove parsing conventions from downstream code.
Stable IDs are probably more useful than putting everything in the filename
The filenames are impressively systematic, but I would still treat the manifest as the authoritative record.
That is especially clear in Pinwheel, where the filename is useful as a human-readable summary but does not encode every transformation parameter.
A simple pattern such as:
asset_id = stable opaque ID
file_name = convenient descriptive filename
metadata = canonical recipe
usually makes later schema changes easier.
If the data eventually exists in several locations/formats, I would also nominate one copy/manifest as the canonical source of truth rather than maintaining independent versions by hand.
This sounds mundane, but benchmark projects do run into real distribution drift when different hosted copies evolve separately.
Provenance and reproducibility are related, but I would keep them as two separate claims
The public CSVs already show something valuable: fairly detailed provenance.
For example, depending on the collection, they contain:
- exact numerical generation settings,
- selected colors,
- transformation recipes,
- source image filenames,
- composition parameters,
- run identifiers/output names.
That alone can be useful even without requiring somebody else to execute the generator.
I would separate that from the stronger statement:
given this record, another run reproduces the same output
because the latter can depend on things outside the row itself:
generator/script version
template version
host application/version
random seed/RNG state, if applicable
fonts/assets/plugins, if applicable
render/export settings
I have not verified the generator itself from the public material, so I would treat exact regeneration as something the workflow may support rather than something independently demonstrated by the CSV alone.
If reproducibility becomes an important selling point, a very cheap demonstration would be a handful of records like:
recorded recipe
↓
regenerate
↓
compare against reference output
with an output checksum when byte-identical output is expected, or an appropriate render comparison when it is not.
You also do not necessarily need to invent all of the provenance vocabulary yourself.
The current Croissant 1.1 specification includes dataset versioning and file checksums, and can connect to provenance relationships such as wasGeneratedBy and wasDerivedFrom.
I would see that as an optional interoperability layer, not as something the collection needs in order to be useful.
One distinction that may help later: generator truth vs. aesthetic truth
One thing I would keep conceptually separate is:
generator truth
from:
design/aesthetic judgment
For example, the generator can tell you exactly that:
blur = 60
twirl = 180
letter spacing = X
blend mode = SCREEN
Those are excellent machine-readable ground-truth values.
It cannot by itself establish:
blur = 60 is aesthetically better
this layout has good hierarchy
this color combination is appropriate
Those are different labels and may require human/designer judgments depending on the task.
That separation can actually be useful.
You could have:
Layer A:
exact construction/generation metadata
Layer B, only when needed:
human preference / aesthetic / task-quality annotations
rather than mixing the two.
So I would not add subjective labels unless a particular research use case calls for them.
If editable/vector sources are retained
This branch only applies if the editable/vector structure is available in addition to the rendered output.
If so, there is another interesting task family:
requested edit:
change stroke width
allowed to change:
stroke width
must remain invariant:
palette
position
other objects
layout
That turns the generator metadata into an edit contract.
The evaluator can test not just whether the requested change happened, but whether unrelated properties stayed unchanged.
This is related to the direction taken by recent SVG editing evaluations, where successful editing and preservation of untouched structure are treated as separate properties.
It would also give the SVG output a different role from the PNG render:
PNG:
perceptual result
SVG/editable source:
structural/editable result
generator metadata:
intended transformation
But I would only go down this route when editable source preservation is already natural for the workflow; I would not add it merely to satisfy a benchmark.
A possible task-selection map
The way I would think about the collections is roughly:
Can one design factor be changed independently?
|
+-- Yes
| |
| +-- Can you render before/after?
| | |
| | +-- Yes
| | | -> controlled change / counterfactual reasoning
| | |
| | +-- No
| | -> single-image parameter extraction
| |
| +-- Can combinations be deliberately held out?
| |
| +-- Yes
| -> compositional / OOD challenge split
|
+-- No
|
+-- Is the structured generation recipe retained?
| |
| +-- Yes
| -> recipe inference / structured generation
|
+-- Is only the rendered output retained?
|
+-- attribute/property recognition
Then there are two useful side branches:
video output
-> animation / temporal property tasks
editable vector source
-> structured editing / preservation tasks
So I do not think all of the collections need to be normalized into one kind of benchmark. Their differences may be useful.
The overall pattern I see is therefore:
your existing generator
+
your existing asset-level metadata
|
+-- broad visual dataset
|
+-- small controlled evaluation dataset
|
+-- optional provenance/reproducibility layer
Those can be separate outputs rather than competing design goals.
If I were trying one thing first, I would probably take Wavy Gradient, generate a small number of matched one-parameter pairs, and publish that together with the existing scalar metadata as a tiny evaluation sample.
That seems like a fairly low-cost way to demonstrate something that is harder to get from ordinary scraped/curated image datasets: not merely knowing what metadata happens to correlate with an image, but knowing exactly what was deliberately changed while the rest of the generation setup stayed fixed.