circlestone-labs

/

Anima

Like 2.43k

Follow

CircleStone Labs 1.08k

7 days and no reply

7 days and no reply

I've just been lost in my work for a while and forgot I asked the question... I don't even really want to train Lore anymore because sometimes on EuLer A or ER SDE, blurriness appears where it shouldn't, and on a regular EuLer, it looks weird. So for now, I'm just sitting there and directing AI agents to study different Lore and look for the best training solutions, without any results so far...

The official recommendations for ai-toolkit anima training guide is really useful: https://www.runcomfy.com/trainer/ai-toolkit/anima-lora-training

Let's have an example lora for character spritesheet with fixed views: front, side, back, few close-ups. If you describe those viewing elements in your dataset, and use an activator on top of it, you train the lora as a suggester / enforcer for an existing concept. You still need to use "multiple views, turnaround, front view, back view..."

But if you got rid of all viewing elements in your dataset, and try to force the concept into your activator, your activator becomes that concept. But, if dataset variety is low AND similarity is high, results will often look bad.

Prompt: @bbf3, (bbaction_card:1.5), goblin, solo, monster, tongue out, long sharp black claws, claws focus
That was my whole prompt on anima turbo model. @bbf3 is the activator for my style lora, unrelated in this case. bbaction_card is the activator for this concept. View is close-up, focuses on one action element + character, has a simple background, background is red (accidental, because most of monster tagged images in concept dataset has red/black background), character is out of frame slightly, and character does not look at the viewer directly but faces the viewer. For my dataset, I've tried to force all those said elements into bbaction_card without describing it in tags / NL. This still can be done with using right tools + tagging methods, but loras are just for this kind of stuff.

If you ask this kind of question, at least provide your dataset size, few example tags + images, and the training interface you're using. It's impossible to help without guessing what might be wrong . My only guess is that you are describing the character / concept tags in your dataset AND use an activator. So, the actual activator for your lora becomes your activator word + description, hence your lora only becomes a suggester / enforcer when only activator is used but behaves exactly like you want when full description (with weighing them) + activator is used. You just might have trained a weak lora.

You can always check how others trained their loras by this kind of tool: https://omervaner.github.io/lora-inspector/ (display options works great with Kohya outputs, slightly worse on Ai-toolkit, and does not display diffusion-pipe results)

Best of lucks to you friend, it sucks that your question got unanswered for a week. Anima is maturing at this point, so beginner questions like this often gets ignored because nearly all lora training questions are asked and answered somewhere else.

The official recommendations for ai-toolkit anima training guide is really useful: https://www.runcomfy.com/trainer/ai-toolkit/anima-lora-training

Let's have an example lora for character spritesheet with fixed views: front, side, back, few close-ups. If you describe those viewing elements in your dataset, and use an activator on top of it, you train the lora as a suggester / enforcer for an existing concept. You still need to use "multiple views, turnaround, front view, back view..."

But if you got rid of all viewing elements in your dataset, and try to force the concept into your activator, your activator becomes that concept. But, if dataset variety is low AND similarity is high, results will often look bad.

Prompt: @bbf3, (bbaction_card:1.5), goblin, solo, monster, tongue out, long sharp black claws, claws focus
That was my whole prompt on anima turbo model. @bbf3 is the activator for my style lora, unrelated in this case. bbaction_card is the activator for this concept. View is close-up, focuses on one action element + character, has a simple background, background is red (accidental, because most of monster tagged images in concept dataset has red/black background), character is out of frame slightly, and character does not look at the viewer directly but faces the viewer. For my dataset, I've tried to force all those said elements into bbaction_card without describing it in tags / NL. This still can be done with using right tools + tagging methods, but loras are just for this kind of stuff.

If you ask this kind of question, at least provide your dataset size, few example tags + images, and the training interface you're using. It's impossible to help without guessing what might be wrong . My only guess is that you are describing the character / concept tags in your dataset AND use an activator. So, the actual activator for your lora becomes your activator word + description, hence your lora only becomes a suggester / enforcer when only activator is used but behaves exactly like you want when full description (with weighing them) + activator is used. You just might have trained a weak lora.

You can always check how others trained their loras by this kind of tool: https://omervaner.github.io/lora-inspector/ (display options works great with Kohya outputs, slightly worse on Ai-toolkit, and does not display diffusion-pipe results)

Best of lucks to you friend, it sucks that your question got unanswered for a week. Anima is maturing at this point, so beginner questions like this often gets ignored because nearly all lora training questions are asked and answered somewhere else.

Well, I've been teaching since May, working primarily with content like Vtubers, and they have different datasets, from small ones like one full 15368x9600 pixel image (which I then manually cut into 3-4 close-up photos) to larger ones like 127 images of different frames. But I'd like to add that these are mostly images with an alpha channel, and if you "generate" backgrounds for them, they usually look illogical, since Vtube models have a fixed "T-shaped pose," and very little usually changes.
Further, from my personal experience:
I taught Lora for Illustrious from about August last year to June of this year. But in May, when the "Base" version of Anima was released, I decided to try teaching. Back then, I had help figuring out how to teach at a basic level. Thanks to kongbai-84 and others who helped me learn how to do them in principle. After almost six months of transferring various characters from previous tutorials, I've come to the conclusion that my Loras, while quite flexible in terms of lighting and placement, have SO many issues with anatomy flexibility. For example, I had a character (I can't disclose this information, it's confidential) whose model has huge breasts and wide hips, but these are limitations of the paid model, and I can't make them smaller. Or, conversely, another character had very small proportions, almost chibi, while I wanted a more masculine character. I found that Illustrous doesn't have this problem; she's completely flexible, even though I trained her with Prodigy + cosine at the starting training power of 1.

Now regarding the training parameters:
Config.toml
gpu_ids = "0"

[training_arguments]
output_name = "testLoraTrainV16"
save_model_as = "safetensors"
max_train_epochs = 10
save_every_n_epochs = 1
save_last_n_epochs_state = 1
sample_every_n_epochs = 1
log_with = "tensorboard"
learning_rate = 0.00005
text_encoder_lr = 0
optimizer_type = "AdamW8bit"
optimizer_args = [ "weight_decay=0.001" ]
use_muon = false
muon_lr = 0.02
muon_lr_scale = 0.05
muon_momentum = 0.95
muon_weight_decay = 0.01
muon_ns_steps = 5
muon_param_filter = "self_attn_mlp_cross"
muon_adam_betas = "0.9,0.95"
muon_adam_eps = "1e-8"
muon_disable_for_llm_adapter = true
muon_disable_for_adaln = true
muon_disable_distributed_allgather = false
muon_fp32_sensitive = false
lr_scheduler = "cosine"
lr_warmup_steps = 100
mixed_precision = "bf16"
save_precision = "bf16"
max_data_loader_n_workers = 4
gradient_accumulation_steps = 1
max_grad_norm = 1
gradient_checkpointing = true
flash_attn = true
torch_compile = false
lowram = false
blocks_to_swap = 0
persistent_data_loader_workers = true
seed = 42
cache_latents_to_disk = true
vae_batch_size = 1
cache_text_encoder_outputs_to_disk = false
skip_cache_check = false
multigpu_mode = "ddp"
deepspeed = false
use_cuda_direct = false
ddp_gradient_as_bucket_view = false
ddp_static_graph = false
ddp_find_unused_parameters = false
ddp_bucket_cap_mb = 25
use_fsdp = false
fsdp_sharding_strategy = "1"
fsdp_offload_params = false
fsdp_reshard_after_forward = false
fsdp_activation_checkpointing = false
fsdp_cpu_ram_efficient_loading = false
fsdp_backward_prefetch = ""
fsdp_forward_prefetch = false
fsdp_use_orig_params = true
fsdp_limit_all_gathers = true
fsdp_auto_wrap_policy = "NO_WRAP"
fsdp_min_num_params = 100_000_000
fsdp_transformer_layer_cls_to_wrap = ""
fsdp2_reshard_after_forward = true
fsdp2_offload_params = false
fsdp2_activation_checkpointing = false
fsdp2_cpu_ram_efficient_loading = false
fsdp2_auto_wrap_policy = "NO_WRAP"
fsdp2_min_num_params = 100_000_000
fsdp2_transformer_layer_cls_to_wrap = ""
freeze_llm_adapter = true
freeze_inserted_only_training = false
step_profile = false
profile_microbatch = false

[network_arguments]
network_module = "networks.lora_anima"
network_dim = 32
network_alpha = 16
network_train_unet_only = true
auto_resume_last_state = true

[anima_arguments]
timestep_sample_method = "logit_normal"
discrete_flow_shift = 3
weighting_scheme = "logit_normal"

dataset.toml
[general]
enable_bucket = true
bucket_no_upscale = true
min_bucket_reso = 512
max_bucket_reso = 1_536
bucket_reso_steps = 64

[[datasets]]
resolution = [ 1_280, 1_280 ]
batch_size = 1
caption_extension = ".txt"

[[datasets.subsets]]
image_dir = "D:\Datasets\TestCharacterDataset\SetV3TestC3"
num_repeats = 12
keep_tokens = 1
flip_aug = false
caption_prefix = ""
caption_dropout_rate = 0.05
caption_tag_dropout_rate = 0
caption_dropout_every_n_epochs = 0
shuffle_caption = false
cache_info = true
alpha_mask = true

Below is a rough example of a character I'm currently trying to train as a "training Lora" for myself. I designed the character entirely in Illustrious, and I'm trying to transfer it to Anima, but so far, it's been unsuccessful.

These are the references I tried to use, and here are the results of training and native transfer:

Regarding the labeling, I've tried different labeling methods. So far, I've developed the following preferences based on training:
Character Lora: training including Unet + Text encoder, at a strength of 0.00004-0.00007 <- Unet, 0 (text encoder freeze) - 0.000006.
Style Lora: 0.0001 Unet, 0 (Unet only), and for datasets of 30-80 images, aim for 1700-1950 steps; for 100-250+ images, aim for 3000 steps.
Concept Lora: So far, there have been few successful attempts at training. For example, I really wanted to train a variant of character breast shape, but I'm still complaining that my dataset is very diverse, which is why it didn't turn out as I would have liked. The parameters are roughly the same as Style Lora.

Here's an example of the markup I've recently started using:
TestAnimFoxAMV13, 1girl, solo, large breasts, looking at viewer, smile, short hair, orange eyes, animal ears, jewelry, bare shoulders, tail, white hair, multicolored hair, pantyhose, cowboy shot, red hair, sleeveless, virtual youtuber, necklace, black dress, two-tone hair, streaked hair, animal ear fluff, black pantyhose, fox ears, bare arms, sleeveless dress, fox tail, short dress, fox girl, arms at sides, turtleneck dress, short red hair with white cream streak in the front bangs, black halter-neck sleeveless bodycon mini dress with mock neck and bare shoulders thin silver chain necklace, cyan straps crossing in an X at the waist, oversized yellow jacket with cyan inner lining and two black flap pockets worn off the shoulders, grey semi-sheer tights, knee-high black lace-up boots with white soles.

Out of desperation, I started collecting various training parameters from Civitai using AI agents and trying to manually test them and see the results. I created a database in Obsidian and began formulating various hypotheses and training experiments for this purpose.

I also don't rule out the possibility that it might be because I use SD Forge Neo for generation. I used Pony tags for the positive and negative presets before, but then I discovered that without them, Lora's stylistic features were much more pronounced and profound. I typically used the following for the positive preset: Score_9, Score_8, Score_7.
I typically used the following for the negative preset: Score_1, Score_2, Score_3.
Also, on the advice of the AI ​​agent, I decided to remove the embeddings I used, like LazyPos and LazyNeg, from the positive and negative presets because, as the quote goes, "anime can't handle them and creates unnecessary noise."

Damn, that's a messy kohya toml, so many unused parameters. Can you make few changes and make a quick training with this:

optimizer_type = "Came"
timestep_sampling = "sigmoid"
sigmoid_scale = 1.0
resolution = 768
network_dim = 16
network_alpha = 8.0

Other can stay the same, but it would be great to reduce lr to 5e-05. And I hope auto_resume_last_state = true is not pointing to something else. Lower resolutions helps concepts and characters more, so training in 1280 might hurt a lot than you think.

And maybe compare that result with Ai-toolkit (with [512, 768, 1024] multi-res + recommended settings) result if you want to test further.

I'm sure that it would also apply to kohya, and maybe test this one too:
When Anima LoRA Training suppresses unrelated artist tags or makes every prompt resemble the dataset, add roughly 20–25% clean, unrelated illustration images to the same dataset folder. Caption them accurately and do not include the trigger. This small preservation slice often restores flexibility without adding another platform setting.

meaning you can add some random characters with the "same style" in your dataset to help generalize your target better. But make sure not to add that trigger to those images. It's a great tactic for smaller OC datasets, and help a lot. Or you can just look into regularization stuff instead. But try to test out one thing at a time, or you will get lost in the process very very quickly.

I also don't rule out the possibility that it might be because I use SD Forge Neo for generation. I used Pony tags for the positive and negative presets before, but then I discovered that without them, Lora's stylistic features were much more pronounced and profound. I typically used the following for the positive preset: Score_9, Score_8, Score_7.
I typically used the following for the negative preset: Score_1, Score_2, Score_3.
Also, on the advice of the AI ​​agent, I decided to remove the embeddings I used, like LazyPos and LazyNeg, from the positive and negative presets because, as the quote goes, "anime can't handle them and creates unnecessary noise."

You can also try out the turbo model with euler_a + simple at 10 step. Base model is really versatile, and that kind of versatility hurts when you slap bunch of stuff together. You don't need any kind of aesthetic tagging / scoring most of the time, that also includes "masterpiece" and "score_Xs". Less is more, and there's no secret recipe. Anima on it's own is a great model.

Out of desperation, I started collecting various training parameters from Civitai using AI agents and trying to manually test them and see the results. I created a database in Obsidian and began formulating various hypotheses and training experiments for this purpose.

I haven't met with anyone that learned anything with this approach. If you can solve any question / problem with AI agents without a problem, they would charge tenfold for tokens lol

I'm sure that it would also apply to kohya, and maybe test this one too:
When Anima LoRA Training suppresses unrelated artist tags or makes every prompt resemble the dataset, add roughly 20–25% clean, unrelated illustration images to the same dataset folder. Caption them accurately and do not include the trigger. This small preservation slice often restores flexibility without adding another platform setting.

meaning you can add some random characters with the "same style" in your dataset to help generalize your target better. But make sure not to add that trigger to those images. It's a great tactic for smaller OC datasets, and help a lot. Or you can just look into regularization stuff instead. But try to test out one thing at a time, or you will get lost in the process very very quickly.

Yes, sorry, I forgot to mention that I teach using Anima Standalone Trainer.

I also don't rule out the possibility that it might be because I use SD Forge Neo for generation. I used Pony tags for the positive and negative presets before, but then I discovered that without them, Lora's stylistic features were much more pronounced and profound. I typically used the following for the positive preset: Score_9, Score_8, Score_7.
I typically used the following for the negative preset: Score_1, Score_2, Score_3.
Also, on the advice of the AI ​​agent, I decided to remove the embeddings I used, like LazyPos and LazyNeg, from the positive and negative presets because, as the quote goes, "anime can't handle them and creates unnecessary noise."

You can also try out the turbo model with euler_a + simple at 10 step. Base model is really versatile, and that kind of versatility hurts when you slap bunch of stuff together. You don't need any kind of aesthetic tagging / scoring most of the time, that also includes "masterpiece" and "score_Xs". Less is more, and there's no secret recipe. Anima on it's own is a great model.

Out of desperation, I started collecting various training parameters from Civitai using AI agents and trying to manually test them and see the results. I created a database in Obsidian and began formulating various hypotheses and training experiments for this purpose.

I haven't met with anyone that learned anything with this approach. If you can solve any question / problem with AI agents without a problem, they would charge tenfold for tokens lol

Maybe I should look into the AI ​​toolkit as a learning tool for Lora...