I wanted to check something I usually take for granted: if effective batch is same, does the per-device batch still make much difference to training time?
I ran Qwen3-1.7B with TRL and LoRA on one T4. I kept the model, data, sequence length, FP16, seed and 100 optimizer steps fixed.
| Per-device batch | Accumulation | Effective batch | Runtime | Step time | Peak reserved |
|---:|---:|---:|---:|---:|---:|
| 1 | 4 | 4 | 287.6s | 2871ms | 5.45 GiB |
| 2 | 2 | 4 | 258.8s | 2585ms | 5.58 GiB |
| 4 | 1 | 4 | 238.2s | 2379ms | 5.81 GiB |
The 4 × 1 run finished about 17% faster than 1 × 4, while using around 0.36 GiB more reserved memory. Most of the measured difference was in forward and backward.
This is not a new rule. The [Hugging Face docs](https://huggingface.co/docs/transformers/grad_accumulation) already recommend using gradient accumulation when a larger physical batch does not fit. But I found the size of difference on this T4 useful to see.
I maintain TraceML and used it here to see where the step time was going. I also put the experiment in a Colab, no Hugging Face login needed:
- Colab: https://colab.research.google.com/github/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb
- Notebook source: https://github.com/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb
If anyone tries it on L4, A10 or A100, I would be interested to compare the numbers.
I like it!
Thanks! If you try the Colab, I would be curious what numbers you get.
For now, I tried running it on Colab (mostly on T4):
I was able to reproduce the main direction pretty cleanly, and the L4 result was especially interesting.
For the T4 part, I ran the same three effective-batch-4 decompositions in three fresh Colab runtimes per configuration and used the median, following the benchmark note in the notebook:
| config | physical batch | grad accumulation | effective batch | Trainer runtime, median | range |
|---|---|---|---|---|---|
| A | 1 | 4 | 4 | 289.57 s | 268.91–290.53 s |
| B | 2 | 2 | 4 | 260.76 s | 238.72–280.18 s |
| C | 4 | 1 | 4 | 245.78 s | 244.96–264.79 s |
So on these T4 runs:
- A → B: about 9.95% shorter
- A → C: about 15.12% shorter
- the medians are ordered
1×4 > 2×2 > 4×1
The A → B result was particularly reassuring because it is very close to the roughly 10% difference in your published T4 run. I would be more cautious about reading too much into the smaller B → C gap by itself, since the per-run ranges overlap quite a bit.
The memory side also moved in the expected direction: peak reserved memory went from about 5.45 GiB at 1×4 to about 5.81 GiB at 4×1. So at least for this workload, the larger physical batch bought a noticeable reduction in runtime for a relatively modest increase in reserved VRAM.
I also tried a deliberately cheap L4 check rather than spending the compute on a full 3×3 benchmark. I used two fresh L4 runtimes and reversed the endpoint order:
| L4 runtime | order | A: 1×4 |
C: 4×1 |
C vs A |
|---|---|---|---|---|
| 1 | A → C | 219.02 s | 125.54 s | 42.68% shorter |
| 2 | C → A | 225.08 s | 124.47 s | 44.70% shorter |
So the direction survived the order reversal, but the effect size was much larger on L4 than on T4.
I would not interpret those percentages as a generic “gradient accumulation penalty.” What looks more useful to me is:
For a fixed desired effective batch, physical micro-batch size is a real performance knob, and the size of the effect can be strongly hardware/workload dependent.
That is also consistent with the current Transformers gradient accumulation documentation, which frames gradient accumulation primarily as a way to reach an effective batch that would not otherwise fit in memory, and notes that it does not improve throughput over using the corresponding true larger batch when that larger batch fits.
For a practical default, I would probably phrase it as: use the largest comfortable physical batch that fits for the effective batch you actually want, then benchmark the nearby decompositions on the target GPU rather than assuming the timing relationship will transfer from another GPU. DeepSpeed treats micro-batch size in essentially this way as a hardware-dependent performance parameter in its autotuning documentation.
Experimental setup / reproducibility
For the T4 measurements I kept the public diagnostic structure intact as much as possible:
- model: Qwen3-1.7B
- LoRA: same adapter configuration
- dataset: Capybara
- effective batch: 4
- physical/GA combinations:
1 × 42 × 24 × 1
- sequence length fixed at 512
- samples padded/truncated to that fixed length
- packing disabled
- FP16
- gradient checkpointing enabled
- 100 optimizer steps
- fixed seed/data seed
- each T4 measurement in a fresh Colab runtime
- each training run executed in its own process
I also recorded the software environment for the runs. The T4 and L4 probes happened to resolve to the same stack:
torch 2.11.0+cu128
transformers 5.15.0
trl 1.10.0
peft 0.20.0
accelerate 1.14.0
traceml-ai 0.3.6
That matters for the cross-GPU comparison because the notebook currently installs current packages rather than pinning a frozen environment. In these particular runs, the large T4/L4 difference cannot simply be explained by having tested two different Transformers/TRL stacks.
The L4 experiment is intentionally weaker statistically than the T4 one. It is not a three-fresh-runtime benchmark for each configuration. It is a low-cost endpoint screen:
fresh L4 #1: A → C
fresh L4 #2: C → A
with A and C launched in separate processes.
I therefore think the safest way to report it is the two raw effects — 42.68% and 44.70% — rather than presenting their average as a precise population estimate.
What the phase breakdown seems to suggest
The phase timing made the result a little more informative than the wall-clock comparison alone.
On the L4 endpoint runs, taking the middle of the two observations for orientation, A → C changed roughly like this:
| measured region | A 1×4 |
C 4×1 |
approximate change |
|---|---|---|---|
| TraceML step | 2218 ms | 1248 ms | -44% |
| forward | 825 ms | 416 ms | -50% |
| backward | 1290 ms | 791 ms | -39% |
| optimizer | 8.7 ms | 8.7 ms | ~unchanged |
| residual | 91 ms | 31 ms | substantially lower |
The optimizer region being almost unchanged is useful context: the number of optimizer updates is fixed at 100 in all of these runs, while the amount of micro-batch repetition inside each update changes.
So the measured difference is concentrated mostly in the regions that are repeated per micro-batch rather than in the optimizer update itself.
I would still avoid turning that into “backward is the root cause.” Gradient checkpointing is enabled, so the measured backward-side region is part of a checkpointed execution where activations are recomputed during backward. The Transformers gradient checkpointing guide is explicit about that recomputation trade-off.
There was also a coarse GPU-utilization clue on L4:
A (batch 1, GA 4): ~37% / ~41%
C (batch 4, GA 1): ~80% / ~63%
in the two fresh runtimes.
I would treat those as run-level sampling clues, not kernel occupancy measurements, but the direction is interesting: the L4 looked considerably more under-utilized with physical batch 1 than with physical batch 4.
Why a larger physical batch can plausibly help this much
One fairly direct mechanism is that changing physical batch size changes the shapes of a large fraction of the GPU work.
NVIDIA’s linear / fully-connected layer performance guide maps the training phases to GEMMs and shows that batch size directly enters the matrix dimensions:
- in forward propagation, batch contributes to the output-matrix dimension;
- in the activation-gradient calculation, it again contributes to the output-matrix dimension;
- in the weight-gradient calculation, it contributes to the accumulation dimension.
So a larger physical batch can expose more tiles / parallel work and make each unit of aggregate computation more efficient.
That gives a useful way to think about this experiment:
1 × GA4:
four smaller forward/backward executions per optimizer update
4 × GA1:
one larger forward/backward execution per optimizer update
The examples processed per optimizer update are the same, but the GPU does not necessarily see an equivalent execution problem.
NVIDIA’s matrix multiplication performance background also gives a useful warning against over-generalizing in the other direction: GEMM efficiency depends on tiling and wave quantization, so performance need not improve smoothly or monotonically with every increase in a dimension. In some shapes, a somewhat smaller matrix can even run more efficiently than a larger one because it maps better onto the available SMs.
That seems like a good reason to treat the T4’s 1 → 2 → 4 result as an observed curve for this workload rather than a general law that every next doubling of physical batch must produce another fixed percentage improvement.
A second plausible contributor is fixed launch/host overhead. NVIDIA has documented that when individual GPU operations become short, the cost of submitting many operations can become significant; this is one of the motivations for CUDA Graphs. Their MLPerf discussion makes the batch-size connection particularly explicit: CPU/kernel-launch work is relatively fixed, while GPU work grows with batch size, so small local batches can expose proportionally more overhead.
I did not capture an Nsight Systems timeline here, though, so I would leave launch overhead as a possible contributor, not an attribution of the measured difference.
Why the L4 result may be so much larger than the T4 result
This part was the most surprising to me.
A simple cross-GPU comparison looks roughly like this:
A / batch 1:
T4 median ~289.6 s
L4 ~219–225 s
C / batch 4:
T4 median ~245.8 s
L4 ~124–126 s
So the L4’s extra capability shows up much more strongly in C than in A.
Together with the utilization samples above, one plausible reading is that physical batch 1 simply does not provide enough useful parallel work for this particular Qwen3-1.7B LoRA workload to exploit the L4 nearly as well as physical batch 4 does.
The hardware context is at least compatible with that interpretation. NVIDIA describes the L4 as an Ada-generation GPU with substantially greater Tensor Core compute capability than the older T4. But this is not a story where every hardware resource is uniformly faster; for example, the L4’s published memory bandwidth is around 300 GB/s. So I would not expect a constant cross-GPU speed ratio across different execution shapes.
This also fits the more general observation from DeepSpeed’s autotuning work: a micro-batch configuration that utilizes one GPU well does not necessarily utilize a different GPU well.
Still, I do not think these runs identify the exact mechanism behind “15% on T4 versus ~43–45% on L4.” Distinguishing GEMM shape/tiling, kernel-launch overhead, attention implementation, checkpoint recomputation behavior, and other kernel-level effects would be a separate profiling experiment.
And I think that separation is useful: the small diagnostic already answers the practical question without having to turn it into a kernel-attribution project.
A few interpretation limits
A few things I would keep separate from the main result:
-
The L4 result is a screen, not a full benchmark.
The order reversal makes the result much more convincing than one run, but it is still only two fresh L4 runtimes. -
The T4 B/C difference is smaller than the run-to-run variation.
The three-run medians are ordered B > C, but those ranges overlap substantially. A/C is the much cleaner T4 endpoint comparison. -
These percentages are not universal GA overheads.
They are measurements for this exact model, sequence length, LoRA setup, software stack, checkpointing mode and GPU. -
The phase labels are measurement regions, not kernel-level root causes.
In particular, checkpointing means the backward-side measurement includes recomputation behavior. -
Missing H2D timing should not be read as zero H2D cost.
I would keep the end-to-end Trainer runtime as the primary performance result and use the phase breakdown mainly as supporting evidence. -
I did not capture the actual attention backend chosen for each operation.
PyTorch’s scaled-dot-product attention can automatically choose among eligible CUDA implementations depending on the inputs. The capability/enabled flags alone do not prove which fused implementation was actually dispatched, so I would not attribute the T4/L4 difference to FlashAttention/SDPA selection from these runs.
So overall, the experiment looks useful to me precisely because the qualitative observation itself is unsurprising, while the size of the effect is not portable.
The T4 reproduction was close to your original numbers, while the inexpensive L4 screen produced a much larger endpoint gap. That makes the idea of collecting the same small diagnostic across GPUs more interesting than I initially expected.
For anyone adding another data point later, a compact report like
GPU
physical batch × gradient accumulation
effective batch
runtime (preferably median + range)
peak memory
software versions
would probably be enough to make the results comparable without turning it into a large benchmarking project.
Thanks for taking the time to run this so carefully. This was really helpful. I tried the same comparison on AWS L4:
| Configuration | Runtime |
|---|---|
1 × 4 |
213.02s |
4 × 1 |
124.76s |
That is around 41.4% lower runtime, which is quite close to the 42.68% and 44.70% you observed. It was good to see the result reproduce on another L4.
I also tested 2 × 2, and interestingly it was slightly faster than 4 × 1 in my run. So I agree with your conclusion: physical microbatch size is a real hardware-dependent performance knob, but nearby configurations still need to be benchmarked rather than assuming that a larger physical batch will always be faster.
Thanks again for trying the notebook and sharing such a detailed breakdown.