FFmpeg benchmark variance

We did not set out to measure this. It fell out of running the same source through ten different benchmark sweeps and noticing that one command appeared in several of them.

Short version. FFmpeg is deterministic and our stopwatch is not. The same command produced the same bytes every time, to the byte, while taking anywhere from 898 ms to 1,912 ms to do it. That is wide enough to have invalidated a claim we had already published, and this post is partly a retraction.

The command#

ffmpeg -i sample.mp4 -c:v libx264 -crf 23 -preset medium -an -t 5 out.mp4

It appears in ten of our sweeps, because libx264 -crf 23 -preset medium is the natural control to compare everything else against. Here is every run:

443,351 bytes, ten times out of ten. Not approximately. Identically.

898 ms to 1,912 ms, a 2.13x spread on the same work.

Why the output is identical and the time is not#

x264 with a fixed seed and fixed settings is a deterministic function. Same input, same flags, same bytes. That is a genuinely useful property and it is why the size column on every benchmark on this site can be trusted without repeating anything.

The time is a property of the machine, not the codec. These jobs run on shared infrastructure, so an encode competes for CPU with whatever else is scheduled, and the container may be warm or cold. None of that changes the output. All of it changes the clock.

Cost follows time, because compute is billed per second. So the cost column inherits the same noise: $0.0023 to $0.0039, a 1.72x spread for identical work.

What this cost us#

We had published, in NVENC vs libx264 compared, that hardware encoding was not faster than libx264 on a short clip. The evidence was NVENC at 1,225 ms against libx264 at 913 ms.

913 ms is the bottom of a band that runs to 1,912 ms. NVENC’s 1,225 ms sits inside it. The comparison cannot be made from these samples, and the claim has been retracted on that page.

The cost claim on the same page survived, but only in a weaker form. We had said hevc_nvenc was 3.8x cheaper. Against a CPU cost that ranges $0.0023 to $0.0039, the true multiple is somewhere between 2.9x and 4.9x. hevc_nvenc at $0.0008 is below the entire band, so the direction is safe. The decimal was not.

The rule we now use#

A difference smaller than about 2x in a single timing sample is not a finding. It is within the noise of one machine on one day.

Applied to what is already published here:

• ultrafast at 238 ms against veryslow at 4,635 ms is a 19x gap. Safe.
• Adjacent x264 presets separated by a few hundred milliseconds are not separable. That page now says to read the size column and not the time column.
• Every size comparison on this site stands unchanged, because sizes do not move.
• The scaling-algorithm sweep, where five algorithms landed within 17 ms of each other, was already reported as “no measurable difference in time”. That conclusion is stronger now, not weaker: the differences were inside the noise floor and we said so.

Why publish the retraction#

Because the alternative is a site full of numbers nobody can check.

Every benchmark on the internet is one sample unless it says otherwise, and almost none of them say otherwise. We only caught this because the same command happened to appear in ten sweeps and the sizes were suspiciously identical while the times were not.

The honest version of a benchmark states which of its columns are repeatable. Ours are: sizes are exact, timings are indicative, costs track timings. Any page here that draws a conclusion from a small timing difference is a page that needs fixing, and we would rather fix them than defend them.

Then we found the cause#

The obvious fix is to repeat each variant and report a median. So we did, five runs per variant, on a sweep whose entire subject is timing. The result was more interesting than a median.

With the thread count pinned, FFmpeg timings are reproducible to within a few percent. The wild variance lives almost entirely in auto.

That makes sense once stated. x264’s default threading adapts to the machine: it asks how many cores are available and sizes itself accordingly. On shared infrastructure that answer changes between runs, so the encoder is doing a different amount of parallel work each time. Pin -threads and you have pinned the workload.

Every one of those ten control runs used default threading. The 2.13x spread was not a mystery about measurement. It was the encoder correctly adapting to a machine whose spare capacity kept moving.

The rule, updated#

If a sweep’s conclusion depends on timing, pin -threads and repeat it. Under those conditions single-digit percentage differences are real.

If it does not, single samples are fine for size, which is deterministic regardless.

That is cheaper than repeating everything. The size-led sweeps on this site, which is most of them, need no repetition at all. The timing-led ones get five runs and a pinned thread count, and pay for it.