# FFmpeg benchmark variance Canonical: https://rendobar.com/blog/measurement-noise-ffmpeg-benchmarks/ Author: Abdelrahman Essawy Published: 2026-08-20 Updated: 2026-08-20 --- ## Key takeaways - One identical libx264 command ran ten times across our sweeps. Output was byte-identical every single time: 443,351 bytes, no variation at all. - Encode time for that same work ranged from 898 ms to 1,912 ms, a 2.13x spread. - Billed cost ranged from $0.0023 to $0.0039 on identical work, a 1.72x spread, because cost tracks time. - This retracted one of our own published claims. We had said a GPU encode was not faster than CPU; its 1,225 ms sits inside the CPU's own 898 to 1,912 ms band, so the comparison cannot be made. - The noise has a cause. Repeating the sweep with -threads pinned gave spreads of 1.4% to 8.9%, while x264's default auto threading swung 918 to 1,699 ms. Pin the threads and FFmpeg benchmarks become reproducible. We did not set out to measure this. It fell out of running the same source through ten different benchmark sweeps and noticing that one command appeared in several of them. Short version. **FFmpeg is deterministic and our stopwatch is not.** The same command produced the same bytes every time, to the byte, while taking anywhere from 898 ms to 1,912 ms to do it. That is wide enough to have invalidated a claim we had already published, and this post is partly a retraction. ## The command ```bash ffmpeg -i sample.mp4 -c:v libx264 -crf 23 -preset medium -an -t 5 out.mp4 ``` It appears in ten of our sweeps, because `libx264 -crf 23 -preset medium` is the natural control to compare everything else against. Here is every run: | Sweep | Encode | Cost | Output | |---|---:|---:|---:| | presets / medium | 1,912 ms | $0.0039 | 443,351 B | | ratecontrol / CRF 23 | 1,596 ms | $0.0031 | 443,351 B | | crf / CRF 23 | 982 ms | $0.0025 | 443,351 B | | gop / -g 250 | 945 ms | $0.0024 | 443,351 B | | ladder / 720p | 944 ms | $0.0023 | 443,351 B | | threads / auto | 924 ms | $0.0023 | 443,351 B | | gpu / libx264 medium | 913 ms | $0.0030 | 443,351 B | | tune / no tune | 912 ms | $0.0025 | 443,351 B | | pixfmt / yuv420p | 899 ms | $0.0024 | 443,351 B | | codecs / H.264 | 898 ms | $0.0024 | 443,351 B | **443,351 bytes, ten times out of ten.** Not approximately. Identically. **898 ms to 1,912 ms**, a 2.13x spread on the same work. ## Why the output is identical and the time is not x264 with a fixed seed and fixed settings is a deterministic function. Same input, same flags, same bytes. That is a genuinely useful property and it is why the size column on every benchmark on this site can be trusted without repeating anything. The time is a property of the machine, not the codec. These jobs run on shared infrastructure, so an encode competes for CPU with whatever else is scheduled, and the container may be warm or cold. None of that changes the output. All of it changes the clock. Cost follows time, because compute is billed per second. So the cost column inherits the same noise: **$0.0023 to $0.0039**, a 1.72x spread for identical work. ## What this cost us We had published, in [NVENC vs libx264 compared](/blog/nvenc-vs-libx264-measured/), that hardware encoding **was not faster** than libx264 on a short clip. The evidence was NVENC at 1,225 ms against libx264 at 913 ms. 913 ms is the bottom of a band that runs to 1,912 ms. NVENC's 1,225 ms sits inside it. **The comparison cannot be made from these samples**, and the claim has been retracted on that page. The cost claim on the same page survived, but only in a weaker form. We had said `hevc_nvenc` was 3.8x cheaper. Against a CPU cost that ranges $0.0023 to $0.0039, the true multiple is somewhere between 2.9x and 4.9x. `hevc_nvenc` at $0.0008 is below the entire band, so the direction is safe. The decimal was not. ## The rule we now use **A difference smaller than about 2x in a single timing sample is not a finding.** It is within the noise of one machine on one day. Applied to what is already published here: - **`ultrafast` at 238 ms against `veryslow` at 4,635 ms** is a 19x gap. Safe. - **Adjacent x264 presets** separated by a few hundred milliseconds are not separable. That page now says to read the size column and not the time column. - **Every size comparison** on this site stands unchanged, because sizes do not move. - **The scaling-algorithm sweep**, where five algorithms landed within 17 ms of each other, was already reported as "no measurable difference in time". That conclusion is stronger now, not weaker: the differences were inside the noise floor and we said so. ## Why publish the retraction Because the alternative is a site full of numbers nobody can check. Every benchmark on the internet is one sample unless it says otherwise, and almost none of them say otherwise. We only caught this because the same command happened to appear in ten sweeps and the sizes were suspiciously identical while the times were not. The honest version of a benchmark states which of its columns are repeatable. Ours are: **sizes are exact, timings are indicative, costs track timings.** Any page here that draws a conclusion from a small timing difference is a page that needs fixing, and we would rather fix them than defend them. ## Then we found the cause The obvious fix is to repeat each variant and report a median. So we did, five runs per variant, on a sweep whose entire subject is timing. The result was more interesting than a median. | Threads | Median | Range | Spread | |---|---:|---|---:| | 1 | 3,644 ms | 3,590 to 3,657 | **1.9%** | | 2 | 2,122 ms | 2,108 to 2,181 | 3.5% | | 4 | 1,510 ms | 1,490 to 1,511 | **1.4%** | | 8 | 1,220 ms | 1,183 to 1,288 | 8.9% | | auto | 1,033 ms | 918 to 1,699 | **85%** | **With the thread count pinned, FFmpeg timings are reproducible to within a few percent.** The wild variance lives almost entirely in `auto`. That makes sense once stated. x264's default threading adapts to the machine: it asks how many cores are available and sizes itself accordingly. On shared infrastructure that answer changes between runs, so the encoder is doing a different amount of parallel work each time. Pin `-threads` and you have pinned the workload. Every one of those ten control runs used default threading. The 2.13x spread was not a mystery about measurement. It was the encoder correctly adapting to a machine whose spare capacity kept moving. ## The rule, updated **If a sweep's conclusion depends on timing, pin `-threads` and repeat it.** Under those conditions single-digit percentage differences are real. **If it does not, single samples are fine for size**, which is deterministic regardless. That is cheaper than repeating everything. The size-led sweeps on this site, which is most of them, need no repetition at all. The timing-led ones get five runs and a pinned thread count, and pay for it.