Two ceilings, neither configurable
Netlify Functions run on AWS Lambda underneath, so a bundled FFmpeg binary does execute. What stops a transcode is not the binary. It is the clock and the CPU cap, and you cannot raise either.
A synchronous function gets 60 seconds. Netlify lists it as not configurable, on every plan. A background function stretches that to 15 minutes, and in exchange it answers 202 and never returns a result, so the storage write and the notification become yours to build.
The harder ceiling is CPU. The vcpu setting tops out at 2, which is what the 4096 MB memory maximum buys. We measured a real VP9 encode holding 2.2 cores with 8 available. FFmpeg takes what it is given, so the same encode still finishes on 2 vCPU. It finishes slower, which is the problem when the budget is counted in seconds.