Process video from an S3 bucket with FFmpeg

The footage is already in S3. The obvious pipeline is an S3 event, a Lambda function with an FFmpeg layer, and the result written back. It works for short clips, and it stops working at a size and a duration that have nothing to do with FFmpeg.

Short version. Lambda ends every run at 900 seconds and gives it at most 10,240 MB of scratch disk. A job that reads the object from the bucket has neither wall. One we ran lasted 1,002 seconds start to finish, completed normally and billed $1.02. The ceiling that applies instead is the plan’s job limit, 1 hour on Free and 9 hours on Pro.

The limits a Lambda function runs into#

Lambda is built for short tasks, and its quotas say so. These are the ones a video pipeline meets first, from AWS’s own quota page:

The timeout is the one that ends the conversation. AWS’s own answer to a question about processing a 40 GB video with Lambda was to use a different service. The AWS media blog’s well-known pattern streams the object through FFmpeg in memory to avoid /tmp, which helps with the disk and does nothing for the clock.

A run past 900 seconds#

To show the difference with a real run rather than a quota table, we submitted one FFmpeg job to the Rendobar API built to last 1,000 seconds:

ffmpeg -re -stream_loop -1 -i https://cdn.rendobar.com/assets/examples/sample.mp4 \
  -t 1000 -c:v libx264 -preset veryfast -crf 28 -c:a aac -b:a 96k out.mp4

-re reads the input at its own frame rate and -stream_loop -1 repeats it, so -t 1000 makes the run last 1,000 seconds regardless of how fast the machine encodes. The point being measured is how long a job may run, not how fast the encoder is, and a CPU-bound encode would have landed anywhere from nine minutes to twenty depending on the machine.

It completed. The job ran 102 seconds past the point where a Lambda function would have been stopped, the output landed, and the run billed $1.02, which works out to about a tenth of a cent per second.

Reading the object from S3#

That run read a public URL. Reading from your bucket is the same job with a different input. Connect the bucket once, then pass the object as storage://<connection>/<key>:

import { createClient } from "@rendobar/sdk";

const rb = createClient({ apiKey: process.env.RENDOBAR_API_KEY });

const job = await rb.jobs.create({
  type: "ffmpeg",
  inputs: { source: "storage://media/raw/interview.mov" },
  params: {
    command: "ffmpeg -i source -c:v libx264 -preset slow -crf 20 -c:a aac -movflags +faststart out.mp4",
  },
  destinations: ["storage://media/exports"],
});

console.log(job.id);

curl -X POST https://api.rendobar.com/jobs \
  -H "Authorization: Bearer $RENDOBAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "ffmpeg",
    "inputs": { "source": "storage://media/raw/interview.mov" },
    "params": { "command": "ffmpeg -i source -c:v libx264 -preset slow -crf 20 -c:a aac -movflags +faststart out.mp4" },
    "destinations": ["storage://media/exports"]
  }'

The reference is checked against your account when you submit, so a typo or a connection you don’t own fails immediately instead of after the job has waited its turn. When the job starts, it gets a signed link to that one object. No key leaves your side, and the link can’t be pointed at anything else in the bucket.

That link has to survive the wait before a job starts as well as the run itself, so it is minted to last as long as the longest job any plan allows. That matters more than it sounds. A presigned URL dies with the credentials that signed it, and a URL signed with an assumed role stops working when the role session ends, 1 hour by default, whatever expiry it requested. Why S3 presigned URLs expire early walks through that trap with numbers.

If the bucket holds footage you never want written to, connect it as read only. A read only connection is admitted as a job source and refused as a destination, so no job can deliver into it by mistake, and no default destination can quietly point at it.

Writing the result back#

The destinations line above delivers the encode to exports/ in the same bucket after the job completes, under the connection’s file name pattern. The finished file also stays downloadable from the job, so a delivery that fails does not cost you the encode. Delivery gets its own webhook per bucket. Save FFmpeg output to an S3 bucket covers the three ways to get output into S3, including this one, and what each costs.

When Lambda is still the right call#

Plenty of video work fits comfortably inside 900 seconds. Probing a file, grabbing a thumbnail or trimming a short clip finishes in seconds, and if your pipeline already lives in AWS there is no reason to move those. Generating a thumbnail when a file lands in S3 is one of them, and even there the function only submits work rather than doing it.

The line is the long tail. User uploads don’t come in a predictable size, and a pipeline that handles 99 files and times out on the hundredth fails the user who uploaded the longest video. A limit measured in hours instead of minutes turns that tail into ordinary jobs.

Where this stops#

The long run was paced with -re, so it proves a job may run for 1,002 seconds, not how fast any particular encode goes. For real encode times across presets and resolutions, how long video transcoding takes has the measurements.

It read a public URL. A storage input changes where the bytes come from, not how long the job may run.

The cost is one sample. Jobs bill by compute time, so a run of the same length costs about the same, and a CPU-heavy encode of a given length is still billed for its length.

Lambda’s limits are the documented defaults. Lambda Managed Instances raise the timeout to 90 minutes for asynchronous invocations, and container services such as ECS and Batch have no 900-second limit at all, in exchange for running and scaling them yourself.

Connecting the bucket the job reads from is covered in the Amazon S3 storage guide, including the role setup that needs no access key.