Connect an Amazon S3 bucket

Connect an Amazon S3 bucket so jobs can read files from it and write outputs back to it. For how connections work across providers, see storage connections.

Rendobar reading inputs from and delivering outputs to an Amazon S3 bucket, alongside the other storage kinds it connects to — Rendobar reading inputs from an S3 bucket and writing outputs back.

Prerequisites

You need an AWS account with permission to create CloudFormation stacks and IAM roles. The bucket can already exist, or the stack can create it. Amazon S3 offers two ways to connect. Choose one in the connect dialog:

Way	Use when	Rendobar stores
One-click role	You can create a CloudFormation stack in the account that owns the bucket	The role’s ARN. No keys
Access keys	You cannot create stacks, or one-click connect does not support the bucket’s region	An access key ID and secret access key, encrypted

This page uses One-click role. To use keys instead, see Connect with access keys.

1. Prepare the role in Rendobar

In Storage, select Connect Storage, then Amazon S3.
Select One-click role.
In Bucket, select or enter the bucket name. Rendobar looks up the bucket and shows its region. Bucket names are global, so check that the bucket is in your AWS account.
Under Access, choose what the role can do:
- Deliver and read: write job outputs to the bucket, and read files from it.
- Read only: read files only, for a bucket you use for job inputs. A read-only connection cannot be a destination.
Neither lets the role delete objects.
(Optional) Under More options, turn on Create the bucket, or change the ID, region, or public URL.
Select Create the Role in AWS. The AWS console opens in a new tab.

The Amazon S3 connect dialog on the One-click role tab, with a bucket found in us-east-1, Deliver and read access selected, and the Create the Role in AWS button — The bucket, the role's access, and Create the Role in AWS.

2. Create the stack

The AWS console opens the stack with every value filled in. Check that the console is signed in to the AWS account that owns the bucket, and leave the values as they are.

AWS CloudFormation's Quick create stack page showing Rendobar's template URL, the stack description, and the stack name rendobar-test-1 — The quick create page, with Rendobar's template and the stack name filled in.

Under Capabilities, select I acknowledge that AWS CloudFormation might create IAM resources with customised names.
Select Create stack.

The Capabilities section of the quick create page with the IAM acknowledgement checked, and the Create stack button — The acknowledgement and Create stack.

The stack creates these resources:

Resource	Purpose
IAM role `rendobar-<id>`	The role Rendobar assumes. It trusts only Rendobar’s identity for this connection, and allows sessions of up to 12 hours
OpenID Connect provider for `https://api.rendobar.com/federation`	Lets the role trust Rendobar without keys. The first stack in an account creates it, and it stays when the stack is deleted
S3 bucket	Only when Create the bucket is on. It blocks public access, and stays when the stack is deleted

The role’s policy allows these actions:

Actions	Resource
`s3:GetObject`, `s3:PutObject`, `s3:AbortMultipartUpload`	Objects in the bucket
`s3:ListBucket`	The bucket
`s3:ListAllMyBuckets`, `s3:GetBucketLocation`	Bucket names and regions in the account, not their contents

With Read only, the role does not get s3:PutObject or s3:AbortMultipartUpload.

3. Wait for the connection

Return to Rendobar. The dialog follows the stack, which usually finishes in under two minutes. When the stack reports, Rendobar assumes the role, runs the access checks, and saves the connection. If you close the dialog, the connection still finishes.

The Amazon S3 dialog with Prepared done, Running in AWS in progress, and Connected still to come — Rendobar waiting for the stack to report.

If the stack reports a problem, the dialog shows it and nothing is saved. Delete the stack in CloudFormation before you try again, because the stack name stays taken.

The finish screen for an S3 bucket with write, read and list passed, delete not granted, and copyable source and destinations snippets — A connected bucket. Delete is not checked, because the role cannot delete objects.

Delete shows a dash because the role has no delete permission, which Rendobar does not need. The empty test object from the access checks stays in the bucket at .rendobar/probe-<uuid>. You can delete it. A read-only connection writes nothing to the bucket, not even a test object, so Write also shows a dash.

4. Use the bucket in a job

Put an object in inputs to read it, and the connection in destinations to write the output:

import { createClient, outputUrl } from "@rendobar/sdk";

const client = createClient({
  apiKey: process.env.RENDOBAR_API_KEY,
});

const job = await client.jobs.run({
  type: "ffmpeg",
  inputs: { source: "storage://test-1/raw/clip.mp4" },
  params: {
    command: "ffmpeg -i source -vf scale=1280:-2 out.mp4",
  },
  destinations: ["storage://test-1/exports"],
});

console.log(outputUrl(job));
console.log(job.deliveries);

curl https://api.rendobar.com/jobs \
  -H "Authorization: Bearer $RENDOBAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "ffmpeg",
    "inputs": { "source": "storage://test-1/raw/clip.mp4" },
    "params": {
      "command": "ffmpeg -i source -vf scale=1280:-2 out.mp4"
    },
    "destinations": ["storage://test-1/exports"]
  }'

With the default output path, this writes exports/clip.mp4. Delivery finishes after the job, so job.deliveries can still show pending when run returns. To change how outputs are named, see delivery settings. If a delivery fails, see delivery errors.

Troubleshooting

Problem	Fix
The stack rolls back because the identity provider already exists	Delete the rolled-back stack. In the Rendobar dialog, select open AWS without creating it
The stack rolls back because the identity provider does not exist	Rendobar skips the provider after your organization’s first stack, and assumes the same AWS account. Delete the rolled-back stack, then select open AWS and create it
The dialog says one-click connect does not reach the bucket’s region	Connect with access keys
The stack is still running after a few minutes	Open the stack’s Events tab in CloudFormation. A stack that fails before it reports cannot tell Rendobar

Create the role another way

To create the role with Terraform or by hand, open Trust policy and Terraform in the waiting dialog and copy either one. Create the role with a maximum session duration of 12 hours, which the Terraform already sets. Then select I created the role another way, and paste the role’s ARN. Rendobar runs the access checks as soon as you finish.

How do I revoke access?

Delete the stack in CloudFormation. This deletes the role, and Rendobar removes the connection. The OpenID Connect provider, and any bucket the stack created, stay in the account. For a role you created another way, delete the role in IAM. Deleting the connection in Rendobar does not delete the role.

Connect with access keys

In the AWS console, create an IAM user, and attach the policy that the connect dialog shows under IAM policy for this bucket.
Create an access key for the user.
In the connect dialog, select Access keys, then enter the bucket name, Access Key ID, and Secret Access Key. Rendobar finds the bucket’s region.
Select Connect.

S3-specific behavior

Name conflicts. S3 refuses a conditional write to a key that exists, so Keep both works. When two outputs share a name, Rendobar adds the end of the job id to the new one.

Delivery settings for an S3 bucket with the By date output path and Name conflicts set to Keep both — Keep both, available on an S3 bucket.

Incomplete uploads. When a large upload fails, Rendobar aborts it so its parts are not left in the bucket. As a safety net for every client that writes to the bucket, add a lifecycle rule that deletes incomplete multipart uploads. Temporary credentials. For a role, Rendobar gets temporary credentials from AWS STS that last up to 12 hours. A link to an input file stops working when the credentials that signed it expire, so a role limited to AWS’s default one-hour session gives jobs input links that last one hour. Each delivery’s credentials are limited to the one object it writes. Why S3 presigned URLs expire early measures this, and giving a service S3 access without an access key walks through the role’s trust and permission policies. Encryption. For a bucket encrypted with SSE-KMS, the role or user also needs kms:GenerateDataKey to write and kms:Decrypt to read, on the bucket’s key.

Next steps

Storage connections

Connection ids, delivery settings, credentials, and error codes.

Jobs

Where inputs and destinations appear in the job payload.

SDK

Submit the job above in TypeScript.

Webhooks

Receive an event when a job completes.

Getting Started

Concepts

Storage

Jobs

Automation

Developer tools

Support

Connect an Amazon S3 bucket

Prerequisites

1. Prepare the role in Rendobar

2. Create the stack

3. Wait for the connection

4. Use the bucket in a job

Troubleshooting

Create the role another way

How do I revoke access?

Connect with access keys

S3-specific behavior

Next steps

Storage connections

Jobs

SDK

Webhooks

​Prerequisites

​1. Prepare the role in Rendobar

​2. Create the stack

​3. Wait for the connection

​4. Use the bucket in a job

​Troubleshooting

​Create the role another way

​How do I revoke access?

​Connect with access keys

​S3-specific behavior

​Next steps

Storage connections

Jobs

SDK

Webhooks

Prerequisites

1. Prepare the role in Rendobar

2. Create the stack

3. Wait for the connection

4. Use the bucket in a job

Troubleshooting

Create the role another way

How do I revoke access?

Connect with access keys

S3-specific behavior

Next steps