Auto Drive S3 Layer Guide

Overview

Auto Drive provides an S3-compatible API layer that allows you to interact with decentralized storage (DSN) using standard Amazon Web Services Simple Storage Service (AWS S3) SDK commands. This bridges the gap between familiar cloud storage patterns and next-generation decentralized infrastructure, giving developers the best of both worlds: the reliability and developer experience of S3 APIs with the permanence and censorship-resistance of decentralized storage.

For those unfamiliar, Amazon Web Services Simple Storage Service (AWS S3) is an industry-standard object storage service that powers much of the modern web’s file storage needs. Auto Drive maintains compatibility with the most-used parts of S3’s APIs while storing your data on a decentralized network instead of centralized servers.

Permanent and immutable by design. Auto Drive storage is permanent. Objects on the Autonomys DSN cannot be modified, overwritten, or deleted. DeleteObject returns 403 Forbidden. Re-uploading the same key creates a new object — the old data persists on the DSN. Plan your data model around this guarantee.

How It Works

Auto Drive maps S3 (bucket, key) pairs to Content Identifiers (CIDs) and MD5 checksums. When you upload via the S3 API, the system:

Stores the file content on the decentralized network (DSN)
Computes the MD5 of the content and records the (bucket, key) → (cid, md5) mapping
Returns the MD5 as the ETag (standard S3 format) and exposes the CID in the x-amz-meta-cid response header
Enables cross-API access between the S3 API and the native Auto Drive API

Key Features

1. Standard S3 SDK and CLI Compatibility

Works with the official AWS S3 SDKs (@aws-sdk/client-s3, boto3), the AWS CLI, and S3-compatible tools like rclone
Supported operations: ListBuckets, ListObjectsV2, PutObject, GetObject, HeadObject, multipart uploads (CreateMultipartUpload, UploadPart, CompleteMultipartUpload)
No code changes required for existing S3 applications other than swapping the endpoint and credentials

2. Buckets

Buckets behave like standard S3 buckets. The first path segment of the object key is the bucket name, and the remainder is the key:

S3 path	Bucket	Key
`my-archive/report.pdf`	`my-archive`	`report.pdf`
`my-archive/sub/file.txt`	`my-archive`	`sub/file.txt`
`test.txt` (no slash)	`default`	`test.txt`

Buckets are created implicitly on first write — there is no CreateBucket call. ListBuckets (GET /) returns every distinct bucket you have written to.

import { ListBucketsCommand } from "@aws-sdk/client-s3";
 
const result = await s3Client.send(new ListBucketsCommand({}));
console.log(result.Buckets); // [{ Name: "my-archive", CreationDate: ... }, ...]

Single-segment keys uploaded before bucket support was introduced remain accessible under the default bucket.

3. ListObjectsV2 with Prefix, Delimiter, and Pagination

Auto Drive implements ListObjectsV2 end-to-end, with the parameters most clients depend on:

Prefix filtering — ?prefix=subdir/ returns only keys starting with subdir/
Delimiter folding — ?delimiter=/ collapses keys into <CommonPrefixes> virtual directories
Pagination — ?max-keys=N plus ?continuation-token=… for cursor-based paging
Object size — <Size> is populated from indexed metadata
MD5 ETag in listings — listing entries return the MD5 ETag, so checksum verification does not require an extra HeadObject per object

Auto Drive implements ListObjectsV2 only. The legacy ListObjects (V1) API is not implemented. rclone users must set list_version = 2 in their remote config (see rclone integration).

import { ListObjectsV2Command } from "@aws-sdk/client-s3";
 
const result = await s3Client.send(
  new ListObjectsV2Command({
    Bucket: "my-archive",
    Prefix: "logs/",
    Delimiter: "/",
    MaxKeys: 100,
  })
);
 
console.log(result.Contents);        // objects directly under "logs/"
console.log(result.CommonPrefixes);  // virtual subdirectories like "logs/2026/"
 
if (result.IsTruncated) {
  // Fetch the next page using result.NextContinuationToken
}

4. MD5 ETags + CID via `x-amz-meta-cid`

PutObject, GetObject, HeadObject, ListObjectsV2, and CompleteMultipartUpload return a standard quoted MD5 ETag (e.g. "d41d8cd98f00b204e9800998ecf8427e") for objects uploaded after the MD5 ETag feature shipped. For multipart uploads, the composite ETag follows the AWS format "<md5_of_part_md5s>-<N>". Legacy objects (those uploaded before the feature) have no stored MD5: HEAD/GET omit the ETag header entirely, and ListObjectsV2 falls back to wrapping the CID as the ETag value. See the migration note below.

The Autonomys CID is exposed on every object response as a custom header:

x-amz-meta-cid: bafkreig...

This makes rclone check, rclone md5sum, AWS CLI checksum verification, and any other MD5-based tooling work out of the box.

Migration note. Objects uploaded before the MD5 ETag feature shipped have a NULL MD5 in the database. They will not return an ETag header on HEAD/GET until they are re-uploaded. The CID is always available via x-amz-meta-cid. For listings of legacy objects, the CID is returned as a fallback.

import { HeadObjectCommand } from "@aws-sdk/client-s3";
 
const head = await s3Client.send(
  new HeadObjectCommand({ Bucket: "my-archive", Key: "report.pdf" })
);
 
console.log(head.ETag);                    // '"d41d8cd98f00b204e9800998ecf8427e"' (MD5)
console.log(head.Metadata?.cid);           // 'bafkreig...' (CID, via x-amz-meta-cid)

5. Deletion is Forbidden by Design

DeleteObject always returns 403 Forbidden with an informative message — Auto Drive storage is permanent. Tools that try to delete (e.g. rclone delete, rclone purge, rclone sync with deletions) will see the 403 and surface it as an error. Re-uploading the same key creates a new object pointing at new content; the prior data still exists on the DSN.

6. Enhanced Metadata Support

Custom user metadata is stored alongside the object and returned on subsequent HEAD/GET calls. Auto Drive recognizes compression and encryption metadata for content stored with those transforms applied client-side.

const command = new PutObjectCommand({
  Bucket: "my-archive",
  Key: "file.txt",
  Body: buffer,
  Metadata: {
    compression: "ZLIB",
    encryption: "AES_256_GCM",
  },
});

7. Range Requests

Partial file downloads using standard HTTP Range headers are supported.

const command = new GetObjectCommand({
  Bucket: "my-archive",
  Key: "large-file.bin",
  Range: "bytes=0-9", // first 10 bytes
});

8. Multipart Uploads

The full multipart workflow (CreateMultipartUpload → UploadPart × N → CompleteMultipartUpload) is supported. UploadPart returns the MD5 of the part body; CompleteMultipartUpload returns the standard AWS composite ETag "<md5_of_part_md5s>-<N>".

const key = "large-file.bin";
const Bucket = "my-archive";
 
// Step 1: Create multipart upload
const { UploadId } = await s3Client.send(
  new CreateMultipartUploadCommand({ Bucket, Key: key })
);
 
// Step 2: Upload one or more parts
const part1 = await s3Client.send(
  new UploadPartCommand({
    Bucket,
    Key: key,
    UploadId: UploadId!,
    PartNumber: 1,
    Body: fileChunk1,
  })
);
 
// Step 3: Complete the upload
const result = await s3Client.send(
  new CompleteMultipartUploadCommand({
    Bucket,
    Key: key,
    UploadId: UploadId!,
    MultipartUpload: {
      Parts: [{ ETag: part1.ETag!, PartNumber: 1 }],
    },
  })
);
 
console.log(result.ETag); // '"<composite_md5>-1"'

Configuration

Endpoint

Environment	Endpoint
Mainnet (public)	`https://public.auto-drive.autonomys.xyz/s3`

Client Setup (AWS JS SDK)

The Auto Drive backend exposes objects under a single /:key(*) route and parses bucket and key from the path, so a path-style client works well in practice:

import { S3Client } from "@aws-sdk/client-s3";
 
const s3Client = new S3Client({
  region: "us-east-1",                                // required by the SDK; ignored by Auto Drive
  endpoint: "https://public.auto-drive.autonomys.xyz/s3",
  credentials: {
    accessKeyId: "your-auto-drive-api-key",          // your Auto Drive API key
    secretAccessKey: "placeholder",                   // any non-empty string; ignored
  },
  forcePathStyle: true,                               // bucket lives in the path
});

Then use bucket names as you would with AWS S3:

await s3Client.send(
  new PutObjectCommand({ Bucket: "my-archive", Key: "report.pdf", Body: buffer })
);

Alternative: bucketEndpoint style. The upstream backend integration tests configure the SDK with bucketEndpoint: true and bake the bucket into the endpoint path (e.g. Bucket: "my-archive/s3"). Either pattern works against the /:key(*) route; forcePathStyle: true with a plain bucket name is shown here because it’s the more familiar S3 configuration.

Authentication

Uses Auto Drive API key-based authentication (the same keys as the native Auto Drive API)
The API key goes in accessKeyId; secretAccessKey must be present but is ignored (use any non-empty placeholder)
Files uploaded via the S3 API are owned by the user the API key belongs to

Get an API key from the Developers section at ai3.storage.

File Ownership & Cross-API Access

Cross-API compatibility — files uploaded via the S3 API are accessible via the native Auto Drive API, and vice versa
Centralized ownership — file ownership is tracked centrally per user, not per API surface
Content deduplication — multiple users uploading identical content share the same underlying CID
Shared access — if different users upload the same file, both can access it through either API

Storage Characteristics

Content Addressing

Files are stored using Content Identifiers (CIDs) on the Autonomys DSN
The CID is exposed via the x-amz-meta-cid response header on every object response (and as a fallback ETag for legacy objects)
Storage is immutable — the same content always produces the same CID

Decentralized Backend

Files are stored on the Distributed Storage Network of the Autonomys Network
Automatic replication and redundancy
No single point of failure

rclone Integration

Auto Drive’s S3 layer is fully rclone-compatible — including rclone check, rclone md5sum, virtual directory listings, multipart uploads, and pagination — so any rclone-driven workflow (archival, scheduled backups, mounts, cloud-to-cloud migration, CI/CD artifact storage) works against Auto Drive.

For the dedicated walkthrough — quickstart, full config reference, every supported command, workflows, and troubleshooting — see the Using rclone guide.

Migrating from AWS S3

For developers moving from traditional AWS S3:

Update the endpoint to https://public.auto-drive.autonomys.xyz/s3
Change credentials to use your Auto Drive API key as accessKeyId; set secretAccessKey to a placeholder
Set forcePathStyle: true in the S3 client configuration
Drop bucket-creation calls — buckets are created implicitly on first write
Remove deletion logic — DeleteObject returns 403; design around immutability
Read CIDs from x-amz-meta-cid, not from the ETag (the ETag is the standard MD5)
Handle longer response times due to network latency vs. AWS edge

// Before (AWS S3)
const s3Client = new S3Client({
  region: "us-east-1",
  credentials: {
    accessKeyId: "AKIA...",
    secretAccessKey: "abc123...",
  },
});
 
// After (Auto Drive)
const s3Client = new S3Client({
  region: "us-east-1",
  endpoint: "https://public.auto-drive.autonomys.xyz/s3",
  credentials: {
    accessKeyId: "your-auto-drive-api-key",
    secretAccessKey: "placeholder",
  },
  forcePathStyle: true,
});

Building Your Own S3-Compatible Layer

If you are building your own S3-compatible service on top of Autonomys storage, the Auto SDK ships reusable, framework-agnostic S3 server-side helpers in @autonomys/file-server:

buildListResult(rows, prefix, delimiter, maxKeys) — ListObjectsV2 delimiter folding into CommonPrefixes plus maxKeys pagination with correct continuation-token placement
computeListObjectsDbLimit(maxKeys, delimiter) — how many rows to fetch from storage for a single page
finalizeListObjects(params, fetchedRows, dbLimit) — wraps buildListResult and applies the full-batch truncation override that prevents duplicate CommonPrefixes across page boundaries
md5Hex, formatETag, multipartETag — S3 ETag computation, including the AWS composite multipart format
Types: S3ObjectListing, ListObjectsParams, ListObjectsResult

import {
  computeListObjectsDbLimit,
  finalizeListObjects,
  multipartETag,
} from "@autonomys/file-server";
 
const dbLimit = computeListObjectsDbLimit(maxKeys, delimiter);
const rows = await fetchSortedObjectsFromStorage(prefix, continuationToken, dbLimit);
 
const result = finalizeListObjects(
  { bucket, prefix, delimiter, maxKeys, continuationToken },
  rows,
  dbLimit,
);
// result is ready to render as ListObjectsV2 XML
 
const completedETag = multipartETag(parts.map((p) => p.etag));

Auto Drive itself uses these helpers, so you get the same delimiter folding, pagination, and composite-ETag behavior the public service is validated against.

Limitations & Considerations

Performance

The DSN has higher write latency than traditional S3 — use multipart uploads for files > 5 MB
Listings paginate at the server’s configured maxKeys (default 1,000); requested values larger than 1,000 are clamped to the 1,000 hard cap
Range requests may have different performance characteristics than AWS S3

Compatibility Notes

Implemented: ListBuckets, ListObjectsV2 (with prefix, delimiter, max-keys, continuation-token), PutObject, GetObject, HeadObject, multipart uploads
Not implemented: ListObjects (V1) — use V2; CreateBucket — buckets are created implicitly; CopyObject, versioning, lifecycle policies, ACLs, bucket policies, presigned URLs
Forbidden by design: DeleteObject, DeleteObjects, DeleteBucket (returns 403)
Effectively no-ops: any operation that assumes mutability

Best Practices

Treat storage as append-only — design data models around immutability rather than fighting it
Use multipart uploads for files larger than 5 MB
Read CIDs from x-amz-meta-cid, not from the ETag
Use the MD5 ETag for integrity verification — it matches md5sum of the content for non-multipart objects
Paginate listings with MaxKeys + NextContinuationToken rather than fetching everything in one call
Implement retry logic for occasional network-induced timeouts
For rclone, always pass --immutable and set list_version = 2

Error Handling

Standard S3 XML error responses
403 Forbidden for any deletion attempt — distinguish this from a permissions error in your client
Network timeouts may be longer than traditional S3; configure your SDK accordingly

This S3 layer provides a familiar interface while leveraging the benefits of decentralized storage, making it easy to migrate existing S3-based applications — and S3-compatible tooling like rclone and the AWS CLI — to Auto Drive.

Pay with AI3 Usage Examples