Blob Storage

Cloudillo’s blob storage holds immutable binary data (files, images, videos) under content-addressed IDs — every blob is named by the SHA-256 hash of its bytes. This gives automatic deduplication, integrity verification, and permanent cacheability. Uploaded media is additionally split into multiple size/quality variants grouped by a file descriptor.

Content-Addressed Storage

File Identifier Format

Cloudillo uses multiple identifier types in its content-addressing system:

{prefix}{version}~{base64url_hash}

Components:

• {prefix}: Resource type indicator (a, f, b, d)
• {version}: Hash algorithm version (currently 1 = SHA-256)
• ~: Separator
• {base64url_hash}: Base64url-encoded hash (43 characters, no padding)

Identifier Types

Important: d2, is not a content-addressed identifier—it’s the actual encoded descriptor string. The file ID (f1~) is the hash of this descriptor.

Examples

Blob ID:       b1~QoEYeG8TJZ2HTGhVlrtTDBpvBGOp6gfGhq4QmD6Z46w
File ID:       f1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM
Descriptor:    d2,vis.tn:b1~xRAVuQtgBx_kLqZnoOSd5XqCK_aQolhq1XeXk73Zn8U:f=avif:s=1960:r=90x128
Action ID:     a1~8kR3mN9pQ2vL6xWpYzT4BjN5FqGxCmK9RsH2VwLnD8P

All file and blob IDs use SHA-256 content-addressing. See Content-Addressing & Merkle Trees for hash computation details.

File Types

Cloudillo supports four file types, each handled by different adapters based on mutability and use case:

File Type Selection

Files are created via different API endpoints based on their type:

Available presets for BLOB uploads: default, profile-picture, cover, high_quality, mobile, archive, podcast, video, orig-only, thumbnail-only, apkg

Per-File Access Control

Each file has independent access control:

See Access Control for detailed permission handling.

File Variants

Concept

A single uploaded image automatically generates multiple variants optimized for different use cases:

• pf (profile): Profile picture icon (~80px)
• tn (thumbnail): Small preview (~256px)
• sd (standard definition): Mobile/low bandwidth (~720px)
• md (medium definition): Desktop viewing (~1280px)
• hd (high definition): High quality display (~1920px)
• xd (extra definition): 4K/maximum quality (~3840px)

File Descriptor Encoding

A file descriptor encodes all available variants in a compact format.

File Descriptor Format Specification

Format

d2,{class}.{variant}:{blob_id}:f={format}:s={size}:r={width}x{height}[:{optional}];...

Components

• d2, - Descriptor prefix (version 2)
• {class} - Media class:
  • vis - Visual (images: jpeg, png, webp, avif)
  • vid - Video (mp4/h264)
  • aud - Audio (opus, mp3)
  • doc - Documents (pdf)
  • raw - Original unprocessed file
• {variant} - Quality tier: pf, tn, sd, md, hd, or xd
• {blob_id} - Content-addressed ID of the blob (b1~...)
• f={format} - Format: avif, webp, jpeg, png, mp4, opus, pdf
• s={size} - File size in bytes (integer, no separators)
• r={width}x{height} - Resolution in pixels (width × height)
• ; - Semicolon separator between variants (no spaces)

The original is encoded as the bare token orig (no class prefix), regardless of its media class — e.g. orig:b1~...:f=jpeg:.... A descriptor may also begin with an optional R={root_id}; field that links the file to its document-tree access-control root.

Optional Fields

For video, audio, and document files:

• dur={seconds} - Duration in seconds (floating point, video/audio only)
• br={kbps} - Bitrate in kbps (integer, video/audio only)
• pg={count} - Page count (integer, documents only)

Example

d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x150;vis.sd:b1~def456:f=avif:s=32768:r=640x480

This descriptor encodes two variants:

• Thumbnail: AVIF format, 4096 bytes, 150×150 pixels, blob ID b1~abc123
• Standard: AVIF format, 32768 bytes, 640×480 pixels, blob ID b1~def456

Video Example

d2,vis.tn:b1~abc:f=avif:s=4096:r=150x84;vid.sd:b1~def:f=mp4:s=5242880:r=720x404:dur=120.5:br=350;vid.hd:b1~ghi:f=mp4:s=20971520:r=1920x1080:dur=120.5:br=1400

This descriptor includes:

• Thumbnail: AVIF image preview
• SD Video: 720p MP4, 120.5 seconds, 350 kbps
• HD Video: 1080p MP4, 120.5 seconds, 1400 kbps

Parsing Rules

1. Check prefix: Verify descriptor starts with d2,
2. Split by semicolon (;): Get individual variant entries
3. For each variant, split by colon (:) to get components:
   • Component [0] = class.variant (vis.tn, vis.sd, vid.hd)
   • Component [1] = blob_id (b1~...)
   • Components [2..] = key=value pairs
4. Parse key=value pairs:
   • f={format} → Format string
   • s={size} → Parse as u64 (bytes)
   • r={width}x{height} → Split by x, parse as u32 × u32
   • dur={seconds} → Parse as f64 (optional)
   • br={kbps} → Parse as u32 (optional)
   • pg={count} → Parse as u32 (optional)

Parsing logic: split by semicolons for variants, then by colons for fields, then parse key=value pairs.

Variant Size Classes - Exact Specifications

Cloudillo generates image variants at specific size targets to optimize bandwidth and storage:

Generation Rules

Which variants are generated depends on the preset configuration. The default preset generates: tn, sd, md, hd. The high_quality preset adds xd. Variants larger than the original image are automatically skipped (smaller originals are never upscaled).

Properties:

• Each variant maintains the original aspect ratio
• Uses Lanczos3 filter for high-quality downscaling
• Maximum dimension constraint prevents oversizing
• Smaller originals don’t get upscaled

Variant Selection

Clients request a specific variant:

GET /api/files/f1~Qo2E3G8TJZ...?variant=hd

Response: Returns HD variant if available, otherwise falls back to smaller variants.

Automatic Fallback

If the requested variant doesn’t exist, the server returns the best available:

1. Try requested variant (e.g., hd)
2. Fall back to next smaller (e.g., md)
3. Continue until variant found
4. Return smallest if none larger

Fallback order: xd → hd → md → sd → tn

Content-Addressing Flow

File storage uses a three-level content-addressing hierarchy:

Level 1: Blob Storage

Upload image → Save as blob → Compute SHA256 of blob bytes → Store blob with ID: b1~{hash}

blob_data = read_file("thumbnail.avif")
blob_id = compute_hash("b", blob_data)
// Result: "b1~abc123..." (thumbnail blob ID)

Example: b1~abc123... identifies the thumbnail AVIF blob

See Content-Addressing & Merkle Trees for hash computation details.

Level 2: Variant Collection

Generate all variants (tn, sd, md, hd) → Each variant gets its own blob ID (b1~...) → Collect all variant metadata → Create descriptor string encoding all variants

variants = [
    { class: "vis.tn", blob_id: "b1~abc123", format: "avif", size: 4096, width: 150, height: 150 },
    { class: "vis.sd", blob_id: "b1~def456", format: "avif", size: 32768, width: 640, height: 480 },
    { class: "vis.md", blob_id: "b1~ghi789", format: "avif", size: 262144, width: 1920, height: 1080 },
]

descriptor = build_descriptor(variants)
// Result: "d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x150;vis.sd:b1~def456:f=avif:s=32768:r=640x480;vis.md:b1~ghi789:f=avif:s=262144:r=1920x1080"

Level 3: File Descriptor

Build descriptor → Compute SHA256 of descriptor string → Final file ID: f1~{hash} → This file ID goes into action attachments

descriptor = "d2,vis.tn:b1~abc:f=avif:s=4096:r=150x150;vis.sd:b1~def:f=avif:s=32768:r=640x480"
file_id = compute_hash("f", descriptor.as_bytes())
// Result: "f1~Qo2E3G8TJZ..." (file ID)

Example Complete Flow

1. User uploads photo.jpg (3MB, 3024x4032px)

2. System generates variants:
   vis.tn:  150x200px → 4KB   → b1~abc123
   vis.sd:  600x800px → 32KB  → b1~def456
   vis.md:  1440x1920px → 256KB → b1~ghi789
   vis.hd:  2880x3840px → 1MB → b1~jkl012

3. System builds descriptor:
   "d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x200;
       vis.sd:b1~def456:f=avif:s=32768:r=600x800;
       vis.md:b1~ghi789:f=avif:s=262144:r=1440x1920;
       vis.hd:b1~jkl012:f=avif:s=1048576:r=2880x3840"

4. System hashes descriptor:
   file_id = f1~Qo2E3G8TJZ2... = SHA256(descriptor)

5. Action references file:
   POST action attachments = ["f1~Qo2E3G8TJZ2..."]

6. Anyone can verify:
   - Download all variants
   - Verify each blob_id = SHA256(blob)
   - Rebuild descriptor
   - Verify file_id = SHA256(descriptor)
   - Cryptographic proof established ✓

File attachments integrate into Cloudillo’s merkle tree structure. See Content-Addressing & Merkle Trees for how files fit into the verification chain.

Image Processing Pipeline

Upload Flow

When a client uploads an image:

1. Client Request

POST /api/files/default/profile-picture.jpg
Authorization: Bearer <access_token>
Content-Type: image/jpeg
Content-Length: 2458624

<binary image data>

2. Dimension Extraction & Variant Selection

The image dimensions are extracted and the preset’s image variant list (e.g. ["vis.tn", "vis.sd", "vis.md", "vis.hd"] for default) is walked from smallest to largest. Each variant’s bounding box is capped at the original’s longest side — the original is never upscaled. A variant is then skipped if its capped size is less than 10% larger than the last variant actually created, so a small original collapses to just the thumbnail plus one or two distinct sizes instead of several near-identical blobs.

The intermediate steps (task scheduling, hash computation, blob storage, variant generation, and metadata storage) are shown in the Complete Upload Flow Diagram below.

3. Response

The upload responds immediately with a temporary local ID (@{f_id}) plus the synchronously-generated thumbnail blob ID and original dimensions. The remaining variants are still being generated asynchronously:

{
  "fileId": "@1234",
  "thumbnailVariantId": "b1~QoE...46w",
  "dim": [3024, 4032]
}

The final content-addressed file ID (f1~...) is only known once all variant tasks finish. The server then pushes a FILE_ID_GENERATED WebSocket event ( { tempId, fileId, rootId } ) so clients can swap the temporary @{f_id} for the permanent f1~ ID.

Complete Upload Flow Diagram

Client uploads image
  ↓
POST /api/files/{preset}/filename.jpg
  ↓
Read image, extract dimensions, allocate local f_id
  ↓
Generate thumbnail synchronously
  ├─ Resize with Lanczos3 → encode → SHA256 → b1~ blob
  ├─ Store blob in BlobAdapter
  └─ Record file_variants row
  ↓
Respond immediately: { fileId: "@<f_id>", thumbnailVariantId, dim }
  ↓
Schedule image.resize task per remaining variant
  ├─ Resize / encode / hash / store blob
  └─ Record file_variants row
  ↓
file.id-generate task (depends on all variant tasks)
  ├─ Collect variant rows → build d2 descriptor
  ├─ file_id = SHA256(descriptor) → f1~...
  ├─ Finalize file (status P → A)
  └─ Broadcast FILE_ID_GENERATED over WebSocket

Download Flow

Client Request

GET /api/files/f1~...?variant=hd
Authorization: Bearer <access_token>

Server Processing

1. Parse Descriptor

variants = parse_file_descriptor(file_id)
# Returns list of VariantInfo

2. Select Best Variant

selected = get_best_file_variant(
    variants,
    requested_variant,   # "hd"
)

# Falls back by quality within the same class if the
# requested variant isn't locally available:
# hd → md → sd → tn

3. Stream from BlobAdapter

stream = blob_adapter.read_blob_stream(tn_id, selected.variant_id)

# Set response headers
response.headers["Content-Type"] = content_type_from_format(selected.format)
response.headers["X-Cloudillo-Variant"] = selected.variant_id
response.headers["X-Cloudillo-Variants"] = descriptor
response.headers["Content-Length"] = selected.size

# Stream response
return stream_response(stream)

Response

HTTP/1.1 200 OK
Content-Type: image/avif
Content-Length: 16384
X-Cloudillo-Variant: b1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM
X-Cloudillo-Variants: d2,vis.tn:b1~xRAVuQtgBx_kLqZnoOSd5XqCK_aQolhq1XeXk73Zn8U:f=avif:s=1960:r=90x128;vis.sd:b1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM:f=avif:s=8137:r=256x364;orig:b1~5gU72rRGiaogZuYhJy853pBd6PsqjPOjS__Kim9-qE0:f=avif:s=15012:r=256x364
Cache-Control: public, max-age=31536000, immutable

<binary image data>

Note: Content-addressed files are immutable, so can be cached forever.

Metadata Structure

File metadata lives in two tables in the MetaAdapter. The files row holds the logical file (type, preset, owner, visibility, folder/document-tree links); file_variants rows hold one entry per generated variant. A local internal f_id integer keys both tables while the file is still being processed; the content-addressed file_id (f1~...) is written only once all variant tasks finish (status transitions P → A).

CREATE TABLE files (
    f_id INTEGER NOT NULL,          -- local internal id (primary key)
    tn_id INTEGER NOT NULL,
    file_id TEXT,                   -- f1~... descriptor hash (set when finalized)
    file_tp CHAR(4),                -- BLOB / CRDT / RTDB
    status CHAR(1),                 -- A active, P pending, D deleted
    preset TEXT,
    content_type TEXT,
    file_name TEXT,
    visibility CHAR(1),             -- NULL / P / V / 2 / F / C
    parent_id TEXT,                 -- folder hierarchy
    root_id TEXT,                   -- document-tree access-control root
    created_at INTEGER,
    PRIMARY KEY(f_id)
);

CREATE TABLE file_variants (
    tn_id INTEGER NOT NULL,
    f_id INTEGER NOT NULL,
    variant_id TEXT,                -- b1~... blob id
    variant TEXT,                   -- 'vis.sd', 'vid.hd', 'orig', ...
    res_x INTEGER,
    res_y INTEGER,
    format TEXT,
    size INTEGER,
    available BOOLEAN,              -- blob present locally
    global BOOLEAN,                 -- stored in shared global cache
    duration REAL,                  -- video/audio
    bitrate INTEGER,                -- video/audio (kbps)
    page_count INTEGER,             -- documents
    PRIMARY KEY(f_id, variant_id, tn_id)
);

The available flag matters for federation: a synced file lists all variants in its descriptor, but only the variants whose blobs have actually been fetched are marked available locally. See Access Control for how visibility and root_id drive permission checks.

File Presets

Presets control which variants are generated and whether the original is stored. Files are uploaded with a preset in the path:

POST /api/files/{preset}/{filename}

POST /api/files/default/avatar.jpg       // standard image variants
POST /api/files/archive/document.pdf     // keep original, minimal processing

Available presets: default, profile-picture, cover, high_quality, mobile, archive, podcast, video, orig-only, thumbnail-only, apkg. See File Processing for the full per-preset variant matrix.

Storage Organization

BlobAdapter Layout

Blobs are stored on disk under a per-tenant directory, sharded into two levels by the first four characters of the hash (after the ~). The filename is the full blob ID:

{data_dir}/
├── {tn_id}/
│   ├── {h0h1}/                      // first 2 hash chars
│   │   └── {h2h3}/                  // next 2 hash chars
│   │       └── b1~QoEYeG8TJ...46w   // blob, filename = full ID
│   └── ...
└── {other_tn_id}/
    └── ...

Each variant (and the original, when stored) is an independent blob with its own ID. The file descriptor — not the filesystem — is what groups variants into a logical file. File metadata is stored separately in the MetaAdapter (see Metadata Structure above).

See Also

• System Architecture - Task system and worker pool
• Actions - File attachments in action tokens
• Access Control - File permission checking