Cloudillo Documentation

Welcome to Cloudillo

Cloudillo is an open-source, decentralized collaboration platform that empowers users to maintain complete control over their data while collaborating globally. Whether you’re building collaborative applications, hosting your own instance, or exploring decentralized architectures, this documentation guides you through every aspect of Cloudillo.

Choose your path below:


πŸ‘₯ For Everyone: Getting Started

New to Cloudillo? Start here to understand what it is and how to get up and running.

  • What is Cloudillo? β€” Understand Cloudillo’s vision: global accessibility with local control, revolutionary authentication, and seamless interconnectivity
  • Prerequisites β€” System requirements and what you need to know before getting started
  • Getting Started β€” Quickstart guide to begin using Cloudillo
  • Install Yourself β€” Self-hosting guide for those who want to run their own instance
  • First Steps β€” Your first interactions and setup on Cloudillo
  • Status Page β€” Service status and availability

πŸ’» For Developers: API & Integration

Build applications on top of Cloudillo using our comprehensive API and client libraries.

Quick Start for Developers

Client Libraries

REST API Reference

  • REST API Overview β€” Complete REST API documentation
    • Authentication β€” Login, registration, and token management
    • Profiles β€” User and community profiles
    • Actions β€” Social interactions and action tokens
    • Files β€” File upload, download, and management
    • Settings β€” User preferences and configuration
    • References β€” Bookmarks and shortcuts

Real-Time APIs

Developer Guides


πŸ—οΈ Technical Deep Dives: Architecture

Understand Cloudillo’s technical architecture, data models, and how the system works internally.

Core Concepts & Fundamentals

Data Storage & Access

Actions & Federation

Runtime Systems


πŸ”§ Implementation & Operations


πŸš€ Next Steps

Subsections of Cloudillo Documentation

Subsections of The Basics

What Is Cloudillo

Early Development

Cloudillo is under active development. The Rust-based server is 90% complete, with core functionality operational but some API endpoints still in progress. The platform is suitable for developers and early adopters but not yet recommended for production use.

Cloudillo is a versatile Decentralized, Self-Hosted, Open-Source Application Platform designed to empower your creativity, facilitate seamless collaboration, and enable easy sharing.

Cloudillo can be:

  • Your Document Store: Store all your documents securely, accessible only to you or shared with selected individuals or groups.
  • Your Personal or Group Knowledge Base: Compile and organize your knowledge effortlessly, whether it’s for personal use or sharing within your team or community.
  • Your Collaboration Platform: Work together with colleagues or friends on projects, documents, or tasks in real-time, enhancing productivity and efficiency.
  • Your Community Network: Build a vibrant community network where members can connect, share ideas, and collaborate on common interests or goals.

What sets Cloudillo apart?

Global Accessibility, Local Control

Cloudillo takes a unique approach by storing all your data locally while making it globally accessible through its unparalleled API. This ensures you have full control over your data while enjoying the benefits of a worldwide reach.

Revolutionary Global Authentication and Authorization

Cloudillo introduces a groundbreaking method for Global Authentication, Authorization, and Verification. Share your data securely without requiring users to register on your instance, mirroring the convenience of Cloud-based platforms.

Seamless Interconnectivity

Thanks to this fresh perspective, Cloudillo-based applications can establish connections that might seem impossible on other platforms. This newfound flexibility opens doors to innovative and seamless collaborations between applications.

Ready to experience collaboration on a whole new level? Explore Cloudillo and witness the difference.

Prerequisites

Before installing or using Cloudillo, ensure your system meets the requirements below.


For Self-Hosting (Rust Version)

System Requirements

Minimum:

  • CPU: 1 core (2+ recommended)
  • RAM: 512MB (1GB+ recommended)
  • Disk: 1GB + storage for user data
  • OS: Linux (x86_64 or ARM64), macOS, Windows with WSL2

Recommended for 10+ users:

  • CPU: 2+ cores
  • RAM: 2GB+
  • Disk: 10GB+ SSD
  • OS: Linux (Ubuntu 22.04+ or Debian 12+)

Recommended for 100+ users:

  • CPU: 4+ cores
  • RAM: 8GB+
  • Disk: 50GB+ SSD
  • OS: Linux on dedicated hardware/VPS

Software Requirements

For Docker Installation (recommended):

  • Docker 20.10+
  • Docker Compose 2.0+ (optional but recommended)

For Building from Source:

  • Rust 1.70+ with cargo
  • Git

For Domain-Based Identity:

  • Domain name with DNS control
  • SSL/TLS certificate (Let’s Encrypt supported via ACME)

Network Requirements

Required Ports:

  • 443 (HTTPS) - Required for application access
  • 80 (HTTP) - Required for ACME/Let’s Encrypt certificate validation
  • OR custom ports if using reverse proxy

DNS Records Required:

  • A record: yourdomain.com β†’ server IP address
  • A record: cl-o.yourdomain.com β†’ server IP address

Example:

yourdomain.com          A    203.0.113.42
cl-o.yourdomain.com     A    203.0.113.42

Firewall:

  • Allow inbound TCP on ports 80 and 443
  • Allow outbound HTTPS (443) for federation

Browser Compatibility (for end users)

Supported:

  • Chrome/Edge 90+
  • Firefox 88+
  • Safari 14+

Recommended:

  • Latest stable versions of above browsers
  • JavaScript enabled
  • Cookies enabled

For App Development

Required Software

  • Node.js: 18+ (20+ recommended)
  • pnpm: 8+
  • TypeScript: 5+
  • Git: Latest stable version
  • VS Code with extensions:
    • TypeScript
    • ESLint
    • Prettier
    • React (if building UI apps)
  • Browser DevTools knowledge
  • Postman or similar API testing tool (optional)

Knowledge Prerequisites

  • JavaScript/TypeScript fundamentals
  • Async/await patterns
  • REST API concepts
  • WebSocket basics (for real-time features)
  • React basics (for UI apps using @cloudillo/react)

For Contributing to Cloudillo

Required

  • Rust: 1.70+ with cargo
  • Git: Latest stable version
  • GitHub account
  • SQLite: 3.x (for testing adapters)
  • Docker: For testing deployments
  • Rust experience:
    • Understanding of Tokio/async Rust
    • Familiarity with Axum web framework
    • Knowledge of Rust error handling
  • Testing tools:
    • cargo test
    • cargo clippy
    • cargo fmt

Useful Background

  • Understanding of web server architecture
  • Database experience (SQLite, key-value stores)
  • WebSocket protocols
  • Authentication systems (JWT, OAuth)
  • CRDT concepts (for collaborative editing features)

Deployment Scenarios

Personal/Family Use

Typical setup:

  • 1-20 users
  • Home server or small VPS
  • 1 vCPU, 512MB-1GB RAM, 50GB disk

Recommended:

  • Raspberry Pi 4 (4GB+)
  • Small VPS ($5-10/month)
  • Docker installation

Small Organization (10-50 users)

Typical setup:

  • 10-50 users
  • VPS or cloud instance
  • 2 vCPU, 1-2GB RAM, 500GB disk

Recommended:

  • DigitalOcean Droplet / AWS t3.small
  • Docker with monitoring
  • Regular backups

Community Server (50-500 users)

Typical setup:

  • 50-500 users
  • Dedicated server or powerful VPS
  • 2-4 vCPU, 4-8GB RAM, 1TB+ SSD

Recommended:

  • Dedicated server or large VPS
  • Load balancing (if high traffic)
  • Database optimization
  • CDN for static assets
  • Professional monitoring

Development/Testing

Typical setup:

  • Local development machine
  • 2+ cores, 4GB+ RAM, 20GB disk
  • Docker or build from source

Recommended:

  • Modern laptop/desktop
  • Fast SSD for rebuilds
  • Multiple browser for testing
  • Local domain setup (hosts file)

Quick Checklist

Before proceeding with installation, verify:

Self-Hosting Checklist

  • Server meets minimum requirements (1 core, 512MB RAM, 1GB disk)
  • Docker 20.10+ installed OR Rust 1.70+ installed
  • Domain name registered and DNS configured
  • Ports 80 and 443 accessible (firewall configured)
  • SSH access to server (if remote)
  • Backup plan in place

Development Checklist

  • Node.js 18+ installed
  • pnpm 8+ installed
  • TypeScript 5+ installed
  • Code editor ready (VS Code recommended)
  • Git configured
  • Access to Cloudillo instance (for testing)

Contributing Checklist

  • Rust 1.70+ installed
  • GitHub account created
  • Repository forked/cloned
  • Tests running (cargo test)
  • Code style configured (cargo fmt, cargo clippy)
  • Architecture documentation reviewed

Verification Commands

Check Docker Installation

docker --version
# Should show: Docker version 20.10+ or higher

docker compose version
# Should show: Docker Compose version 2.0+ or higher

Check Rust Installation

rustc --version
# Should show: rustc 1.70.0 or higher

cargo --version
# Should show: cargo 1.70.0 or higher

Check Node.js Installation

node --version
# Should show: v18.0.0 or higher

pnpm --version
# Should show: 8.0.0 or higher

Check DNS Configuration

# Check A records
dig yourdomain.com A
dig cl-o.yourdomain.com A

# Or use nslookup
nslookup yourdomain.com
nslookup cl-o.yourdomain.com

Check Port Accessibility

# Check if ports are open (from another machine)
nc -zv yourserver.com 80
nc -zv yourserver.com 443

# Or use telnet
telnet yourserver.com 80
telnet yourserver.com 443

Common Issues

“Cannot connect to Docker daemon”

Solution: Start Docker service

sudo systemctl start docker
sudo systemctl enable docker

“Port 443 already in use”

Solution:

  • Check what’s using port 443: sudo lsof -i :443
  • Stop conflicting service or use reverse proxy
  • Configure Cloudillo to use alternate port

“DNS records not propagating”

Solution:

  • Wait 24-48 hours for DNS propagation
  • Use TTL of 300 seconds for faster updates
  • Verify with multiple DNS checkers online

“Rust compiler version too old”

Solution: Update Rust

rustup update stable
rustup default stable

Next Steps

Once your system meets the prerequisites:

  1. Self-Hosting: Install Cloudillo β†’
  2. Development: Getting Started Guide β†’
  3. Contributing: Architecture Documentation β†’

Need Help?

Feature Status

Version: v0.1.0-alpha (Rust) | Progress: 90% | Last Updated: 2025-10-29

Info

Cloudillo is under active development. Features marked 🟒 are production-ready. 🟑 partial. πŸ”΄ not started.


Implementation Status

Component Status Progress Notes
Backend Infrastructure
Core Server & Adapters 🟒 Complete 100% All 5 storage adapters fully implemented
Response Envelope System 🟒 Complete 100% Structured responses with error codes
Security & Auth 🟒 Complete 100% JWT, bcrypt, ABAC, TLS
API Endpoints
Authentication 🟑 Partial 50% Register/login working, token endpoints legacy
Profile Management 🟒 Complete 100% Get, update, media upload all working
Actions & Social 🟑 Partial 60% List/create working, others in progress
File Storage 🟑 Partial 40% Upload/download working, management incomplete
Real-Time Systems
RTDB Backend 🟒 Complete 100% Fully implemented with redb
CRDT Backend 🟒 Complete 100% Yjs-compatible collaborative editing
Message Bus 🟒 Complete 100% WebSocket pub/sub working
Client Libraries (TypeScript)
@cloudillo/types 🟒 Stable 100% Type definitions with validation
@cloudillo/core 🟑 Partial 95% API client works with Rust backend
@cloudillo/react 🟑 Partial 95% React hooks & components
@cloudillo/rtdb 🟒 Complete 95% Real-time database client, needs testing
Frontend & Apps
Shell/UI 🟒 Production 90% Dashboard, profile, files, messaging
Quillo (Editor) 🟒 Active 100% Collaborative rich text editor
Prello (Presentations) 🟒 Active 100% Presentation tool
Sheello (Spreadsheets) 🟒 Active 100% Spreadsheet application
Todollo (RTDB Demo) 🟒 Active 100% Todo list with real-time sync
Formillo (Forms) 🟒 Active 100% Form builder & survey tool
Federation 🟑 Partial 60% Basic federation working, envelope updates needed

Legend: 🟒 Production-ready | 🟑 Partial/In-progress | πŸ”΄ Not started


Known Issues

  • Some API endpoints still use legacy response format (Phase 1 migration in progress)
  • Some action operations incomplete (reactions, stats, etc.)
  • File management incomplete (update, delete, tags)
  • Client library integration with RTDB/CRDT needs testing

For detailed API endpoint status, feature breakdowns, and roadmap timelines, check the repository status.


Support

Getting started

Development Status

Cloudillo is under active development (90% complete). This guide explains core concepts that apply to all versions. For installation, see Installing Cloudillo for the Rust version, and check the Status Page for available features.

When you are ready to join the Cloudillo Network, you’ll need to make two decisions:

1. Identity

You identity is like your unique username on the network. Cloudillo uses the Domain Name System β€” the backbone of the internet β€” to create profile identities. This approach eliminates the need for a central provider, although it might seem overwhelming for newcomers. To simplify the process, you have two options:

  • Option 1: If you already have a domain name, you can use it to create your identity. This option may require technical know-how β€” but we’ve got you covered.

  • Option 2: If you don’t have a domain name or prefer to avoid the hassle, you can choose a Cloudillo Identity Provider. These services make it super easy to create your identity.

    Note

    cloudillo.net identity provider is planned for launch in Q1 2026 (Phase 4). Until then, you’ll need to use your own domain or wait for community providers to emerge.

2. Storage Provider

This is where your data will be stored.

  • Option 1: If you have your own server or a home NAS (Network Attached Storage), you can use that as your storage provider. You can even invite your family and friends to use it.

  • Option 2: Don’t want to manage your own storage? No problem. You can opt for a Cloudillo Storage Provider. They handle all the technical stuff for you.

Why Two Services?

Cloudillo is all about giving you control and keeping your data secure. Separating your identity from your storage provider makes it easy to switch storage options whenever you need to.

Using Your Own Domain

Many people and organizations already have domain names. You can use yours as your identity and even create sub-identities for different purposes. This helps organizations give their members identities tied to their domain.

Changing Your Storage Provider

If you control your identity, switching storage providers is simple. Whether you move to self-hosting or another provider, your identity remains intact, making the transition seamless.

Can You Use the Same Provider for Both?

Technically, yes. However, for security reasons, it’s better to choose different providers. Using separate providers avoids potential conflicts of interest if you ever need to switch storage providers.

Trust Your Identity Provider

Your identity provider is crucial. Ensure you trust them, as they could theoretically take control of your identity. If this concerns you, registering your own domain name can offer maximum security.

Warning

Keep in mind, on Cloudillo you own not only your data, but also your network, followers, and likes. No one can take these away from you, unless they gain control of your identity.

Installing Cloudillo Yourself

Rust Version

This guide is for the Rust implementation of Cloudillo (v0.1.0-alpha). The Node.js version is deprecated and no longer maintained.

Alpha Software

The Rust version is 90% complete. Core functionality works, but some API endpoints are still in development. See Status Page for details.

Before You Begin

  • Review Prerequisites to ensure your system meets requirements
  • Have a domain name ready with DNS control
  • Decide on deployment mode (standalone vs proxy)

Installation Options

You have several options for installing Cloudillo:

  1. Docker (recommended) - Simplest and most versatile
  2. Build from source - For development or custom builds

You also need to decide whether to run in:

  • Standalone mode - Cloudillo handles HTTPS with Let’s Encrypt
  • Proxy mode - Run behind a reverse proxy (nginx, Caddy, etc.)

Running Cloudillo using Docker in Standalone Mode

Warning

Docker images for the Rust version may not be published yet. Check the cloudillo-rs repository for latest status. If not available, use the build from source method below.

Here is the Docker command (adjust image name when available):

docker run -d --name cloudillo \
  -v /var/vol/cloudillo:/data \
  -p 443:443 -p 80:80 \
  -e LOCAL_IPS=1.2.3.4 \
  -e BASE_ID_TAG=agatha.example.com \
  -e BASE_APP_DOMAIN=agatha.example.com \
  -e BASE_PASSWORD=SomeSecret \
  -e ACME_EMAIL=user@example.com \
  -e ACME_TERMS_AGREED=true \
  cloudillo/cloudillo-rs:latest

You basically need a local directory on your server and mount it to /data inside the container and publish port 443 (HTTPS) and 80 (HTTP). You can read about the configuration environment variables below.

Cloudillo in standalone mode uses TLS certificates managed by Let’s Encrypt. If you want to use different certificates then you have to use Proxy Mode.

DNS Required

For Let’s Encrypt certificate issuance to work you have to ensure that $BASE_APP_DOMAIN and cl-o.$BASE_ID_TAG both have DNS A records pointing to one of $LOCAL_IPS before first starting the container.

Here is a docker-compose for the same:

version: "3"
services:
  cloudillo:
    image: cloudillo/cloudillo-rs:latest  # Adjust when image is published
    container_name: cloudillo
    volumes:
      - /var/vol/cloudillo:/data
    ports:
      - 443:443
      - 80:80
    environment:
      - LOCAL_IPS: 1.2.3.4
      - BASE_ID_TAG: agatha.example.com
      - BASE_APP_DOMAIN: agatha.example.com
      - BASE_PASSWORD: SomeSecret
      - ACME_EMAIL: user@example.com
      - ACME_TERMS_AGREED: true
Security

You should change the default password as soon as possible!

Configuration Environment Variables

Variable Value Default
MODE “standalone” or “proxy” “standalone”
LOCAL_IPS Comma separated list of IP addresses this node serves *
BASE_ID_TAG ID tag for the admin user to create *
BASE_APP_DOMAIN App domain for the admin user to create *
BASE_PASSWORD Password for the admin user to create *
ACME_EMAIL Email address for ACME registration -
ACME_TERMS_AGREED Whether to agree to ACME terms -
DATA_DIR Path to the data directory /data
PRIVATE_DATA_DIR Path to the private data directory $DATA/priv
PUBLIC_DATA_DIR Path to the public data directory $DATA/pub

In case you are not familiar with the Cloudillo Identity System we provide some information about the notions used above:

The ID tag is the unique identifier of a Cloudillo profile. It can be any domain name associated with a user.

The App Domain is the domain address used by the user to access their Cloudillo shell. Currently the App Domain is also unique for every user, but it might change in the future. It is preferably the same as the ID tag of the user, but it can be different when the domain is used by an other site.

For a domain to work as a Cloudillo Identity it must serve a Cloudillo API endpoint accessable at the cl-o subdomain. In the above example you have to create the following DNS records:

Name Type Value Purpose
agatha.example.com A 1.2.3.4 App domain (APP_DOMAIN)
cl-o.agatha.example.com A 1.2.3.4 API domain (cl-o.ID_TAG)

Running Cloudillo using Docker in Proxy Mode

An other one-liner:

docker run -d --name cloudillo \
  -v /var/vol/cloudillo:/data \
  -p 1443:1443 \
  -e MODE=proxy \
  -e LOCAL_IPS=1.2.3.4 \
  -e BASE_ID_TAG=agatha.example.com \
  -e BASE_APP_DOMAIN=agatha.example.com \
  -e BASE_PASSWORD=SomeSecret

In proxy mode you have to provide your own solution for TLS certificates. In proxy mode Cloudillo serves its API port on 1443 but using HTTP and it doesn’t serve the HTTP port by default. You should provide your own redirections. However, you can turn it on with the LISTEN_HTTP=1080 environment variable.

We provide an example nginx configuration:

server {
	listen   80;
	server_name  agatha.example.com cl-o.agatha.example.com;

	location /.well-known/ {
		root    /var/www/certbot;
		autoindex off;
	}
	location / {
		return 301 https://$host$request_uri;
	}
}

server {
	listen 443 ssl;
	server_name agatha.example.com cl-o.agatha.example.com;
	ssl_certificate /etc/letsencrypt/live/agatha.example.com/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/agatha.example.com/privkey.pem;

	location /.well-known/cloudillo/id-tag {
		add_header 'Access-Control-Allow-Origin' '*';
		return 200 '{"idTag":"agatha.example.com"}\n';
	}
	location /api {
		rewrite /api/(.*) /api/$1 break;
		proxy_pass http://localhost:1443/;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto https;
		proxy_set_header X-Forwarded-Host $host;
		client_max_body_size 100M;
	}
	location /ws {
		proxy_pass http://localhost:1443;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto https;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
	}
	location / {
		# You can serve the cloudillo shell locally, or proxy it.
		root /home/agatha/cloudillo/shell;
		try_files $uri /index.html;
		autoindex off;
		expires 0;
	}
}

Building from Source

For the latest features or if Docker images aren’t available yet, build from source:

Prerequisites

# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

# Verify installation
rustc --version  # Should be 1.70+
cargo --version

Clone and Build

# Clone the Rust repository
git clone https://github.com/cloudillo/cloudillo-rs.git
cd cloudillo-rs

# Build release version
cargo build --release

# Binary will be at:
./target/release/cloudillo-basic-server

Run the Server

# Set environment variables
export DATA_DIR=/var/vol/cloudillo
export BASE_ID_TAG=agatha.example.com
export BASE_APP_DOMAIN=agatha.example.com
export BASE_PASSWORD=SomeSecret
export ACME_EMAIL=user@example.com
export ACME_TERMS_AGREED=true

# Create data directory
mkdir -p $DATA_DIR

# Run the server
./target/release/cloudillo-basic-server

Next Steps

After installation:

  1. Verify Installation: Access https://your-domain.com
  2. Configure First User: Login with credentials you set
  3. Review Features: Check Status Page for available features
  4. Follow First Steps: First Steps Guide β†’

More Information

First Steps

Once you’ve set up your Cloudillo Identity and Storage, it’s time to get creative and start sharing content.

Accessing your Cloudillo Space

To access your Cloudillo Space, use your own URL, which usually matches your Identity. First, authenticate yourself using the method you set up during installation or registration.

Important

Remember, always access Cloudillo using your own URL, and never give your credentials on any other site!

Inside your Cloudillo Space you’ll run various applications from other spaces. Cloudillo will always let you know when this happens and shows the trust level of the application.

The Cloudillo Shell

When you enter your Cloudillo Space, you’ll see the Cloudillo Shell. This is the main application and it typically displays a header on the top of the window, unless you are running a full-screen application.

Connecting with Others

Cloudillo doesn’t have a central database, so you’ll need to know the identities of other users, organizations, or groups to connect with them. Once you’ve joined some groups, you can connect with others through those groups.

You can start by following the cloudillo.net identity, which is the official identity of the Cloudillo Community. From there you can find and join other groups.

Core Concept & Architecture

Cloudillo is an open-source collaboration platform that allows users to store their own data wherever they choose. Users can:

  • Self-host their data
  • Store data with community-hosted servers
  • Use third-party storage providers offering Cloudillo as a service

This ensures privacy-conscious users maintain full control while enabling seamless onboarding for less technical users. The design also ensures full user autonomy, preventing vendor lock-in by allowing seamless migration between storage providers or self-hosting at any time.

Architecture Diagram

Cloudillo Architecture Cloudillo Architecture

Identity System & User Profiles

Cloudillo decouples identity from storage through a Domain Name System (DNS)-based identity system. Each user and community has a stable, trusted identifier independent of storage location.

Users can create identities using their own domain name, allowing them to maintain full control and leverage their trusted brand within the platform.

Cloudillo Identity Providers (CIP) help users create and manage identities without manual DNS setup. Any domain owner can implement a CIP, with the first provider being cloudillo.net.

Each Cloudillo Identity has a publicly accessible profile containing:

  • Identity Name
  • Identity Public Key
  • Optional additional metadata

The Identity Public Key verifies user authenticity within the Cloudillo network.

Content-Addressing & Merkle Trees

Cloudillo uses content-addressing throughout its architecture, where every action, file, and data blob is identified by the cryptographic hash of its content. This creates a merkle tree structure that provides:

  • Cryptographic proof of authenticity: Anyone can verify content integrity
  • Immutability: Content cannot be modified without changing its identifier
  • Tamper-evidence: Any modification is immediately detectable
  • Deduplication: Identical content produces identical identifiers
  • Trustless verification: No need to trust storage providers

Every resource in Cloudilloβ€”from action tokens to image blobsβ€”is part of this merkle tree, creating a verifiable chain of trust from user actions down to individual bytes of data.

Learn more: Content-Addressing & Merkle Trees

Actions

Cloudillo supports event-driven communication between nodes. Actions are event-driven interactions that users perform, such as:

  • Connecting with another user
  • Following a user
  • Posting, commenting, or reacting
  • Sharing a document

When an action occurs, Cloudillo generates a cryptographically signed Action Token, distributing it to involved parties. This mechanism prevents spam and unauthorized actions.

Action tokens are content-addressed, meaning each action has a unique identifier derived from the hash of its content. This ensures actions are immutable and verifiable.

Access Control & Resource Sharing

When a user wants to access a resource stored on another node (e.g., editing a document hosted by another user), the following process occurs:

  • The user’s node requests access using their Identity Key.
  • The remote node validates the request and grants an Access Token.
  • The application uses this token to interact with the resource on behalf of the user.

This process ensures secure, decentralized access control without requiring direct trust between storage providers. This process is designed to be seamless, requiring no additional user interactionβ€”offering the ease of centralized cloud platforms while maintaining full decentralization.

Authentication & Authorization Tokens

Cloudillo utilizes cryptographic tokens for authentication and authorization:

  • Action Tokens: Signed by users, representing their activities (e.g., posting, following).
  • Access Tokens: Used for resource access, granted by the data owner’s node.
  • Verification: All tokens are cryptographically signed and validated by the recipient nodes before processing.

Resource Types & Storage Model

Cloudillo supports multiple resource types:

  • Extensible Actions: Developers can define new action types that integrate seamlessly with Cloudillo’s decentralized architecture, allowing for future expansion of platform capabilities.
  • Immutable Resources: Content such as images, videos, and published documents. These are content-addressed using cryptographic hashes to ensure integrity and enable deduplication.
  • Editable Documents: Built-in collaborative structures allow real-time multi-user editing of documents.
  • Databases: Cloudillo provides a real-time database API, enabling developers to create custom applications on top of the platform.

This design ensures Cloudillo remains flexible and adaptable to evolving user and developer needs.

Architectural Documentation

Dive deeper into Cloudillo’s architecture:

Fundamentals

Data Storage & Access

Actions & Federation

Runtime Systems

Subsections of Core Concept & Architecture

Fundamentals

The foundational concepts underlying Cloudillo’s architecture. These documents explain the core principles, system design patterns, identity mechanisms, and security considerations that enable Cloudillo’s federated, decentralized design.

Core Topics

  • System Architecture - Technical overview of the core patterns and components
  • Identity System - DNS-based identity and profile key management
  • Content-Addressing & Merkle Trees - Cryptographic hashing and proof of authenticity
  • Network & Security - Security architecture and inter-instance communication

These fundamentals form the basis for understanding Cloudillo’s data storage, action systems, and federation mechanisms.

Subsections of Fundamentals

System Architecture Overview

This document provides a technical overview of the Cloudillo system architecture, explaining the core patterns and components that enable its federated, privacy-focused design.

Workspace Structure

Cloudillo is organized as a Rust workspace with the following crates:

cloudillo-rs/
β”œβ”€β”€ server/                      # Core library (cloudillo)
β”œβ”€β”€ basic-server/                # Reference implementation
└── adapters/
    β”œβ”€β”€ auth-adapter-sqlite/     # Authentication & cryptography (SQLite)
    β”œβ”€β”€ meta-adapter-sqlite/     # Metadata storage (SQLite)
    β”œβ”€β”€ blob-adapter-fs/         # Binary blob storage (Filesystem)
    β”œβ”€β”€ rtdb-adapter-redb/       # Real-time database (Redb)
    └── crdt-adapter-redb/       # Collaborative editing CRDT (Redb)

Crate Responsibilities

  • server: Core business logic, HTTP handlers, federation, task system
  • basic-server: Executable reference implementation using SQLite, Redb, and filesystem
  • adapters: Five pluggable storage backends implementing the core adapter traits

Adapter Types

The five adapters separate concerns and enable flexible deployments:

  1. AuthAdapter - Authentication, JWT tokens, certificate management, cryptographic operations
  2. MetaAdapter - Tenant/profile data, action tokens, file metadata, tasks
  3. BlobAdapter - Content-addressed immutable binary data (files, images, snapshots)
  4. RtdbAdapter - Real-time hierarchical JSON database with queries and subscriptions
  5. CrdtAdapter - Collaborative document editing with conflict-free merges

Core Architecture Patterns

The Five-Adapter Architecture

Cloudillo’s architecture is built on five fundamental adapters that separate concerns and enable flexible deployments:

1. AuthAdapter

Purpose: Authentication, authorization, and cryptographic operations

Responsibilities:

  • JWT/token validation and creation
  • TLS certificate management (ACME integration)
  • Profile signing key storage and rotation
  • VAPID keys for push notifications
  • Password hashing and WebAuthn credentials
  • Tenant management

Key Operations: Validate tokens, create access tokens, manage TLS certificates, handle profile keys, store passwords/credentials

Why separate? Authentication and cryptography require special security considerations and may need different storage backends (HSM, vault services, etc.).

2. MetaAdapter

Purpose: Structured metadata storage

Responsibilities:

  • Tenant and profile information
  • Action tokens (posts, comments, reactions, etc.)
  • File metadata with variants
  • Task persistence for the scheduler
  • Database metadata (for RTDB)

Key Operations: Create/query tenants, store/retrieve actions, manage file metadata, persist tasks, handle profiles

Why separate? Metadata can be stored in different databases (SQLite, PostgreSQL, MongoDB) based on scale and requirements.

3. BlobAdapter

Purpose: Immutable binary data storage

Responsibilities:

  • Content-addressed blob storage
  • File data (images, videos, documents)
  • Database snapshots (for RTDB)
  • Both buffered and streaming I/O

Key Operations: Write/read blobs (buffered or streaming), check blob existence and size

Why separate? Blob storage can use filesystem, S3, CDN, or specialized object storage based on scale and cost considerations.

4. RtdbAdapter

Purpose: Real-time structured database

Responsibilities:

  • Path-based hierarchical storage (Firebase-like)
  • Query with filtering, sorting, pagination
  • Real-time subscriptions via WebSocket
  • Transaction support for atomic writes
  • Secondary indexes for performance
  • Multi-tenant database isolation

Key Operations: Query/create/update/delete documents, transactions, real-time subscriptions

Why separate? Real-time database functionality can use different backends (redb, PostgreSQL, MongoDB) based on query requirements and scale.

Learn more: RTDB with redb

5. CrdtAdapter

Purpose: Collaborative document storage with CRDTs

Responsibilities:

  • Binary CRDT update storage (Yjs protocol)
  • Document metadata management
  • Real-time change subscriptions
  • Conflict-free merge guarantees
  • Awareness state (presence, cursors)
  • Multi-tenant document isolation

Key Operations: Create/read documents, append updates, retrieve update history, subscribe to changes

Why separate? CRDT storage can use different backends (redb, dedicated CRDT stores) and has different performance characteristics than traditional databases.

Learn more: CRDT Collaborative Editing

Benefits of the Adapter Pattern

βœ… Flexible Deployment: Switch storage backends without changing core logic βœ… Separation of Concerns: Security, metadata, and binary data have different requirements βœ… Testing: Easy to create in-memory adapters for testing βœ… Scalability: Can distribute adapters across different services βœ… Cost Optimization: Use appropriate storage for each data type

Content-Addressed Architecture

Cloudillo uses content-addressing throughout its architecture, where resource identifiers are cryptographic hashes of their content. This creates a merkle tree structure that provides cryptographic proof of authenticity and immutability.

Hash-Based Identifiers

All resource IDs are SHA-256 hashes with versioned prefixes:

Prefix Resource Type Hash Input Example
a1~ Action Entire JWT token (header + payload + signature) a1~8kR3mN9pQ2vL6xW...
f1~ File File descriptor string f1~Qo2E3G8TJZ2HTGh...
b1~ Blob Blob bytes (actual image/video data) b1~abc123def456ghi...
d2, Descriptor (not a hash, the encoded string itself) d2,vis.tn:b1~abc:f=avif:...

Version Scheme

Format: {prefix}{version}~{base64_encoded_hash}

  • Version 1: SHA-256 with base64url encoding (no padding)
  • Future versions: Can upgrade to SHA-3, BLAKE3, etc. without breaking old content
  • Backward compatibility: Old content remains valid forever
  • Algorithm agility: Migrate to new algorithms without breaking existing references

Example upgrade path:

a1~...  (SHA-256)
a2~...  (SHA-3)
a3~...  (BLAKE3)

Six-Level Merkle Tree

Content-addressing creates a hierarchical merkle tree:

Level 1: Blob Data (raw bytes)
   ↓ SHA-256 hash
Level 2: Blob ID (b1~hash)
   ↓ collected in descriptor
Level 3: File Descriptor (d2,class.variant:b1~hash:format:size:resolution;...)
   ↓ SHA-256 hash of descriptor
Level 4: File ID (f1~hash)
   ↓ referenced in action
Level 5: Action Token (JWT with content, parent, attachments)
   ↓ SHA-256 hash of entire JWT
Level 6: Action ID (a1~hash)

Properties

βœ… Immutable: Content cannot change without changing the ID βœ… Tamper-Evident: Any modification is immediately detectable βœ… Deduplicatable: Identical content produces identical IDs βœ… Verifiable: Anyone can recompute and verify hashes βœ… Cacheable: Content-addressed data can be cached forever βœ… Trustless: No need to trust storage providersβ€”verify the hash

Integration with Adapters

Content-addressing is implemented across multiple adapters:

BlobAdapter:

  • Stores blobs indexed by blob_id (b1~...)
  • Blob IDs are SHA-256 hashes of the blob bytes
  • Enables deduplication and integrity verification

MetaAdapter:

  • Stores action tokens indexed by action_id (a1~...)
  • Action IDs are SHA-256 hashes of the JWT token
  • Stores file metadata with file_id (f1~...)
  • File IDs are SHA-256 hashes of the descriptor

AuthAdapter:

  • Signs action tokens with profile keys (ES384)
  • Signature + content hash = complete authenticity proof

Security Benefits

Content-addressing provides multiple security layers:

  1. Integrity Verification: SHA-256 ensures data hasn’t been tampered with
  2. Deduplication: Same content = same hash, prevents storage waste
  3. Cryptographic Binding: Parent references create immutable chains
  4. Federation Trust: Remote instances can verify data integrity
  5. Cache Safety: Content-addressed data can be cached without trust

Performance Benefits

  1. Caching: Immutable content can be cached forever (max-age=31536000)
  2. Deduplication: Identical blobs stored only once across all tenants
  3. Parallel Verification: Hash verification can be parallelized
  4. CDN-Friendly: Content-addressed resources perfect for CDN distribution

Learn more: Content-Addressing & Merkle Trees

Task-Based Asynchronous Processing

Complex operations in Cloudillo are modeled as persistent tasks that can execute asynchronously, survive restarts, and depend on other tasks.

Task System Components

Tasks

Tasks implement the Task<S> trait:

pub trait Task<S>: Debug + Send + Sync {
    fn kind() -> &'static str;
    async fn run(&self, state: &S) -> Result<()>;
    fn priority(&self) -> Priority { Priority::Medium }
    fn dependencies(&self) -> Vec<TaskId> { vec![] }
}

Built-in Task Types:

  • ActionCreatorTask: Creates and signs action tokens for federation
  • ActionVerifierTask: Validates incoming federated actions
  • ActionDeliveryTask: Delivers actions to remote instances with retry
  • FileIdGeneratorTask: Generates content-addressed file IDs
  • ImageResizerTask: Creates image variants (thumbnails, etc.)
  • VideoTranscoderTask: Transcodes video to web-optimized formats
  • AudioExtractorTask: Extracts audio metadata
  • PdfProcessorTask: Extracts text and metadata from PDFs
  • EmailSenderTask: Sends emails asynchronously
  • CertRenewalTask: Handles ACME certificate renewal
  • ProfileRefreshBatchTask: Batch-refreshes remote profiles
  • TenantImageUpdaterTask: Updates tenant avatar images

Scheduler

The scheduler manages task lifecycle with dependency resolution:

Features:

  • Task registry with dynamic builders
  • Dependency resolution (DAG-based)
  • Scheduled execution (cron-like)
  • Persistence via MetaAdapter (survives restarts)
  • Notification system for task completion

Example Flow:

ActionCreatorTask (depends on FileIdGeneratorTask)
    ↓
    waits for file processing to complete
    ↓
FileIdGeneratorTask completes
    ↓
ActionCreatorTask auto-starts
    ↓
Creates signed JWT, stores in MetaAdapter

Worker Pool

A priority-based thread pool for CPU-intensive and blocking operations:

Architecture:

  • Three priority tiers: High > Medium > Low
  • Each tier has configurable worker thread count
  • Uses flume MPMC channels for work distribution
  • Returns futures for async integration

Default Configuration (basic-server):

  • 1 high-priority worker
  • 2 medium-priority workers
  • 1 low-priority worker

Use Cases:

  • Image processing (CPU-intensive)
  • Cryptographic operations
  • File compression
  • Blocking I/O

Why Task-Based Processing?

βœ… Resilience: Tasks survive server restarts βœ… Dependencies: Complex workflows with ordered execution βœ… Scheduling: Cron-like execution for periodic tasks βœ… Observability: Track task progress and failures βœ… Concurrency: Priority-based execution

Application State Management

AppState Structure

The core application state contains: scheduler, worker pool, HTTP client, TLS certificates, and all five adapters (auth, meta, blob, rtdb, crdt).

AppBuilder Pattern

Configuration uses a fluent builder API with mode, identity, domain, data directory, and adapter selections.

Configuration Options:

  • Server mode (Standalone, Proxy, StreamProxy)
  • Network binding (HTTPS/HTTP ports)
  • Domain configuration
  • Directory paths (dist, tmp, data)
  • Adapter injection
  • Worker pool sizing

Module Organization

The server crate is organized by functional domain:

core/ - Infrastructure Layer

Core system components providing foundational services:

  • app.rs: Application state, builder, bootstrap logic
  • acme.rs: Let’s Encrypt/ACME certificate management
  • webserver.rs: Axum/Rustls HTTPS server with SNI
  • middleware.rs: Authentication middleware
  • extract.rs: Custom Axum extractors (TnId, IdTag, Auth)
  • scheduler.rs: Task scheduling with dependencies
  • worker.rs: Thread pool with priority levels
  • websocket.rs: WebSocket infrastructure
  • request.rs: HTTP client for federation
  • hasher.rs: Content-addressable storage (SHA256)
  • utils.rs: Random ID generation

auth/ - Authentication Module

  • handler.rs: Login endpoints, token generation, password management

action/ - Action/Activity Subsystem

Implements the federated activity system:

  • action.rs: Action creation/verification tasks
  • process.rs: JWT verification, token processing
  • handler.rs: Action CRUD endpoints, federation inbox

profile/ - Profile Management

  • handler.rs: Tenant profile endpoints

file/ - File Storage & Processing

  • file.rs: File descriptor encoding, variant selection
  • handler.rs: File upload/download endpoints
  • image.rs: Image resizing tasks
  • store.rs: Storage layer abstraction

rtdb/ - Real-Time Database

Note: In development, see RTDB Architecture for details

  • handler.rs: Database CRUD endpoints
  • websocket.rs: WebSocket connection handler
  • manager.rs: Database instance lifecycle

routes/ - HTTP Routing

  • Separates public and protected route groups
  • API endpoints (/api/*)
  • Static file serving for frontend
  • CORS configuration

Concurrency Model

Multi-threaded Tokio Runtime

Cloudillo uses Tokio’s multi-threaded async runtime for handling concurrent requests.

Concurrency Layers

  1. Async Layer (Tokio): HTTP request handling, WebSocket connections, I/O
  2. Worker Pool: CPU-intensive tasks, blocking operations
  3. Scheduler: Background task execution with dependencies

Interaction Example:

HTTP Request β†’ Tokio async handler
    ↓
Spawn ImageResizerTask on scheduler
    ↓
Scheduler dispatches to worker pool (CPU-intensive)
    ↓
Worker completes, updates MetaAdapter
    ↓
Scheduler notifies waiting tasks
    ↓
Response returned to client

Request Handling Flow

Middleware Pipeline

HTTP requests flow through these layers:

  1. HTTPS/SNI Resolution: CertResolver selects TLS certificate by domain
  2. Tracing/Logging: Request tracing with structured logging
  3. Authentication: require_auth or optional_auth middleware
  4. Custom Extractors: TnId, IdTag, Auth extract from request context
  5. Handler: Business logic execution
  6. Response: JSON serialization, error handling

Custom Extractors

Axum extractors provide typed access to request context:

  • TnId: Tenant ID (database primary key, i64)
  • IdTag: Tenant identifier string (e.g., “alice.example.com”)
  • Auth: Full authentication context (tn_id, id_tag, scope, etc.)

Error Handling

Custom error enum with automatic HTTP response conversion: NotFound (404), PermissionDenied (403), DbError/Unknown/Parse/Io (500).

Server Modes

Cloudillo supports different deployment modes:

Standalone (Default)

Self-contained single instance:

  • HTTPS on configured port
  • Optional HTTP for ACME challenges
  • All adapters run locally

Use case: Personal servers, small communities

Proxy

Used if Cloudillo is behind a reverse proxy:

  • Listens on HTTP port
  • Certificate handled is the responsibility of the proxy

Use case: Managed hosting providers, self-hosting with multiple services on one IP address

Security Architecture

Implemented in Rust

Maximal memory and concurrency safety. Minimal attack surface.

No Unsafe Code

Cloudillo enforces memory safety:

#![forbid(unsafe_code)]

ABAC Permission System

Cloudillo uses Attribute-Based Access Control (ABAC) for fine-grained permissions across all resources. ABAC provides flexible permission rules based on:

  • User attributes (identity, roles, relationships)
  • Resource attributes (owner, visibility, type)
  • Contextual factors (time, environment)

Key Features:

  • Six visibility levels: Public (P), Verified (V), SecondDegree (2), Follower (F), Connected (C), Direct (NULL)
  • Policy-based access control (TOP/BOTTOM policies)
  • Relationship-aware (following, connections)
  • Time-based permissions
  • Custom policy support

Learn more: ABAC Permission System

Cryptographic Algorithms

  • P384: Elliptic curve for action signing
  • ES384: JWT signature algorithm
  • SHA256: Content addressing and hashing
  • bcrypt: Password hashing

Security Layers

  1. TLS/HTTPS: All connections encrypted (Rustls)
  2. ACME: Automatic certificate management (Let’s Encrypt)
  3. JWT: Cryptographically signed tokens
  4. Content Addressing: Tamper detection via SHA256
  5. Permission Checks: Authorization at every access point

Bootstrap Process

Initial setup when starting a new instance:

  1. Create tenant with base_id_tag
  2. Set password (if provided)
  3. Generate profile signing key (P384)
  4. Initiate ACME for TLS certificates (if email provided)
  5. Start scheduler and worker pool
  6. Start HTTP/HTTPS servers

Example configuration:

BASE_ID_TAG=alice.example.com        # Required
BASE_PASSWORD=secret                 # Initial password
ACME_EMAIL=alice@example.com         # Let's Encrypt email
MODE=standalone                      # Server mode
LISTEN=0.0.0.0:8443                  # HTTPS binding
DATA_DIR=./data                      # storage path

Key Dependencies

Web Framework

  • axum (0.8): Async web framework
  • tower: Service abstractions
  • tower-http: CORS, static files

TLS & Crypto

  • rustls (0.23): Pure Rust TLS
  • instant-acme (0.8): ACME client
  • jsonwebtoken (9.3): JWT handling
  • p384: Elliptic curve operations

Async Runtime

  • tokio (1.48): Multi-threaded async

Serialization

  • serde (1.0): Serialization framework
  • serde_json: JSON support

Database

  • sqlx (0.8): Async SQL (SQLite support)

Utilities

  • image (0.25): Image processing
  • sha2: SHA256 hashing
  • croner (3.0): Cron expressions
  • flume: MPMC channels

Architectural Strengths

βœ… Pluggable Adapters: Easy to swap storage backends βœ… Self-Contained: No external dependencies required βœ… Federated: Communicates with other instances βœ… Task-Based: Persistent, resumable async execution βœ… Type-Safe: Leverages Rust’s type system βœ… Memory-Safe: Complete #![forbid(unsafe_code)] βœ… Observable: Built-in tracing integration

Next Steps

Identity System & User Profiles

Cloudillo Profiles are located using a DNS-based identity system. Each Cloudillo Identity is associated with a specific API endpoint, which can be accessed via the “cl-o” subdomain of the identity. For example, the API domain of the cloudillo.net identity is available at https://cl-o.cloudillo.net/.

Retrieving a Cloudillo Profile

You can retrieve a Cloudillo profile by making an API request to the /api/me endpoint of the identity’s API domain:

curl https://cl-o.cloudillo.net/api/me

Example response:

{
  "idTag": "cloudillo.net",
  "name": "Cloudillo",
  "type": "community",
  "profilePic": "QoEYeG8TJZ2HTGhVlrtTDBpvBGOp6gfGhq4QmD6Z46w",
  "keys": [
    {
      "keyId": "20250205",
      "publicKey": "MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAENSq6EZZ+xCypWNkm+0+MHIZfLX7I01wTT+SOw7DoOUOuDAWKMkhBVG+SUb9AzCxEOlmkefuEW5zmXNwmH2MphEQW/r18RDjd+Nt5nbemBsoQzsm2Wg/mUyWBsKYs1oe5"
    }
  ]
}

Profile Fields Explained

  • idTag: The Identity Tag associated with the profile.
  • name: The display name of the profile.
  • type: Either empty for a personal profile or contains “community” for community profiles.
  • profilePic: The identifier for the profile picture, which can be retrieved via the storage API.
  • keys: A list of cryptographic keys associated with the profile.

Profile Picture Retrieval

The profilePic field contains an identifier that allows retrieval of the profile’s profile picture using the Cloudillo Storage API. A request to retrieve the profile picture may look like this:

curl https://cl-o.nikita.cloudillo.net/api/store/NMzE4gNI29aEkiV6Q7I1UNWh4x2gFZ7753Pl74veYtU

Key Management

A profile supports multiple cryptographic keys, enabling periodic key rotation for enhanced security. Older keys can be deprecated while maintaining account integrity.

Key Structure

Each profile key contains:

pub struct ProfileKey {
    key_id: String,        // Identifier (e.g., "20250205")
    public_key: String,    // Base64-encoded P384 public key
    created_at: i64,       // Unix timestamp
    expires_at: Option<i64>,  // Optional expiration
    key_type: KeyType,     // Purpose of the key
}

pub enum KeyType {
    Signing,    // For action token signatures
    VAPID,      // For push notifications
    // Future: Encryption, Authentication, etc.
}

Cryptographic Algorithms

Cloudillo uses elliptic curve cryptography for signing:

  • Curve: P384 (NIST P-384, also known as secp384r1)
  • Algorithm: ES384 (ECDSA using P-384 and SHA-384)
  • Key Size: 384 bits
  • Signature Size: ~96 bytes

Why P384?

  • Strong security (192-bit security level)
  • Widely supported
  • Good performance
  • Standards-compliant (FIPS 186-4)

Key Rotation

Best practices for key rotation:

  1. Generate new key with current date as keyId
  2. Add to profile alongside existing keys
  3. Start using new key for new signatures
  4. Keep old keys active for verification (30-90 days)
  5. Mark old keys as expired after grace period
  6. Remove expired keys after verification period

Example key rotation timeline:

Day 0:   Generate key "20250301"
Day 0:   Add to profile, start signing with new key
Day 30:  Mark old key "20250101" as expiring soon
Day 90:  Set expires_at on old key
Day 180: Remove old key from profile

Key Generation

Keys are generated during tenant bootstrap:

generate_p384_keypair()
encode_public_key()
key_id = current_date (YYYYMMDD format)
store in AuthAdapter (private key stored securely)

VAPID Keys

VAPID (Voluntary Application Server Identification) keys are used for web push notifications:

pub struct VapidKey {
    key_id: String,
    public_key: String,   // Base64-encoded
    private_key: String,  // Stored in AuthAdapter, never exposed
    contact: String,      // mailto: or https: URI
}

These are separate from signing keys to isolate concerns and enable different rotation policies.

Identity Resolution

DNS-Based Discovery

Cloudillo uses DNS to discover the API endpoint for an identity:

  1. Identity: alice.example.com
  2. API Domain: cl-o.alice.example.com
  3. API Endpoint: https://cl-o.alice.example.com/api/me

Cloudillo Identity Providers (CIP)

Users without their own domain can use a Cloudillo Identity Provider:

CIP Responsibilities:

  • Domain management (subdomains or custom domains)
  • DNS configuration
  • Dynamic DNS support
  • CIPs never store or access any user data, nor have any rescponsibilities about it.

Example CIP: cloudillo.net

  • Provides identities like alice.cloudillo.net
  • API available at https://cl-o.alice.cloudillo.net
  • Users can migrate to their own domain later

Custom Domain Setup

To use your own domain with Cloudillo:

  1. DNS Configuration:
cl-o.alice.example.com.  A      <your-server-ip>
cl-o.alice.example.com.  AAAA   <your-server-ipv6>
app.example.com.  A             <your-server-ip>
app.example.com.  AAAA          <your-server-ipv6>
  1. TLS Certificate: Automatic via ACME (Let’s Encrypt)
ACME_EMAIL=admin@example.com
  1. Bootstrap: Configure tenant
BASE_ID_TAG=alice.example.com
BASE_APP_DOMAIN=app.example.com
BASE_PASSWORD=<secure-password>

Profile Metadata

Storage

Profile metadata is stored in the MetaAdapter:

pub struct ProfileMetadata {
    tn_id: TnId,           // Internal tenant ID
    id_tag: String,        // Public identifier
    name: String,          // Display name
    profile_type: ProfileType,
    profile_pic: Option<String>,  // File ID
    bio: Option<String>,
    created_at: i64,
    updated_at: i64,
}

pub enum ProfileType {
    Personal,
    Community,
}

Synchronization

Profiles can be synchronized across instances for caching:

GET https://cl-o.{remote_id_tag}/api/me
Parse JSON response
Cache locally in MetaAdapter

Security Considerations

Public Key Infrastructure

  • Public keys are publicly accessible via /api/me
  • Private keys are stored securely in AuthAdapter
  • No private key export - keys never leave the server
  • Key verification happens on every action token

Identity Trust Model

Trust is established through:

  1. DNS ownership: Control of domain proves identity ownership
  2. Key signatures: Private key proves control of identity
  3. HTTPS: TLS proves server authenticity
  4. Action signatures: ES384 signatures prove action authenticity

Attack Mitigation

DNS Hijacking:

  • Old signatures remain valid (past actions can’t be forged)
  • Users can verify key changes

Key Compromise:

  • Rotate immediately to new key
  • Mark compromised key as expired
  • Past actions become invalid

Server Compromise:

  • Private keys in AuthAdapter may need HSM/vault for high-security deployments
  • Separate AuthAdapter storage from other adapters
  • Regular security audits

Bootstrap Process Details

When initializing a new Cloudillo instance:

1. Create tenant in MetaAdapter
   tn_id = meta_adapter.create_tenant(base_id_tag)

2. Set password (if provided)
   hash = bcrypt.hash(password)
   auth_adapter.create_password(tn_id, hash)

3. Generate profile signing key
   (public_key, private_key) = generate_p384_keypair()
   key_id = current_date (YYYYMMDD format)
   auth_adapter.create_profile_key(tn_id, key_id, public_key, private_key)

4. Initiate ACME certificate (if email provided)
   cert = acme.request_certificate(domain, email)
   auth_adapter.create_cert(domain, cert)

API Reference

GET /api/me

Retrieve public profile information.

Request:

curl https://cl-o.example.com/api/me

Response (200 OK):

{
  "idTag": "example.com",
  "name": "Alice",
  "type": "",
  "profilePic": "f1~Qo...46w",
  "keys": [
    {
      "keyId": "20250205",
      "publicKey": "MHYwEAYHKoZIzj0CAQ..."
    }
  ]
}

GET /api/me/keys

Retrieve all public keys (for verification).

Request:

curl https://cl-o.example.com/api/me/keys

Response (200 OK):

{
  "keys": [
    {
      "keyId": "20250205",
      "publicKey": "MHYwEAYHKoZIzj0CAQ...",
      "createdAt": 1738704000,
      "expiresAt": null,
      "keyType": "signing"
    },
    {
      "keyId": "20250101",
      "publicKey": "MHYwEAYHKoZIzj0CAQ...",
      "createdAt": 1735689600,
      "expiresAt": 1743465600,
      "keyType": "signing"
    }
  ]
}

See Also

Content-Addressing & Merkle Trees

Cloudillo implements a merkle tree structure using content-addressed identifiers throughout its architecture. Every action, file, and data blob is identified by the cryptographic hash of its content, creating an immutable, verifiable chain of trust.

What is Content-Addressing?

Content-addressing means identifying data by what it is (its content) rather than where it is (its location). Instead of using arbitrary IDs or URLs, Cloudillo computes a cryptographic hash of the content itself and uses that hash as the identifier.

Benefits

βœ… Immutable: Content cannot change without changing its identifier βœ… Tamper-Evident: Any modification is immediately detectable βœ… Deduplicatable: Identical content produces identical identifiers βœ… Verifiable: Anyone can recompute and verify hashes independently βœ… Cacheable: Content-addressed data can be cached forever βœ… Trustless: No need to trust storage providersβ€”verify the hash

Hash Function

Cloudillo uses SHA-256 for all content-addressing:

  • Algorithm: SHA-256 (256-bit Secure Hash Algorithm)
  • Encoding: Base64url without padding (URL-safe)
  • Output: 43-character base64-encoded string
  • Collision Resistance: Cryptographically secure
compute_hash(prefix, data):
    hash = SHA256(data)
    encoded = base64url_encode(hash)  // URL-safe, no padding
    return "{prefix}1~{encoded}"

// Example:
compute_hash("b", blob_bytes) β†’ "b1~abc123def456..." (43 chars)
compute_hash("f", descriptor)  β†’ "f1~Qo2E3G8TJZ..." (43 chars)
compute_hash("a", jwt_token)   β†’ "a1~8kR3mN9pQ2vL..." (43 chars)

Merkle Tree Structure

Cloudillo’s content-addressing creates a variable-depth merkle tree where actions can reference other actions recursively. The example below shows a six-level hierarchy for a POST action with image attachments:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Level 6: Action ID (a1~8kR3mN9pQ2vL6xW...)              β”‚
β”‚   ↑ SHA-256 hash of ↓                                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 5: Action Token (JWT)                             β”‚
β”‚   Header: {"alg":"ES384","typ":"JWT"}                   β”‚
β”‚   Payload: {                                            β”‚
β”‚     "iss": "alice.example.com",                         β”‚
β”‚     "t": "POST:IMG",                                    β”‚
β”‚     "c": "Amazing photo!",                              β”‚
β”‚     "a": ["f1~Qo2E3G8TJZ..."],                          β”‚
β”‚     "iat": 1738483100                                   β”‚
β”‚   }                                                     β”‚
β”‚   Signature: <ES384 signature>                          β”‚
β”‚   ↓ references ↓                                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 4: File ID (f1~Qo2E3G8TJZ2HTGhVlrtTDBp...)        β”‚
β”‚   ↑ SHA-256 hash of ↓                                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 3: File Descriptor (d2,...)                       β”‚
β”‚   "d2,vis.tn:b1~abc:f=avif:s=4096:r=150x150;            β”‚
β”‚        vis.sd:b1~def:f=avif:s=32768:r=640x480;          β”‚
β”‚        vis.md:b1~ghi:f=avif:s=262144:r=1920x1080"       β”‚
β”‚   ↓ references ↓                                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 2: Variant IDs                                    β”‚
β”‚   b1~abc123... (vis.tn)                                 β”‚
β”‚   b1~def456... (vis.sd)                                 β”‚
β”‚   b1~ghi789... (vis.md)                                 β”‚
β”‚   ↑ SHA-256 hash of ↓                                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 1: Blob Data (raw bytes)                          β”‚
β”‚   <AVIF encoded image data - tn variant>                β”‚
β”‚   <AVIF encoded image data - sd variant>                β”‚
β”‚   <AVIF encoded image data - md variant>                β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Level 1: Blob Data

Raw bytes of actual content (images, videos, documents).

  • No identifier yetβ€”just the binary data
  • This is what gets hashed to create variant IDs

Level 2: Variant IDs (b1~…)

Content-addressed blob identifiers computed as SHA256(blob_bytes).

Properties:

  • Identifies a single image/video variant
  • Changing even one byte changes the entire ID
  • Multiple actions can reference the same variant_id (deduplication)

Level 3: File Descriptor (d2,…)

Encoded string listing all available variants of a file.

Format:

d2,{class}.{variant}:{variant_id}:f={format}:s={size}:r={width}x{height};...

Example:

d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x150;vis.sd:b1~def456:f=avif:s=32768:r=640x480

This descriptor says:

  • Thumbnail variant: b1~abc123, AVIF format, 4KB, 150Γ—150px
  • Standard variant: b1~def456, AVIF format, 32KB, 640Γ—480px

Level 4: File ID (f1~…)

Content-addressed file identifier computed as SHA256(descriptor_string).

Properties:

  • Identifies the complete file with all its variants
  • Changing any variant changes the descriptor, which changes the file_id
  • Used in action token attachments

Level 5: Action Token (JWT)

Cryptographically signed JSON Web Token representing a user action.

Structure:

{
  "header": {
    "alg": "ES384",
    "typ": "JWT"
  },
  "payload": {
    "iss": "alice.example.com",
    "k": "20250101",
    "t": "POST:IMG",
    "c": "Check out this amazing photo!",
    "a": ["f1~Qo2E3G8TJZ..."],
    "iat": 1738483100
  },
  "signature": "<ES384 signature with alice's private key>"
}

Encoding: {base64(header)}.{base64(payload)}.{base64(signature)}

Level 6: Action ID (a1~…)

Content-addressed action identifier computed as SHA256(complete_jwt_token).

Properties:

  • Identifies the complete action immutably
  • Changing any field (even whitespace) changes the action_id
  • Used as parent references in replies/reactions

Hash Versioning Scheme

All identifiers use a versioned prefix format for future-proofing:

{prefix}{version}~{base64_encoded_hash}

Current Prefixes

Prefix Resource Type Hash Input Example
a1~ Action Entire JWT token a1~8kR3mN9pQ2vL6xW...
f1~ File File descriptor string f1~Qo2E3G8TJZ2HTGh...
d2, Descriptor (not a hash, the encoded format itself) d2,vis.tn:b1~abc:f=avif:...
b1~ Blob Blob bytes (raw data) b1~abc123def456ghi...

Version Scheme

  • Version 1: SHA-256 with base64url encoding (no padding)
  • Future versions: Can upgrade to SHA-3, BLAKE3, etc.
  • Backward compatibility: Old content remains valid forever
  • Algorithm agility: Migrate to new algorithms without breaking existing references

Example upgrade path:

a1~...  (SHA-256)
a2~...  (SHA-3)
a3~...  (BLAKE3)

Merkle Tree Properties

Cloudillo’s content-addressing creates a merkle tree with these properties:

Immutability

Once created, content cannot be modified without changing its identifier.

Example:

Original Post:
  content: "Hello World"
  action_id: a1~abc123...

Modified Post:
  content: "Hello Universe"
  action_id: a1~xyz789...  ← DIFFERENT ID!

Any attempt to modify content results in a completely new action with a different ID. The original remains unchanged.

Tamper-Evidence

Any modification anywhere in the tree is immediately detectable.

Example:

Post (a1~abc...)
  └─ Attachment: f1~Qo2...
      └─ Variants: b1~tn..., b1~sd..., b1~md...

If someone modifies the thumbnail image:
  βœ— New variant_id (b1~xyz...)
  βœ— Descriptor changes
  βœ— New file_id (f1~uvw...)
  βœ— Post attachment no longer matches
  βœ— Verification fails!

Deduplication

Identical content produces identical identifiers, enabling automatic deduplication.

Example:

Alice posts photo.jpg β†’ file_id: f1~abc123...
Bob posts photo.jpg (same file) β†’ file_id: f1~abc123... (same!)

Storage: Only one copy of the blob bytes needed
Bandwidth: Can skip downloading if already cached

Verifiability

Anyone can independently verify the entire chain.

Process:

  1. Download action token
  2. Verify JWT signature (proves author identity)
  3. Recompute action_id = SHA256(JWT)
  4. Compare with claimed action_id
  5. For each attachment:
    • Download file descriptor
    • Download all variant blobs
    • Verify each variant_id = SHA256(blob)
    • Recompute file_id = SHA256(descriptor)
    • Compare with attachment reference
  6. βœ“ Complete verification

No trust required: Pure mathematics ensures integrity.

Chain of Trust

Parent references create an immutable chain.

Example:

Post (a1~abc...)
  ↑ parent_id
Comment (a1~def...)
  ↑ parent_id
Reply (a1~ghi...)
  • Reply references Comment by content hash
  • Comment references Post by content hash
  • Cannot modify Post without breaking Comment reference
  • Cannot modify Comment without breaking Reply reference
  • Entire thread is cryptographically bound together

Proof of Authenticity

Cloudillo provides two complementary layers of proof:

Layer 1: Author Identity (Cryptographic Signatures)

Action tokens are signed with ES384 (ECDSA with P-384 curve and SHA-384 hash):

Action Token = Header.Payload.Signature

Signature = ECDSA_sign(SHA384(Header.Payload), alice_private_key)

Verification:

  1. Fetch alice’s public key from https://cl-o.alice.example.com/api/me/keys
  2. Verify signature using public key
  3. βœ“ Proves Alice created this action

Layer 2: Content Integrity (Content Hashes)

All identifiers are SHA-256 hashes:

action_id = SHA256(entire JWT token)
file_id = SHA256(descriptor string)
variant_id = SHA256(blob bytes)

Verification:

  1. Download content
  2. Recompute hash
  3. Compare with claimed identifier
  4. βœ“ Proves content hasn’t been tampered with

Combined Proof

Author + Content = Complete Authenticity

For a post with image attachment:

  1. βœ“ Signature proves Alice authored the post
  2. βœ“ Action hash proves post content is intact
  3. βœ“ File hash proves descriptor is intact
  4. βœ“ Variant hashes prove image bytes are intact
  5. βœ“ Complete chain of authenticity established

No trusted intermediaries neededβ€”pure cryptographic proof.

Verification Example

Scenario: Verify a LIKE action on a POST with image attachment.

verify_like_action(like_id):
    // 1. Verify LIKE action
    like_token = fetch_action_token(like_id)
    verify_signature(like_token, bob_public_key)
    verify like_id == SHA256(like_token)
    βœ“ Bob authored this LIKE, content is intact

    // 2. Verify parent POST action
    post_id = like_token.parent_id
    post_token = fetch_action_token(post_id)
    verify_signature(post_token, alice_public_key)
    verify post_id == SHA256(post_token)
    βœ“ Alice authored this POST, content is intact

    // 3. Verify file attachment
    file_id = post_token.attachments[0]
    descriptor = fetch_descriptor(file_id)
    verify file_id == SHA256(descriptor)
    βœ“ File descriptor is intact

    // 4. Verify each variant blob
    variants = parse_descriptor(descriptor)
    for each variant:
        blob_data = fetch_blob(variant.blob_id)
        verify variant.blob_id == SHA256(blob_data)
        verify blob_data.size == variant.size
    βœ“ All image variants are intact

    RESULT: Complete chain verified!

What this proves:

  • Bob signed the LIKE (authentication)
  • Alice signed the POST (authentication)
  • No content was tampered with at any level (integrity)
  • The LIKE references this specific POST (linkage)
  • The POST references this specific image file (linkage)
  • All image bytes are authentic (end-to-end verification)

DAG Structure

Cloudillo’s action system forms a Directed Acyclic Graph (DAG) with these properties:

Multiple Roots

Unlike a traditional tree with one root, Cloudillo has multiple independent threads:

Post 1 (a1~abc...)                Post 2 (a1~xyz...)
β”‚                                  β”‚
β”œβ”€ Comment 1.1 (a1~def...)        β”œβ”€ Comment 2.1 (a1~uvw...)
β”‚  β”‚                               β”‚
β”‚  β”œβ”€ Reply 1.1.1 (a1~ghi...)     └─ Like 2.1 (a1~rst...)
β”‚  β”‚                                  (parent: a1~uvw...)
β”‚  └─ Reply 1.1.2 (a1~jkl...)
β”‚
β”œβ”€ Comment 1.2 (a1~mno...)
β”‚
└─ Like 1 (a1~pqr...)
   (parent: a1~abc...)

Each top-level post is a root node. Comments and reactions form child nodes.

Shared Attachments

Multiple actions can reference the same file:

Post 1 (a1~abc...)
  └─ Attachment: f1~photo123

Post 2 (a1~xyz...)
  └─ Attachment: f1~photo123  ← SAME FILE!

Repost (a1~uvw...)
  └─ Attachment: f1~photo123  ← SAME FILE!

Benefits:

  • Storage efficiency: Only one copy of blob data needed
  • Bandwidth efficiency: Download once, use everywhere
  • Consistency: Everyone sees exactly the same image
  • Verification: Single verification proves authenticity for all uses

Acyclic Property

The graph has no cycles (no circular references):

βœ“ Valid:
  Post β†’ Comment β†’ Reply (linear chain)
  Post β†’ Comment1, Post β†’ Comment2 (branching)

βœ— Invalid:
  Post β†’ Comment β†’ Reply β†’ Post (cycle!)

Cycles are prevented because:

  • Parent references use content hashes
  • Cannot reference an action that doesn’t exist yet
  • Cannot create action hash without knowing full content
  • Mathematical impossibility to create circular references

Efficient Traversal

Forward traversal (find children):

-- Find all comments on a post
SELECT * FROM actions
WHERE parent_id = 'a1~abc123...'
AND type LIKE 'CMNT%';

Backward traversal (find parents):

find_root(action_id):
    action = fetch_action(action_id)
    if action.parent_id exists:
        return find_root(action.parent_id)  // Recursive
    else:
        return action  // Found root

Root ID optimization: To avoid repeated traversal, the root_id is computed once and stored in the database (see Actions: Root ID Handling).

Performance Implications

Caching Strategy

Content-addressed data is immutable, enabling aggressive caching:

GET /api/files/f1~Qo2E3G8TJZ...

Response Headers:
  Cache-Control: public, max-age=31536000, immutable
  Content-Type: image/avif

Benefits:

  • Browsers cache forever (max-age = 1 year)
  • CDN cache forever
  • No cache invalidation needed
  • Reduces bandwidth for federated instances

Storage Deduplication

Identical content is stored only once:

Alice uploads photo.jpg β†’ b1~abc123 (1MB stored)
Bob uploads photo.jpg β†’ b1~abc123 (0MB stored, reuse!)
Carol uploads photo.jpg β†’ b1~abc123 (0MB stored, reuse!)

Total storage: 1MB instead of 3MB

Automatic deduplication at multiple levels:

  • Variant level (same image blob)
  • File level (same set of variants)
  • Action level (impossible to duplicate due to signatures and timestamps)

Security Considerations

Collision Resistance

SHA-256 provides 256-bit security:

  • Probability of collision: ~2^-256 (effectively zero)
  • Preimage attack: computationally infeasible
  • Second preimage attack: computationally infeasible

In practice: Finding a collision would require more energy than exists in the observable universe.

Tamper Detection

Any modification anywhere in the tree is immediately detectable:

Attack scenario: Attacker tries to modify an image in alice’s post

1. Attacker modifies image blob
   β†’ New variant_id (hash mismatch!)
2. Attacker updates descriptor with new variant_id
   β†’ New file_id (hash mismatch!)
3. Attacker updates post attachment with new file_id
   β†’ Breaks JWT signature (alice didn't sign this!)
4. Attacker creates new JWT with new attachment
   β†’ New action_id (different post!)

Result: Attacker cannot tamper without detection. They can only create NEW actions, not modify existing ones.

Trust Model

Cloudillo’s merkle tree creates a trustless verification model:

What Verification Method Trust Required
Author identity JWT signature (ES384) DNS + Public key infrastructure
Content integrity SHA-256 hash None (pure mathematics)
Parent references Content hashes None (pure mathematics)
Attachment integrity SHA-256 hash chain None (pure mathematics)

Storage providers don’t need to be trusted: Even if a storage provider is malicious, they cannot:

  • Modify content without breaking hashes βœ—
  • Forge signatures without private keys βœ—
  • Create false parent references βœ—
  • Tamper with attachments without detection βœ—

Users verify everything cryptographicallyβ€”no trust required.

Attack Resistance

Known attacks and mitigations:

Attack Mitigation
Modify action content Hash mismatch detected
Forge author signature Signature verification fails
Swap file attachment Hash mismatch detected
Modify parent reference Breaks cryptographic chain
Replay old actions Timestamp validation, deduplication
Storage provider tampering Hash verification fails

See Also

Network & Security

Cloudillo’s network architecture emphasizes security, privacy, and ease of deployment. The system uses modern TLS with automatic certificate management via ACME (Let’s Encrypt) and supports both HTTP/1.1 and HTTP/2.

TLS/HTTPS Architecture

Rustls Integration

Cloudillo uses Rustls, a modern, memory-safe TLS library written in pure Rust:

Advantages:

  • βœ… Memory-safe: No buffer overflows or memory corruption
  • βœ… No unsafe code: Aligns with Cloudillo’s #![forbid(unsafe_code)]
  • βœ… Modern protocols: TLS 1.2 and TLS 1.3
  • βœ… High performance: Zero-copy design
  • βœ… Small footprint: Minimal dependencies

Version:

[dependencies]
rustls = "0.23"
tokio-rustls = "0.26"

HTTP/2 Support

Cloudillo supports HTTP/2 via ALPN (Application-Layer Protocol Negotiation):

TLS Configuration Steps:
1. Create ServerConfig builder with no client auth
2. Set certificate resolver
3. Configure ALPN protocols:
   - "h2" (HTTP/2, preferred)
   - "http/1.1" (fallback for compatibility)
4. Client selects protocol during TLS handshake

Benefits:

  • Multiplexing (multiple requests over single connection)
  • Server push (future feature)
  • Header compression
  • Binary protocol efficiency

SNI (Server Name Indication)

SNI enables hosting multiple domains on one IP address:

CertResolver Structure:

  • certs: RwLock - In-memory cache of domain β†’ CertifiedKey
  • auth_adapter: Reference to persistent certificate storage

Resolution Algorithm:

  1. Extract domain from ClientHello SNI field
  2. Look up domain in in-memory cache
  3. Return cached certificate if found
  4. (On-demand loading if not cached - see Dynamic Loading)

Flow:

Client β†’ TLS ClientHello with SNI="alice.example.com"
  ↓
CertResolver extracts "alice.example.com"
  ↓
Looks up certificate for alice.example.com
  ↓
Returns appropriate certificate
  ↓
TLS handshake completes

ACME (Let’s Encrypt) Integration

Automatic Certificate Management

Cloudillo uses instant-acme for automatic certificate provisioning:

Algorithm: Request ACME Certificate

1. Create ACME account:
   - Contact email: mailto:{email}
   - Agree to terms of service
   - Connect to Let's Encrypt production server

2. Create order for domain
   - Specify domain identifiers

3. Get authorization challenges
   - Find HTTP-01 challenge (not DNS-01)
   - Extract challenge token

4. Store challenge response in AuthAdapter
   - key: challenge.token
   - value: key_authorization(challenge)

5. Validate challenge
   - ACME server verifies HTTP endpoint

6. Poll for order completion (max 10 attempts, 5s intervals)
   - Retry until OrderStatus::Ready

7. Generate Certificate Signing Request (CSR)
   - Create certificate parameters for domain
   - Serialize to DER format

8. Finalize order with CSR

9. Download signed certificate chain

10. Store certificate in AuthAdapter
    - Persist for reuse and renewal

HTTP-01 Challenge

Cloudillo uses HTTP-01 challenge for domain validation:

Challenge Endpoint:

HTTP Route: GET /.well-known/acme-challenge/{token}

Handler Algorithm:
1. Extract token from URL path
2. Query AuthAdapter for stored challenge response
3. Return challenge value (plain text)

This endpoint must be publicly accessible over HTTP (port 80)
for ACME validation.

Challenge Flow:

ACME server β†’ GET http://example.com/.well-known/acme-challenge/{token}
  ↓
Cloudillo β†’ Lookup token in AuthAdapter
  ↓
Cloudillo β†’ Return challenge response
  ↓
ACME server validates
  ↓
Domain ownership confirmed
  ↓
Certificate issued

Certificate Renewal

Certificates auto-renew before expiration:

Algorithm: Auto-Renew Certificates (runs every 12 hours)

1. Loop continuously:
   a. Sleep 12 hours
   b. Get all domains from AuthAdapter
   c. For each domain:
      - Read certificate from storage
      - Parse expiration date
      - Calculate days until expiry
      - If < 30 days:
        * Request new certificate via ACME
        * On success:
          - Log certificate renewed
          - Update in-memory certificate cache
        * On error:
          - Log error
          - Continue to next domain

Certificate Storage

Certificates stored in AuthAdapter via trait interface:

AuthAdapter Methods:

  • create_cert(domain, cert_chain) - Store new certificate chain
  • read_cert(domain) - Retrieve certificate (returns Option<Vec>)
  • delete_cert(domain) - Remove certificate
  • list_domains() - Get all registered domains

SQLite Schema:

CREATE TABLE certificates (
    domain TEXT PRIMARY KEY,
    cert_chain BLOB NOT NULL,
    private_key BLOB NOT NULL,
    created_at INTEGER NOT NULL,
    expires_at INTEGER NOT NULL
);

CREATE INDEX idx_certs_expiry ON certificates(expires_at);

Certificate Resolver

In-Memory Cache

Certificates cached in memory for performance:

Data Structures:

  • App.certs: RwLock<HashMap<domain, CertifiedKey»
  • CertResolver.certs: RwLock<HashMap<domain, CertifiedKey»
  • CertResolver.auth_adapter: Persistent storage reference

load_certificate(domain) Algorithm:

  1. Query AuthAdapter for certificate bytes
  2. Parse certificate chain and private key
  3. Acquire write lock on certs
  4. Insert domain β†’ CertifiedKey into cache

reload_all() Algorithm:

  1. Get all domains from AuthAdapter
  2. For each domain, call load_certificate
  3. Repopulate entire in-memory cache

Dynamic Loading

Certificates loaded on-demand if not in cache:

Algorithm: ResolvesServerCert::resolve(client_hello)

1. Extract domain from ClientHello SNI
2. Try read lock on cache:
   - If domain found: Return cached certificate
   - If not found: Continue to step 3

3. Load from storage (async I/O in blocking task):
   a. Read certificate bytes from AuthAdapter
   b. Parse certificate and key
   c. Acquire write lock on cache
   d. Insert into cache
   e. Return certificate

This pattern ensures:
- Fast path (cached): Lock-free read
- Slow path (on-demand): Blocking async I/O off async runtime

Server Architecture

Dual-Server Setup

Cloudillo runs two servers:

HTTPS Server (primary, always running):

Algorithm: HTTPS Server

1. Configure TLS with Rustls:
   - No client authentication
   - Dynamic certificate resolver (SNI-based)
2. Bind TCP listener to configured address (e.g., 0.0.0.0:443)
3. Accept connections in loop:
   a. Accept TCP stream
   b. Upgrade to TLS connection
   c. Spawn async task to handle connection

HTTP Server (conditional, for ACME only):

Algorithm: HTTP Server (if ACME enabled)

1. Bind TCP listener to 0.0.0.0:80
2. Spawn separate async task to run in parallel
3. Accept connections in loop:
   a. Accept TCP stream
   b. Parse HTTP request
   c. If path is /.well-known/acme-challenge/{token}:
      - Handle ACME challenge request
   d. Else:
      - Send HTTP 301 redirect to HTTPS

Graceful Shutdown

Algorithm: Server Shutdown

1. Create oneshot channel for shutdown signal (tx, rx)
2. Spawn server task (runs in background)
3. Wait for either:
   a. Ctrl+C signal (SIGINT), or
   b. Shutdown signal sent to tx
4. On shutdown trigger:
   a. Log "Shutting down gracefully..."
   b. Abort server task
   c. Allow in-flight connections to complete
   d. Return success

This allows clean shutdown without dropping active requests.

Cryptography

Algorithms

TLS:

  • TLS 1.2 (minimum)
  • TLS 1.3 (preferred)
  • Cipher suites: Modern, secure defaults from Rustls

Action Signatures:

  • Algorithm: ES384 (ECDSA with P-384 and SHA-384)
  • Key size: 384 bits
  • Signature size: ~96 bytes

Content Hashing:

  • Algorithm: SHA256
  • Used for: File IDs, action IDs, content addressing
  • Output: 256 bits (32 bytes)

Password Hashing:

  • Algorithm: bcrypt
  • Work factor: 12 (default)
  • Salt: Random, unique per password

Key Management

Profile Signing Keys (ES384):

Algorithm: Generate and Store Profile Key

1. Generate random key pair using P-384 curve
   - Use OS random number generator
   - Both signing_key (private) and verifying_key (public)

2. Store in AuthAdapter:
   - tenant_id: Identifies user/organization
   - key_id: Identifies specific key version
   - public_key: Shared publicly, used for verification
   - private_key: Kept secret, used for signing

TLS Certificates (RSA or ECDSA):

  • Generated by Let’s Encrypt
  • Stored in AuthAdapter
  • Auto-renewed before expiration

Secure Random Generation

Algorithm: Generate Secure Session ID

1. Request 32 random bytes from OS entropy source (OsRng)
2. Encode bytes as URL-safe base64 string
3. Return base64-encoded session ID

This produces 256-bit random session identifiers suitable for
authentication tokens and nonces.

Security Best Practices

No Unsafe Code

Cloudillo enforces memory safety:

#![forbid(unsafe_code)]

All dependencies must also be free of unsafe code where possible.

Input Validation

URL Validation Algorithm:

  1. Parse URL string
  2. Verify scheme is “https” (reject “http”, “ftp”, etc.)
  3. Extract domain/host from URL
  4. Validate domain format (valid characters, not IP, proper TLD)
  5. Return error if any check fails

JWT Validation Algorithm:

  1. Decode token (checks base64 format)
  2. Verify signature using ES384 algorithm with public key
  3. Validate expiration time (exp claim < current time)
  4. Extract and return claims

Size Limits:

  • MAX_REQUEST_SIZE: 100 MB (entire HTTP request)
  • MAX_JSON_SIZE: 10 MB (JSON request body)
  • MAX_ACTION_SIZE: 1 MB (individual action token)

Rate Limiting

Per-IP Rate Limiter Algorithm:

Data: RateLimiter with HashMap<IpAddr β†’ VecDeque<Instant>>

check(ip) Algorithm:
1. Acquire write lock on limits map
2. Get or create rate limit entry for IP
3. Get current time
4. Remove all requests older than 1 minute
5. If requests >= max_per_minute: return false (rate limited)
6. Record new request timestamp
7. Return true (request allowed)

This sliding-window approach counts requests in the last 60 seconds.

Per-User Rate Limits (hardcoded):

  • MAX_ACTIONS_PER_HOUR: 100 actions (limit posting/interactions)
  • MAX_FILE_UPLOADS_PER_HOUR: 50 uploads (limit bandwidth)
  • MAX_DB_QUERIES_PER_MINUTE: 1000 queries (limit database load)

CORS Configuration

CORS Layer Configuration:

1. Allow Origin: Mirror request (echo Origin header)
   - Allows any origin to make requests
   - Each response includes Access-Control-Allow-Origin: <requesting origin>

2. Allowed Methods: GET, POST, PATCH, DELETE
   - Other methods (PUT, HEAD) are rejected

3. Allowed Headers: Authorization, Content-Type
   - Custom headers must be explicitly allowed

4. Credentials: Enabled
   - Allows cookies and credentials in cross-origin requests

5. Max Age: 3600 seconds (1 hour)
   - Browsers cache CORS preflight responses for 1 hour

6. Apply to API routes
   - All routes inherit CORS configuration

Monitoring & Logging

Structured Logging

Tracing Library Format:

  • Each log entry includes message + contextual fields
  • Fields use structured format (field = value)
  • Levels: debug, info, warn, error

Request Logging Example:

  • method: HTTP verb
  • path: Request URI
  • remote_addr: Client IP
  • Message: “Handling request”

Error Logging Example:

  • error: Exception message
  • context: Category (e.g., “certificate_renewal”)
  • domain: Relevant domain

Security Events

Key security-related events logged with structured context:

Rate Limit Hit:

  • ip: Client IP address
  • reason: “rate_limit_exceeded”

Token Rejection:

  • token_issuer: Who signed the token
  • reason: “signature_verification_failed”

Authorization Failure:

  • ip: Client IP
  • path: Request URL
  • reason: “permission_denied”

Metrics

SecurityMetrics Structure:

  • failed_auth_attempts (AtomicU64) - Failed login attempts
  • rate_limit_hits (AtomicU64) - Rate limit violations
  • cert_renewals (AtomicU64) - Successful certificate renewals
  • signature_failures (AtomicU64) - Token signature verification failures
  • blocked_ips (AtomicUsize) - Currently rate-limited IP addresses

Deployment Considerations

Firewall Configuration

Required Ports:

  • 443 (HTTPS) - Primary server
  • 80 (HTTP) - ACME challenges only (optional redirect to HTTPS)

Firewall Rules:

# Allow HTTPS
sudo ufw allow 443/tcp

# Allow HTTP for ACME
sudo ufw allow 80/tcp

# Deny all other inbound
sudo ufw default deny incoming
sudo ufw default allow outgoing

Reverse Proxy

If using nginx or similar:

server {
    listen 443 ssl http2;
    server_name *.example.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass https://localhost:8443;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # WebSocket support
    location /ws/ {
        proxy_pass https://localhost:8443;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Docker Deployment

FROM rust:1.75 AS builder
WORKDIR /app
COPY . .
RUN cargo build --release

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates
COPY --from=builder /app/target/release/basic-server /usr/local/bin/

EXPOSE 8443 80

CMD ["basic-server"]

Docker Compose:

version: '3.8'

services:
  cloudillo:
    build: .
    ports:
      - "443:8443"
      - "80:80"
    environment:
      MODE: standalone
      BASE_ID_TAG: example.com
      ACME_EMAIL: admin@example.com
      DB_DIR: /data/db
      DATA_DIR: /data/blobs
    volumes:
      - ./data:/data
    restart: unless-stopped

Security Checklist

Before Deployment

  • TLS certificates configured (ACME or manual)
  • Firewall rules in place
  • Rate limiting enabled
  • CORS properly configured
  • Logging enabled
  • Backups configured
  • Password policies enforced
  • Security headers set

Ongoing Maintenance

  • Monitor certificate expiration
  • Review security logs weekly
  • Update dependencies monthly
  • Backup data daily
  • Test disaster recovery quarterly
  • Audit access controls quarterly
  • Review rate limits as needed

See Also

Data Storage & Access

Cloudillo’s data storage systems and access control mechanisms. These documents explain how data is stored, organized, queried, and protected across the decentralized network.

Storage Types

Cloudillo provides three storage systems for different use cases:

Feature Blob RTDB CRDT
Primary Use Static files Structured data with queries Collaborative editing
Data Model Immutable binary Collections of JSON documents Shared types (text, maps, arrays)
Mutability Immutable (new version = new blob) Mutable with real-time sync Mutable with automatic merge
Conflict Resolution N/A (content-addressed) Last-write-wins Automatic merge (no conflicts)
Offline Support Cache only Reconnection sync Full offline with local persistence
Query Capabilities By ID, metadata, tags Rich queries (where, orderBy, limit) Read entire document
Best For Images, videos, PDFs, attachments Todos, settings, lists, forms Text editors, whiteboards, real-time docs

Blob Storage

Blob Storage - Content-addressed immutable binary data. Every file is identified by its SHA-256 hash, enabling deduplication and integrity verification. Supports automatic variant generation (thumbnails, transcoded video). Choose Blob for static files that don’t change frequently.

RTDB (Real-Time Database)

RTDB - Firebase-like API for structured data. Choose RTDB when you need to query and filter data, or when your application works with structured records (users, posts, settings). Changes sync in real-time, but concurrent edits use last-write-wins semantics.

CRDT (Collaborative Editing)

CRDT - Conflict-free replicated data types using Yjs. Choose CRDT when multiple users edit the same content simultaneously (documents, spreadsheets, presentations). All changes merge automatically without conflicts, even when users are offline.

Choosing the Right System

Use Blob when:

  • Storing static files (images, videos, PDFs)
  • Content is immutable or versioned
  • You need content-addressing and deduplication
  • Generating variants (thumbnails, transcodes)

Use RTDB when:

  • You need to query/filter data (e.g., “show incomplete todos”)
  • Data is structured as records/documents
  • Users typically edit different records
  • You need server-side validation

Use CRDT when:

  • Multiple users edit the same content simultaneously
  • You’re building a collaborative editor
  • Offline-first is critical
  • Character-level or element-level merging is needed

Many applications use all three: Blob for attachments, CRDT for document content, RTDB for metadata and settings.

Access Control

Access Control - How resources are protected and shared while maintaining user privacy and sovereignty through token-based authentication and attribute-based permissions.

Subsections of Data Storage & Access

Blob Storage

Cloudillo’s blob storage uses content-addressed storage for immutable binary data (files, images, videos). Intelligent variant generation for images ensures data integrity, deduplication, and efficient delivery across different use cases.

Why Content-Addressed Storage?

Traditional file storage asks “where is this file?” Content-addressed storage asks “what is this file?” This simple shift enables powerful features:

Real-world analogy: Imagine a library where books are organized by their content fingerprint rather than shelf location. Two copies of the same book have the same fingerprintβ€”you only need to store one. If someone claims to have “the original,” you can verify it instantly by checking the fingerprint.

Benefits you’ll notice:

  • Upload once, access anywhere: The same image uploaded by different users is stored only once
  • Verification without trust: Anyone can confirm a file hasn’t been modified
  • Efficient caching: Files can be cached foreverβ€”they never change
  • Automatic deduplication: Storage costs decrease as the network grows

Benefits for developers:

  • Simple URLs: File ID = file content = permanent reference
  • No cache invalidation: Content-addressed files are immutable
  • Built-in integrity: Hash verification catches corruption instantly

Content-Addressed Storage

Concept

Files are identified by the SHA256 hash of their content, making identifiers:

  • Immutable: Content cannot change without changing the ID
  • Verifiable: Recipients can verify integrity
  • Deduplicat able: Identical content gets same ID
  • Tamper-proof: Any modification is immediately detectable

File Identifier Format

Cloudillo uses multiple identifier types in its content-addressing system:

{prefix}{version}~{base64url_hash}

Components:

  • {prefix}: Resource type indicator (a, f, b, d)
  • {version}: Hash algorithm version (currently 1 = SHA-256)
  • ~: Separator
  • {base64url_hash}: Base64url-encoded hash (43 characters, no padding)

Identifier Types

Prefix Resource Type Hash Input Example
b1~ Blob Blob bytes (raw image/video data) b1~abc123def456...
f1~ File File descriptor string f1~QoEYeG8TJZ2HTGh...
d2, Descriptor (not a hash, the encoded format itself) d2,vis.tn:b1~abc:f=avif:...
a1~ Action Complete JWT token a1~8kR3mN9pQ2vL...

Important: d2, is not a content-addressed identifierβ€”it’s the actual encoded descriptor string. The file ID (f1~) is the hash of this descriptor.

Examples

Blob ID:       b1~QoEYeG8TJZ2HTGhVlrtTDBpvBGOp6gfGhq4QmD6Z46w
File ID:       f1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM
Descriptor:    d2,vis.tn:b1~xRAVuQtgBx_kLqZnoOSd5XqCK_aQolhq1XeXk73Zn8U:f=avif:s=1960:r=90x128
Action ID:     a1~8kR3mN9pQ2vL6xWpYzT4BjN5FqGxCmK9RsH2VwLnD8P

All file and blob IDs use SHA-256 content-addressing. See Content-Addressing & Merkle Trees for hash computation details.

File Types

Cloudillo supports four file types, each handled by different adapters based on mutability and use case:

Type Adapter Mutability Description
BLOB BlobAdapter Immutable Binary content (images, videos, documents)
CRDT CrdtAdapter Mutable Collaborative documents (Yjs-based real-time editing)
RTDB RtdbAdapter Mutable Real-time database files for app state
FLDR MetaAdapter Mutable Folder/directory metadata

Why Different File Types?

Different collaboration scenarios require different storage strategies:

  • BLOB: When you upload a photo or video, it never changesβ€”if you want a different version, you upload a new file. This immutability enables powerful caching and deduplication across the network.
  • CRDT: When editing a document with others in real-time (like Google Docs), changes from all participants must merge seamlessly. CRDTs (Conflict-free Replicated Data Types) make this possible.
  • RTDB: Apps need to store changing state (todo lists, game scores, form data). RTDB provides real-time synchronization with WebSocket subscriptions.
  • FLDR: Organizing files into folders requires mutable metadata without changing the files themselves.

File Type Selection

Files are created via different API endpoints based on their type:

Endpoint File Type Use Case
/api/files/image/* BLOB Image uploads with auto-variant generation
/api/files/raw/* BLOB Raw file uploads (no processing)
/api/files/crdt/* CRDT Collaborative document creation
/api/files/rtdb/* RTDB Real-time database file creation

Per-File Access Control

Each file has independent access control:

Field Values Description
Visibility P/V/F/C/null Who can discover this file
Access Level R/W Read-only vs read-write access

See Access Control for detailed permission handling.

File Variants

Concept

A single uploaded image automatically generates multiple variants optimized for different use cases:

  • tn (thumbnail): Tiny preview (~128x96px)
  • sd (standard definition): Social media size (~640x480px)
  • md (medium definition): Web display (~1280x720px)
  • hd (high definition): Full screen (~1920x1080px)
  • xd (extra definition): Original/4K+ (~3840x2160px+)

File Descriptor Encoding

A file descriptor encodes all available variants in a compact format.

File Descriptor Format Specification

Format

d2,{class}.{variant}:{blob_id}:f={format}:s={size}:r={width}x{height}[:{optional}];...

Components

  • d2, - Descriptor prefix (version 2)
  • {class} - Media class:
    • vis - Visual (images: jpeg, png, webp, avif)
    • vid - Video (mp4/h264)
    • aud - Audio (opus, mp3)
    • doc - Documents (pdf)
    • raw - Original unprocessed file
  • {variant} - Quality tier: pf, tn, sd, md, hd, xd, or orig
  • {blob_id} - Content-addressed ID of the blob (b1~...)
  • f={format} - Format: avif, webp, jpeg, png, mp4, opus, pdf
  • s={size} - File size in bytes (integer, no separators)
  • r={width}x{height} - Resolution in pixels (width Γ— height)
  • ; - Semicolon separator between variants (no spaces)

Optional Fields

For video, audio, and document files:

  • dur={seconds} - Duration in seconds (floating point, video/audio only)
  • br={kbps} - Bitrate in kbps (integer, video/audio only)
  • pg={count} - Page count (integer, documents only)

Example

d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x150;vis.sd:b1~def456:f=avif:s=32768:r=640x480

This descriptor encodes two variants:

  • Thumbnail: AVIF format, 4096 bytes, 150Γ—150 pixels, blob ID b1~abc123
  • Standard: AVIF format, 32768 bytes, 640Γ—480 pixels, blob ID b1~def456

Video Example

d2,vis.tn:b1~abc:f=avif:s=4096:r=150x84;vid.sd:b1~def:f=mp4:s=5242880:r=720x404:dur=120.5:br=350;vid.hd:b1~ghi:f=mp4:s=20971520:r=1920x1080:dur=120.5:br=1400

This descriptor includes:

  • Thumbnail: AVIF image preview
  • SD Video: 720p MP4, 120.5 seconds, 350 kbps
  • HD Video: 1080p MP4, 120.5 seconds, 1400 kbps

Parsing Rules

  1. Check prefix: Verify descriptor starts with d2,
  2. Split by semicolon (;): Get individual variant entries
  3. For each variant, split by colon (:) to get components:
    • Component [0] = class.variant (vis.tn, vis.sd, vid.hd)
    • Component [1] = blob_id (b1~...)
    • Components [2..] = key=value pairs
  4. Parse key=value pairs:
    • f={format} β†’ Format string
    • s={size} β†’ Parse as u64 (bytes)
    • r={width}x{height} β†’ Split by x, parse as u32 Γ— u32
    • dur={seconds} β†’ Parse as f64 (optional)
    • br={kbps} β†’ Parse as u32 (optional)
    • pg={count} β†’ Parse as u32 (optional)

Parsing logic: split by semicolons for variants, then by colons for fields, then parse key=value pairs.

Variant Size Classes - Exact Specifications

Cloudillo generates image variants at specific size targets to optimize bandwidth and storage:

Class Name Target Resolution Max Dimension Use Case
tn Thumbnail ~150Γ—150px 200px List views, previews, avatars
sd Standard Definition ~640Γ—480px 800px Mobile devices, low bandwidth
md Medium Definition ~1920Γ—1080px 2000px Desktop viewing, full screen
hd High Definition ~3840Γ—2160px 4000px 4K displays, high quality
xd Extra Definition Original size No limit Archival, original quality

Generation Rules

Generated variants based on maximum dimension (largest of width or height):

  • max_dim β‰₯ 3840px: tn, sd, md, hd, xd (all variants)
  • max_dim β‰₯ 1920px: tn, sd, md, hd
  • max_dim β‰₯ 1280px: tn, sd, md
  • max_dim < 1280px: tn, sd

Properties:

  • Each variant maintains the original aspect ratio
  • Uses Lanczos3 filter for high-quality downscaling
  • Maximum dimension constraint prevents oversizing
  • Smaller originals don’t get upscaled

Variant Selection

Clients request a specific variant:

GET /api/files/f1~Qo2E3G8TJZ...?variant=hd

Response: Returns HD variant if available, otherwise falls back to smaller variants.

Automatic Fallback

If the requested variant doesn’t exist, the server returns the best available:

  1. Try requested variant (e.g., hd)
  2. Fall back to next smaller (e.g., md)
  3. Continue until variant found
  4. Return smallest if none larger

Fallback order: xd β†’ hd β†’ md β†’ sd β†’ tn

Content-Addressing Flow

File storage uses a three-level content-addressing hierarchy:

Level 1: Blob Storage

Upload image β†’ Save as blob β†’ Compute SHA256 of blob bytes β†’ Store blob with ID: b1~{hash}

blob_data = read_file("thumbnail.avif")
blob_id = compute_hash("b", blob_data)
// Result: "b1~abc123..." (thumbnail blob ID)

Example: b1~abc123... identifies the thumbnail AVIF blob

See Content-Addressing & Merkle Trees for hash computation details.

Level 2: Variant Collection

Generate all variants (tn, sd, md, hd) β†’ Each variant gets its own blob ID (b1~...) β†’ Collect all variant metadata β†’ Create descriptor string encoding all variants

variants = [
    { class: "vis.tn", blob_id: "b1~abc123", format: "avif", size: 4096, width: 150, height: 150 },
    { class: "vis.sd", blob_id: "b1~def456", format: "avif", size: 32768, width: 640, height: 480 },
    { class: "vis.md", blob_id: "b1~ghi789", format: "avif", size: 262144, width: 1920, height: 1080 },
]

descriptor = build_descriptor(variants)
// Result: "d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x150;vis.sd:b1~def456:f=avif:s=32768:r=640x480;vis.md:b1~ghi789:f=avif:s=262144:r=1920x1080"

Level 3: File Descriptor

Build descriptor β†’ Compute SHA256 of descriptor string β†’ Final file ID: f1~{hash} β†’ This file ID goes into action attachments

descriptor = "d2,vis.tn:b1~abc:f=avif:s=4096:r=150x150;vis.sd:b1~def:f=avif:s=32768:r=640x480"
file_id = compute_hash("f", descriptor.as_bytes())
// Result: "f1~Qo2E3G8TJZ..." (file ID)

Example Complete Flow

1. User uploads photo.jpg (3MB, 3024x4032px)

2. System generates variants:
   vis.tn:  150x200px β†’ 4KB   β†’ b1~abc123
   vis.sd:  600x800px β†’ 32KB  β†’ b1~def456
   vis.md:  1440x1920px β†’ 256KB β†’ b1~ghi789
   vis.hd:  2880x3840px β†’ 1MB β†’ b1~jkl012

3. System builds descriptor:
   "d2,vis.tn:b1~abc123:f=avif:s=4096:r=150x200;
       vis.sd:b1~def456:f=avif:s=32768:r=600x800;
       vis.md:b1~ghi789:f=avif:s=262144:r=1440x1920;
       vis.hd:b1~jkl012:f=avif:s=1048576:r=2880x3840"

4. System hashes descriptor:
   file_id = f1~Qo2E3G8TJZ2... = SHA256(descriptor)

5. Action references file:
   POST action attachments = ["f1~Qo2E3G8TJZ2..."]

6. Anyone can verify:
   - Download all variants
   - Verify each blob_id = SHA256(blob)
   - Rebuild descriptor
   - Verify file_id = SHA256(descriptor)
   - Cryptographic proof established βœ“

Integration with Action Merkle Tree

File attachments create an extended merkle tree:

Action (a1~8kR...)
  β”œβ”€ Signed by user (ES384)
  β”œβ”€ Content-addressed (SHA256 of JWT)
  └─ Attachments: [f1~Qo2...]
       └─ File (f1~Qo2...)
            β”œβ”€ Content-addressed (SHA256 of descriptor)
            └─ Descriptor: "d2,vis.tn:b1~abc...;vis.sd:b1~def..."
                 β”œβ”€ Blob vis.tn (b1~abc...)
                 β”‚   └─ Content-addressed (SHA256 of blob)
                 β”œβ”€ Blob vis.sd (b1~def...)
                 β”‚   └─ Content-addressed (SHA256 of blob)
                 └─ Blob vis.md (b1~ghi...)
                     └─ Content-addressed (SHA256 of blob)

Benefits:

  • Entire tree is cryptographically verifiable
  • Cannot modify image without changing all parent hashes
  • Deduplication: same image = same file_id
  • Federation: remote instances can verify integrity

See Content-Addressing & Merkle Trees for how file content-addressing integrates with the action system.

Image Processing Pipeline

Upload Flow

When a client uploads an image:

  1. Client Request
POST /api/files/image/profile-picture.jpg
Authorization: Bearer <access_token>
Content-Type: image/jpeg
Content-Length: 2458624

<binary image data>
  1. Dimension Extraction

Extract image dimensions to determine which variants to generate:

img = load_image_from_memory(data)
(width, height) = img.dimensions()
max_dim = max(width, height)

if max_dim >= 3840:
    variants = ["tn", "sd", "md", "hd", "xd"]
else if max_dim >= 1920:
    variants = ["tn", "sd", "md", "hd"]
else if max_dim >= 1280:
    variants = ["tn", "sd", "md"]
else:
    variants = ["tn", "sd"]
  1. FileIdGeneratorTask

Create a task to generate the content-addressed ID:

task = FileIdGeneratorTask(
    tn_id,
    temp_file_path="/tmp/upload-abc123",
    original_filename="profile-picture.jpg"
)

task_id = scheduler.schedule(task)
  1. ImageResizerTask (Multiple)

For each variant, create a resize task:

for variant in variants:
    task = ImageResizerTask(
        tn_id,
        source_file_id=original_id,
        variant=variant,
        target_dimensions=get_variant_dimensions(variant),
        format="avif",  # Primary format
        quality=85,
        dependencies=[file_id_task_id]  # Wait for ID generation
    )

    scheduler.schedule(task)
  1. Hash Computation

FileIdGeneratorTask computes SHA256 hash:

file_id = compute_content_hash("f", file_contents)
# See merkle-tree.md for hash computation details
  1. Blob Storage

Store original in BlobAdapter:

blob_adapter.create_blob_stream(tn_id, file_id, file_stream)
  1. Variant Generation

Each ImageResizerTask runs in worker pool (CPU-intensive):

# Execute in worker pool
img = load_image(source_path)

# Resize with Lanczos3 filter (high quality)
resized = img.resize(target_width, target_height, filter=Lanczos3)

# Encode to AVIF
buffer = encode_avif(resized, quality)

# Store variant
variant_id = compute_file_id(buffer)
blob_adapter.create_blob(tn_id, variant_id, buffer)
  1. Metadata Storage

Store file metadata with all variants:

file_metadata = FileMetadata(
    tn_id,
    file_id=descriptor_id,
    original_filename="profile-picture.jpg",
    mime_type="image/jpeg",
    size=original_size,
    variants=[
        Variant(name="tn", blob_id="b1~QoE...46w", format="avif",
                size=4096, width=200, height=200),
        Variant(name="sd", blob_id="b1~xyz...789", format="webp",
                size=32768, width=640, height=480),
        # ... more variants
    ],
    created_at=current_timestamp()
)

meta_adapter.create_file_metadata(tn_id, file_metadata)
  1. Response

Return descriptor ID to client:

{
  "file_id": "f1~QoE...46w",
  "descriptor": "d2,vis.tn:b1~QoE...46w:f=avif:s=4096:r=128x96;vis.sd:b1~xyz...789:f=avif:s=8192:r=640x480",
  "variants": [
    {"name": "vis.tn", "format": "avif", "size": 4096, "dimensions": "128x96"},
    {"name": "vis.sd", "format": "avif", "size": 8192, "dimensions": "640x480"}
  ],
  "processing": true
}

Complete Upload Flow Diagram

Client uploads image
  ↓
POST /api/files/image/filename.jpg
  ↓
Save to temp file
  ↓
Extract dimensions
  ↓
Determine variants to generate
  ↓
Create FileIdGeneratorTask
  β”œβ”€ Compute SHA256 hash
  β”œβ”€ Move to permanent storage (BlobAdapter)
  └─ Generate file_id
  ↓
Create ImageResizerTask (for each variant)
  β”œβ”€ Depends on FileIdGeneratorTask
  β”œβ”€ Load source image
  β”œβ”€ Resize with Lanczos3
  β”œβ”€ Encode to AVIF/WebP/JPEG
  β”œβ”€ Compute variant ID (SHA256)
  └─ Store in BlobAdapter
  ↓
Create file descriptor
  β”œβ”€ Collect all variant IDs
  β”œβ”€ Encode as descriptor
  └─ Store metadata in MetaAdapter
  ↓
Return descriptor ID to client

Download Flow

Client Request

GET /api/files/f1~...?variant=hd
Authorization: Bearer <access_token>

Server Processing

  1. Parse Descriptor
variants = parse_file_descriptor(file_id)
# Returns list of VariantInfo
  1. Select Best Variant
selected = select_best_variant(
    variants,
    requested_variant,   # "hd"
)

# Falls back if exact match not available:
# hd/avif β†’ hd/webp β†’ md/avif β†’ md/webp β†’ sd/avif β†’ ...
  1. Stream from BlobAdapter
stream = blob_adapter.read_blob_stream(tn_id, selected.file_id)

# Set response headers
response.headers["Content-Type"] = f"image/{selected.format}"
response.headers["X-Cloudillo-Variant"] = selected.blob_id
response.headers["X-Cloudillo-Descriptor"] = descriptor
response.headers["Content-Length"] = selected.size

# Stream response
return stream_response(stream)

Response

HTTP/1.1 200 OK
Content-Type: image/avif
Content-Length: 16384
X-Cloudillo-Variant: b1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM
X-Cloudillo-Descriptor: d2,vis.tn:b1~xRAVuQtgBx_kLqZnoOSd5XqCK_aQolhq1XeXk73Zn8U:f=avif:s=1960:r=90x128;vis.sd:b1~m8Z35EIa3prvb3bhjsVjdg9SG98xd0bkoWomOHQAwCM:f=avif:s=8137:r=256x364;vis.orig:b1~5gU72rRGiaogZuYhJy853pBd6PsqjPOjS__Kim9-qE0:f=avif:s=15012:r=256x364
Cache-Control: public, max-age=31536000, immutable

<binary image data>

Note: Content-addressed files are immutable, so can be cached forever.

Metadata Structure

FileMetadata

Stored in MetaAdapter:

FileMetadata {
    tn_id: TnId
    file_id: String           # Descriptor ID
    original_filename: String
    mime_type: String
    size: u64                 # Original size
    width: Optional[u32]
    height: Optional[u32]
    variants: List[VariantInfo]
    created_at: i64
    owner: String             # Identity tag
    permissions: FilePermissions
}

VariantInfo {
    name: String              # "tn", "sd", "md", "hd", "xd"
    file_id: String           # Content-addressed ID
    format: String            # "avif", "webp", "jpeg", "png"
    size: u64                 # Bytes
    width: u32
    height: u32
}

FilePermissions {
    public_read: bool
    shared_with: List[String]  # Identity tags
}

File Presets

Concept

Presets define how files should be processed:

FilePreset:
    Image      # Auto-generate variants
    Video      # Future: transcode, thumbnails
    Document   # Future: preview generation
    Database   # RTDB database files
    Raw        # No processing, store as-is

Upload with Preset

POST /api/files/{preset}/{filename}

Examples:
POST /api/files/image/avatar.jpg      // Generate image variants
POST /api/files/raw/document.pdf      // Store as-is

Storage Organization

BlobAdapter Layout

{data_dir}/
β”œβ”€β”€ blobs/
β”‚   β”œβ”€β”€ {tn_id}/
β”‚   β”‚   β”œβ”€β”€ f1~QoE...46w           // Original file
β”‚   β”‚   β”œβ”€β”€ f1~xyz...789           // Variant 1
β”‚   β”‚   β”œβ”€β”€ f1~abc...123           // Variant 2
β”‚   β”‚   └── ...
β”‚   └── {other_tn_id}/
β”‚       └── ...

MetaAdapter (SQLite)

CREATE TABLE files (
    id INTEGER PRIMARY KEY,
    tn_id INTEGER NOT NULL,
    file_id TEXT NOT NULL,
    original_filename TEXT,
    mime_type TEXT,
    size INTEGER,
    width INTEGER,
    height INTEGER,
    variants TEXT,  -- JSON array
    created_at INTEGER,
    owner TEXT,
    permissions TEXT,  -- JSON object
    UNIQUE(tn_id, file_id)
);

CREATE INDEX idx_files_owner ON files(owner);
CREATE INDEX idx_files_created ON files(created_at);

Performance Considerations

Worker Pool Usage

Image processing is CPU-intensive, so uses worker pool:

# Priority levels
Priority.High   β†’ User-facing operations (thumbnail)
Priority.Medium β†’ Background tasks (other image variants)
Priority.Low    β†’ Longer operations (video upload)

Parallel Processing

Multiple variants can be generated in parallel:

# Create all resize tasks at once
task_ids = []

for variant in ["tn", "sd", "md", "hd"]:
    task_id = scheduler.schedule(ImageResizerTask(
        variant=variant,
        # ...
    ))

    task_ids.append(task_id)

# Wait for all to complete
scheduler.wait_all(task_ids)

Caching Strategy

Content-addressed files are immutable:

Cache-Control: public, max-age=31536000, immutable
  • Browsers cache forever
  • CDN can cache forever
  • No cache invalidation needed

See Also

RTDB (Real-Time Database)

Query-oriented database system providing Firebase-like functionality for structured JSON data with real-time subscriptions.

Overview

The RTDB system enables:

  • JSON document storage and retrieval
  • Query filters (equals, greater than, less than)
  • Sorting and pagination
  • Computed values (increment, aggregate, functions)
  • Atomic transactions
  • Real-time subscriptions via WebSocket

Documents

  • Overview - Introduction to RTDB architecture and features
  • redb Implementation - How RTDB is implemented using the lightweight redb embedded database

Use Cases

  • User profiles and settings
  • Task lists and project management
  • E-commerce catalogs
  • Analytics and reporting
  • Structured forms and surveys

Subsections of RTDB (Real-Time Database)

RTDB Overview

Cloudillo’s RTDB (Real-Time Database) provides Firebase-like functionality for structured JSON data with queries, subscriptions, and real-time synchronization. It integrates seamlessly with Cloudillo’s federated architecture while maintaining privacy and user control.

Overview

The RTDB system provides:

  • Real-time synchronization: Changes propagate to all connected clients instantly
  • Offline support: Works offline, syncs when connection returns
  • Collaborative editing: Multiple users can edit the same data concurrently
  • Query capabilities: Filter, sort, and paginate data
  • WebSocket-based: Efficient, bidirectional communication
  • Privacy-focused: Data stored on user’s chosen node

Real-Time Database (RTDB)

The RTDB system provides Firebase-like functionality for structured data:

Technology: redb - Lightweight embedded database (171 KiB package size)

Features:

  • JSON document storage
  • Query filters (equals, greater than, less than, in, not-in, array-contains, array-contains-any, array-contains-all)
  • Sorting and pagination
  • Computed values (increment, decrement, multiply, concat, min, max, aggregate, functions)
  • Atomic transactions with temporary references
  • Real-time subscriptions via WebSocket
  • Document locking (soft/advisory and hard/enforced)
  • Aggregate queries with groupBy (sum, avg, min, max)

Use Cases:

  • User profiles and settings
  • Task lists and project management
  • E-commerce catalogs
  • Analytics and reporting
  • Structured forms and surveys

Learn more: RTDB with redb

CRDT Collaborative Editing (Separate System)

Cloudillo also provides a separate CRDT API for collaborative editing:

Technology: Yrs - Rust implementation of Yjs CRDT

Features:

  • Conflict-free replicated data types (CRDTs)
  • Rich data structures (Text, Map, Array, XML)
  • Automatic conflict resolution
  • Time-travel and versioning
  • Awareness (presence, cursors)
  • Yjs ecosystem compatibility

Use Cases:

  • Collaborative text editors (Google Docs-like)
  • Shared whiteboards
  • Real-time collaborative forms
  • Collaborative spreadsheets
  • Multiplayer game state

Learn more: CRDT Collaborative Editing

Comparison: RTDB vs CRDT

Feature RTDB (redb) CRDT (Yrs)
Purpose Structured data storage Concurrent editing
Queries Rich (filter, sort, paginate, aggregate) Limited (document-based)
Conflict Resolution Last-write-wins + document locking Automatic merge (CRDT)
Locking Soft (advisory) and hard (enforced) Not applicable
Aggregations Server-side (sum, avg, min, max, groupBy) Not applicable
Best For Traditional database needs Collaborative editing
API Style Firebase-like Yjs-compatible

Note: These are separate, complementary systems. Use RTDB for structured data with queries, and CRDT for collaborative editing scenarios.

Core Concept: Database-as-File

Both systems use the same foundational concept: databases/documents are special files in the Cloudillo file system.

How It Works

  1. File Metadata (MetaAdapter) stores:

    • Database ID, name, owner
    • Creation timestamp, last accessed
    • Permission rules
    • Configuration (max size, retention policy)
  2. Database Content (RtdbAdapter or CrdtAdapter) stores:

    • Actual data (documents, CRDT state)
    • Indexes (for query performance)
    • Snapshots (for fast loading)
  3. File ID serves as database identifier:

    /ws/rtdb/:fileId  // WebSocket connection endpoint

Benefits

βœ… Natural Integration: Databases managed like files βœ… Permission Reuse: File permissions apply to databases βœ… Federation Ready: Databases can be shared across instances βœ… Content Addressing: Database snapshots are tamper-proof βœ… Discoverable: Find databases through file APIs

Example

// Create database file
const response = await fetch('/api/db', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${token}` },
  body: JSON.stringify({
    name: 'My Tasks',
    type: 'redb',  // or 'yrs' for CRDT
    permissions: {
      public_read: false,
      readers: ['bob.example.com'],
      writers: ['bob.example.com']
    }
  })
});

const { fileId } = await response.json();
// fileId: "f1~abc123..."

// Connect to database via WebSocket
const ws = new WebSocket(`wss://cl-o.alice.example.com/ws/rtdb/${fileId}`);

Architecture Overview

Components

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Client Application                                  β”‚
β”‚  - JavaScript/TypeScript                            β”‚
β”‚  - React hooks / Vue composables                    β”‚
β”‚  - WebSocket connection                             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                          ↓ WebSocket
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Cloudillo Server                                    β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚ WebSocket Handler                           β”‚    β”‚
β”‚  β”‚  - Authentication                           β”‚    β”‚
β”‚  β”‚  - Message routing                          β”‚    β”‚
β”‚  β”‚  - Subscription management                  β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚                        ↓                            β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚ Database Manager                            β”‚    β”‚
β”‚  β”‚  - Instance lifecycle (load/evict)          β”‚    β”‚
β”‚  β”‚  - Snapshot management                      β”‚    β”‚
β”‚  β”‚  - Memory limits                            β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β”‚         ↓                           ↓               β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”         β”‚
β”‚  β”‚ RtdbAdapter β”‚           β”‚ CrdtAdapter  β”‚         β”‚
β”‚  β”‚  (redb)     β”‚           β”‚  (Yrs)       β”‚         β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜           β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜         β”‚
β”‚         ↓                           ↓               β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚
β”‚  β”‚ Storage Layer                                β”‚   β”‚
β”‚  β”‚  - MetaAdapter (metadata)                    β”‚   β”‚
β”‚  β”‚  - BlobAdapter (snapshots, data)             β”‚   β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Request Flow

Query-Based (redb):

Client β†’ WebSocket message (query/subscribe)
  ↓
WebSocket Handler β†’ Authenticate
  ↓
Database Manager β†’ Get or load instance
  ↓
RtdbAdapter β†’ Execute query
  ↓
Return results + subscribe to changes
  ↓
Client receives data + real-time updates

CRDT-Based (Yrs):

Client β†’ WebSocket connection
  ↓
WebSocket Handler β†’ Authenticate
  ↓
Database Manager β†’ Get or load instance
  ↓
Yrs Sync Protocol β†’ Exchange state vectors
  ↓
Bidirectional updates β†’ Merge with CRDT algorithm
  ↓
Both clients stay in sync

Permission Model

File-Level Permissions

Inherited from file metadata:

pub struct DatabasePermissions {
    pub owner: String,              // Identity tag
    pub readers: Vec<String>,       // Can query data
    pub writers: Vec<String>,       // Can modify data
    pub public_read: bool,
    pub public_write: bool,
}

Runtime Permission Checking

Permissions checked on every operation:

fn check_permission(auth: &Auth, db_meta: &DatabaseMetadata, action: Action) -> bool {
    // Owner can do anything
    if db_meta.owner == auth.id_tag {
        return true;
    }

    match action {
        Action::Read => {
            db_meta.permissions.public_read ||
            db_meta.permissions.readers.contains(&auth.id_tag)
        }
        Action::Write => {
            db_meta.permissions.public_write ||
            db_meta.permissions.writers.contains(&auth.id_tag)
        }
    }
}

Future: Fine-Grained Permissions

Planned for future releases:

  • Per-collection permissions: Different access per table
  • Per-document permissions: Filter queries by ownership
  • Runtime rules: JavaScript-like expressions evaluated at runtime
  • Attribute-based: Permissions based on user attributes

WebSocket Protocol

Both systems (RTDB and CRDT) use WebSocket for real-time communication, though with different protocols:

Connection

const ws = new WebSocket(
  `wss://cl-o.example.com/ws/rtdb/${fileId}`,
  {
    headers: {
      'Authorization': `Bearer ${accessToken}`
    }
  }
);

Message Format

JSON messages with type field:

// Client β†’ Server
{
  "type": "query",      // or "subscribe", "create", "update", "delete"
  "id": 123,            // Request ID for correlation
  // ... type-specific fields
}

// Server β†’ Client
{
  "type": "queryResult", // or "change", "error"
  "id": 123,             // Matches request ID
  // ... response data
}

Lifecycle

Client connects
  ↓
Server authenticates
  ↓
Server loads database instance
  ↓
Client sends queries/subscriptions
  ↓
Server sends results + change notifications
  ↓
Client disconnects
  ↓
Server cleans up subscriptions

Performance Characteristics

Query-Based RTDB (redb)

  • Package size: ~171 KiB
  • Database file size: ~50 KiB minimum
  • Query speed: ~10,000 queries/second (in-memory)
  • Write speed: ~1,000 writes/second
  • Connection capacity: ~1,000 concurrent per database
  • Memory usage: ~10 MB per active database instance

CRDT-Based (Yrs)

  • Sync speed: ~50 ms for typical documents
  • Conflict resolution: Automatic, deterministic
  • Memory usage: ~5-20 MB per active document
  • Update latency: <10 ms for local network
  • Scalability: Tested with 100+ concurrent editors

Optimization Strategies

  1. Snapshots: Periodic full-state saves reduce sync time
  2. Compression: zstd compression for storage
  3. Eviction: LRU eviction for inactive databases
  4. Indexing: Secondary indexes for fast queries
  5. Batching: Batch updates for efficiency

Storage Strategy

Snapshots

Periodic full-state backups:

pub struct SnapshotStrategy {
    update_threshold: u32,      // Snapshot every N updates
    time_threshold: Duration,   // Snapshot every T duration
    compression: u8,            // zstd compression level (0-9)
    retention: u32,             // Keep N historical snapshots
}

Delta Updates

Incremental changes stored separately:

{file_id}~snapshot~v001      // Full state
{file_id}~updates~v001       // Changes since snapshot
{file_id}~metadata~v001      // Database config

File Variants

Reuse file variant system:

  • snapshot: Full database state (primary)
  • delta: Incremental updates
  • compressed: Compressed snapshot
  • exported: Human-readable export (JSON)

Federation Support

Databases can be shared across Cloudillo instances:

Read-Only Federation

pub struct FederatedDatabase {
    origin_instance: String,    // Original instance
    local_replica: bool,        // Keep local copy
    sync_mode: SyncMode,
}

pub enum SyncMode {
    ReadOnly,          // Subscribe to updates only
    ReadWrite,         // Full bidirectional sync
    Periodic(Duration), // Sync every N seconds
}

Sync Protocol

Using existing action/inbox mechanism:

pub struct DatabaseSyncAction {
    db_file_id: String,
    updates: Vec<u8>,       // Serialized updates
    state_vector: Vec<u8>,  // CRDT state or query timestamp
}

// POST /api/inbox with action type "db.sync"

Security Considerations

Authentication

  • WebSocket connections require valid access token
  • Token validated on connection establishment
  • Token can expire during connection (disconnected)

Authorization

  • Permissions checked on connection
  • Every read/write operation validated
  • Subscriptions filtered by permissions

Data Validation

  • Schema validation (optional)
  • Size limits per database
  • Rate limiting per user
  • Malicious update detection

Encryption

  • TLS/WSS for all connections
  • Optional client-side encryption (future)
  • Content-addressed snapshots prevent tampering

Choosing Between RTDB and CRDT

Use RTDB (redb) When:

βœ… You need structured data with schemas βœ… Complex queries are important (filters, sorts, aggregates) βœ… Computed values and aggregations are needed βœ… Document locking for exclusive editing is required βœ… Traditional database patterns fit your use case βœ… Atomic transactions are required βœ… You want minimal package size

Use CRDT (Yrs) When:

βœ… Multiple users edit the same data concurrently βœ… Conflict-free merging is critical βœ… Rich text editing is needed βœ… Offline-first design is important βœ… You want Yjs ecosystem compatibility βœ… Time-travel/versioning is valuable

Can You Use Both?

Yes! Many applications benefit from both:

  • Yrs for collaborative document editing
  • redb for user profiles, settings, and structured data

Example: A collaborative task management app might use:

  • Yrs for the task description (rich text, concurrent editing)
  • redb for task metadata (assignee, due date, status)

API Overview

Database Management

POST /api/db                  # Create database
GET /api/db                   # List databases
GET /api/db/:fileId          # Get metadata
PATCH /api/db/:fileId        # Update metadata
DELETE /api/db/:fileId       # Delete database

WebSocket Connection

GET /ws/rtdb/:fileId
Upgrade: websocket
Authorization: Bearer <token>

Export/Import

GET /api/db/:fileId/export?format=json
POST /api/db/:fileId/import

Next Steps

redb Implementation

The query-based RTDB uses redb, a lightweight embedded database, to provide Firebase-like functionality with minimal overhead. This approach is ideal for structured data, complex queries, and traditional database operations.

Why redb?

redb is chosen for its exceptional characteristics:

  • Tiny footprint: 171 KiB package size (vs. 1+ MB for most databases)
  • Pure Rust: Memory-safe, no unsafe code
  • ACID transactions: Full transactional guarantees
  • Zero-copy reads: Excellent performance
  • Embedded: No separate database server
  • LMDB-inspired: Proven B-tree architecture

Architecture

Layered Design

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Client Application (JavaScript)      β”‚
β”‚  - Query API                         β”‚
β”‚  - Subscriptions                     β”‚
β”‚  - Transactions                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                ↓ WebSocket
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ WebSocket Handler                    β”‚
β”‚  - Message parsing                   β”‚
β”‚  - Authentication                    β”‚
β”‚  - Subscription tracking             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ RtdbAdapter Trait                    β”‚
β”‚  - query()                           β”‚
β”‚  - create()                          β”‚
β”‚  - update()                          β”‚
β”‚  - delete()                          β”‚
β”‚  - transaction()                     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ redb Implementation                  β”‚
β”‚  - Key-value storage                 β”‚
β”‚  - Secondary indexes                 β”‚
β”‚  - Query execution                   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Real-Time Layer                      β”‚
β”‚  - tokio::broadcast channels         β”‚
β”‚  - Change event propagation          β”‚
β”‚  - Subscription filtering            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

RtdbAdapter Trait

The core interface for database operations:

#[async_trait]
pub trait RtdbAdapter: Send + Sync {
    /// Query documents
    async fn query(
        &self,
        db_id: &str,
        path: &str,
        options: QueryOptions,
    ) -> Result<Vec<Document>>;

    /// Create document
    async fn create(
        &self,
        db_id: &str,
        path: &str,
        data: Value,
    ) -> Result<String>;  // Returns document ID

    /// Update document
    async fn update(
        &self,
        db_id: &str,
        path: &str,
        doc_id: &str,
        data: Value,
    ) -> Result<()>;

    /// Delete document
    async fn delete(
        &self,
        db_id: &str,
        path: &str,
        doc_id: &str,
    ) -> Result<()>;

    /// Execute transaction
    async fn transaction(
        &self,
        db_id: &str,
        operations: Vec<Operation>,
    ) -> Result<TransactionResult>;

    /// Subscribe to changes
    fn subscribe(
        &self,
        db_id: &str,
        path: &str,
        filter: Option<QueryFilter>,
    ) -> broadcast::Receiver<ChangeEvent>;
}

Data Model

Collections and Documents

Data is organized into collections containing JSON documents:

database/
β”œβ”€β”€ users/
β”‚   β”œβ”€β”€ user_001
β”‚   β”œβ”€β”€ user_002
β”‚   └── ...
β”œβ”€β”€ posts/
β”‚   β”œβ”€β”€ post_abc
β”‚   β”œβ”€β”€ post_def
β”‚   └── ...
└── comments/
    └── ...

Document Structure

Documents are JSON objects with auto-generated IDs:

{
  "_id": "user_001",
  "_createdAt": 1738483200,
  "_updatedAt": 1738486800,
  "name": "Alice",
  "email": "alice@example.com",
  "age": 30,
  "active": true
}

System Fields (auto-managed):

  • _id: Unique document identifier
  • _createdAt: Creation timestamp (Unix)
  • _updatedAt: Last modification timestamp

Path Syntax

Paths use slash-separated segments:

users                    // Collection
users/user_001          // Specific document
posts/post_abc/comments // Sub-collection

Query System

QueryOptions

pub struct QueryOptions {
    pub filter: Option<QueryFilter>,
    pub sort: Option<Sort>,
    pub limit: Option<u32>,
    pub offset: Option<u32>,
}

pub struct Sort {
    pub field: String,
    pub direction: SortDirection,  // Asc | Desc
}

QueryFilter

Filters support various comparison operators:

pub enum QueryFilter {
    Equals { field: String, value: Value },
    NotEquals { field: String, value: Value },
    GreaterThan { field: String, value: Value },
    GreaterThanOrEqual { field: String, value: Value },
    LessThan { field: String, value: Value },
    LessThanOrEqual { field: String, value: Value },
    Contains { field: String, value: String },
    In { field: String, values: Vec<Value> },
    NotIn { field: String, values: Vec<Value> },
    ArrayContains { field: String, value: Value },
    ArrayContainsAny { field: String, values: Vec<Value> },
    ArrayContainsAll { field: String, values: Vec<Value> },
    And(Vec<QueryFilter>),
    Or(Vec<QueryFilter>),
}

Query Examples

Simple query:

{
  "type": "query",
  "id": 1,
  "path": "users",
  "filter": {
    "equals": { "field": "active", "value": true }
  },
  "sort": { "field": "name", "direction": "asc" },
  "limit": 50
}

Complex query:

{
  "type": "query",
  "id": 2,
  "path": "posts",
  "filter": {
    "and": [
      { "equals": { "field": "published", "value": true } },
      { "greaterThan": { "field": "views", "value": 100 } }
    ]
  },
  "sort": { "field": "createdAt", "direction": "desc" },
  "limit": 20
}

WebSocket Protocol

Message Types

Client β†’ Server

1. Query

{
  "type": "query",
  "id": 1,
  "path": "users",
  "filter": { "equals": { "field": "active", "value": true } },
  "limit": 50
}

2. Subscribe

{
  "type": "subscribe",
  "id": 2,
  "path": "posts",
  "filter": { "equals": { "field": "published", "value": true } }
}

3. Get (single document)

{
  "type": "get",
  "id": 3,
  "path": "users/user_001"
}

4. Transaction (wraps create/update/delete operations)

{
  "type": "transaction",
  "id": 4,
  "operations": [
    {
      "type": "create",
      "path": "posts",
      "ref": "$post",
      "data": { "title": "Hello", "author": "alice" }
    },
    {
      "type": "update",
      "path": "users/alice",
      "data": { "postCount": { "$op": "increment", "by": 1 } }
    },
    {
      "type": "delete",
      "path": "posts/post_old"
    }
  ]
}
Info

All write operations (create, update, delete) must be wrapped in a transaction message. There are no standalone create/update/delete message types.

5. Lock

{
  "type": "lock",
  "id": 5,
  "path": "todos/task_123",
  "mode": "hard"
}

6. Unlock

{
  "type": "unlock",
  "id": 6,
  "path": "todos/task_123"
}

7. Create Index

{
  "type": "createIndex",
  "id": 7,
  "path": "users",
  "field": "email"
}

8. Ping

{
  "type": "ping",
  "id": 8
}

Server β†’ Client

1. Query Result

{
  "type": "queryResult",
  "id": 1,
  "data": [
    { "_id": "user_001", "name": "Alice", "active": true },
    { "_id": "user_002", "name": "Bob", "active": true }
  ],
  "total": 2
}

2. Get Result

{
  "type": "getResult",
  "id": 3,
  "data": { "_id": "user_001", "name": "Alice", "active": true }
}

3. Subscribe Result

{
  "type": "subscribeResult",
  "id": 2,
  "subscriptionId": "sub_abc123",
  "data": [
    { "_id": "post_001", "title": "Hello", "published": true }
  ]
}

4. Change Event

Change events use a single event object with an action field, not a changes array:

{
  "type": "change",
  "subscriptionId": "sub_abc123",
  "event": {
    "action": "create",
    "path": "posts",
    "data": { "_id": "post_002", "title": "New Post", "published": true }
  }
}

Possible action values: create, update, delete, lock, unlock, ready

The ready action is sent after the initial subscription data has been delivered.

5. Transaction Result

{
  "type": "transactionResult",
  "id": 4,
  "results": [
    { "ref": "$post", "id": "post_new_001" },
    { "id": "users/alice" },
    { "id": "posts/post_old" }
  ]
}

6. Lock Result

{
  "type": "lockResult",
  "id": 5,
  "locked": true
}

If the lock is denied:

{
  "type": "lockResult",
  "id": 5,
  "locked": false,
  "holder": "bob@example.com",
  "mode": "hard"
}

7. Error

{
  "type": "error",
  "id": 4,
  "code": "permission_denied",
  "message": "Insufficient permissions to update this document"
}

8. Pong

{
  "type": "pong",
  "id": 8
}

Real-Time Subscriptions

Subscription Flow

Client sends subscribe message
  ↓
Server validates permissions
  ↓
Server creates broadcast channel
  ↓
Server executes initial query
  ↓
Server sends subscribeResult with data
  ↓
Server watches for changes matching filter
  ↓
On change: Server sends change event
  ↓
Client updates local state

Implementation

Subscription Structure:

  • id: Unique subscription identifier
  • path: Collection path being subscribed to
  • filter: Optional query filter to match changes
  • sender: Broadcast channel for sending change events

Notification Algorithm:

Algorithm: Notify Subscribers on Change

Input: db_id, path, change_event
Output: None (side effect: sends to all matching subscribers)

1. For each active subscription:
   a. Check if subscription.path matches change path
   b. If path matches:
      - Evaluate if change data matches subscription filter
      - If matches: send change_event through subscriber's broadcast channel
      - If no match: skip this subscriber
2. Return

This ensures:
- Only subscribed collections receive notifications
- Filter conditions prevent unnecessary updates
- All matching subscribers notified in parallel via broadcast channels

Change Event Types

pub enum ChangeType {
    Added,    // New document created
    Modified, // Existing document updated
    Removed,  // Document deleted
}

pub struct ChangeEvent {
    pub change_type: ChangeType,
    pub path: String,
    pub doc_id: String,
    pub data: Option<Value>,  // None for Removed
}

Transactions

Atomic Operations

Transactions ensure multiple operations execute atomically:

{
  "type": "transaction",
  "id": 10,
  "operations": [
    {
      "type": "update",
      "path": "accounts/alice",
      "data": { "balance": { "$op": "increment", "by": -100 } }
    },
    {
      "type": "update",
      "path": "accounts/bob",
      "data": { "balance": { "$op": "increment", "by": 100 } }
    }
  ]
}

Guarantees:

  • All operations succeed or all fail
  • Intermediate states never visible
  • Sequential consistency

Temporary References

Reference documents created within the same transaction:

{
  "type": "transaction",
  "id": 11,
  "operations": [
    {
      "type": "create",
      "path": "posts",
      "ref": "$post",
      "data": { "title": "My Post", "author": "alice" }
    },
    {
      "type": "create",
      "path": "comments",
      "data": {
        "postId": { "$ref": "$post" },
        "text": "First comment!",
        "author": "alice"
      }
    }
  ]
}

How it works:

  1. First operation creates post, saves ID as $post
  2. Second operation references $post, replaced with actual ID
  3. Comment gets correct post ID even though it wasn’t known initially

Document Locking

The RTDB supports document-level locking for exclusive or advisory editing access.

Lock Modes

  • Soft lock (advisory): Other clients can still write but receive a notification that the document is locked. Useful for signaling editing intent.
  • Hard lock (enforced): The server rejects writes from other clients while the lock is held. Only the lock holder (identified by conn_id) can modify the document.

Lock/Unlock Messages

Client β†’ Server:

{ "type": "lock", "id": 1, "path": "todos/task_123", "mode": "soft" }
{ "type": "unlock", "id": 2, "path": "todos/task_123" }

Server β†’ Client:

{ "type": "lockResult", "id": 1, "locked": true }
{ "type": "lockResult", "id": 1, "locked": false, "holder": "bob@example.com", "mode": "hard" }

TTL-Based Expiration

Locks expire automatically after a TTL (time-to-live) period. This prevents permanently locked documents when clients disconnect unexpectedly or crash without releasing their locks. The server cleans up expired locks during its periodic maintenance cycle.

Connection-Based Echo Suppression

The server tracks lock ownership by conn_id. When a lock change event is broadcast to subscribers, the originating connection is excluded from the notification (echo suppression), similar to how write operations suppress echoes. This prevents the client that acquired the lock from receiving its own lock notification.

Lock Status in Change Events

Active subscriptions receive lock/unlock events as part of the change stream:

{
  "type": "change",
  "subscriptionId": "sub_abc123",
  "event": {
    "action": "lock",
    "path": "todos/task_123",
    "data": { "holder": "alice@example.com", "mode": "hard" }
  }
}

Aggregate Queries

The RTDB supports server-side aggregate computations on query results.

Aggregate Request

Add the aggregate option to a query or subscribe message:

{
  "type": "query",
  "id": 1,
  "path": "tasks",
  "filter": { "equals": { "field": "completed", "value": false } },
  "aggregate": {
    "groupBy": "status",
    "ops": [
      { "op": "sum", "field": "hours" },
      { "op": "avg", "field": "hours" }
    ]
  }
}

Aggregate Operations

Operation Description
sum Sum of a numeric field
avg Average of a numeric field
min Minimum value of a field
max Maximum value of a field

Each group always includes a count of matching documents.

Aggregate Response

{
  "type": "queryResult",
  "id": 1,
  "aggregate": {
    "groups": [
      { "group": "todo", "count": 12, "sum_hours": 36, "avg_hours": 3.0 },
      { "group": "in_progress", "count": 5, "sum_hours": 20, "avg_hours": 4.0 }
    ]
  }
}

Incremental Aggregate Subscriptions

When aggregate is used with a subscribe message, the server computes aggregates incrementally. On each change event that affects the subscribed path and filter, the server recalculates the affected groups and sends an updated aggregate snapshot rather than the full document set. This keeps aggregate subscriptions efficient even for large collections.

Computed Values

Field Operations

Modify field values with special operations:

Increment:

{
  "views": { "$op": "increment", "by": 1 }
}

Append (to array):

{
  "tags": { "$op": "append", "value": "javascript" }
}

Remove (from array):

{
  "tags": { "$op": "remove", "value": "draft" }
}

Decrement:

{
  "stock": { "$op": "decrement", "by": 1 }
}

Multiply:

{
  "price": { "$op": "multiply", "by": 1.1 }
}

Concat (string concatenation):

{
  "fullName": { "$op": "concat", "values": ["firstName", " ", "lastName"] }
}

Min (set to minimum of current and given value):

{
  "lowestScore": { "$op": "min", "value": 42 }
}

Max (set to maximum of current and given value):

{
  "highScore": { "$op": "max", "value": 99 }
}

Set if not exists:

{
  "createdAt": { "$op": "setIfNotExists", "value": 1738483200 }
}

Query Operations

Aggregate data within queries:

Count:

{
  "type": "query",
  "id": 12,
  "path": "posts",
  "query": { "$query": "count" },
  "filter": { "equals": { "field": "published", "value": true } }
}

Sum:

{
  "type": "query",
  "id": 13,
  "path": "orders",
  "query": { "$query": "sum", "field": "total" }
}

Average:

{
  "type": "query",
  "id": 14,
  "path": "reviews",
  "query": { "$query": "avg", "field": "rating" }
}

Function Operations

Server-side functions for computed values:

Now (current timestamp):

{
  "createdAt": { "$fn": "now" }
}

Slugify (URL-safe string):

{
  "slug": { "$fn": "slugify", "input": "Hello World!" }
  // Results in: "hello-world"
}

Lowercase (convert string to lowercase):

{
  "emailNormalized": { "$fn": "lowercase", "input": "Alice@Example.COM" }
  // Results in: "alice@example.com"
}

Hash (SHA256):

{
  "passwordHash": { "$fn": "hash", "input": "password123" }
}

Indexing

Secondary Indexes

Improve query performance:

Index Definition:

  • name: Unique name for the index (e.g., “idx_email”)
  • fields: Vector of field names to index (single or compound)
  • unique: Boolean flag ensuring no duplicate values (for unique constraints)

Create Index Algorithm:

Algorithm: Create Index

Input: db_id, collection_path, index_definition
Output: Result<()>

1. Validate index definition:
   - Check name uniqueness (not duplicate of existing index)
   - Verify all fields exist in collection schema
   - If unique=true: verify collection has no duplicate values for these fields

2. Build index structure:
   - Scan existing documents in collection
   - For each document, extract values for indexed fields
   - Build index data structure (B-tree for efficient lookups)

3. Store index metadata:
   - Save index definition in metadata adapter
   - Record index name, fields, and unique flag

4. Return success

This ensures:
- Efficient lookups on indexed fields
- Query optimizer can use index automatically
- Unique constraints enforced at index level

Index Usage

Queries automatically use indexes when available:

{
  "type": "query",
  "path": "users",
  "filter": { "equals": { "field": "email", "value": "alice@example.com" } }
}
// Uses idx_email if available, otherwise full scan

Index Strategies

Single-field indexes:

fields: vec!["email"]           // For email lookups
fields: vec!["createdAt"]       // For sorting by date

Compound indexes:

fields: vec!["category", "price"]  // For category + price queries

Unique indexes:

unique: true  // Ensures no duplicates (e.g., email, username)

Client SDK Example

JavaScript/TypeScript

import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

// Create RTDB client
const rtdb = new RtdbClient({
  dbId: 'my-database-id',
  auth: {
    getToken: () => bus.accessToken
  },
  serverUrl: getRtdbUrl(bus.idTag!, 'my-database-id', bus.accessToken!)
})

// Connect
await rtdb.connect()

// Query data
const users = await rtdb.collection('users')
  .where('active', '==', true)
  .get()

console.log(users.docs.map(doc => doc.data()))

// Subscribe to changes
const unsubscribe = rtdb.collection('posts')
  .where('published', '==', true)
  .onSnapshot((snapshot) => {
    snapshot.docChanges().forEach((change) => {
      if (change.type === 'added') {
        console.log('New post:', change.doc.data())
      }
      if (change.type === 'modified') {
        console.log('Modified post:', change.doc.data())
      }
      if (change.type === 'removed') {
        console.log('Removed post:', change.doc.id)
      }
    })
  })

// Create document via batch
const batch = rtdb.batch()
batch.create(rtdb.collection('users'), {
  name: 'Charlie',
  email: 'charlie@example.com',
  age: 28
})
const results = await batch.commit()

// Update document via batch
const batch2 = rtdb.batch()
batch2.update(rtdb.ref('users/' + results[0].id), {
  age: 29
})
await batch2.commit()

// Cleanup
unsubscribe()
await rtdb.disconnect()

React Example

import { useEffect, useState } from 'react'
import { useAuth } from '@cloudillo/react'
import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

interface Task {
  title: string
  completed: boolean
  priority: number
}

function TaskList({ dbId }: { dbId: string }) {
  const [auth] = useAuth()
  const [tasks, setTasks] = useState<(Task & { id: string })[]>([])
  const [rtdb, setRtdb] = useState<RtdbClient | null>(null)

  // Initialize RTDB client
  useEffect(() => {
    if (!auth?.token || !auth?.idTag) return

    const client = new RtdbClient({
      dbId,
      auth: { getToken: () => auth.token },
      serverUrl: getRtdbUrl(auth.idTag, dbId, auth.token!)
    })

    client.connect()
    setRtdb(client)

    return () => { client.disconnect() }
  }, [auth?.token, auth?.idTag, dbId])

  // Subscribe to incomplete tasks
  useEffect(() => {
    if (!rtdb) return

    const unsubscribe = rtdb.collection<Task>('tasks')
      .where('completed', '==', false)
      .onSnapshot((snapshot) => {
        setTasks(snapshot.docs.map(doc => ({
          id: doc.id,
          ...doc.data()
        })))
      })

    return () => unsubscribe()
  }, [rtdb])

  return (
    <ul>
      {tasks.map(task => (
        <li key={task.id}>{task.title} (priority: {task.priority})</li>
      ))}
    </ul>
  )
}

Performance Optimization

Query Optimization

Use indexes:

// Create index for frequently queried fields
await db.createIndex('users', { fields: ['email'], unique: true });

Limit results:

// Always use limit for large collections
const recent = await db.collection('posts')
  .orderBy('createdAt', 'desc')
  .limit(20)
  .get();

Use pagination:

const page1 = await db.collection('posts').limit(20).get();
const page2 = await db.collection('posts').limit(20).offset(20).get();

Subscription Optimization

Filter subscriptions:

// Only subscribe to relevant data
db.collection('messages')
  .where('conversationId', '==', conversationId)
  .onSnapshot(handler);  // Not all messages

Cleanup subscriptions:

// Always unsubscribe when component unmounts
useEffect(() => {
  const unsubscribe = db.collection('...').onSnapshot(handler);
  return () => unsubscribe();
}, []);

Memory Management

Database eviction:

pub struct EvictionPolicy {
    max_instances: usize,        // Max databases in memory
    idle_timeout: Duration,      // Evict after N minutes idle
    lru_eviction: bool,          // Use LRU when at max
}

Connection limits:

pub struct ConnectionLimits {
    max_sessions_per_db: usize,  // Per database
    max_total_sessions: usize,   // Server-wide
    max_sessions_per_user: usize,
}

Error Handling

Error Types

pub enum RtdbError {
    PermissionDenied,
    DocumentNotFound,
    InvalidQuery,
    TransactionFailed,
    DatabaseNotFound,
    ConnectionClosed,
    RateLimitExceeded,
}

Client Handling

try {
  await db.collection('users').doc(id).update(data);
} catch (error) {
  if (error.code === 'permission_denied') {
    console.error('No permission to update');
  } else if (error.code === 'document_not_found') {
    console.error('Document does not exist');
  } else {
    console.error('Unknown error:', error);
  }
}

Security Best Practices

Permission Rules

// Create database with strict permissions
await fetch('/api/db', {
  method: 'POST',
  body: JSON.stringify({
    name: 'Private Data',
    type: 'redb',
    permissions: {
      public_read: false,
      public_write: false,
      readers: [],  // Only owner can read
      writers: []   // Only owner can write
    }
  })
});

Input Validation

// Validate on client before sending
function validateUser(data) {
  if (!data.email || !/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(data.email)) {
    throw new Error('Invalid email');
  }
  if (!data.name || data.name.length < 2) {
    throw new Error('Name too short');
  }
  return true;
}

if (validateUser(userData)) {
  await db.collection('users').add(userData);
}

Rate Limiting

Server-Side Rate Limits:

To prevent abuse and ensure fair resource allocation, the server enforces per-user/per-IP rate limits:

Query Limits:

  • MAX_QUERIES_PER_SECOND: 100 queries per user per second
    • Prevents overwhelming the database with excessive read queries
    • Typical usage: <10 queries/second for normal applications

Write Limits:

  • MAX_WRITES_PER_SECOND: 10 writes per user per second
    • More restrictive than reads to protect data consistency
    • Typical usage: <1 write/second for normal applications

Implementation Pattern:

  • Sliding window counter: Track requests in last 1 second
  • Per-user tracking: Rate limits apply to authenticated users
  • Per-IP fallback: Anonymous requests rate-limited by IP address
  • When exceeded: Return HTTP 429 (Too Many Requests) with retry-after header

See Also

CRDT (Collaborative Editing)

Conflict-free replicated data types enabling true collaborative editing with automatic conflict resolution.

Overview

The CRDT system provides:

  • Conflict-free replicated data types (CRDTs)
  • Rich data structures (Text, Map, Array, XML)
  • Automatic conflict resolution
  • Real-time synchronization
  • Offline editing support
  • Yrs/Yjs ecosystem compatibility
For App Developers

If you’re building collaborative applications, see the CRDT Design Guide for schema design patterns, best practices, and common pitfalls when working with Yjs/CRDTs.

Documents

  • Overview - Introduction to CRDTs and Yrs implementation
  • redb Implementation - How CRDT updates are stored persistently using redb

Subsections of CRDT (Collaborative Editing)

CRDT Overview

Cloudillo’s CRDT system uses Yrs, a Rust implementation of the Yjs CRDT (Conflict-free Replicated Data Type), to enable true collaborative editing with automatic conflict resolution. This is a separate API from RTDB, optimized specifically for concurrent editing scenarios where multiple users modify the same data simultaneously.

What are CRDTs?

Conflict-free Replicated Data Types (CRDTs) are data structures that can be replicated across multiple nodes and modified independently, then merged automatically without conflicts.

Key Properties

  1. Eventual Consistency: All replicas converge to the same state
  2. No Central Authority: No server needed to resolve conflicts
  3. Deterministic Merging: Same operations always produce same result
  4. Commutative: Order of operations doesn’t matter
  5. Idempotent: Applying same operation twice has no extra effect

Example: Concurrent Editing

Alice's editor:
  "Hello"
  ↓ (adds " World")
  "Hello World"

Bob's editor (at same time):
  "Hello"
  ↓ (adds "!")
  "Hello!"

After sync (both converge to):
  "Hello World!"
  ↑ CRDT algorithm automatically merges

Why Yrs?

Yrs is the Rust port of Yjs, the most battle-tested CRDT implementation:

Advantages

βœ… Production-Ready: Used in Google Docs-like applications βœ… Pure Rust: Memory-safe, high performance βœ… Rich Data Types: Text, Map, Array, XML βœ… Ecosystem Compatible: Works with Yjs clients (JavaScript) βœ… Battle-Tested Protocol: Proven sync algorithm βœ… Awareness API: Presence, cursors, selections βœ… Time-Travel: Built-in versioning support

Performance

  • Sync speed: ~50 ms for typical documents (local network)
  • Memory efficiency: Compact binary encoding
  • Conflict resolution: Deterministic, instantaneous
  • Scalability: Tested with 100+ concurrent editors

Architecture

Components

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Client Application                           β”‚
β”‚  - Yjs Document (Y.Doc)                      β”‚
β”‚  - Shared types (Y.Text, Y.Map, Y.Array)     β”‚
β”‚  - Awareness (presence, cursors)             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
               ↓ WebSocket (binary protocol)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Cloudillo Server                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ WebSocket Handler                      β”‚  β”‚
β”‚  β”‚  - Yrs sync protocol                   β”‚  β”‚
β”‚  β”‚  - Binary message parsing              β”‚  β”‚
β”‚  β”‚  - Authentication                      β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                    ↓                         β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Database Manager                       β”‚  β”‚
β”‚  β”‚  - Y.Doc instances (in-memory)         β”‚  β”‚
β”‚  β”‚  - Session tracking                    β”‚  β”‚
β”‚  β”‚  - Snapshot management                 β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                    ↓                         β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Yrs Document (yrs::Doc)                β”‚  β”‚
β”‚  β”‚  - CRDT state                          β”‚  β”‚
β”‚  β”‚  - Update log                          β”‚  β”‚
β”‚  β”‚  - Awareness state                     β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                    ↓                         β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚  β”‚ Storage Layer                           β”‚ β”‚
β”‚  β”‚  - CrdtAdapter (snapshots, updates)     β”‚ β”‚
β”‚  β”‚  - Compressed binary format             β”‚ β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Database Instance

Each collaborative document is a DatabaseInstance:

pub struct DatabaseInstance {
    file_id: Box<str>,
    tn_id: TnId,
    doc: Arc<RwLock<yrs::Doc>>,                    // Yrs document
    sessions: RwLock<HashMap<SessionId, Session>>,  // Connected clients
    awareness: Arc<RwLock<yrs::sync::Awareness>>,  // Presence info
    last_accessed: AtomicTimestamp,
    auto_save_handle: Option<TaskHandle>,
}

Database Manager

Manages database instances lifecycle:

DatabaseManager Structure:

  • instances: RwLock-protected HashMap of file_id β†’ DatabaseInstance
  • blob_adapter: Persistent storage for snapshots and incremental updates
  • meta_adapter: Metadata storage for database information
  • max_instances: Maximum concurrent documents in memory
  • eviction_policy: Strategy for removing inactive instances
  • snapshot_interval: Duration between automatic snapshots

Core Methods:

get_or_load(file_id, tn_id): Retrieve or load a database instance

  • Check if instance exists in memory cache
  • If cached and recently accessed: return immediately
  • If not cached: load latest snapshot from blob storage, apply incremental updates, cache in memory
  • Return loaded instance

create_database(tn_id, options): Create new collaborative document

  • Generate unique file_id
  • Initialize empty Y.Doc with Yrs
  • Store metadata in meta_adapter
  • Register instance in memory cache
  • Return new file_id

snapshot(file_id): Save full document state

  • Retrieve document from cache
  • Encode full state as compressed binary update
  • Store snapshot with versioning
  • Prune old snapshots based on retention policy
  • Return success

evict(file_id): Remove instance from memory

  • Get write lock on instances map
  • Remove instance from cache
  • Instance data persists in blob storage for later reload
  • Release memory

Yrs Data Types

Y.Text

For collaborative text editing:

import * as Y from 'yjs'

const doc = new Y.Doc()
const text = doc.getText('content')

// Insert text
text.insert(0, 'Hello World!')

// Format text
text.format(0, 5, { bold: true })  // Bold "Hello"

// Delete text
text.delete(6, 6)  // Remove "World!"

// Observe changes
text.observe(event => {
  event.changes.delta.forEach(change => {
    if (change.insert) {
      console.log('Inserted:', change.insert)
    }
    if (change.delete) {
      console.log('Deleted:', change.delete, 'characters')
    }
  })
})

Y.Map

For key-value data structures:

const map = doc.getMap('config')

// Set values
map.set('theme', 'dark')
map.set('fontSize', 14)
map.set('notifications', { email: true, push: false })

// Get values
const theme = map.get('theme')  // 'dark'

// Delete keys
map.delete('fontSize')

// Iterate
map.forEach((value, key) => {
  console.log(key, value)
})

// Observe changes
map.observe(event => {
  event.changes.keys.forEach((change, key) => {
    if (change.action === 'add') {
      console.log('Added:', key, map.get(key))
    } else if (change.action === 'update') {
      console.log('Updated:', key, map.get(key))
    } else if (change.action === 'delete') {
      console.log('Deleted:', key)
    }
  })
})

Y.Array

For ordered lists:

const array = doc.getArray('tasks')

// Push items
array.push([
  { title: 'Task 1', completed: false },
  { title: 'Task 2', completed: false }
])

// Insert at position
array.insert(0, [{ title: 'Urgent Task', completed: false }])

// Delete
array.delete(1, 1)  // Delete 1 item at index 1

// Iterate
array.forEach((item, index) => {
  console.log(index, item)
})

// Observe changes
array.observe(event => {
  event.changes.delta.forEach(change => {
    if (change.insert) {
      console.log('Inserted:', change.insert)
    }
    if (change.delete) {
      console.log('Deleted:', change.delete, 'items')
    }
  })
})

Y.XML

For structured document editing:

const xmlFragment = doc.getXmlFragment('document')

// Create element
const paragraph = new Y.XmlElement('p')
paragraph.setAttribute('class', 'intro')
paragraph.insert(0, [new Y.XmlText('Hello World!')])

// Add to document
xmlFragment.push([paragraph])

// Convert to HTML
const html = xmlFragment.toDOM().outerHTML

WebSocket Sync Protocol

Connection Flow

Client                          Server
  |                               |
  |--- GET /ws/db/:fileId ------->|
  |    (Authorization: Bearer...) |
  |                               |--- Validate token
  |                               |--- Load database instance
  |                               |--- Create session
  |<-- 101 Switching Protocols ---|
  |                               |
  |<====== WebSocket Open =======>|
  |                               |
  |<-- Sync Step 1 (State Vec) ---|
  |--- Sync Step 2 (State Vec) -->|
  |<-- Update (missing data) -----|
  |--- Update (local changes) --->|
  |                               |
  |<====== Synchronized =========>|
  |                               |
  |--- Update (user edits) ------>|
  |                               |--- Broadcast to sessions
  |<-- Update (remote edits) -----|
  |                               |
  |--- Awareness Update --------->|
  |<-- Awareness Update ----------|

Message Types

All messages use binary format (not JSON):

Sync Step 1 (Server β†’ Client)

Server sends its state vector:

[message_type: u8, state_vector: Vec<u8>]

State vector represents the version of the document the server has.

Sync Step 2 (Client β†’ Server)

Client sends its state vector:

[message_type: u8, state_vector: Vec<u8>]

Update (Bidirectional)

Document updates (changes):

[message_type: u8, update: Vec<u8>]

Updates are binary-encoded CRDT operations.

Awareness Update

Presence information (cursors, selections):

[message_type: u8, awareness_update: Vec<u8>]

WebSocket Connection Handler

Algorithm: Handle WebSocket Database Connection

Input: WebSocket, DatabaseInstance, Auth context
Output: Result<()>

1. Session Setup:
   - Generate unique session_id
   - Add session to db_instance.sessions map
   - Associate auth context with session

2. Send Sync Step 1:
   - Read document from db_instance
   - Encode full state as binary (state vector)
   - Send binary message to client

3. Message Loop (while socket connected):
   - Receive binary message from client
   - Extract message type from first byte

   a. SYNC_STEP_2 (client state vector):
      - Decode state vector from bytes [1..]
      - Read document snapshot
      - Compute missing updates needed for client
      - Send update binary to client

   b. UPDATE (document changes):
      - Decode update from bytes [1..]
      - Acquire write lock on document
      - Apply update to Yrs document
      - Release lock
      - Broadcast update to all other active sessions

   c. AWARENESS_UPDATE (presence/cursor):
      - Decode awareness update from bytes [1..]
      - Acquire write lock on awareness state
      - Apply update to awareness
      - Release lock
      - Broadcast awareness update to all other sessions

   d. Other message types:
      - Ignore or log unknown types

4. Connection Close:
   - Exit message loop on close frame
   - Remove session from sessions map
   - Return success (cleanup implicit via Arc drop)

This pattern ensures:
- Efficient binary protocol (smaller than JSON)
- State vector sync for minimal data transfer
- Broadcasting to multiple concurrent editors
- Clean session cleanup on disconnect

Awareness (Presence)

What is Awareness?

Awareness tracks ephemeral state like:

  • User presence (online/offline)
  • Cursor positions
  • Text selections
  • Custom user state (name, color, etc.)

Setting Local State

import { WebsocketProvider } from 'y-websocket'

const provider = new WebsocketProvider(
  'wss://cl-o.example.com/ws/db/file_id',
  doc
)

// Set local user state
provider.awareness.setLocalStateField('user', {
  name: 'Alice',
  color: '#ff0000',
  avatar: 'https://...'
})

// Update cursor position
provider.awareness.setLocalStateField('cursor', {
  anchor: 42,
  head: 50
})

Observing Awareness

provider.awareness.on('change', ({ added, updated, removed }) => {
  // New users joined
  added.forEach(clientId => {
    const state = provider.awareness.getStates().get(clientId)
    console.log('User joined:', state.user.name)
  })

  // Existing users updated state
  updated.forEach(clientId => {
    const state = provider.awareness.getStates().get(clientId)
    console.log('User updated:', state.user.name)
  })

  // Users left
  removed.forEach(clientId => {
    console.log('User left:', clientId)
  })
})

Rendering Cursors

function renderCursors() {
  const states = provider.awareness.getStates()

  states.forEach((state, clientId) => {
    if (state.cursor && clientId !== provider.awareness.clientID) {
      const cursorEl = document.createElement('div')
      cursorEl.className = 'remote-cursor'
      cursorEl.style.backgroundColor = state.user.color
      cursorEl.style.left = `${getCursorPosition(state.cursor.anchor)}px`
      cursorEl.textContent = state.user.name

      document.body.appendChild(cursorEl)
    }
  })
}

Storage Strategy

Snapshots

Periodic full-state saves reduce sync time for new connections:

Snapshot Strategy Configuration:

  • update_threshold: Trigger snapshot after N updates (e.g., 1000)
  • time_threshold: Trigger snapshot after duration (e.g., every 5 minutes)
  • compression: zstd compression level 0-9 (default: 3 for speed/ratio balance)
  • retention: Keep N historical snapshots (e.g., 5 recent snapshots)

Algorithm: Create Snapshot

Input: file_id, database_instance
Output: Result<()>

1. Get Write Lock:
   - Acquire exclusive lock on document
   - (Prevents new updates during snapshot)

2. Encode Full State:
   - Create default state vector (represents empty state)
   - Encode entire Y.Doc as binary update
   - This captures all document content

3. Compress (CPU-intensive work):
   - Offload to worker pool with Priority::Low
   - Use zstd compression level 3
   - Results in binary blob

4. Store in Persistent Storage:
   - Generate version string (e.g., "v001", "v002")
   - Store compressed snapshot in BlobAdapter:
     * key format: "{file_id}~snapshot~{version}"
     * value: compressed binary
   - This persists document state across restarts

5. Manage Retention:
   - List all existing snapshots for this file_id
   - If count > retention_limit:
     * Delete oldest snapshots
     * Keep most recent N snapshots

6. Release Write Lock
7. Return success

Benefits:
- New connections load snapshot instead of replaying all updates
- Significantly faster initial sync for large documents
- Binary format is compact and efficient

Loading from Snapshot

Algorithm: Load Database from Storage

Input: file_id, tn_id (tenant)
Output: Result<Arc<DatabaseInstance>>

1. Retrieve Latest Snapshot:
   - Query BlobAdapter for "{file_id}~snapshot~latest"
   - If found: proceed to step 2
   - If not found: create empty document (new database)

2. Decompress Snapshot:
   - Use zstd decompression on snapshot bytes
   - Produces binary Yrs update data

3. Initialize Yrs Document:
   - Create new Y.Doc instance
   - Decode compressed binary as Yrs update
   - Apply update to empty document
   - Document now contains snapshot state

4. Load Incremental Updates:
   - Query all updates created since snapshot timestamp
   - Sorted in chronological order
   - For each update:
     * Decode binary update
     * Apply to document (applies changes on top of snapshot)
   - Document now contains latest state

5. Create DatabaseInstance:
   - Wrap Y.Doc in Arc<RwLock> (shared, thread-safe access)
   - Initialize empty sessions HashMap
   - Create Awareness for presence tracking
   - Record current timestamp (for eviction tracking)
   - Set auto_save_handle (optional optimization)

6. Start Auto-Save Task:
   - Schedule background task for periodic snapshots
   - Uses snapshot_interval from DatabaseManager config
   - Task runs in background without blocking

7. Register in Memory Cache:
   - Add instance to DatabaseManager.instances map
   - File_id β†’ Arc<DatabaseInstance>

8. Return Arc-wrapped instance
   - Callers can clone Arc for shared access

Optimization:
- Snapshot avoids replaying thousands of individual updates
- Typically reduces load time from seconds β†’ milliseconds for large docs

Client Integration

Basic Setup

import * as Y from 'yjs'
import { WebsocketProvider } from 'y-websocket'

// Create document
const doc = new Y.Doc()

// Connect to Cloudillo
const provider = new WebsocketProvider(
  `wss://cl-o.example.com/ws/db/${fileId}`,
  'document-name',  // Room name (arbitrary)
  doc,
  {
    params: {
      token: accessToken  // Auth token as query param
    }
  }
)

// Get shared types
const text = doc.getText('content')
const map = doc.getMap('metadata')

// Make changes
text.insert(0, 'Hello World!')
map.set('title', 'My Document')

// Changes automatically sync!

Editor Integration: ProseMirror

import { ySyncPlugin, yCursorPlugin, yUndoPlugin } from 'y-prosemirror'
import { EditorState } from 'prosemirror-state'
import { EditorView } from 'prosemirror-view'
import { schema } from 'prosemirror-schema-basic'

const ytext = doc.getText('prosemirror')

const state = EditorState.create({
  schema,
  plugins: [
    ySyncPlugin(ytext),
    yCursorPlugin(provider.awareness),
    yUndoPlugin(),
  ]
})

const view = new EditorView(document.querySelector('#editor'), {
  state
})

Editor Integration: Monaco (VS Code)

import * as monaco from 'monaco-editor'
import { MonacoBinding } from 'y-monaco'

const ytext = doc.getText('monaco')

const editor = monaco.editor.create(document.getElementById('monaco'), {
  value: '',
  language: 'javascript'
})

const binding = new MonacoBinding(
  ytext,
  editor.getModel(),
  new Set([editor]),
  provider.awareness
)

Editor Integration: Quill

import Quill from 'quill'
import { QuillBinding } from 'y-quill'

const ytext = doc.getText('quill')

const quill = new Quill('#editor', {
  theme: 'snow'
})

const binding = new QuillBinding(ytext, quill, provider.awareness)

Use Cases

Collaborative Text Editor

const doc = new Y.Doc()
const provider = new WebsocketProvider(url, doc)
const text = doc.getText('content')

// Bind to editor
const editor = new ProseMirrorEditor(text, provider.awareness)

Shared Whiteboard

const doc = new Y.Doc()
const provider = new WebsocketProvider(url, doc)
const shapes = doc.getArray('shapes')

// Add shape
shapes.push([{
  type: 'rectangle',
  x: 100,
  y: 100,
  width: 200,
  height: 150,
  color: '#ff0000'
}])

// Observe changes
shapes.observe(event => {
  redrawCanvas(shapes.toArray())
})

Collaborative Form

const doc = new Y.Doc()
const provider = new WebsocketProvider(url, doc)
const form = doc.getMap('form')

// Update fields
form.set('name', 'Alice')
form.set('email', 'alice@example.com')
form.set('preferences', { newsletter: true, notifications: false })

// Observe changes
form.observe(event => {
  event.changes.keys.forEach((change, key) => {
    updateFormField(key, form.get(key))
  })
})

Performance Optimization

Connection Pooling

Reuse WebSocket connections:

class CloudilloProvider {
  static connections = new Map()

  static getProvider(fileId, doc) {
    if (!this.connections.has(fileId)) {
      const provider = new WebsocketProvider(url, doc)
      this.connections.set(fileId, provider)
    }
    return this.connections.get(fileId)
  }
}

Batching Updates

Group rapid changes:

let pending = []
let timeout = null

function batchUpdate(update) {
  pending.push(update)

  clearTimeout(timeout)
  timeout = setTimeout(() => {
    // Apply all pending updates
    doc.transact(() => {
      pending.forEach(u => u())
    })
    pending = []
  }, 50)  // Batch for 50ms
}

Memory Management

Manage in-memory database instances efficiently:

EvictionPolicy Configuration:

  • max_instances: Maximum number of documents in memory (e.g., 100)
  • idle_timeout: Evict documents unused for duration (e.g., 30 minutes)
  • lru_eviction: Use Least Recently Used strategy when at max capacity
  • pinned: HashSet of file_ids never evicted (for critical documents)

Eviction Algorithm:

  1. Track last_accessed timestamp for each instance
  2. On timer (every 1 minute):
    • Find all instances exceeding idle_timeout
    • If instance count > max_instances:
      • Mark least recently accessed for eviction
      • Respect pinned set (never evict these)
  3. For each instance marked for eviction:
    • Trigger snapshot (save state to persistent storage)
    • Remove from instances map
    • Document can be reloaded later from snapshot

Security Considerations

Authentication

Algorithm: Authenticate WebSocket Connection

Input: HTTP headers from WebSocket upgrade request
Output: Result<Auth>

1. Extract Authorization Header:
   - Look for "authorization" header
   - If missing: Return Unauthorized error

2. Parse Bearer Token:
   - Convert header value to string
   - Check for "Bearer " prefix
   - Extract token (everything after "Bearer ")
   - If format invalid: Return InvalidToken error

3. Validate Token:
   - Call token validation function
   - Verifies signature using profile key
   - Checks expiration
   - Extracts claims (tn_id, id_tag, scope)

4. Return Auth context
   - Contains user identity and permissions

Permission Enforcement

Algorithm: Check Database Permission

Input: Auth context, DatabaseMetadata, Action (read/write)
Output: Result<()>

1. Check Ownership:
   - If auth.id_tag == db_meta.owner: Return Ok
   - Owner has all permissions

2. Check Action-Specific Permissions:
   - For Read action:
     * If auth.id_tag in db_meta.permissions.readers: Return Ok
   - For Write action:
     * If auth.id_tag in db_meta.permissions.writers: Return Ok

3. Default Deny:
   - If no permission found: Return PermissionDenied

Rate Limiting

Server-Side Rate Limits:

  • MAX_UPDATES_PER_SECOND: 100 updates per user per second

    • Prevents overwhelming with document changes
    • Typical usage: 5-20 updates/second for collaborative editing
  • MAX_AWARENESS_UPDATES_PER_SECOND: 10 awareness updates per user per second

    • Controls cursor/presence update frequency
    • Prevents spam of position updates

Implementation:

  • Per-user sliding window counter
  • Track requests in last 1 second
  • When exceeded: Return HTTP 429 with retry-after header

See Also

redb Implementation

The CRDT adapter stores collaborative document updates persistently using redb, enabling conflict-free replicated data types to survive server restarts while maintaining real-time synchronization capabilities.

Architecture Overview

The CRDT adapter bridges between the Yrs CRDT engine (in-memory) and persistent storage, storing binary update streams that can be replayed to reconstruct document state.

Client 1 ─┐
          β”œβ”€β–Ί Yrs CRDT ─► Binary Updates ─► redb Storage
Client 2 β”€β”˜      β”‚                              β”‚
                 │◄──────── Load on startup β”€β”€β”€β”€β”˜
                 └─► Broadcast ─► Subscribed Clients

Key Insight

CRDT systems work by accumulating operation updates rather than storing full document state. Each update is a binary-encoded operation (insert, delete, format, etc.) that can be:

  • Applied to reconstruct current document state
  • Sent to new subscribers for synchronization
  • Replayed in any order (commutative property)

Storage Layout

The adapter uses three redb tables per database:

1. Updates Table (crdt_updates)

Stores binary CRDT update blobs indexed by document and sequence number.

Schema: (doc_id:seq) β†’ update_bytes

Key Format:    "{doc_id}:{sequence}"
Value:         Binary CRDT update blob (from Yrs)

Example Keys:
  "doc_abc123:0"      First update
  "doc_abc123:1"      Second update
  "doc_abc123:2"      Third update

Properties:

  • Sequential numbering ensures order preservation
  • Each update is a self-contained binary blob
  • Updates are append-only (immutable)
  • Prefix scan retrieves all updates for a document

2. Metadata Table (crdt_metadata)

Stores document metadata as JSON.

Schema: doc_id β†’ metadata_json

{
  "created_at": 1738483200,
  "updated_at": 1738486800,
  "owner": "alice.example.com",
  "permissions": {...},
  "title": "Collaborative Document"
}

Purpose:

  • Track document ownership and permissions
  • Store human-readable metadata
  • Enable document discovery and listing

3. Stats Table (crdt_stats)

Tracks update counts and storage metrics (currently defined but not actively used in core operations).

Schema: doc_id β†’ stats_json

Potential metrics:

  • Total update count
  • Total bytes stored
  • Last compaction timestamp

Multi-Tenancy Storage Modes

The adapter supports two storage strategies configured at initialization:

Per-Tenant Files Mode (per_tenant_files=true)

Each tenant gets a dedicated redb file:

storage/
β”œβ”€β”€ tn_1.db      (Tenant 1 documents)
β”œβ”€β”€ tn_2.db      (Tenant 2 documents)
└── tn_3.db      (Tenant 3 documents)

Advantages:

  • βœ… Complete isolation between tenants
  • βœ… Independent backups per tenant
  • βœ… Easier to delete/archive specific tenants
  • βœ… Better fault isolation

Trade-offs:

  • ⚠️ More file handles required
  • ⚠️ Slightly higher disk overhead

Use case: Multi-tenant SaaS deployments where tenant isolation is critical

Single File Mode (per_tenant_files=false)

All tenants share one database:

storage/
└── crdt.db      (All tenants)

Advantages:

  • βœ… Fewer file handles
  • βœ… Simpler operational management
  • βœ… Easier bulk operations

Trade-offs:

  • ⚠️ No physical isolation between tenants
  • ⚠️ Tenant deletion requires filtering

Use case: Single-user deployments or trusted environments

In-Memory Document Instances

The adapter caches document instances in memory to optimize performance and enable real-time subscriptions.

DocumentInstance Structure

struct DocumentInstance {
    broadcaster: tokio::sync::broadcast::Sender<CrdtChangeEvent>,
    last_accessed: AtomicU64,  // For LRU eviction
    update_count: AtomicU64,   // Sequence counter
}

Each instance provides:

  • Broadcast channel: Real-time notifications to subscribed clients
  • Sequence counter: Monotonically increasing update numbers
  • LRU tracking: Timestamp for idle document eviction

Caching Strategy

Cache population:

  1. Document accessed β†’ Check cache (DashMap)
  2. If missing β†’ Create instance with broadcast channel
  3. Store in cache with initial timestamp

LRU eviction (configurable):

  • max_instances: Maximum cached documents (default: 100)
  • idle_timeout_secs: Evict after N seconds idle (default: 300s)
  • auto_evict: Enable/disable automatic eviction (default: true)

Benefits:

  • Avoid reopening redb transactions repeatedly
  • Enable efficient pub/sub without polling
  • Reduce memory usage for inactive documents

Core Operations

Storing Updates

When a client modifies a CRDT document:

1. Client sends binary update β†’ Server
2. Adapter fetches/creates DocumentInstance
3. Sequence number assigned (atomic increment)
4. Update stored: updates["{doc_id}:{seq}"] = update_bytes
5. Broadcast update to all subscribers
6. Return success

Atomicity: Each update is stored in a redb write transaction, ensuring crash consistency.

Key generation:

fn make_update_key(doc_id: &str, seq: u64) -> String {
    format!("{}:{}", doc_id, seq)
}

Loading Updates

When a client opens a document:

1. Prefix scan: updates.range("{doc_id}:"..)
2. Collect all matching keys (while prefix matches)
3. Read binary blobs in sequence order
4. Return Vec<CrdtUpdate>
5. Client applies updates to Yrs document β†’ reconstructs state

Performance: Prefix scans are efficient in redb’s B-tree structure.

Subscriptions

Clients can subscribe to document changes:

Without snapshot (new updates only):

let mut rx = instance.broadcaster.subscribe();
while let Ok(event) = rx.recv().await {
    // Send update to client
}

With snapshot (full sync):

// 1. Send all existing updates (from redb)
for update in get_updates(doc_id).await? {
    yield update;
}

// 2. Then stream new updates (from broadcaster)
let mut rx = instance.broadcaster.subscribe();
while let Ok(event) = rx.recv().await {
    yield event;
}

Snapshot mode enables new clients to:

  1. Receive complete document history
  2. Reconstruct current state
  3. Continue receiving live updates

Deleting Documents

1. Begin write transaction
2. Prefix scan to find all updates: "{doc_id}:"
3. Delete each update key
4. Delete metadata entry
5. Commit transaction
6. Remove from instance cache

Note: Currently no compaction strategy to reduce update count.

Configuration

struct AdapterConfig {
    max_instances: usize,        // Default: 100
    idle_timeout_secs: u64,      // Default: 300 (5 minutes)
    broadcast_capacity: usize,   // Default: 1000 messages
    auto_evict: bool,            // Default: true
}

Tuning guidance:

  • High traffic: Increase max_instances and broadcast_capacity
  • Memory constrained: Reduce max_instances, lower idle_timeout_secs
  • Long-running docs: Disable auto_evict or increase timeout

Update Compaction (Future Enhancement)

Currently, updates accumulate indefinitely. Potential future optimization:

Compaction strategy:

  1. When update count exceeds threshold (e.g., 1000)
  2. Load all updates and apply to Yrs document
  3. Encode current state as single snapshot update
  4. Replace all updates with snapshot
  5. Reset sequence counter

Benefits:

  • Faster document loading
  • Reduced storage usage
  • Shorter sync times for new clients

Trade-off: Loses granular edit history

Comparison with RTDB Storage

Aspect CRDT (crdt-adapter-redb) RTDB (rtdb-adapter-redb)
Data model Binary operation stream Structured JSON documents
Storage Sequential update blobs Hierarchical key-value
Queries None (replays all updates) Filters, sorting, pagination
Concurrency Automatic conflict resolution Last-write-wins
Use case Collaborative text editing Structured data with queries
Sync protocol Full update stream Partial updates via queries

Performance Characteristics

Write performance:

  • Insert update: O(log N) (B-tree insert)
  • Broadcast: O(M) where M = subscriber count
  • Typical: <1ms for small updates

Read performance:

  • Load document: O(K log N) where K = update count
  • Subscription: O(1) after initial load
  • Typical: 50-200ms for 1000 updates

Memory usage:

  • Per instance: ~1KB overhead
  • Broadcast buffer: broadcast_capacity Γ— avg_update_size
  • Total: ~100KB for 100 cached docs

See Also

Access Control

Cloudillo’s access control and permission systems for protecting resources while maintaining user privacy and enabling secure resource sharing across the decentralized network.

Core Subsystems

  • Access Control & Resource Sharing - Token-based authentication and authorization mechanisms
  • ABAC Permissions - Attribute-based access control (ABAC) system for fine-grained permissions

Key Concepts

When a user wants to access a resource stored on another node, Cloudillo uses cryptographic tokens to grant access without requiring direct trust between storage providers. This process is designed to be seamless, requiring no additional user interaction while maintaining full decentralization.

Subsections of Access Control

Access Control & Resource Sharing

Access tokens are used to authenticate and authorize requests to the API. They are usually bound to a resource, which can reside on any node within the Cloudillo network.

Token Types

Cloudillo uses different token types for different purposes:

AccessToken

Session tokens for authenticated API requests.

Purpose: Grant a client access to specific resources Format: JWT (JSON Web Token) Lifetime: 1-24 hours (configurable)

JWT Claims:

{
  "sub": "alice.example.com",   // Subject (user identity)
  "aud": "bob.example.com",     // Audience (target identity)
  "exp": 1738483200,            // Expiration timestamp
  "iat": 1738396800,            // Issued at timestamp
  "scope": "resource_id"        // Scope (resource identifier)
}

ActionToken

Cryptographically signed tokens representing user actions (see Actions.

Purpose: Federated activity distribution Format: JWT signed with profile key (ES384) Lifetime: Permanent (immutable, content-addressed)

ProxyToken

Inter-instance authentication tokens.

Purpose: Get access tokens from a remote instance (on behalf of a user) Format: JWT Lifetime: Short-lived (1-5 minutes)

JWT Claims:

{
  "sub": "alice.example.com",  // Requesting identity
  "aud": "bob.example.com",    // Target identity
  "exp": 1738400000,
  "iat": 1738396800,
  "scope": "resource_id"       // Scope (resource identifier)
}

Access Token Lifecycle

1. Token Request

Client requests an access token from their node.

2. Token Generation

The AuthAdapter creates a JWT with appropriate claims.

3. Token Validation

Every API request validates the token before processing.

4. Token Expiration

Tokens expire and must be refreshed.

Requesting an Access Token

When a user wants to access a resource, they follow this process:

  1. The user’s node requests an access token.
  2. If the resource is local, the node issues the token directly.
  3. If the resource is remote, the node authenticates with the remote node and requests a token on behalf of the user.
  4. The access token is returned to the user, allowing them to interact with the resource directly on its home node.

Security & Trust Model

  • Access tokens are cryptographically signed to prevent tampering.
  • Tokens have expiration times and scopes to limit misuse.
  • Nodes validate access tokens before granting access to a resource.

Example 1: Request access to own resource

sequenceDiagram
    box Alice frontend
        participant Alice shell
        participant Alice app
    end
    participant Alice node
    Alice shell ->>+Alice node: Initiate access token request
    Note right of Alice node: Create access token
    Alice node ->>+Alice shell: Access token granted
    deactivate Alice node
    Alice shell ->>+Alice app: Open resource with this token
    deactivate Alice shell
    Alice app ->+Alice node: Use access token
    loop Edit resource
        Alice app --> Alice node: Edit resource
    end
    deactivate Alice app
  • Alice opens a resource using her Cloudillo Shell
  • Her shell initiates an access token request at her node
  • Her node creates an access token and sends it to her shell
  • Her shell gives the access token to the App Alice uses to open the resource
  • The App uses the access token to edit the resource

Example 2: Request access to resource of an other identity

sequenceDiagram
    box Alice frontend
        participant Alice shell
        participant Alice app
    end
    participant Alice node
    participant Bob node
    Alice shell ->>+Alice node: Initiate access token request
    Note right of Alice node: Create signed request
    Alice node ->>+Bob node: Request access token
    Note right of Bob node: Verify signed request
    Note right of Bob node: Create access token
    deactivate Alice node
    Bob node ->>+Alice node: Grant access token
    deactivate Bob node
    Alice node ->>+Alice shell: Access token granted
    deactivate Alice node
    Alice shell ->>+Alice app: Open resource with this token
    deactivate Alice shell
    Alice app ->+Bob node: Use access token
    loop Edit resource
        Alice app --> Bob node: Edit resource
    end
    deactivate Alice app
    deactivate Bob node
  • Alice opens a resource using her Cloudillo Shell
  • Her shell initiates an access token request through her node
  • Her node creates a signed request and sends it to Bob’s node
  • Bob’s node creates an access token and sends it back to Alice’s node
  • Alice’s node sends the access token to her shell
  • Her shell gives the access token to the App Alice uses to open the resource
  • The App uses the access token to edit the resource

Token Validation Process

Authentication Middleware

Cloudillo uses Axum middleware to validate tokens on protected routes:

Handler Patterns:

Pattern 1: Required Authentication
async fn protected_handler(auth: Auth) -> Result<Response> {
    // auth.tn_id, auth.id_tag, auth.scope available
    // Access granted only if middleware validated token
}

Pattern 2: Optional Authentication
async fn public_handler(auth: Option<Auth>) -> Result<Response> {
    if let Some(auth) = auth {
        // Authenticated user - access Auth context
    } else {
        // Anonymous access - no Auth context
    }
}

The Axum extractor validates token before passing to handler.
If validation fails on required routes, request is rejected.

Validation Steps

When a request includes an Authorization: Bearer <token> header:

  1. Extract Token: Parse JWT from Authorization header
  2. Decode JWT: Parse header and claims (no verification yet)
  3. Verify Signature: Validate using AuthAdapter-stored secret
  4. Check Expiration: Ensure exp > current time
  5. Validate Claims: Check aud, scope, tid
  6. Create Auth Context: Build Auth struct for handler
pub struct Auth {
    pub tn_id: TnId,        // Tenant ID (database key)
    pub id_tag: String,     // Identity tag (e.g., "alice.example.com")
    pub scope: Vec<String>, // Permissions (e.g., ["read", "write"])
    pub token_type: TokenType,
}

Custom Extractors

Axum extractors provide typed access to authentication context:

TnId Extractor:

  • struct TnId(pub i64) - Wraps internal tenant ID
  • Usage: handler(TnId(tn_id): TnId) extracts from Auth context

IdTag Extractor:

  • struct IdTag(pub String) - Wraps user identity domain
  • Usage: handler(IdTag(id_tag): IdTag) extracts from Auth context

Auth Extractor (Full Context):

  • tn_id: Internal tenant identifier
  • id_tag: User identity (e.g., “alice.example.com”)
  • scope: Permission vector (e.g., [“read”, “write”])
  • token_type: Type of token (AccessToken, ProxyToken, etc.)

Usage: Check auth.scope.contains(&"write") for permission checks

Permission System

Cloudillo uses ABAC (Attribute-Based Access Control) for comprehensive permission management. Access tokens work in conjunction with ABAC policies to determine what actions users can perform.

Learn more: ABAC Permission System

Scope-Based Permissions

Access tokens include a scope claim that specifies permissions.

Resource-Level Permissions

Permissions are checked at multiple levels:

  1. File-Level: Who can access a file
  2. Database-Level: Who can access a database (RTDB)
  3. Action-Level: Who can see an action token
  4. API-Level: Rate limiting, quota enforcement

Permission Checking

Algorithm: Check Permission

Input: auth context, resource, required_scope
Output: Result<()>

1. Check token scope:
   - If required_scope NOT in auth.scope: Return PermissionDenied

2. Load resource metadata:
   - Fetch metadata by tn_id + resource_id

3. Check ownership:
   - If metadata.owner == auth.id_tag: Return OK (owner has all)

4. Check sharing list:
   - If auth.id_tag in metadata.shared_with: Return OK

5. Default: Return PermissionDenied

This pattern combines:
- Token-level permissions (scope)
- Resource-level ownership
- Resource-level sharing permissions

Cross-Instance Authentication

ProxyToken Flow

When Alice (on instance A) wants to access Bob’s resource (on instance B):

  1. Alice’s client requests access from instance A
  2. Instance A creates a ProxyToken signed with its profile key
  3. Instance A sends ProxyToken to instance B: POST /api/auth/proxy
  4. Instance B validates ProxyToken:
    • Fetches instance A’s public key
    • Verifies signature
    • Checks expiration
  5. Instance B creates AccessToken for Alice
  6. Instance B returns AccessToken to instance A
  7. Instance A returns AccessToken to Alice’s client
  8. Alice’s client uses AccessToken to access Bob’s resource directly on instance B

ProxyToken Verification

Algorithm: Verify ProxyToken

Input: JWT token string, requester_id_tag
Output: Result<ProxyTokenClaims>

1. Decode JWT without verification (read claims)

2. Fetch requester's profile:
   - GET /api/me from requester's instance
   - Extract public keys from profile

3. Find signing key:
   - Look up key by key_id (kid) in claims
   - If not found: Return KeyNotFound error

4. Verify signature:
   - Use requester's public key to verify JWT signature

5. Check expiration:
   - If exp < current_time: Return TokenExpired

6. Return verified claims

Token Generation

Creating an Access Token

Algorithm: Create Access Token

Input: tn_id, id_tag, resource_id, scope array, duration
Output: JWT token string

1. Build AccessTokenClaims:
   - sub: User identity (id_tag)
   - aud: Resource identifier (resource_id)
   - exp: current_time + duration
   - iat: current_time
   - scope: scope array joined as space-separated string
   - tid: Tenant ID (tn_id)

2. Sign JWT:
   - Use AuthAdapter to create JWT
   - Signed with instance's private key

3. Return token string

Token Refresh

Access tokens can be refreshed before expiration:

POST /api/auth/refresh
Authorization: Bearer <expiring_token>

Response:
{
  "access_token": "eyJhbGc...",
  "expires_in": 3600
}

API Reference

POST /api/auth/token

Request an access token.

Request:

POST /api/auth/token
Content-Type: application/json

{
  "resource_id": "f1~abc123...",
  "scope": "read write",
  "duration": 3600
}

Response (200 OK):

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "token_type": "Bearer",
  "expires_in": 3600,
  "scope": "read write"
}

POST /api/auth/refresh

Refresh an expiring access token.

Request:

POST /api/auth/refresh
Authorization: Bearer <current_token>

Response (200 OK):

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expires_in": 3600
}

POST /api/auth/proxy

Request an access token on behalf of a user (cross-instance).

Request:

POST /api/auth/proxy
Content-Type: application/json
Authorization: Bearer <proxy_token>

{
  "user_id_tag": "alice.example.com",
  "resource_id": "f1~xyz789...",
  "scope": "read"
}

Response (200 OK):

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expires_in": 3600
}

Security Considerations

Token Storage

Client-Side:

  • Store in memory or sessionStorage (not localStorage for better security)
  • Never log tokens
  • Clear on logout

Server-Side:

  • JWT signing secrets in AuthAdapter (never exposed)
  • Rotate signing secrets periodically
  • Use strong secrets (256+ bits)

Token Transmission

  • Always use HTTPS/TLS
  • Include in Authorization: Bearer <token> header
  • Never in URL query parameters

Token Revocation

Since JWTs are stateless, revocation requires:

  1. Short Lifetimes: Limit damage window (1-24 hours)
  2. Blacklisting: Maintain revoked token list (for critical cases)
  3. Key Rotation: Invalidates all tokens signed with old key

Rate Limiting

Protect token endpoints from abuse:

Token Endpoint Rate Limits:

  • MAX_TOKEN_REQUESTS_PER_HOUR: 100 (per user, per hour)

    • Prevents token request floods
    • Typical usage: <10 requests/hour
  • MAX_REFRESH_PER_TOKEN: 10 (per token)

    • Limits how many times a single token can be refreshed
    • Prevents refresh abuse
    • Encourages re-authentication after N refreshes

See Also

ABAC Permission System

Cloudillo uses Attribute-Based Access Control (ABAC) to provide flexible, fine-grained permissions across all resources. This system moves beyond simple role-based access control to enable sophisticated permission rules based on attributes of users, resources, and context.

What is ABAC?

Attribute-Based Access Control determines access by evaluating attributes rather than fixed roles. Instead of asking “Does this user have the admin role?”, ABAC asks “Does this user’s attributes match the policy rules for this resource?”

Benefits Over RBAC

Role-Based Access Control (RBAC):

if user.role == "admin" β†’ allow
if user.role == "editor" β†’ allow read/write
if user.role == "viewer" β†’ allow read

Fixed roles, limited flexibility

Attribute-Based Access Control (ABAC):

if user.connected_to(resource.owner) AND resource.visibility == "connected" β†’ allow
if resource.visibility == "public" β†’ allow
if current_time < resource.expires_at β†’ allow

Flexible rules based on any attributes

Key Advantages

βœ… Fine-Grained Control: Permission rules can use any attribute βœ… Context-Aware: Decisions based on time, location, relationship status βœ… Scalable: No need to create roles for every permission combination βœ… Decentralized: Resource owners define their own permission rules βœ… Expressive: Complex boolean logic with AND/OR operators


The Four-Object Model

ABAC decisions involve four types of objects:

1. Subject (Who)

The user or entity requesting access.

pub struct AuthCtx {
    pub tn_id: TnId,              // Tenant ID (database key)
    pub id_tag: String,           // Identity (e.g., "alice.example.com")
    pub roles: Vec<String>,       // Roles (e.g., ["user", "moderator"])
}

Attributes Available:

  • tn_id - Internal tenant identifier
  • id_tag - Public identity tag
  • roles - Assigned roles
  • Relationship to resource owner (computed at runtime)

Example:

Subject:
    id_tag: "bob.example.com"
    roles: ["user"]

2. Action (What)

The operation being attempted.

Format: resource:operation

Common Actions:

file:read        β†’ Read a file
file:write       β†’ Modify a file
file:delete      β†’ Delete a file
action:create    β†’ Create an action token
action:delete    β†’ Delete an action token
profile:update   β†’ Update a profile
profile:admin    β†’ Administrative access to profile

Format: resource:operation (e.g., file:read, action:create, profile:admin)

3. Object (Resource)

The resource being accessed. Must implement AttrSet trait to provide queryable attributes.

pub trait AttrSet {
    fn get_attr(&self, key: &str) -> Option<Value>;
}

Example - File Resource:

struct FileMetadata {
    file_id: String,
    owner: String,           // "alice.example.com"
    visibility: Visibility,  // public, private, followers, etc.
    created_at: i64,
    size: u64,
    shared_with: Vec<String>,
}

Implementing AttrSet:

FileMetadata.get_attr(key):
    switch key:
        case "owner":
            return Value::String(self.owner)
        case "visibility":
            return Value::String(self.visibility)
        case "created_at":
            return Value::Number(self.created_at)
        case "size":
            return Value::Number(self.size)
        default:
            return None

Attributes Available (depends on resource type):

  • owner - Resource owner’s identity
  • visibility - Visibility level
  • created_at - Creation timestamp
  • audience - Intended audience (for actions)
  • shared_with - List of identities with explicit access
  • Any custom attributes defined by the resource

4. Environment (Context)

Contextual factors like time, location, or system state.

pub struct Environment {
    pub current_time: i64,        // Unix timestamp
    pub request_origin: Option<String>,  // Future: IP, location
    pub system_load: Option<f32>, // Future: rate limiting context
}

Attributes Available:

  • current_time - Current Unix timestamp
  • Future: IP address, geographic location, system load

Example:

Environment:
    current_time: 1738483200
    request_origin: null
    system_load: null

Visibility Levels

Cloudillo defines six visibility levels that determine who can access a resource. Stored as a single character in the database (or NULL for direct).

Hierarchy (Most to Least Permissive)

Code Level Description
P Public Anyone, including unauthenticated users
V Verified Any authenticated user from any federated instance
2 SecondDegree Friend of friend (reserved for voucher token system)
F Follower Authenticated users who follow the owner
C Connected Authenticated users with mutual connection
NULL Direct Only owner + explicit audience

1. P - Public

Everyone can access the resource, even unauthenticated users.

Use Cases:

  • Public blog posts
  • Open documentation
  • Community announcements
  • Shared files with public links

Permission Logic:

if resource.visibility == 'P':
    return ALLOW

Example:

FileMetadata:
    owner: "alice.example.com"
    visibility: 'P'
    # Anyone can read this file

2. V - Verified

Authenticated users from any federated instance can access the resource.

Use Cases:

  • Semi-public posts requiring login
  • Community content for logged-in users
  • Content protected from anonymous scraping

Permission Logic:

if resource.visibility == 'V':
    if subject.is_authenticated:
        return ALLOW
    else:
        return DENY

Example:

ActionToken:
    owner: "alice.example.com"
    visibility: 'V'
    type: "POST"
    content: "Only for logged-in users"
    # Accessible to any authenticated user

3. 2 - SecondDegree (Reserved)

Friend of friend access - authenticated users who are connected to someone who is connected to the owner.

Info

This level is reserved for future implementation via a voucher token system.

Use Cases:

  • Extended network visibility
  • Gradual trust expansion
  • Community discovery

4. F - Follower

Followers of the owner can access the resource (plus the owner).

Use Cases:

  • Social media posts visible to followers
  • Updates shared with audience
  • Blog posts for subscribers

Permission Logic:

if resource.owner == subject.id_tag:
    return ALLOW
if subject.follows(resource.owner):
    return ALLOW
else:
    return DENY

Checking Following Status:

is_following = meta_adapter.has_action(subject.tn_id, "FLLW", resource.owner)
# Checks if subject has a FLLW action token for the resource owner

Example:

ActionToken:
    owner: "alice.example.com"
    visibility: 'F'
    type: "POST"
    content: "Hello followers!"
    # Accessible to anyone who follows alice

5. C - Connected

Connected users can access the resource (mutually connected users + owner).

A connection exists when both parties have issued CONN action tokens to each other.

Use Cases:

  • Private conversations
  • Shared documents between colleagues
  • Connection-only updates
  • Close friends content

Permission Logic:

if resource.owner == subject.id_tag:
    return ALLOW
if bidirectional_connection(subject.id_tag, resource.owner):
    return ALLOW
else:
    return DENY

Checking Connection Status:

# Check if both users have CONN tokens for each other
alice_to_bob = meta_adapter.has_action(alice_tn_id, "CONN", "bob.example.com")
bob_to_alice = meta_adapter.has_action(bob_tn_id, "CONN", "alice.example.com")
connected = alice_to_bob AND bob_to_alice

Example:

FileMetadata:
    owner: "alice.example.com"
    visibility: 'C'
    file_name: "project-proposal.pdf"
    # Only users connected to alice can access

6. NULL - Direct (Audience-Based)

Most restrictive - only the owner and users explicitly listed in the audience field can access.

Use Cases:

  • Direct messages to specific users
  • Files shared with specific people
  • Invitations to specific identities
  • Private resources

Permission Logic:

if resource.owner == subject.id_tag:
    return ALLOW
if resource.audience.contains(subject.id_tag):
    return ALLOW
else:
    return DENY

Example:

ActionToken:
    owner: "alice.example.com"
    type: "MSG"
    content: "Hi Bob!"
    audience: ["bob.example.com"]
    visibility: NULL
    # Only alice and bob can see this

Subject Access Levels

While Visibility Levels (P/V/2/F/C/NULL) define who can see a resource, Subject Access Levels are computed attributes that describe the relationship between a requesting user and a resource owner. These levels are used during permission evaluation.

Access Level Hierarchy

Level Code Description
Owner 6 The user owns the resource
Connected 5 Mutual connection with owner (bidirectional CONN)
Follower 4 Follows the owner (unidirectional FLLW)
SecondDegree 3 Friend of a friend (reserved for future voucher system)
Verified 2 Authenticated user from any instance
Public 1 Unauthenticated or anonymous user
None 0 No access (blocked or unknown)

How Access Levels Are Computed

compute_access_level(subject, resource_owner):
    # Check if subject is the owner
    if subject.id_tag == resource_owner:
        return AccessLevel::Owner

    # Check for mutual connection
    if has_bidirectional_connection(subject.id_tag, resource_owner):
        return AccessLevel::Connected

    # Check if subject follows owner
    if subject_follows(subject.id_tag, resource_owner):
        return AccessLevel::Follower

    # Check second-degree connection (future)
    if has_second_degree_connection(subject.id_tag, resource_owner):
        return AccessLevel::SecondDegree

    # Check if authenticated
    if subject.is_authenticated:
        return AccessLevel::Verified

    # Unauthenticated user
    return AccessLevel::Public

Access Level vs Visibility

The permission check compares the computed access level against the required visibility:

check_access(subject, resource):
    access_level = compute_access_level(subject, resource.owner)
    required_level = visibility_to_access_level(resource.visibility)

    return access_level >= required_level

Visibility to Access Level mapping:

Visibility Required Access Level
P (Public) Public (1)
V (Verified) Verified (2)
2 (SecondDegree) SecondDegree (3)
F (Follower) Follower (4)
C (Connected) Connected (5)
NULL (Direct) Owner or explicit audience

Attribute Set Implementations

Cloudillo implements the AttrSet trait for different resource types, providing consistent attribute access for permission evaluation.

ProfileAttrs

Attributes for user profile resources:

struct ProfileAttrs {
    id_tag: String,       // "alice.example.com"
    owner: String,        // Same as id_tag for profiles
    visibility: char,     // Profile visibility setting
    created_at: i64,      // Profile creation timestamp
    verified: bool,       // Is identity verified?
    roles: Vec<String>,   // Assigned roles
}

impl AttrSet for ProfileAttrs {
    fn get_attr(&self, key: &str) -> Option<Value> {
        match key {
            "id_tag" | "owner" => Some(Value::String(self.id_tag)),
            "visibility" => Some(Value::Char(self.visibility)),
            "created_at" => Some(Value::Number(self.created_at)),
            "verified" => Some(Value::Bool(self.verified)),
            "roles" => Some(Value::Array(self.roles)),
            _ => None
        }
    }
}

ActionAttrs

Attributes for action token resources:

struct ActionAttrs {
    action_id: String,    // "a1~xyz789..."
    owner: String,        // Issuer identity
    action_type: String,  // "POST", "CMNT", "REACT", etc.
    audience: Option<String>, // Intended recipient
    visibility: Option<char>, // P/V/2/F/C/NULL
    parent_id: Option<String>, // Parent action (for threading)
    created_at: i64,      // iat claim timestamp
    status: char,         // A/C/N/D
}

impl AttrSet for ActionAttrs {
    fn get_attr(&self, key: &str) -> Option<Value> {
        match key {
            "id" => Some(Value::String(self.action_id)),
            "owner" | "issuer" => Some(Value::String(self.owner)),
            "type" => Some(Value::String(self.action_type)),
            "audience" => self.audience.as_ref().map(|a| Value::String(a)),
            "visibility" => self.visibility.map(|v| Value::Char(v)),
            "parent" => self.parent_id.as_ref().map(|p| Value::String(p)),
            "created_at" => Some(Value::Number(self.created_at)),
            "status" => Some(Value::Char(self.status)),
            _ => None
        }
    }
}

FileAttrs

Attributes for file/blob resources:

struct FileAttrs {
    file_id: String,      // "f1~abc123..."
    owner: String,        // File owner identity
    visibility: char,     // P/V/F/C/NULL
    mime_type: String,    // "image/jpeg", "application/pdf"
    size: u64,            // File size in bytes
    created_at: i64,      // Upload timestamp
    shared_with: Vec<String>, // Explicit share list
}

impl AttrSet for FileAttrs {
    fn get_attr(&self, key: &str) -> Option<Value> {
        match key {
            "id" => Some(Value::String(self.file_id)),
            "owner" => Some(Value::String(self.owner)),
            "visibility" => Some(Value::Char(self.visibility)),
            "mime_type" => Some(Value::String(self.mime_type)),
            "size" => Some(Value::Number(self.size as i64)),
            "created_at" => Some(Value::Number(self.created_at)),
            "shared_with" => Some(Value::Array(self.shared_with)),
            _ => None
        }
    }
}

SubjectAttrs

Attributes for the requesting subject (user context):

struct SubjectAttrs {
    id_tag: String,       // "bob.example.com"
    tn_id: TnId,          // Internal tenant ID
    is_authenticated: bool,
    roles: Vec<String>,   // ["user", "moderator"]
    access_level: AccessLevel, // Computed for current resource
}

impl AttrSet for SubjectAttrs {
    fn get_attr(&self, key: &str) -> Option<Value> {
        match key {
            "id_tag" => Some(Value::String(self.id_tag)),
            "authenticated" => Some(Value::Bool(self.is_authenticated)),
            "roles" => Some(Value::Array(self.roles)),
            "access_level" => Some(Value::Number(self.access_level as i64)),
            _ => None
        }
    }
}

Using Attribute Sets in Policies

# Policy rule using attribute sets
Rule:
    Condition:
        (object.visibility == 'F') AND
        (subject.access_level >= 4)  # Follower or higher
    Effect: ALLOW

# Another rule combining multiple attributes
Rule:
    Condition:
        (object.type == 'POST') AND
        (object.owner == subject.id_tag OR subject.HasRole('moderator'))
    Effect: ALLOW_DELETE

Policy Structure

ABAC uses two-level policies to define permission boundaries:

TOP Policy (Constraints)

Defines maximum permissions - what is never allowed.

Example:

TopPolicy:
    Rule 1:
        Condition: visibility == "public" AND size > 100MB
        Effect: DENY
        # Files larger than 100MB cannot be shared publicly

    Rule 2:
        Condition: created_at < (current_time - 86400)
        Effect: DENY_WRITE
        # Action tokens cannot be modified after 24 hours

BOTTOM Policy (Guarantees)

Defines minimum permissions - what is always allowed.

Example:

BottomPolicy:
    Rule 1:
        Condition: subject.id_tag == resource.owner
        Effect: ALLOW
        # Owner can always access their own resources

    Rule 2:
        Condition: visibility == "public" AND action == "read"
        Effect: ALLOW
        # Public resources are always readable

Default Rules

Between TOP and BOTTOM policies, default rules apply based on visibility and ownership:

default_permission_check(subject, action, object):
    1. Check ownership:
       if object.owner == subject.id_tag
           return ALLOW

    2. Check visibility:
       switch object.visibility:
           case 'P':  # Public
               if action ends with ":read"
                   return ALLOW
           case 'V':  # Verified
               if subject.is_authenticated
                   return ALLOW
               return DENY
           case '2':  # SecondDegree
               return check_second_degree(subject, object)
           case 'F':  # Follower
               return check_following(subject, object)
           case 'C':  # Connected
               return check_connection(subject, object)
           case NULL:  # Direct
               return check_audience(subject, object)

    3. Default deny
       return DENY

Policy Operators

ABAC supports various operators for building permission rules:

Comparison Operators

Equals

visibility == "public"
subject.id_tag == resource.owner

NotEquals

status != "deleted"

GreaterThan / LessThan

size > 1,000,000
created_at < current_time

GreaterThanOrEqual / LessThanOrEqual

age >= 18
priority <= 5

Set Operators

Contains

"public" IN tags
subject.id_tag IN shared_with

NotContains

subject.id_tag NOT IN blocked_users

In

subject.role IN ["admin", "moderator"]

Role Operator

HasRole

subject.HasRole("admin")
subject.HasRole("moderator")

Logical Operators

And

(published == true) AND (visibility == "public")

Or

(subject.id_tag == resource.owner) OR (subject.HasRole("admin"))

Permission Evaluation Flow

When a permission check is requested:

1. Load Subject (user context from JWT)
   ↓
2. Load Object (resource with attributes)
   ↓
3. Load Environment (current time, etc.)
   ↓
4. Check TOP Policy (maximum permissions)
   β”œβ”€ If denied β†’ return Deny
   └─ If allowed β†’ continue
   ↓
5. Check BOTTOM Policy (minimum permissions)
   β”œβ”€ If allowed β†’ return Allow
   └─ If not matched β†’ continue
   ↓
6. Check Default Rules
   β”œβ”€ Ownership check
   β”œβ”€ Visibility check
   └─ Relationship checks
   ↓
7. Return Decision (Allow or Deny)

Evaluation Example

Request: Bob wants to read Alice’s file

Subject:
    id_tag: "bob.example.com"
    roles: ["user"]

Action: "file:read"

Object:
    owner: "alice.example.com"
    visibility: "connected"
    file_id: "f1~abc123"

Environment:
    current_time: 1738483200

Evaluation:
1. TOP Policy: No blocking rules β†’ continue
2. BOTTOM Policy: Not owner β†’ continue
3. Default Rules:
   a. Is owner? No (alice β‰  bob)
   b. Visibility = "connected"
   c. Check connection:
      - Alice has CONN to Bob? Yes
      - Bob has CONN to Alice? Yes
      - Result: Connected!
   d. Action is "read"? Yes
   β†’ ALLOW

Integration with Routes

Cloudillo uses permission middleware to enforce ABAC on HTTP routes:

Permission Middleware Layers

Routes are configured with permission middleware to enforce access control:

Protected Routes:
    # Actions
    GET  /api/action           + check_perm_action("read")
    POST /api/action           + check_perm_action("create")
    DEL  /api/action/:id       + check_perm_action("delete")

    # Files
    GET  /api/file/:id         + check_perm_file("read")
    PATCH /api/file/:id        + check_perm_file("write")
    DEL  /api/file/:id         + check_perm_file("delete")

    # Profiles
    PATCH /api/profile/:id     + check_perm_profile("update")
    PATCH /api/admin/profile/:id + check_perm_profile("admin")

Each middleware checks permissions before the handler executes.

Middleware Implementation

The permission middleware follows this flow:

check_perm_action(action):
    1. Extract auth context from request
    2. Extract resource ID from request path
    3. Load resource from storage adapter
    4. Call abac::check_permission(auth, "action:{action}", resource, environment)
    5. If allowed: proceed to next middleware/handler
    6. If denied: return Error::PermissionDenied

This middleware is applied to each route requiring permission checks (see Permission Middleware Layers above).


Examples

Example 1: Public File Access

Alice creates a public file:

POST /api/file/image/logo.png
Authorization: Bearer <alice_token>
Body: <image data>

Response:
    file_id: "f1~abc123"
    visibility: "public"

Bob reads the file (no connection needed):

GET /api/file/f1~abc123

Permission Check:
    Subject: bob.example.com
    Action: file:read
    Object: { owner: alice, visibility: public }
    Decision: ALLOW (public resources readable by anyone)

Example 2: Connected-Only File

Alice creates a connected-only file:

POST /api/file/document/private-notes.pdf
Authorization: Bearer <alice_token>
Body: { visibility: "connected" }

Response:
    file_id: "f1~xyz789"
    visibility: "connected"

Bob tries to read (not connected):

GET /api/file/f1~xyz789
Authorization: Bearer <bob_token>

Permission Check:
    Subject: bob
    Action: file:read
    Object: { owner: alice, visibility: connected }
    Connection: alice ↔ bob? NO
    Decision: DENY

Charlie tries to read (connected):

GET /api/file/f1~xyz789
Authorization: Bearer <charlie_token>

Permission Check:
    Subject: charlie
    Action: file:read
    Object: { owner: alice, visibility: connected }
    Connection: alice ↔ charlie? YES
    Decision: ALLOW

Example 3: Role-Based Access

Admin deletes any profile:

DELETE /api/admin/profile/bob.example.com
Authorization: Bearer <admin_token>

Permission Check:
    Subject: admin (roles: [admin])
    Action: profile:admin
    Object: { owner: bob }
    HasRole("admin")? YES
    Decision: ALLOW

Regular user tries same action:

DELETE /api/admin/profile/bob.example.com
Authorization: Bearer <alice_token>

Permission Check:
    Subject: alice (roles: [user])
    Action: profile:admin
    HasRole("admin")? NO
    Decision: DENY

Example 4: Time-Based Access

Define policy: documents expire after 30 days

TopPolicy:
    Rule:
        Condition: (expires_at < current_time) AND (action == "read")
        Effect: DENY

Bob tries to read expired document:

GET /api/file/f1~old123

Permission Check:
    Subject: bob
    Action: file:read
    Object: {
        owner: alice
        visibility: public
        expires_at: 1738400000  (in the past)
    }
    Environment: { current_time: 1738483200 }
    TOP Policy check:
        1738400000 < 1738483200? YES
    Decision: DENY (expired)

Implementing Custom Policies

Adding a Custom Policy

Define a custom policy for team files:

create_team_policy():
    top_rules:
        Rule 1:
            Condition: (type == "team") AND (visibility == "public")
            Effect: DENY
            Description: "Team files must not be public"

    bottom_rules:
        Rule 1:
            Condition: (type == "team") AND (subject.id_tag IN members) AND (action == "read")
            Effect: ALLOW
            Description: "Team members can read team files"

Apply in handler:

get_team_file(auth, file_id):
    file = meta_adapter.read_file(file_id)

    allowed = abac::check_permission_with_policy(
        auth,
        "file:read",
        file,
        Environment::current(),
        create_team_policy()
    )

    if NOT allowed:
        return Error::PermissionDenied

    return serve_file(file)

Defining Custom Attributes

struct TeamFile {
    file_id: String
    owner: String
    team_id: String
    members: List[String]
    file_type: String
}

Implementing AttrSet trait:

TeamFile.get_attr(key):
    switch key:
        case "owner":
            return Value::String(self.owner)
        case "team_id":
            return Value::String(self.team_id)
        case "members":
            return Value::Array(self.members)
        case "type":
            return Value::String(self.file_type)
        default:
            return None

Performance Considerations

Caching Relationship Checks

Following/connection checks can be expensive. Cache results to avoid repeated database queries:

RelationshipCache:
    ttl: Duration (cache validity)
    cache: HashMap[(user_a, user_b) β†’ (bool, timestamp)]

check_connection(user_a, user_b):
    1. Check cache for (user_a, user_b)
       if found AND cached_at.elapsed() < ttl
           return cached_result

    2. Query database
       connected = check_connection_db(user_a, user_b)

    3. Cache result
       cache[(user_a, user_b)] = (connected, now)

    4. Return result

Optimizing Policy Evaluation

For complex policies, evaluate cheapest conditions first:

// Bad: Expensive database check first
And(vec![
    Attr("members").Contains(subject.id_tag),  // DB query
    Attr("type").Equals("team"),               // Cheap
])

// Good: Cheap check first
And(vec![
    Attr("type").Equals("team"),               // Cheap, fails fast
    Attr("members").Contains(subject.id_tag),  // Only if needed
])

Permission Check Batching

When checking permissions for multiple resources, batch-fetch relationships to minimize database queries:

check_permissions_batch(auth, action, resources):
    1. Extract all resource owners
       owners = {r.owner for r in resources}

    2. Pre-fetch all relationships
       relationships = fetch_relationships_batch(auth.id_tag, owners)
       # Fetches all FLLW/CONN tokens in one query

    3. Check each resource
       for resource in resources:
           permission = check_permission_with_cache(
               auth, action, resource, relationships
           )
           results.append(permission)

    4. Return results

This avoids N+1 queries where each permission check would require separate lookups.


Security Best Practices

1. Default Deny

Always default to denying access unless explicitly allowed:

Good Pattern:
    check_permission(...):
        check all explicit allow conditions
        if any match: return ALLOW
        otherwise: return DENY

Bad Pattern:
    check_permission(...):
        check some conditions
        forget to handle unknown cases
        accidentally allows access

2. Validate on Both Sides

Check permissions in both locations:

  • Client-side: For UX (show/hide UI elements)

    • Don’t rely on this for security
    • Users can modify client-side checks
  • Server-side: For security (enforce access control)

    • Always validate, even if client already checked
    • Trust only server-side permission checks

Example flow:

Client:
    if canDeleteFile(auth, file):
        show delete button (UX convenience)

Server:
    delete_file(auth, file_id):
        file = load_file(file_id)
        if NOT check_permission(auth, "file:delete", file):
            return Error::PermissionDenied
        // Safe to proceed with deletion

3. Audit Permission Denials

Log all permission denials for security monitoring:

if NOT allowed:
    log warning:
        subject: auth.id_tag
        action: action
        resource: resource.id
        timestamp: current_time
        reason: "ABAC permission denial"

    return Error::PermissionDenied

Log fields should include subject identity, action attempted, resource ID, and timestamp for audit trails.

4. Test Permission Policies

Write comprehensive tests for all permission scenarios:

test_connected_file_access():
    # Setup
    alice = create_user("alice")
    bob = create_user("bob")
    charlie = create_user("charlie")

    create_connection(alice, bob)  # Alice ↔ Bob
    file = create_file(alice, visibility="connected")

    # Test cases
    assert check_permission(alice, "file:read", file)      # Owner β†’ ALLOW
    assert check_permission(bob, "file:read", file)        # Connected β†’ ALLOW
    assert NOT check_permission(charlie, "file:read", file) # Not connected β†’ DENY

Test all visibility levels (P, V, 2, F, C, NULL) and edge cases (expired resources, role-based access, custom attributes).


Troubleshooting

Permission Denied But Should Be Allowed

Debugging steps:

  1. Check visibility level:

    visibility = resource.visibility
    # Expected: 'P' | 'V' | '2' | 'F' | 'C' | NULL
    # (Public | Verified | SecondDegree | Follower | Connected | Direct)
  2. Check ownership:

    owner = resource.owner
    subject_id = subject.id_tag
    # If owner == subject_id, should have full access
  3. Check relationship status:

    following = check_following(subject, resource)
    connected = check_connection(subject, resource)
    # Verify FLLW and/or CONN tokens exist if needed
  4. Check action format:

    Wrong:  Action("read")           # Missing resource type
    Correct: Action("file:read")      # Includes resource type format
  5. Enable debug logging:

    RUST_LOG=cloudillo::core::abac=debug cargo run
    # Shows decision flow and matched rules

Relationship Checks Not Working

Common issues:

  1. Missing action tokens: Ensure FLLW/CONN tokens exist

    Check for FLLW token: meta_adapter.read_action(tn_id, "FLLW", target)
    Check for CONN token: meta_adapter.read_action(tn_id, "CONN", target)
    # If not found, relationship check will return false
  2. Unidirectional connection: Both sides need CONN tokens

    Alice β†’ Bob: CONN token exists
    Bob β†’ Alice: CONN token MISSING
    Result: NOT connected (requires bidirectional tokens)
  3. Cache staleness: Clear relationship cache if stale

    relationship_cache.clear()
    # Cache holds results for TTL duration; manually clear if needed

See Also

File Processing Pipeline

Overview

Cloudillo processes uploaded files through an asynchronous pipeline that generates multiple variants optimized for different use cases. The system uses FFmpeg for multimedia processing and supports images, videos, audio, and PDFs.

Processing Architecture

File Upload
    ↓
Store original blob
    ↓
Create FileIdGeneratorTask
    ↓
Detect file type (MIME)
    ↓
Generate variants (async)
    β”œβ”€ Image: thumbnails, SD, MD, HD
    β”œβ”€ Video: transcoded variants, thumbnails
    β”œβ”€ Audio: normalized, compressed
    └─ PDF: text extraction, thumbnails
    ↓
Create file descriptor
    ↓
Content-address all variants
    ↓
Return file ID (f1~...)

Supported File Types

Images

Format Extensions Processing
JPEG .jpg, .jpeg Resize, AVIF/WebP conversion
PNG .png Resize, AVIF/WebP conversion
GIF .gif First frame extraction, resize
WebP .webp Resize only
AVIF .avif Resize only
Image Format Selection

Thumbnails use AVIF for best compression. Larger variants (SD, MD, HD) use WebP for faster encoding while maintaining good quality.

Video

Format Extensions Processing
MP4 .mp4 H.264 transcode, thumbnails
WebM .webm H.264 transcode, thumbnails
MOV .mov H.264 transcode, thumbnails
MKV .mkv H.264 transcode, thumbnails

Audio

Format Extensions Processing
MP3 .mp3 OPUS conversion
WAV .wav OPUS conversion
OGG .ogg OPUS conversion
FLAC .flac OPUS conversion
M4A .m4a OPUS conversion
OPUS .opus Normalization only

Documents

Format Extensions Processing
PDF .pdf Text extraction, page thumbnails

Variant System

Cloudillo uses a two-level variant system with format <class>.<quality>:

vis.sd   β†’  class: visual (image), quality: standard definition
vid.hd   β†’  class: video, quality: high definition
aud.md   β†’  class: audio, quality: medium

Variant Classes

Class Code Description Source Types
Visual vis Static images JPEG, PNG, WebP, AVIF, GIF
Video vid Video content MP4, WebM, MKV, AVI, MOV
Audio aud Audio tracks MP3, WAV, OGG, FLAC, AAC, OPUS
Document doc Documents PDF
Raw raw Original file Any (unprocessed)

Quality Levels

Quality Code Description
Profile pf 80px - Profile pictures
Thumbnail tn 128px - Small previews
Standard sd 720px - Mobile/low bandwidth
Medium md 1280px - Desktop viewing
High hd 1920px - High quality
Extra xd 3840px - 4K/maximum quality
Original orig Unprocessed source file

Visual Variants (Images)

Variant Max Size Format Use Case
vis.pf 80Γ—80 AVIF Profile pictures
vis.tn 128Γ—128 AVIF Thumbnails, listings
vis.sd 720px WebP Mobile, previews
vis.md 1280px WebP Desktop viewing
vis.hd 1920px WebP High quality display
vis.xd 3840px WebP 4K displays
orig - Original Source file

Video Variants

Variant Max Resolution Bitrate Use Case
vid.sd 720px 1.5 Mbps Mobile, low bandwidth
vid.md 1280px 3 Mbps Desktop
vid.hd 1920px 5 Mbps High quality
vid.xd 3840px 15 Mbps 4K playback

Video processing also extracts a vis.tn thumbnail from the first few seconds.

Audio Variants

Variant Format Bitrate Use Case
aud.sd OPUS 64 kbps Low bandwidth
aud.md OPUS 128 kbps Normal playback
aud.hd OPUS 256 kbps High quality

Document Variants

Variant Description
doc.orig Original PDF
vis.tn Thumbnail of first page

Variant Fallback

When a requested variant isn’t available, the system falls back to lower quality:

Request: vis.hd
Fallback chain: vis.md β†’ vis.sd β†’ vis.tn

File Descriptor Format

File descriptors encode all variant information:

d2,vis.tn:b1~abc123:f=avif:s=4096:r=128x128;vis.sd:b1~def456:f=webp:s=32768:r=720x540;vid.hd:b1~xyz789:f=mp4:s=5242880:r=1920x1080:dur=120.5:br=5000
Component Description
d2, Descriptor version prefix
; Variant separator
vis.tn, vid.hd Two-level variant code
b1~... Blob ID (SHA-256 hash)
f= Format (avif, webp, mp4, opus)
s= Size in bytes
r= Resolution (WxH)
dur= Duration in seconds (video/audio)
br= Bitrate in kbps (video/audio)
pg= Page count (PDFs)

Processing Presets

Presets define which variants to generate for different use cases:

Preset Visual Video Audio Use Case
default vis.tn, vis.sd, vis.md, vis.hd vid.sd, vid.md, vid.hd aud.sd, aud.md, aud.hd General uploads
profile_picture vis.pf, vis.tn, vis.sd - - Profile images
cover vis.tn, vis.sd, vis.md, vis.hd - - Cover/banner images
high_quality vis.tn β†’ vis.xd vid.sd β†’ vid.xd aud.md, aud.hd Maximum quality
mobile vis.tn, vis.sd, vis.md vid.sd, vid.md aud.sd Optimized for mobile
archive vis.tn only - - Minimal (keeps original)
podcast - - aud.sd, aud.md, aud.hd Audio extraction
video vis.tn (thumbnail) vid.sd β†’ vid.hd - Video-focused

All presets preserve the original file as orig unless configured otherwise.

FFmpeg Integration

Video and audio processing uses FFmpeg:

Video Transcoding

ffmpeg -i input.mov \
  -c:v libx264 -preset medium -crf 23 \
  -c:a aac -b:a 128k \
  -vf "scale=1280:720:force_original_aspect_ratio=decrease" \
  output.mp4

Audio Transcoding

ffmpeg -i input.mp3 \
  -c:a libopus -b:a 96k \
  output.opus

Audio Extraction

From video files:

ffmpeg -i input.mp4 \
  -vn -c:a libopus -b:a 96k \
  output.opus

Thumbnail Generation

ffmpeg -i input.mp4 \
  -ss 00:00:01 -vframes 1 \
  -vf "scale=150:150:force_original_aspect_ratio=decrease" \
  thumbnail.jpg

Content-Addressing

All variants are content-addressed:

  1. Blob level: Raw bytes β†’ b1~{SHA256(bytes)}
  2. Descriptor level: Descriptor string β†’ f1~{SHA256(descriptor)}

This enables:

  • Deduplication: Identical files share blobs
  • Verification: Hashes prove integrity
  • Caching: Immutable content can be cached forever

Task Scheduling

File processing uses the task scheduler:

FileUploadTask
    ↓
Creates FileIdGeneratorTask (depends on upload)
    ↓
FileIdGeneratorTask generates variants
    ↓
Action can reference file (depends on FileIdGeneratorTask)

Dependencies ensure actions only reference fully processed files.

Federation Sync

When syncing files across instances:

Metadata-Only Sync

For efficiency, only file descriptors are synced initially:

  1. Receive action with file attachment
  2. Fetch file descriptor from origin
  3. Store descriptor locally
  4. Fetch variants on demand

Variant Fetching

Client requests file
    ↓
Check if variant exists locally
    ↓
Yes β†’ Serve from local storage
    ↓
No β†’ Fetch from origin server
    Store locally
    Serve to client

Error Handling

Error Action
Unsupported format Reject upload with error
FFmpeg failure Log, mark as failed, allow retry
Storage full Queue for retry, alert admin
Timeout Retry with extended timeout

See Also

Actions & Federation

Cloudillo’s event-driven action system and federation architecture. These documents explain how users perform actions (posting, following, connecting), how actions are distributed across the network, and how independent instances communicate.

Core Subsystems

Action Tokens

Cryptographically signed events representing user activities and interactions. Actions enable event-driven communication between nodes in a federated network.

  • Actions & Action Tokens - Overview of the action token system and token types
  • Action Token Categories:
    • User Relationships - Connection and follow tokens
    • Content Actions - Posts, reposts, comments, reactions, endorsements
    • Communication - Direct messages and conversations
    • Metadata - Statistics, acknowledgments, file sharing

Federation

Cloudillo’s federated architecture enables independent instances to communicate, share content, and enable collaboration while maintaining user sovereignty and privacy.

  • Federation Architecture - Decentralization principles, inter-instance communication, and federation protocols

Subsections of Actions & Federation

Actions & Action Tokens

An Action Token represents a user action within Cloudillo. Examples of actions include creating a post, adding a comment, leaving a like, or performing other interactions.

Why Action Tokens?

Traditional social platforms store your posts, likes, and comments in their private databases. If the platform disappears, so does your content. Cloudillo takes a different approach: your actions are portable, verifiable, and truly yours.

Think of action tokens like signed letters:

  • Anyone can verify who wrote them (cryptographic signature)
  • They can be delivered to any server (federation)
  • They can’t be tampered with without detection (content-addressing)
  • They belong to you, not to any platform (decentralization)

Real-world example: When Alice posts a photo, her server creates a signed action token. This token can be delivered to Bob’s server (federation), verified as authentic (no trust required), and displayed in Bob’s feed. If Alice’s server goes offline, Bob still has a cryptographic proof that Alice created that post.

Key benefits:

  • Portable identity: Your actions follow your identity, not a server
  • Trustless verification: Anyone can verify authenticity without trusting intermediaries
  • Censorship-resistant: No single entity controls your content
  • Offline-capable: Actions can be verified without network access

Each Action Token is:

  • Cryptographically signed by it’s creator.
  • Time-stamped with an issue time.
  • Structured with relevant metadata about the action.

Action tokens are implemented as JSON web tokens (JWTs).

Action Token Fields

Standard JWT Claims

Field Type Required Description
iss identity * The identity of the creator of the Action Token.
aud identity The audience of the Action Token (recipient for directed actions).
sub identity The subject of the Action Token (references content/user WITHOUT creating hierarchy).
iat timestamp * The time when the Action Token was issued.
exp timestamp The time when the Action Token will expire.

Cloudillo-Specific Claims

Field Type Required Description
k string * The ID of the key the identity used to sign the Token.
t string * The type of the Action Token (e.g., POST, CMNT, FLLW).
c string / object The content of the Action Token (specific to the token type).
p string The ID of the parent token (creates hierarchical threading - CMNT only).
a string[] The IDs of the attachments (file references).
vis char Visibility level: P=Public, V=Verified, F=Follower, C=Connected, null=Direct.
f string Capability flags: R/r (reactions enabled/disabled), C/c (comments), O/o (open).
_ string Nonce for proof-of-work (used in CONN actions for rate limiting).

Field Semantics

Parent (p) vs Subject (sub):

  • p (parent): Creates TRUE hierarchy (threading). Only used for CMNT tokens to form comment chains.
  • sub (subject): References content WITHOUT creating hierarchy. Used for reactions, follows, connections.

Visibility (vis):

  • P - Public: Anyone can view
  • V - Verified: Only authenticated users
  • F - Follower: Only user’s followers
  • C - Connected: Only mutual connections
  • null - Direct: Only owner + explicit audience

Flags (f):

  • Uppercase = enabled, lowercase = disabled
  • R/r - Reactions allowed
  • C/c - Comments allowed
  • O/o - Open (anyone can interact)

Action Status Codes

Each action has a lifecycle status that determines how it appears in the UI and whether it requires user interaction:

Code Name Description
A Active Action is active, accepted, or approved. Normal operational state.
C Confirmation Awaiting user confirmation (e.g., connection requests, invitations). Shows in notifications.
N Notification Informational only, auto-processed. No user action required.
D Deleted Action has been deleted or rejected. Excluded from most queries.
P Pending (Files only) Awaiting processing before becoming active.

Status Transitions

         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
         β”‚                                              β”‚
         β–Ό                                              β”‚
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”     user accepts      β”Œβ”€β”€β”€β”€β”€β”€β”€β”          β”‚
 ───►│   C   β”‚ ─────────────────────►│   A   β”‚          β”‚
     β”‚ Conf  β”‚                       β”‚Active β”‚          β”‚
     β””β”€β”€β”€β”¬β”€β”€β”€β”˜                       β””β”€β”€β”€β”€β”€β”€β”€β”˜          β”‚
         β”‚                               β”‚              β”‚
         β”‚ user rejects                  β”‚ user deletes β”‚
         β”‚                               β”‚              β”‚
         β–Ό                               β–Ό              β”‚
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”                       β”Œβ”€β”€β”€β”€β”€β”€β”€β”          β”‚
     β”‚   D   │◄──────────────────────│   D   β”‚β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
     β”‚Delete β”‚                       β”‚Delete β”‚
     β””β”€β”€β”€β”€β”€β”€β”€β”˜                       β””β”€β”€β”€β”€β”€β”€β”€β”˜

Examples:

  • CONN request: Arrives as C β†’ user accepts β†’ becomes A (mutual connection)
  • INVT to conversation: Arrives as C β†’ user accepts β†’ creates SUBS with status A
  • Mutual CONN: When both users have sent CONN β†’ auto-accepted β†’ status N
  • DELETE subtype: Changes target action status to D

Merkle Tree Structure

Cloudillo’s action system implements a merkle tree structure where every action, file, and attachment is content-addressed using SHA-256 hashing. This creates cryptographic proof of authenticity and immutability through a six-level hierarchy:

  1. Blob Data β†’ hashed to create Variant IDs (b1~...)
  2. File Descriptor β†’ hashed to create File IDs (f1~...)
  3. Action Token JWT β†’ hashed to create Action IDs (a1~...)
  4. Parent References β†’ create immutable chains between actions
  5. Attachment References β†’ bind files to actions cryptographically
  6. Complete DAG β†’ forms a verifiable directed acyclic graph

Each level is tamper-evident: modifying any content changes all parent hashes, making tampering immediately detectable.

See Content-Addressing & Merkle Trees for complete details on how this creates proof of authenticity for all resources.

Attachment and Token IDs

Attachment and token IDs are derived using SHA256 hashes of their content, creating a multi-level content-addressing hierarchy:

All identifiers use SHA-256 content-addressing:

  • Action IDs (a1~): Hash of entire JWT token β†’ a1~8kR3mN9pQ2vL6xW...
  • File IDs (f1~): Hash of descriptor string β†’ f1~Qo2E3G8TJZ2HTGh...
  • Variant IDs (b1~): Hash of raw blob bytes β†’ b1~abc123def456ghi...

See Content-Addressing & Merkle Trees for detailed hash computation.

Verification Chain

This creates a verifiable chain of hashes:

Action (a1~8kR...)
  β”œβ”€ Signed by user (ES384)
  β”œβ”€ Content-addressed (SHA256 of JWT)
  └─ Attachments: [f1~Qo2...]
       └─ File (f1~Qo2...)
            β”œβ”€ Content-addressed (SHA256 of descriptor)
            └─ Descriptor: "d2,vis.tn:b1~abc...;vis.sd:b1~def..."
                 β”œβ”€ Variant vis.tn (b1~abc...)
                 β”‚   └─ Content-addressed (SHA256 of blob)
                 β”œβ”€ Variant vis.sd (b1~def...)
                 β”‚   └─ Content-addressed (SHA256 of blob)
                 └─ Variant vis.md (b1~ghi...)
                     └─ Content-addressed (SHA256 of blob)

Properties:

  • Immutable: Content cannot change without changing all IDs
  • Verifiable: Anyone can recompute hashes to verify integrity
  • Deduplicate: Identical content produces identical IDs
  • Tamper-evident: Any modification breaks the hash chain

Overriding Action Tokens

  • Each token type is linked to a database key, allowing previous tokens to be overridden where applicable.
  • The database key always contains the “iss” (issuer) field and may include other relevant fields.
  • Example: A REACT token (representing a reaction to a post) uses a key composed of “iss” and “p” (parent post ID). If a user reacts to the same post multiple times, the latest reaction replaces the previous one.

Root ID Handling

Important: The root_id field is NOT included in the action token JWT.

  • root_id is stored in the database for query optimization
  • It is a computed field, derived by traversing the parent chain to find the root action
  • It is NOT cryptographically signed (not in the JWT payload)
  • Recipients must compute root_id by following parent references

Why Root ID is Computed

This design choice has several benefits:

  • Smaller JWT payload: Keeps tokens compact and efficient
  • Avoids redundancy: Root ID can be derived from the parent chain
  • Maintains flexibility: Thread structure can be recomputed if needed
  • No signature burden: No need to sign derived data

Finding the Root

To find the root of an action thread:

find_root_id(action_id):
    action = fetch_action(action_id)
    if action.parent_id exists:
        return find_root_id(action.parent_id)  // Recursive
    else:
        return action.id  // Found root

Database Optimization

The computed root_id is stored in the database to enable efficient thread queries:

-- Find all actions in a thread
SELECT * FROM actions
WHERE root_id = 'a1~abc123...'
ORDER BY created_at;

-- Without root_id, this would require recursive parent traversal!

Action Creation Pipeline

Local Action Creation

When a user creates an action (e.g., posting, commenting), the following process occurs:

  1. Client Request
POST /api/actions
Authorization: Bearer <access_token>
Content-Type: application/json

{
  "type": "POST",
  "content": "Hello, Cloudillo!",
  "attachments": ["f1~abc123..."],
  "audience": "alice.example.com"
}
  1. Task Scheduling

Server creates a task and waits for dependencies (e.g., file attachments to be processed).

  1. JWT Creation & Signing
Build JWT with claims:
  iss: user's identity
  t: action type (POST, CMNT, etc.)
  c: content payload
  p: parent action ID (if reply/reaction)
  a: file attachment IDs
  iat: timestamp
  k: signing key ID

Sign JWT with ES384 (ECDSA P-384 curve)
  1. Storage & Response

Return the action ID to the client:

{
  "action_id": "a1~xyz789...",
  "token": "eyJhbGc..."
}

Complete Flow Diagram

Client
  ↓ POST /api/actions
Server creates ActionCreatorTask
  ↓
Scheduler checks dependencies
  ↓
Wait for FileIdGeneratorTask (if attachments)
  ↓
ActionCreatorTask runs
  β”œβ”€ Build JWT claims
  β”œβ”€ Fetch private key from AuthAdapter
  β”œβ”€ Sign JWT (ES384)
  β”œβ”€ Compute action ID (SHA256)
  └─ Store in MetaAdapter
  ↓
Response to client

Action Verification Pipeline

Federated Action Reception

When a remote instance sends an action (federation), it arrives at /api/inbox:

  1. Inbound Request
POST /api/inbox
Content-Type: application/json

{
  "token": "eyJhbGc..."
}
  1. Verification Steps
Decode JWT (extract issuer ID)
  ↓
Fetch issuer's public key
  GET https://cl-o.{issuer}/api/me
  ↓
Verify JWT signature (ES384)
  ↓
Check permissions:
  - Comments/reactions: verify parent ownership
  - Connections/follows: verify audience matches
  - Posts: verify following/connected status
  ↓
Sync attachments (if any)
  GET https://cl-o.{issuer}/api/file/{id}
  ↓
Store action locally

Complete Verification Flow

Remote Instance
  ↓ POST /api/inbox
Create ActionVerifierTask
  ↓
Decode JWT (unverified)
  ↓
Fetch issuer's public keys
  ↓ GET https://cl-o.{issuer}/api/me
Verify JWT signature (ES384)
  ↓
Check expiration
  ↓
Verify permissions
  β”œβ”€ Following/Connected status
  β”œβ”€ Audience matches
  └─ Parent ownership (for replies)
  ↓
Sync attachments (if any)
  ↓ GET https://cl-o.{issuer}/api/file/{id}
Store in MetaAdapter
  ↓
Trigger hooks (notifications, etc.)

Detailed Processing Pipelines

The following sections describe the complete processing pipelines implemented in the codebase.

Inbound Pipeline (12 Steps)

When receiving a federated action at /api/inbox:

Step Operation Description
1 Decode Parse JWT token without verification
2 PoW Check Verify proof-of-work nonce (for CONN actions)
3 Signature Verify Fetch public key, verify ES384 signature
4 DSL Validation Validate against action type definition schema
5 Permission Check Verify sender has permission (following/connected)
6 Subscription Check For subscribable actions, verify subscription exists
7 Store Action Persist to MetaAdapter with computed action ID
8 Execute Hooks Run on_receive hooks from DSL definition
9 WebSocket Forward Broadcast to connected tenant clients
10 Fan-out Create delivery tasks for related actions
11 Related Actions Process APRV fan-out to followers
12 ACK Response Generate acknowledgment token if required

Outbound Pipeline (8 Steps)

When a user creates an action via /api/actions:

Step Operation Description
1 Validate Check request parameters against DSL schema
2 Serialize Build JWT claims (iss, iat, t, c, p, a, etc.)
3 Generate Sign JWT with user’s private key (ES384)
4 Compute ID Calculate action ID as SHA256(token)
5 Store Persist to MetaAdapter
6 Execute Hooks Run on_create hooks from DSL definition
7 WebSocket Forward Notify local connected clients
8 Delivery Tasks Schedule ActionDeliveryTask for each recipient

Action Retrieval

GET /api/actions

Retrieve actions owned by or visible to the authenticated user:

Request:

GET /api/actions?type=POST&limit=50&offset=0
Authorization: Bearer <access_token>

Response (200 OK):

{
  "actions": [
    {
      "id": "a1~xyz789...",
      "type": "POST",
      "issuer": "alice.example.com",
      "content": "Hello, Cloudillo!",
      "attachments": ["f1~abc123..."],
      "created_at": 1738483200,
      "token": "eyJhbGc..."
    }
  ],
  "total": 150,
  "limit": 50,
  "offset": 0
}

GET /api/actions/:id

Retrieve a specific action by ID:

Request:

GET /api/actions/a1~xyz789...
Authorization: Bearer <access_token>

Response (200 OK):

{
  "id": "a1~xyz789...",
  "type": "POST",
  "issuer": "alice.example.com",
  "content": "Hello, Cloudillo!",
  "attachments": ["f1~abc123..."],
  "parent": null,
  "created_at": 1738483200,
  "token": "eyJhbGc..."
}

Federation & Distribution

Outbound Distribution

Determine recipients based on action type:
  - POST: send to all followers
  - CMNT/REACT: send to parent action owner
  - CONN/FLLW: send to audience

For each recipient:
  POST https://cl-o.{recipient}/api/inbox
  Body: {"token": "eyJhbGc..."}

The /api/inbox endpoint is public (no authentication required) because the action token itself contains the cryptographic proof of authenticity.

Security Considerations

Action Token Immutability

Action tokens are content-addressed using SHA-256: action_id = SHA256(entire_jwt_token).

This includes:

  • Header (algorithm, type)
  • Payload (issuer, content, attachments, timestamps, etc.)
  • Signature (cryptographic proof of authorship)

Immutability properties:

  • Tokens cannot be modified without changing the ID
  • Duplicate actions are automatically deduplicated
  • References to actions are tamper-proof
  • Parent references create immutable chains
  • Attachment references are cryptographically bound

Merkle Tree Verification

The content-addressing system creates a merkle tree that can be verified at multiple levels:

  1. Signature Verification: Verify the action was created by the claimed author using their public key
  2. Action ID Verification: Recompute action_id and verify it matches (proves no tampering)
  3. Parent Chain Verification: Recursively verify parent actions exist and are valid
  4. Attachment Verification: Verify file descriptors and blob variants match their hashes

See Content-Addressing & Merkle Trees for complete verification examples.

Complete Verification

After all levels are verified:

  • βœ… Author identity confirmed (signature)
  • βœ… Action content confirmed (action_id hash)
  • βœ… Parent references confirmed (recursive verification)
  • βœ… File attachments confirmed (file and variant hashes)
  • βœ… Complete merkle tree verified

See Content-Addressing & Merkle Trees for detailed verification examples.

Signature Verification

Every federated action undergoes cryptographic verification:

  1. Signature: Proves the issuer created the action
  2. Key Ownership: Public key fetched from issuer’s /api/me
  3. Expiration: Optional exp claim prevents token replay
  4. Audience: Optional aud claim ensures intended recipient

Spam Prevention

Multiple mechanisms prevent spam:

  1. Relationship Requirements: Only receive actions from connected/followed users
  2. Rate Limiting: Limit actions per user per time period
  3. Proof of Work: Required for CONN (connection request) actions via the _ nonce field, preventing spam connection requests
  4. Reputation: (Future) Trust scores based on user behavior

Action Token Types

User Relationships

CONN - Connect
Represents one side of a connection between two profiles. A connection is established when both parties issue a connection token to each other.
FLLW - Follow
Represents a follow relationship (a profile follows another profile).

Content

POST - Post
Represents a post created by a profile. Can include text, images, videos, or other attachments.
REPOST - Repost/Share
Represents the reposting/sharing of another user’s content to your profile.
CMNT - Comment
Represents a comment attached to another token (post, comment, etc.).
REACT - Reaction
Represents a reaction (like, emoji, etc.) to another token.

Communication

MSG - Message
Represents a direct message sent from one profile to another, or a message within a conversation.
CONV - Conversation
Represents a group conversation (group chat) between multiple participants.
SUBS - Subscription
Represents a subscription to a subscribable action (e.g., joining a conversation).
INVT - Invitation
Represents an invitation for a user to join a subscribable action.

Metadata

STAT - Statistics
Represents statistics about another token (number of reactions, comments, views, etc.).
APRV - Approval
Represents an approval of another user’s content, enabling federated fan-out to followers.
PRES - Presence
Represents ephemeral real-time presence (typing indicators, online status). NOT persisted to database.

File Sharing

FSHR - File Share
Represents sharing a file with another user. Requires acceptance. Supports subtypes: WRITE (grant write permission), DEL (revoke share).

Identity Provider

IDP:REG - Identity Registration
Represents an identity registration request to an Identity Provider instance. Enables federated identity creation for community-owned identities.

Native Hook Registry

Action types have lifecycle hooks that execute during specific events. These hooks are implemented in native Rust code for performance and security:

Action Type on_create on_receive on_accept on_reject
CONN βœ“ βœ“ βœ“ βœ“
FLLW βœ“ βœ“ - -
REACT βœ“ βœ“ - -
CMNT βœ“ βœ“ - -
FSHR - βœ“ βœ“ -
SUBS βœ“ βœ“ - -
CONV βœ“ βœ“ - -
INVT βœ“ βœ“ βœ“ -
APRV - βœ“ - -
IDP:REG - βœ“ - -

Hook Descriptions:

  • on_create: Executes when the local user creates this action type
  • on_receive: Executes when a federated action of this type arrives at /api/inbox
  • on_accept: Executes when the user accepts a confirmation action (CONN, FSHR, INVT)
  • on_reject: Executes when the user rejects a confirmation action (CONN only)
Info

POST, MSG, STAT, and PRES action types do not have native hooks - they use the default action processing pipeline.

TODO

Permissions

  • Public
  • Followers
  • Connections
  • Tags

Flags

  • Can react
  • Can comment
  • With permissions?

Tokens

Review
Represents a review post with a rating, attached to something
Patch
Represents a patch of a token (modify the content, or flags for example)
Resource
Represents a resource that can be booked (e.g. a conference room)
Book
Represents a booking of a resource.

Complete Example: LIKE β†’ POST β†’ Attachments β†’ Variants

This example demonstrates the complete merkle tree structure from a LIKE action down to the individual image blob bytes.

Example Data

LIKE Action (Bob reacts to Alice's post)
β”œβ”€ Action ID: a1~m9K7nP2qR8vL3xWpYzT4BjN...
β”œβ”€ Type: REACT:LIKE
β”œβ”€ Issuer: bob.example.com
β”œβ”€ Subject: a1~8kR3mN9pQ2vL6xW... (Alice's POST)
└─ Created: 2025-01-02T10:30:00Z

POST Action (Alice's post with 3 images)
β”œβ”€ Action ID: a1~8kR3mN9pQ2vL6xWpYzT4BjN...
β”œβ”€ Type: POST:IMG
β”œβ”€ Issuer: alice.example.com
β”œβ”€ Content: "Check out these amazing photos from our trip!"
β”œβ”€ Attachments:
β”‚   β”œβ”€ f1~Qo2E3G8TJZ2HTGhVlrtTDBpvBGOp6gfGhq4QmD6Z46w (Image 1)
β”‚   β”œβ”€ f1~7xW4Y9K5LM8Np2Qr3St6Uv8Xz9Ab1Cd2Ef3Gh4Ij5 (Image 2)
β”‚   └─ f1~9mN1P6Q8RS2Tu3Vw4Xy5Za6Bc7De8Fg9Hi0Jk1Lm2 (Image 3)
└─ Created: 2025-01-02T09:15:00Z

Image 1 File Descriptor
β”œβ”€ File ID: f1~Qo2E3G8TJZ2HTGhVlrtTDBpvBGOp6gfGhq4QmD6Z46w
β”œβ”€ Descriptor: d2,vis.tn:b1~abc123...:f=avif:s=4096:r=150x150;
β”‚              vis.sd:b1~def456...:f=avif:s=32768:r=640x480;
β”‚              vis.md:b1~ghi789...:f=avif:s=262144:r=1920x1080
└─ Variants:
    β”œβ”€ vis.tn: b1~abc123def456ghi789... (4KB, 150Γ—150px)
    β”œβ”€ vis.sd: b1~def456ghi789jkl012... (32KB, 640Γ—480px)
    └─ vis.md: b1~ghi789jkl012mno345... (256KB, 1920Γ—1080px)

Image 2 File Descriptor
β”œβ”€ File ID: f1~7xW4Y9K5LM8Np2Qr3St6Uv8Xz9Ab1Cd2Ef3Gh4Ij5
└─ Variants: tn, sd, md (similar structure)

Image 3 File Descriptor
β”œβ”€ File ID: f1~9mN1P6Q8RS2Tu3Vw4Xy5Za6Bc7De8Fg9Hi0Jk1Lm2
└─ Variants: tn, sd, md, hd (4K image, has HD variant)

Merkle Tree Visualization

flowchart TB
    subgraph "Action Layer"
        LIKE[LIKE Action<br/>a1~m9K7nP2qR8vL3xW...<br/>Type: REACT:LIKE<br/>Issuer: bob.example.com]
        POST[POST Action<br/>a1~8kR3mN9pQ2vL6xW...<br/>Type: POST:IMG<br/>Issuer: alice.example.com<br/>Content: Check out these photos!]
    end

    subgraph "Subject Reference"
        LIKE -->|subject| POST
    end

    subgraph "Attachment References"
        POST -->|attachments[0]| FILE1
        POST -->|attachments[1]| FILE2
        POST -->|attachments[2]| FILE3
    end

    subgraph "File Descriptor Layer"
        FILE1[File 1<br/>f1~Qo2E3G8TJZ2...]
        FILE2[File 2<br/>f1~7xW4Y9K5LM8...]
        FILE3[File 3<br/>f1~9mN1P6Q8RS2...]
    end

    subgraph "File 1 Variants"
        FILE1 --> V1TN[tn variant<br/>b1~abc123def456...<br/>AVIF, 4KB<br/>150Γ—150px]
        FILE1 --> V1SD[sd variant<br/>b1~def456ghi789...<br/>AVIF, 32KB<br/>640Γ—480px]
        FILE1 --> V1MD[md variant<br/>b1~ghi789jkl012...<br/>AVIF, 256KB<br/>1920Γ—1080px]
    end

    subgraph "File 2 Variants"
        FILE2 --> V2TN[tn variant<br/>b1~jkl012mno345...<br/>AVIF, 4KB<br/>150Γ—150px]
        FILE2 --> V2SD[sd variant<br/>b1~mno345pqr678...<br/>AVIF, 28KB<br/>640Γ—480px]
        FILE2 --> V2MD[md variant<br/>b1~pqr678stu901...<br/>AVIF, 248KB<br/>1920Γ—1080px]
    end

    subgraph "File 3 Variants"
        FILE3 --> V3TN[tn variant<br/>b1~stu901vwx234...<br/>AVIF, 4KB<br/>150Γ—150px]
        FILE3 --> V3SD[sd variant<br/>b1~vwx234yza567...<br/>AVIF, 35KB<br/>640Γ—480px]
        FILE3 --> V3MD[md variant<br/>b1~yza567bcd890...<br/>AVIF, 280KB<br/>1920Γ—1080px]
        FILE3 --> V3HD[hd variant<br/>b1~bcd890efg123...<br/>AVIF, 1.2MB<br/>3840Γ—2160px]
    end

    subgraph "Hash Computation"
        COMP1[Action ID = SHA256 of JWT token]
        COMP2[File ID = SHA256 of descriptor string]
        COMP3[Blob ID = SHA256 of blob bytes]
    end

    style LIKE fill:#ffcccc
    style POST fill:#ccffcc
    style FILE1 fill:#ccccff
    style FILE2 fill:#ccccff
    style FILE3 fill:#ccccff
    style V1TN fill:#ffffcc
    style V2TN fill:#ffffcc
    style V3TN fill:#ffffcc
    style V1SD fill:#ffeecc
    style V2SD fill:#ffeecc
    style V3SD fill:#ffeecc
    style V1MD fill:#ffddcc
    style V2MD fill:#ffddcc
    style V3MD fill:#ffddcc
    style V3HD fill:#ffcccc

Verification Steps

To verify this complete chain:

  1. Verify LIKE action signature and action_id
  2. Verify parent POST action (signature + action_id)
  3. Verify each file attachment (file_id = SHA256(descriptor))
  4. Verify all variants for each file (blob_id = SHA256(blob_data))

Complete verification example: see Content-Addressing & Merkle Trees.

Result: βœ…

  • Bob’s LIKE signature verified
  • LIKE action_id verified
  • Alice’s POST signature verified
  • POST action_id verified
  • All 3 file IDs verified
  • All 10 blob IDs verified (3+3+4 variants)
  • Complete merkle tree authenticated

Properties of This Structure

Immutability:

  • Cannot change Bob’s reaction without changing LIKE action_id
  • Cannot change Alice’s post content without changing POST action_id
  • Cannot swap images without changing file_ids
  • Cannot modify image bytes without changing blob_ids

Verifiability:

  • Anyone can recompute all hashes
  • No trusted third party needed
  • Pure cryptographic proof of authenticity

Deduplication:

  • If Alice uses the same image in another post, same file_id is reused
  • If Bob also posts the same image, same blob_ids are reused
  • Storage and bandwidth savings across the network

Federation:

  • Remote instances can verify the complete chain
  • Cannot tamper with any level without detection
  • Trustless content distribution

See Also

Subsections of Actions & Action Tokens

User Relationships

Action tokens representing connections and relationships between users on the Cloudillo network.

Contains:

  • CONN - Connect tokens representing one side of a connection
  • FLLW - Follow tokens representing follow relationships

Subsections of User Relationships

Connect Token

This token represents a connection intent created by a user.

The connect token can contain a content (c) field in markdown format. The token must contain an audience (aud) field which points to the identity the user is connecting with. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

Database Key

The database key for a connect token is [iss, t, aud]

Purpose: This key ensures that a user can only have one active connection intent to a specific identity. The key components are:

  • iss: Issuer identity (who is connecting)
  • t: Token type (“CONN”)
  • aud: Audience (who they’re connecting with)

Example:

  • Alice sends connection to Bob β†’ Stored with key [alice.example.com, "CONN", bob.example.com]
  • Alice updates the connection message β†’ New token with same key, previous one is marked deleted
  • Only ONE connection intent from Alice to Bob at a time

Example

User @alice.cloudillo.net wants to connect with @bob.cloudillo.net:

Field Value
iss alice.cloudillo.net
aud bob.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240101
t CONN
c Hi! Remember me? We met last week on the bus.

Follow Token

This token represents a follow intent created by a user.

The token must contain an audience (aud) field which points to the identity the user is following. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

Database Key

The database key for a follow token is [iss, t, aud]

Purpose: This key ensures that a user can only have one active follow relationship to a specific identity. The key components are:

  • iss: Issuer identity (who is following)
  • t: Token type (“FLLW”)
  • aud: Audience (who they’re following)

Example:

  • Alice follows Bob β†’ Stored with key [alice.example.com, "FLLW", bob.example.com]
  • If Alice unfollows and re-follows β†’ New token with same key, previous one is marked deleted
  • Only ONE follow relationship from Alice to Bob at a time

Example

User @alice.cloudillo.net follows @bob.cloudillo.net:

Field Value
iss alice.cloudillo.net
aud bob.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240101
t FLLW

Content Actions

Action tokens representing content creation, sharing, and interaction on the Cloudillo network. These tokens enable users to post, share, comment on, and react to content.

Contains:

  • POST - Posts created by users
  • REPOST - Reposting/sharing of user content
  • CMNT - Comments attached to other tokens
  • REACT - Reactions (likes, emojis, etc.)

Subsections of Content Actions

Post Token

This token represents a post created by a user.

The post token must contain a content (c) field. For other constraints see the Action Tokens.

There can be multiple subtypes of this token, which can represent the content of the post in different ways.

POST

A simple text only post. The content must be in markdown format.

POST:IMG

A post with an image. The content must be in markdown format, the attachment (a) field must contain exactly one item which must be an image.

POST:VID

A post with a video. The content must be in markdown format, the attachment (a) field must contain exactly one item which must be a video.

Content-Addressing

This token is content-addressed using SHA-256:

  • The entire JWT token (header + payload + signature) is hashed
  • Action ID format: a1~{base64_hash}
  • Changing any field invalidates the action_id
  • See Content-Addressing & Merkle Trees for details

Immutability: Once created, a POST token cannot be modified without changing its action ID.

Attachments

The a (attachments) field can contain file references:

  • Each entry is a file_id (f1~...)
  • File IDs are content-addressed (SHA256 of file descriptor)
  • Files contain multiple variants (different resolutions)
  • See File Storage for details

Properties:

  • Attachments are cryptographically bound to the post
  • Cannot swap images without breaking the action signature
  • Deduplication: same image in multiple posts = same file_id
  • Federation: remote instances can verify attachment integrity

Database Key

The database key for a post token is [iss, "POST", id]

Purpose: The database key is used to identify and potentially invalidate previous versions of this action type. For POST tokens, the key includes the action ID itself, meaning each post is unique and not overridden by subsequent posts.

Example

User @someuser.cloudillo.net writes a post on the wall of @somegroup.cloudillo.net, attaching an image:

Field Value
iss someuser.cloudillo.net
aud somegroup.cloudillo.net
iat 2024-04-13T00:00:00.000Z
k 20240109
t POST:IMG
c “Love U All <3”
a [“ohst:51mp8Oe5gekbAualO6jydbOriq0OfuZ5zpBY-I30U00,CFN6hm21Z73m12CK2igjFy8bVDhSV8oFZS4xOrzHE98,rk9n8iz–t0ov4sJAnBzEktmyZVsLmcKkPEVhkK4688,nfpr7eTtApLNTRS5qDokBsodo4UQ_zj7kzNWwvj7oEs”]

React Token

This token represents a reaction created by a user.

The react token must not contain a content (c) field. The token must contain a subject (sub) field which points to the action being reacted to. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

  • The entire JWT token (header + payload + signature) is hashed
  • Action ID format: a1~{base64_hash}
  • Changing any field invalidates the action_id
  • See Content-Addressing & Merkle Trees for details

Immutability: Once created, a REACT token cannot be modified without changing its action ID.

Subject Reference

The sub (subject) field references the action being reacted to:

  • Contains the target action’s action_id (a1~...)
  • Target action must exist and be verified
  • Creates a non-hierarchical reference (reactions don’t create visible threading)
  • Cannot modify subject without breaking reference

Why Subject Instead of Parent:

  • parent (p) is used for hierarchical threading (comments create visible child hierarchy)
  • subject (sub) is used for non-hierarchical references (reactions reference without creating hierarchy)
  • Reactions don’t create visible child actions in the timeline
  • This semantic distinction keeps threading clean

Properties:

  • Subject references are immutable
  • Cannot change which post you’re reacting to
  • Merkle tree ensures subject hasn’t been tampered with
  • Federation: remote instances can verify the complete reference

Database Key

The database key for a react token is {type}:{sub}:{iss}

Purpose: This key ensures that a user can only have one active reaction of each type to a specific action. The key components are:

  • {type}: Full type including subtype (e.g., “REACT:LIKE”)
  • {sub}: Subject ID (what they’re reacting to)
  • {iss}: Issuer identity (who is reacting)

Example:

  • Alice LIKEs a post β†’ Stored with key REACT:LIKE:a1~post123:alice.example.com
  • Alice changes to LOVE β†’ New LOVE token, previous LIKE is marked deleted
  • Only ONE reaction of each type per user per post
  • Changing reaction type creates a new action but invalidates the previous one

Audience Resolution

The aud (audience) field is automatically resolved from the subject action:

  • Set to the issuer of the subject action (the post owner)
  • Ensures the reaction notification reaches the content creator
  • Does not need to be explicitly provided when creating the reaction

Example

User @someotheruser.cloudillo.net likes a post:

Field Value
iss someotheruser.cloudillo.net
aud somegroup.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240301
t REACT:LIKE
sub NAado5PS4j5+abYtRpBELU0e5OQ+zGf/tuuWvUwQ6PA=

Subtypes

REACT tokens use subtypes to indicate the reaction type:

Subtype Description
REACT:LIKE Standard like reaction
REACT:LOVE Love/heart reaction
REACT:LAUGH Laughing reaction
REACT:WOW Surprised/amazed reaction
REACT:SAD Sad reaction
REACT:ANGRY Angry reaction
REACT:DEL Delete/remove the reaction

See Also

Comment Token

This token represents a comment created by a user.

The comment token must contain a content (c) field which contains the text of the comment in markdown format. The token must also contain a parent (p) field which points to the parent object the comment is referring to.

For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

Parent Reference

The p (parent) field references the parent action:

  • Contains the parent’s action_id (a1~...)
  • Creates immutable parent-child relationship
  • Parent can be a POST, another CMNT, or any commentable action
  • Federation: remote instances can verify the complete chain

Database Key

The database key for a comment token is [iss, t, id]

Purpose: Each comment is unique and not overridden by subsequent comments from the same user. The key includes the action ID itself, allowing multiple comments from the same user on the same parent.

Example

User @someotheruser.cloudillo.net writes a comment on a post:

Field Value
iss someotheruser.cloudillo.net
aud somegroup.cloudillo.net
iat 2024-04-13T00:01:00.000Z
k 20240301
t CMNT
c “I love U too!”
p NAado5PS4j5+abYtRpBELU0e5OQ+zGf/tuuWvUwQ6PA=

Repost Token

This token represents a repost (share) of another user’s content.

A repost allows a user to share someone else’s post with their own followers, optionally adding their own commentary. This is similar to “retweeting” in Twitter or “sharing” in other social platforms.

The token must contain a parent (p) field pointing to the original post being reposted. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

Parent Reference

The p (parent) field references the original post being reposted:

  • Contains the original post’s action_id (a1~...)
  • Parent must exist and be verified
  • Creates immutable link between repost and original content
  • Federation: remote instances can verify the complete chain

Database Key

The database key for a repost token is [iss, t, p]

Purpose: This key ensures that a user can only repost the same content once. The key components are:

  • iss: Issuer identity (who is reposting)
  • t: Token type (“REPOST”)
  • p: Parent ID (what’s being reposted)

Example:

  • Alice reposts Bob’s post β†’ Stored with key [alice.example.com, "REPOST", a1~post123]
  • Alice reposts the same post again β†’ New token with same key, previous one is marked deleted
  • Only ONE repost of the same content per user

Types of Reposts

Simple Repost

Sharing content without additional commentary:

{
  "iss": "alice.example.com",
  "iat": 1738483200,
  "k": "20240101",
  "t": "REPOST",
  "p": "a1~xyz789..."
}

Repost with Commentary

Sharing content with your own thoughts added:

{
  "iss": "alice.example.com",
  "iat": 1738483200,
  "k": "20240101",
  "t": "REPOST",
  "p": "a1~xyz789...",
  "c": "This is an excellent analysis! Everyone should read this."
}

Quote Repost

Reposting with substantial commentary (quote post/quote tweet):

{
  "iss": "alice.example.com",
  "iat": 1738483200,
  "k": "20240101",
  "t": "REPOST",
  "p": "a1~xyz789...",
  "c": "While I agree with the main points, I think the conclusion overlooks an important aspect..."
}

Fields

Field Required Description
iss βœ“ The identity reposting the content
iat βœ“ Timestamp when repost was created
k βœ“ Key ID used to sign the token
t βœ“ Token type (always “REPOST”)
p βœ“ Parent token ID (the original post being reposted)
c Optional commentary on the repost (markdown)
aud Optional audience restriction

Example

User @alice.example.com reposts @bob.example.com’s post with commentary:

Field Value
iss alice.example.com
iat 2024-04-13T00:01:10.000Z
k 20240101
t REPOST
p a1~xyz789abc…
c Great insights on distributed systems!

Visibility and Federation

Repost tokens are broadcast actions, meaning they are:

  • Sent to all followers of the reposter
  • Displayed in the reposter’s timeline/feed
  • Credit the original author
  • Link back to the original post

Federation Flow

When Alice reposts Bob’s post:

  1. Alice’s instance creates a REPOST token referencing Bob’s original POST token
  2. The REPOST is broadcast to Alice’s followers
  3. The original POST token is fetched/synchronized if not already available locally
  4. Followers see the repost in Alice’s timeline with proper attribution to Bob

Permission Checks

When creating a repost:

  1. Original post exists: Verify the parent post ID is valid
  2. Permission to view: Ensure the reposter can access the original post
  3. Repost allowed: Check if original author allows reposts (future feature)
  4. Audience restrictions: Honor any audience limitations on original post

Statistics Impact

Reposts affect the statistics of the original post:

  • Original post’s STAT token includes repost count
  • Reposts increase content visibility and reach
  • Original author can see who reposted their content

Undo/Delete Repost

To remove a repost:

  • Create a new action that overwrites the REPOST (using same database key)
  • Or delete the REPOST token entirely
  • This removes the repost from the reposter’s timeline
  • Original post’s repost count is decremented

Comparison with Other Actions

Action Purpose Visibility
REPOST Share content with followers Broadcast to your followers
REACT Express opinion privately Only visible to post author/viewers
CMNT Add threaded discussion Attached to original post

See Also

Communication

Action tokens for direct communication and group collaboration between users on the Cloudillo network, including direct messages, conversations, subscriptions, and invitations.

Contains:

  • MSG - Direct messages sent from one profile to another, or messages within conversations
  • CONV - Conversation threads and group messaging sessions
  • SUBS - Subscriptions to subscribable actions (e.g., joining a conversation)
  • INVT - Invitations for users to join subscribable actions

Subsections of Communication

Message Token

This token represents a direct message sent from one profile to another.

A message token enables private, one-on-one communication between users. Unlike posts (which are broadcast to followers), messages are sent directly to a specific recipient and are not federated to other instances.

The token must contain an audience (aud) field pointing to the recipient of the message. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

Attachments

The a (attachments) field can contain file references:

  • Each entry is a file_id (f1~...)
  • File IDs are content-addressed (SHA256 of file descriptor)
  • Files contain multiple variants (different resolutions)
  • See File Storage for details

Parent Reference

The optional p (parent) field enables message threading:

  • Contains the parent message’s action_id (a1~...)
  • Creates conversation threads and reply chains
  • Parent must exist and be verified

Database Key

The database key for a message token is [iss, t, id]

Purpose: Each message has a unique ID, allowing multiple messages in a conversation thread. The key includes the action ID itself, so messages are never overridden.

Message Types

Simple Text Message

Basic text communication:

{
  "iss": "alice.example.com",
  "aud": "bob.example.com",
  "iat": 1738483200,
  "k": "20240101",
  "t": "MSG",
  "c": "Hey Bob, want to grab coffee tomorrow?"
}

Message with Attachments

Messages can include files, images, or other attachments:

{
  "iss": "alice.example.com",
  "aud": "bob.example.com",
  "iat": 1738483200,
  "k": "20240101",
  "t": "MSG",
  "c": "Here's the photo from our trip!",
  "a": ["f1~abc123..."]
}

Reply to Message

Threading messages in a conversation:

{
  "iss": "bob.example.com",
  "aud": "alice.example.com",
  "iat": 1738483210,
  "k": "20240101",
  "t": "MSG",
  "p": "a1~xyz789...",
  "c": "Sounds great! How about 10am?"
}

Fields

Field Required Description
iss βœ“ The identity sending the message
aud βœ“ The identity receiving the message
iat βœ“ Timestamp when message was sent
k βœ“ Key ID used to sign the token
t βœ“ Token type (always “MSG”)
c βœ“ Message content (markdown)
p Parent message ID (for threading/replies)
a Attachments (files, images, etc.)
exp Optional expiration (for disappearing messages)

Example

User @alice.example.com sends a message to @bob.example.com:

Field Value
iss alice.example.com
aud bob.example.com
iat 2024-04-13T00:01:10.000Z
k 20240101
t MSG
c Hey Bob! Are you coming to the meetup tonight?

Visibility and Federation

Message tokens are direct actions, meaning they are:

  • Sent only to the specified audience (recipient)
  • NOT broadcast to followers
  • Private between sender and recipient
  • Delivered to POST https://cl-o.{recipient}/api/inbox

Federation Flow

When Alice sends a message to Bob:

  1. Alice’s instance creates a MSG token with aud: bob.example.com
  2. The token is signed by Alice’s private key
  3. The message is sent to https://cl-o.bob.example.com/api/inbox
  4. Bob’s instance verifies the signature
  5. Bob’s instance stores the message for Bob to read
  6. Bob receives a notification (via WebSocket bus or push notification)

Permission Checks

When receiving a message token:

  1. Verify signature: Ensure the token is signed by the claimed issuer
  2. Check audience: Verify aud field matches the local user
  3. Check relationship: Verify sender is connected or followed (configurable)
  4. Spam prevention: Check sender is not blocked
  5. Rate limiting: Enforce message rate limits per sender

Message Delivery

Messages use a different delivery mechanism than broadcast actions:

Aspect Broadcast Actions (POST) Direct Messages (MSG)
Delivery To all followers To specific recipient only
Federation Sent to multiple instances Sent to one instance
Storage Stored publicly (with permissions) Stored privately
Visibility Timeline/feed Message inbox
Retry Best-effort Reliable delivery with retry

Message Threading

Messages can be threaded using the parent (p) field:

Message 1 (no parent)
  β”œβ”€ Reply 1 (p: Message 1)
  β”œβ”€ Reply 2 (p: Message 1)
  β”‚   └─ Reply 3 (p: Reply 2)
  └─ Reply 4 (p: Message 1)

This allows for:

  • Conversation continuity
  • Quote/reply functionality
  • Context preservation

Encryption (Future)

While action tokens are already signed for authentication, end-to-end encryption of message content is a planned feature:

  • Message content (c field) encrypted with recipient’s public key
  • Only sender and recipient can decrypt
  • Server cannot read message content
  • Forward secrecy with ephemeral keys

Message Storage

Messages are stored in MetaAdapter with:

  • Sender and recipient identity tags
  • Read/unread status
  • Delivery status
  • Optional expiration timestamp

Privacy considerations:

  • Messages stored locally at sender and recipient instances
  • Not replicated to other instances
  • Can be deleted by either party
  • Retention policies configurable per user

Notifications

New messages trigger notifications via:

  1. WebSocket Bus: Real-time notification if recipient is online
  2. Push Notifications: If configured and recipient is offline
  3. Email: Optional email notification for important messages

Notification payload includes:

  • Sender identity
  • Message preview (first ~100 characters)
  • Timestamp
  • Attachment indicator

Comparison with Conversations

Feature MSG (Message) CONV (Conversation)
Participants 1-to-1 Group (multiple participants)
Status βœ… Implemented ⚠️ Future/Planned
Use Case Direct messaging Group chats
Audience Single recipient Multiple recipients

See Also

Conversation Token

This token represents a group conversation (group chat) between multiple participants.

The conversation token must contain a content (c) field with conversation metadata. For other constraints see the Action Tokens.

Note: For 1-to-1 direct messaging, see Message Token (MSG). CONV is for group conversations with multiple participants.

Content-Addressing

This token is content-addressed using SHA-256:

Immutability: Once created, a CONV token cannot be modified without changing its action ID.

Content Structure

The content (c) field must be a JSON object with the following structure:

{
  "name": "Project Discussion",
  "description": "Weekly sync about the project"
}
Property Type Required Description
name string Yes Display name for the conversation
description string No Optional description of the conversation purpose

Flags

CONV tokens use the flags (f) field to control conversation behavior:

Flag Default Description
R/r r (disabled) Reactions allowed on the conversation itself
C/c c (disabled) Comments allowed on the conversation itself
O/o o (closed) Open (anyone can join) vs Closed (invite-only)

Default flags: rco (no reactions, no comments, closed/invite-only)

To create an open conversation that anyone can join, use flags O.

Subtypes

Subtype Description
CONV Create a new conversation
CONV:UPD Update conversation metadata (name, description, flags)
CONV:DEL Delete/archive the conversation

Subscribable Behavior

CONV is a subscribable action type, meaning:

  • Users can subscribe to receive messages in the conversation
  • Subscription is managed via SUBS tokens
  • Users can be invited via INVT tokens
  • Messages sent to the conversation (MSG with parent=CONV) are delivered to all subscribers

Auto-Subscription

When a CONV is created, the creator is automatically subscribed as an admin:

  1. CONV token is created and signed
  2. System automatically creates a SUBS token for the creator
  3. Creator’s SUBS has role = “admin”

Participant Roles

Conversation participants can have different roles, managed via SUBS tokens:

Role Permissions
observer Can read messages, cannot send
member Can read and send messages
moderator Can read, send, and invite/remove members
admin Full control including conversation settings

Federation

Creating a Conversation

Creator creates CONV token
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ ActionCreatorTask runs      β”‚
β”‚ - Signs CONV token          β”‚
β”‚ - Creates admin SUBS        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    CONV ready for messages

Adding Participants

Participants join via invitation or (for open conversations) self-subscription:

Admin creates INVT for new participant
         β”‚
         β–Ό
INVT delivered to invitee (includes CONV token)
         β”‚
         β–Ό
Invitee creates SUBS token to accept
         β”‚
         β–Ό
SUBS auto-accepted (INVT exists)
         β”‚
         β–Ό
Participant now receives MSG in conversation

Message Fan-Out

When a message is sent to a conversation:

  1. Local CONV: If the CONV owner sends/receives a message, they fan it out to all subscribers
  2. Remote CONV: If a subscriber sends a message, it goes to the CONV owner who fans it out

See Subscriber Fan-Out for detailed federation flow.

Example

Field Value
iss alice.cloudillo.net
aud
iat 2024-04-13T00:00:00.000Z
k 20240109
t CONV
c {“name”: “Project Team”, “description”: “Discussion for project X”}
f rco

Conversation Flow Example

CONV: @alice.cloudillo.net creates "Project Team"
  β”‚
  β”œβ”€ SUBS: @alice.cloudillo.net (auto-created, role=admin)
  β”‚
  β”œβ”€ INVT: @alice.cloudillo.net β†’ @bob.cloudillo.net
  β”‚   └─ SUBS: @bob.cloudillo.net (accepts invitation, role=member)
  β”‚
  β”œβ”€ INVT: @alice.cloudillo.net β†’ @charlie.cloudillo.net
  β”‚   └─ SUBS: @charlie.cloudillo.net (accepts invitation, role=member)
  β”‚
  β”œβ”€ MSG: @alice.cloudillo.net "Hi everyone!"
  β”‚   └─ Fan-out to bob, charlie
  β”‚
  β”œβ”€ MSG: @bob.cloudillo.net "Hello!"
  β”‚   β”œβ”€ REACT:LIKE @alice.cloudillo.net
  β”‚   └─ Fan-out to alice, charlie
  β”‚
  └─ SUBS:DEL: @charlie.cloudillo.net (leaves conversation)

See Also

Subscription Token

This token represents a subscription to a subscribable action, such as a conversation (CONV). Subscriptions control who receives messages and updates from group activities.

The subscription token must contain both a subject (sub) field referencing the subscribable action and an audience (aud) field specifying the owner of that action. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

  • The entire JWT token (header + payload + signature) is hashed
  • Action ID format: a1~{base64_hash}
  • Changing any field invalidates the action_id
  • See Content-Addressing & Merkle Trees for details

Immutability: Once created, a SUBS token cannot be modified without changing its action ID.

Purpose

SUBS tokens serve several important purposes:

  1. Membership Management: Track who is part of a conversation or group
  2. Message Delivery: Determine who receives messages sent to the group
  3. Role Assignment: Assign permissions (observer, member, moderator, admin)
  4. Participation Control: Enable joining, leaving, and role changes

Required Fields

Field Required Description
iss Yes The subscriber’s identity
aud Yes The owner of the subject action (e.g., CONV creator)
sub Yes The action_id being subscribed to (a1~...)
t Yes “SUBS”, “SUBS:UPD”, or “SUBS:DEL”
c Optional Subscription metadata (role, invitedBy, message)

Subtypes

Subtype Description
SUBS Create a new subscription (join)
SUBS:UPD Update subscription (change role)
SUBS:DEL Delete subscription (leave)

Content Structure

The optional content (c) field contains subscription metadata:

{
  "role": "member",
  "invitedBy": "alice.cloudillo.net",
  "message": "Joining the team discussion"
}
Property Type Required Description
role string No Participant role: “observer”, “member”, “moderator”, “admin”
invitedBy string No Identity of the user who invited this subscriber
message string No Optional message when joining

Database Key

The database key for a subscription token is {type}:{sub}:{iss}

Purpose: This key ensures that a user can only have one active subscription to a specific action. The key components are:

  • {type}: “SUBS” (base type)
  • {sub}: Subject ID (what they’re subscribing to)
  • {iss}: Issuer identity (who is subscribing)

Example:

  • Alice subscribes to a CONV β†’ Stored with key SUBS:a1~conv123:alice.example.com
  • Alice updates her subscription β†’ New SUBS:UPD with same key pattern
  • Only ONE active subscription per user per subject

Auto-Accept Logic

When a SUBS token is received, the system determines whether to accept it automatically:

Subscription received
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Check auto-accept conditionsβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β–Ό               β–Ό               β–Ό               β–Ό
Subject has    INVT exists     Issuer is      Otherwise
'O' flag?      for user?       subject creator?
    β”‚               β”‚               β”‚               β”‚
    β–Ό               β–Ό               β–Ό               β–Ό
 Accept          Accept          Accept          Reject
 (Open)         (Invited)       (Creator)      (status='R')

Auto-accept conditions (any one is sufficient):

  1. Subject action has the ‘O’ flag (open - anyone can join)
  2. An INVT token exists for this user + subject combination
  3. The subscriber is the creator of the subject action

If none of these conditions are met, the subscription is rejected (status=‘R’).

Participant Roles

Subscriptions can specify different access levels:

Role Can Read Can Write Can Invite Can Moderate Can Admin
observer Yes No No No No
member Yes Yes No No No
moderator Yes Yes Yes Yes No
admin Yes Yes Yes Yes Yes

Default role: member (if not specified)

Federation Flow

Subscribing to a Conversation

User creates SUBS token
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Local processing            β”‚
β”‚ - Validate subject exists   β”‚
β”‚ - Check for INVT or 'O'     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_delivery()         β”‚
β”‚ - Send to subject owner     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    Owner receives SUBS
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ on_receive hook             β”‚
β”‚ - Check auto-accept logic   β”‚
β”‚ - Accept or reject          β”‚
β”‚ - Update subscription list  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Receiving Messages After Subscription

Once subscribed, messages sent to the conversation are delivered via fan-out:

Message sent to CONV
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_subscriber_fanout()β”‚
β”‚ - Walk parent chain to CONV β”‚
β”‚ - Get all SUBS for CONV     β”‚
β”‚ - Deliver to each subscriberβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Leaving a Conversation

User creates SUBS:DEL token
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Delivered to subject owner  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ on_receive hook             β”‚
β”‚ - Mark SUBS as deleted      β”‚
β”‚ - Remove from subscriber    β”‚
β”‚   list                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    User no longer receives
    messages from conversation

Example

User @bob.cloudillo.net subscribes to a conversation owned by @alice.cloudillo.net:

Field Value
iss bob.cloudillo.net
aud alice.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240301
t SUBS
sub a1~NAado5PS4j5+abYtRpBELU0e5OQ+zGf/tuuWvUwQ6PA=
c {“role”: “member”, “invitedBy”: “alice.cloudillo.net”}

Status Values

Status Description
A (Active) Subscription is active, user receives messages
R (Rejected) Subscription was rejected (no invitation, not open)
D (Deleted) User left or was removed from the group

See Also

Invitation Token

This token represents an invitation for a user to join a subscribable action, such as a conversation (CONV). Invitations enable controlled access to closed groups.

The invitation token must contain a subject (sub) field referencing the subscribable action and an audience (aud) field specifying the invited user. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

  • The entire JWT token (header + payload + signature) is hashed
  • Action ID format: a1~{base64_hash}
  • Changing any field invalidates the action_id
  • See Content-Addressing & Merkle Trees for details

Immutability: Once created, an INVT token cannot be modified without changing its action ID.

Purpose

INVT tokens serve several important purposes:

  1. Access Control: Enable users to join closed/private groups
  2. Role Assignment: Pre-assign roles for invited users
  3. Discoverability: Notify users about groups they can join
  4. Subject Delivery: Include the subject action (CONV) with the invitation

Required Fields

Field Required Description
iss Yes The inviter’s identity (must have moderator+ role)
aud Yes The invited user’s identity
sub Yes The action_id being invited to (a1~...)
t Yes “INVT” or “INVT:DEL”
c Optional Invitation metadata (role, message)

Subtypes

Subtype Description
INVT Create a new invitation
INVT:DEL Revoke an invitation

Content Structure

The optional content (c) field contains invitation metadata:

{
  "role": "member",
  "message": "Welcome to our project discussion!"
}
Property Type Required Description
role string No Role to assign when accepting: “observer”, “member”, “moderator”, “admin”
message string No Optional invitation message

Database Key

The database key for an invitation token is {type}:{sub}:{aud}

Purpose: This key ensures that only one active invitation exists per user per subject. The key components are:

  • {type}: “INVT” (base type)
  • {sub}: Subject ID (what they’re invited to)
  • {aud}: Audience identity (who is invited)

Example:

  • Alice invites Bob to a CONV β†’ Stored with key INVT:a1~conv123:bob.example.com
  • A new invitation to the same user replaces the previous one

Permission Requirements

To create an invitation, the inviter must have appropriate permissions:

Inviter Role Can Invite
observer No
member No
moderator Yes
admin Yes

The system validates that the inviter has at least moderator-level access to the subject action.

Subject Delivery

INVT has deliver_subject=true behavior, meaning:

  • When an invitation is delivered, the subject action (e.g., CONV) is included
  • The invitee receives both the INVT and the CONV token
  • This allows the invitee to see conversation details before accepting
{
  "token": "eyJhbGciOi...INVT_TOKEN",
  "related": ["eyJhbGciOi...CONV_TOKEN"]
}

Federation Flow

Sending an Invitation

Moderator creates INVT token
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Validate permissions        β”‚
β”‚ - Check inviter has mod+    β”‚
β”‚ - Validate subject exists   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_delivery()         β”‚
β”‚ - Deliver to invitee (aud)  β”‚
β”‚ - Include subject action    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    Invitee receives INVT + CONV

Accepting an Invitation

Invitee receives INVT + CONV
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Invitee creates SUBS token  β”‚
β”‚ - sub = CONV action_id      β”‚
β”‚ - aud = CONV owner          β”‚
β”‚ - role from INVT (optional) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ SUBS sent to CONV owner     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Auto-accept logic           β”‚
β”‚ - INVT exists? β†’ Accept!    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    Invitee is now a subscriber
    and receives messages

Revoking an Invitation

Moderator creates INVT:DEL token
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Delivered to invitee        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Original INVT marked deletedβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    User can no longer use
    invitation to subscribe

Invitation Lifecycle

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”     Create      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  None    │────────────────▢│  Active  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                   β”‚
                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                    β–Ό              β–Ό              β–Ό
              SUBS created    INVT:DEL       Expires
                    β”‚              β”‚              β”‚
                    β–Ό              β–Ό              β–Ό
              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
              β”‚  Used    β”‚  β”‚ Revoked  β”‚  β”‚ Expired  β”‚
              β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Example

User @alice.cloudillo.net invites @bob.cloudillo.net to a conversation:

Field Value
iss alice.cloudillo.net
aud bob.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240301
t INVT
sub a1~NAado5PS4j5+abYtRpBELU0e5OQ+zGf/tuuWvUwQ6PA=
c {“role”: “member”, “message”: “Join our project discussion!”}

Delivery includes:

  • INVT token (the invitation)
  • CONV token (the conversation being invited to)

Flow:

  1. Alice (moderator or admin) creates INVT for Bob
  2. Bob receives INVT + CONV details
  3. Bob creates SUBS to accept
  4. SUBS auto-accepted because INVT exists
  5. Bob now receives messages in the conversation

See Also

Action Type DSL

The Action Type DSL (Domain-Specific Language) is Cloudillo’s system for defining action types declaratively. Instead of hardcoding behavior for each action type, the DSL allows flexible configuration of validation rules, processing behavior, and lifecycle hooks.

Overview

The DSL system serves several purposes:

  • Validation: Define what fields are required/optional for each action type
  • Behavior Configuration: Control how actions are processed, delivered, and stored
  • Extensibility: Add new action types without modifying core code
  • Consistency: Ensure all actions of a type follow the same rules

ActionDefinition Structure

Each action type is defined by an ActionDefinition:

struct ActionDefinition {
    action_type: String,      // e.g., "POST", "CMNT", "FLLW"
    version: u32,             // Schema version
    metadata: Metadata,       // Display name, description, etc.
    subtypes: Vec<String>,    // e.g., POST:IMG, POST:VID
    fields: Vec<FieldDef>,    // Field definitions
    schema: Schema,           // JSON Schema for validation
    behavior: Behavior,       // Processing behavior flags
    hooks: Hooks,             // Lifecycle hooks
    permissions: Permissions, // Permission requirements
    key_pattern: String,      // Database key pattern for overriding
}

Metadata

struct Metadata {
    display_name: String,     // Human-readable name
    description: String,      // What this action represents
    category: Category,       // content, user, communication, metadata
}

Fields

Each field in an action is defined with validation rules:

struct FieldDef {
    name: String,             // Field name in JWT (e.g., "c", "p", "vis")
    field_type: FieldType,    // string, object, array, number, boolean
    required: bool,           // Is this field required?
    validate: Option<Schema>, // Additional validation rules
}

Behavior Flags

Behavior flags control how actions are processed:

Flag Description Example Types
broadcast Send to multiple recipients (followers) POST, REPOST
allow_unknown Accept actions from unknown senders CONN
ephemeral Don’t persist to database PRES (presence)
approvable Can be approved/rejected by recipient CONN, FLLW
requires_subscription Sender must be subscribed MSG, CONV
deliver_subject Deliver to subject owner CMNT, REACT
deliver_to_subject_owner Alternative delivery to subject owner CMNT

Behavior Examples

POST Action:

Behavior {
    broadcast: true,              // Send to all followers
    allow_unknown: false,         // Must be from followed/connected user
    ephemeral: false,             // Persist to database
    approvable: false,            // No approval needed
    requires_subscription: false, // No subscription required
    deliver_subject: false,       // Not a reply
}

CONN Action:

Behavior {
    broadcast: false,             // Send only to audience
    allow_unknown: true,          // Accept from anyone (with PoW)
    ephemeral: false,             // Persist to database
    approvable: true,             // Recipient can accept/reject
    requires_subscription: false, // No subscription required
    deliver_subject: false,       // Direct to audience
}

MSG Action:

Behavior {
    broadcast: false,             // Send to conversation participants
    allow_unknown: false,         // Must be conversation member
    ephemeral: false,             // Persist to database
    approvable: false,            // No approval needed
    requires_subscription: true,  // Must be subscribed to conversation
    deliver_subject: false,       // Uses subscription delivery
}

PRES (Presence) Action:

Behavior {
    broadcast: true,              // Send to connections
    allow_unknown: false,         // Must be connected
    ephemeral: true,              // DO NOT persist
    approvable: false,            // No approval needed
    requires_subscription: false, // No subscription required
    deliver_subject: false,       // Direct broadcast
}

Hook System

Hooks allow custom logic at different points in the action lifecycle:

struct Hooks {
    on_create: Option<HookFn>,   // Called when action is created locally
    on_receive: Option<HookFn>,  // Called when action is received from federation
    on_accept: Option<HookFn>,   // Called when approvable action is accepted
    on_reject: Option<HookFn>,   // Called when approvable action is rejected
}

Hook Use Cases

on_create:

  • Validate business logic
  • Generate related actions
  • Update statistics

on_receive:

  • Trigger notifications
  • Update local state
  • Forward to WebSocket clients

on_accept:

  • Establish connection (CONN)
  • Add to subscriber list (SUBS)
  • Grant permissions

on_reject:

  • Clean up pending state
  • Notify sender of rejection

Hook Example: CONN Action

on_create:
    # Validate PoW nonce
    verify_proof_of_work(action.nonce)
    # Store as pending connection
    store_pending_connection(action)

on_receive:
    # Notify user of connection request
    send_notification(action.audience, "New connection request")
    # Forward to WebSocket
    forward_to_websocket(action)

on_accept:
    # Create bidirectional connection
    create_connection(action.issuer, action.audience)
    # Update follower counts
    update_connection_counts()

on_reject:
    # Remove pending connection
    remove_pending_connection(action.id)
    # Optionally notify sender

Permissions

Define who can perform which operations:

struct Permissions {
    create: PermissionRule,    // Who can create this action type
    read: PermissionRule,      // Who can read actions of this type
    delete: PermissionRule,    // Who can delete (owner only typically)
}

Permission Rules

Rule Description
Owner Only the action issuer
Audience The action’s audience field
Connected Mutually connected users
Follower Users who follow the issuer
Verified Any authenticated user
Public Anyone, including unauthenticated

Key Pattern

The key_pattern determines how actions are stored and whether they can be overridden:

Action Type Key Pattern Override Behavior
POST {iss}:{id} Each post is unique
CMNT {iss}:{p}:{id} Each comment is unique (p = parent)
REACT {iss}:{sub} One reaction per user per subject
FLLW {iss}:{aud} One follow per user per target
CONN {iss}:{aud} One connection request per pair

Override example:

User reacts to post with LIKE
  β†’ Stored at key: "alice.example.com:a1~post123"

User changes reaction to LOVE
  β†’ Same key: "alice.example.com:a1~post123"
  β†’ Previous LIKE is REPLACED by LOVE

Example Definitions

POST Definition

ActionDefinition {
    action_type: "POST",
    version: 1,
    metadata: Metadata {
        display_name: "Post",
        description: "A public post or status update",
        category: Category::Content,
    },
    subtypes: vec!["IMG", "VID", "LNK", "TXT"],
    fields: vec![
        FieldDef { name: "c", field_type: String, required: true },
        FieldDef { name: "a", field_type: Array, required: false },
        FieldDef { name: "vis", field_type: String, required: false },
        FieldDef { name: "f", field_type: String, required: false },
    ],
    behavior: Behavior {
        broadcast: true,
        allow_unknown: false,
        ephemeral: false,
        approvable: false,
        requires_subscription: false,
        deliver_subject: false,
    },
    hooks: Hooks {
        on_create: Some(post_on_create),
        on_receive: Some(post_on_receive),
        on_accept: None,
        on_reject: None,
    },
    permissions: Permissions {
        create: PermissionRule::Owner,
        read: PermissionRule::ByVisibility,
        delete: PermissionRule::Owner,
    },
    key_pattern: "{iss}:{id}",
}

CMNT Definition

ActionDefinition {
    action_type: "CMNT",
    version: 1,
    metadata: Metadata {
        display_name: "Comment",
        description: "A comment on another action",
        category: Category::Content,
    },
    subtypes: vec![],
    fields: vec![
        FieldDef { name: "c", field_type: String, required: true },
        FieldDef { name: "p", field_type: String, required: true },  // Parent required
        FieldDef { name: "a", field_type: Array, required: false },
    ],
    behavior: Behavior {
        broadcast: false,
        allow_unknown: false,
        ephemeral: false,
        approvable: false,
        requires_subscription: false,
        deliver_subject: true,  // Deliver to parent owner
    },
    hooks: Hooks {
        on_create: Some(cmnt_on_create),
        on_receive: Some(cmnt_on_receive),
        on_accept: None,
        on_reject: None,
    },
    permissions: Permissions {
        create: PermissionRule::Connected,
        read: PermissionRule::ByVisibility,
        delete: PermissionRule::Owner,
    },
    key_pattern: "{iss}:{p}:{id}",
}

CONV Definition

ActionDefinition {
    action_type: "CONV",
    version: 1,
    metadata: Metadata {
        display_name: "Conversation",
        description: "A group conversation",
        category: Category::Communication,
    },
    subtypes: vec![],
    fields: vec![
        FieldDef { name: "c", field_type: Object, required: true },  // {name, participants}
    ],
    behavior: Behavior {
        broadcast: false,
        allow_unknown: false,
        ephemeral: false,
        approvable: false,
        requires_subscription: false,
        deliver_subject: false,
    },
    hooks: Hooks {
        on_create: Some(conv_on_create),  // Creates SUBS for participants
        on_receive: Some(conv_on_receive),
        on_accept: None,
        on_reject: None,
    },
    permissions: Permissions {
        create: PermissionRule::Verified,
        read: PermissionRule::Subscriber,
        delete: PermissionRule::Owner,
    },
    key_pattern: "{iss}:{id}",
}

DSL Validation

When an action is created or received, it’s validated against its DSL definition:

validate_action(action, definition):
    1. Check required fields exist
    2. Validate field types
    3. Run schema validation
    4. Check behavior constraints:
       - If requires_subscription: verify sender is subscribed
       - If !allow_unknown: verify sender is known (following/connected)
       - If approvable: set initial status to pending
    5. Return validation result

Extending with New Action Types

To add a new action type:

  1. Define the ActionDefinition with all fields, behavior, and hooks
  2. Register the definition in the action registry
  3. Implement hooks for custom behavior (optional)
  4. Add client support for creating/displaying the action

Example: Adding a new POLL action type:

ActionDefinition {
    action_type: "POLL",
    version: 1,
    metadata: Metadata {
        display_name: "Poll",
        description: "A poll with voting options",
        category: Category::Content,
    },
    subtypes: vec![],
    fields: vec![
        FieldDef { name: "c", field_type: Object, required: true },  // {question, options, expires}
    ],
    behavior: Behavior {
        broadcast: true,
        allow_unknown: false,
        ephemeral: false,
        approvable: false,
        requires_subscription: false,
        deliver_subject: false,
    },
    hooks: Hooks {
        on_create: Some(poll_on_create),
        on_receive: Some(poll_on_receive),
        on_accept: None,
        on_reject: None,
    },
    permissions: Permissions {
        create: PermissionRule::Verified,
        read: PermissionRule::ByVisibility,
        delete: PermissionRule::Owner,
    },
    key_pattern: "{iss}:{id}",
}

See Also

Metadata Actions

Action tokens for metadata, system operations, and auxiliary information about other tokens and actions on the Cloudillo network.

Contains:

  • STAT - Statistics about other tokens (reaction counts, comment counts, etc.)
  • FSHR - File sharing tokens for sharing files with others
  • APRV - Approval tokens for endorsing and redistributing content
  • PRES - Presence tokens for ephemeral real-time status (typing, online, etc.)

Subsections of Metadata Actions

Statistics Token

This token represents the statistics of reactions on an object (post, comment, etc.)

The issuer (iss) of the token must be the audience (aud) of the parent token (p). A statistics token must not contain an audience (aud) field. The token must contain a parent (p) field which points to the parent object the statistics are referring to. The statistics token must contain a content (c) field which is a JSON object in the following format:

Field Value Description
c number The number of comments (optional)
r number The number of reactions (optional)

For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

Parent Reference

The p (parent) field references the action being counted:

  • Contains the parent action’s action_id (a1~...)
  • Statistics token issued by the audience of the parent token
  • Creates aggregated view of reactions and comments

Database Key

The database key for a statistics token is [iss, t, p]

Purpose: This key ensures that only one statistics token exists per parent action. The key components are:

  • iss: Issuer identity (audience of the parent token)
  • t: Token type (“STAT”)
  • p: Parent ID (what’s being counted)

Example:

  • Group posts statistics for a post β†’ Stored with key [somegroup.cloudillo.net, "STAT", a1~post123]
  • Statistics update β†’ New token with same key, previous one is marked deleted
  • Only ONE statistics token per parent action

Example

User @someotheruser.cloudillo.net also likes the post:

Field Value
iss somegroup.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240301
t STAT
p NAado5PS4j5+abYtRpBELU0e5OQ+zGf/tuuWvUwQ6PA=
c { “c”: 1, “r”: 2 }

Fileshare Token

Overview

The FSHR (Fileshare) token represents sharing a file with another user. It grants the recipient access to view or edit the specified file.

Token Structure

Field Type Required Description
iss identity Yes The identity sharing the file
aud identity Yes The recipient of the share
iat timestamp Yes Issue time
k string Yes Signing key ID
t string Yes FSHR or FSHR:WRITE or FSHR:DEL
sub string Yes File ID being shared (e.g., f1~abc123)
c object Yes Share content (see below)

Subtypes

Subtype Type String Description
Read (default) FSHR View-only access to the file
Write FSHR:WRITE Edit access to the file
Delete FSHR:DEL Revokes a previous share

Content Schema

Field Type Required Description
contentType string Yes MIME type (e.g., application/pdf)
fileName string Yes Original filename
fileTp string Yes File type: BLOB, CRDT, or RTDB

File Types

Type Description
BLOB Static binary file (images, PDFs, etc.)
CRDT Collaborative document (real-time editing)
RTDB Real-time database document

Content-Addressing

This token is content-addressed using SHA-256:

Database Key

The database key for a fileshare token is [iss, t, sub]

Purpose: This key ensures that a user can only have one active fileshare for a given file. The key components are:

  • iss: Issuer identity (who is sharing)
  • t: Token type (FSHR)
  • sub: Subject (the file being shared)

Example:

  • Alice shares file with Bob β†’ Stored with key [alice.example.com, "FSHR", f1~file123]
  • Alice updates the share β†’ New token with same key, previous one is marked deleted
  • Only ONE fileshare of the same file from the same user to the same recipient

Examples

Read-Only Share

User @alice.cloudillo.net shares a PDF with @bob.cloudillo.net (read-only):

Field Value
iss alice.cloudillo.net
aud bob.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240101
t FSHR
sub f1~7NtuTab_K4FwYmARMNuk4
c.contentType application/pdf
c.fileName report.pdf
c.fileTp BLOB

Write Access Share

User @alice.cloudillo.net grants edit access to a collaborative document:

Field Value
iss alice.cloudillo.net
aud bob.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240101
t FSHR:WRITE
sub f1~collaborative_doc_id
c.contentType application/vnd.cloudillo.doc
c.fileName Team Notes.quillo
c.fileTp CRDT

Revoke Share

User @alice.cloudillo.net revokes Bob’s access:

Field Value
iss alice.cloudillo.net
aud bob.cloudillo.net
iat 2024-04-13T00:05:00.000Z
k 20240101
t FSHR:DEL
sub f1~7NtuTab_K4FwYmARMNuk4

Integration with References

File shares can be combined with references for public sharing:

  1. Create FSHR action to share with specific user
  2. Or create a reference with type: share.file for public/guest access
  3. Reference tokens support accessLevel: read or accessLevel: write

Access Control Flow

Owner creates FSHR action
  ↓
Action sent to recipient via federation
  ↓
Recipient's server verifies:
  - JWT signature
  - Issuer owns the file
  - Recipient matches audience
  ↓
Access granted based on subtype:
  - FSHR β†’ Read access
  - FSHR:WRITE β†’ Write access
  ↓
Access revoked on FSHR:DEL receipt

See Also

IDP:REG Token

Overview

The IDP:REG (Identity Provider Registration) token enables federated identity registration. It allows a registrar to request identity creation on a remote Identity Provider (IDP) instance, enabling community-owned identities and cross-instance identity management.

Use Cases

  • Community-Owned Identities: A community owner creates identities under their domain on a remote IDP
  • Federated Registration: Registrar on instance A creates identities on IDP instance B
  • Bulk Identity Provisioning: Automated identity creation for organizations

Token Structure

Field Type Required Description
iss identity Yes The registrar’s identity
aud identity Yes The IDP instance to register on
iat timestamp Yes Issue time
k string Yes Signing key ID
t string Yes IDP:REG
c object Yes Registration content (see below)

Content Schema

Field Type Required Description
id_tag string Yes Identity to register (e.g., alice.cloudillo.net)
email string No Email address (required if no owner_id_tag)
owner_id_tag string No Owner identity for community-owned identities
issuer string No Issuer role: registrar (default) or owner
expires_at number No Identity expiration timestamp

Content-Addressing

This token is content-addressed using SHA-256:

Database Key

The database key for an IDP:REG token is [type, issuer, audience, content.id_tag]

Purpose: This key ensures that only one registration request exists for a given identity from a given registrar. The key components are:

  • type: Token type (IDP:REG)
  • issuer: Registrar identity
  • audience: Target IDP instance
  • content.id_tag: The identity being registered

Behavior Flags

Flag Value Description
broadcast false Not broadcast to followers
allow_unknown true Allows additional content fields
requires_acceptance false Processed immediately on receipt

Authorization

Role Can Create Can Receive
Registrar Yes (authenticated) -
IDP Instance - Yes (any)

Issuer Roles

The issuer field in content determines the relationship:

Role Description
registrar Default. Registrar manages identity until activation
owner Issuer is the owner of the identity

After activation, registrar control is revoked - only the owner retains access.

Example

A community owner (community@cloudillo.net) registers a new member identity:

{
  "iss": "community@cloudillo.net",
  "aud": "cl-o.cloudillo.net",
  "iat": 1735000000,
  "k": "20250101",
  "t": "IDP:REG",
  "c": {
    "id_tag": "alice.cloudillo.net",
    "email": "alice@example.com",
    "owner_id_tag": "community@cloudillo.net",
    "issuer": "owner",
    "expires_at": 1766536000
  }
}
Field Value
iss community@cloudillo.net
aud cl-o.cloudillo.net
iat 2025-01-02T00:00:00Z
k 20250101
t IDP:REG
c.id_tag alice.cloudillo.net
c.owner_id_tag community@cloudillo.net

Processing Flow

1. Registrar creates IDP:REG action
   ↓
2. Action sent to IDP instance (/api/inbox)
   ↓
3. IDP verifies:
   - JWT signature
   - Issuer has permission to create identities
   - Domain matches IDP's domain
   ↓
4. Identity created with status "pending"
   ↓
5. IDP sends activation link directly to identity owner (via email)
   ↓
6. Identity owner activates via /api/idp/activate
   ↓
7. Identity status becomes "active"

Federation Diagram

sequenceDiagram
    participant R as Registrar
    participant IDP as IDP Instance
    participant O as Identity Owner

    R->>R: Create IDP:REG JWT
    R->>IDP: POST /api/inbox {token}
    IDP->>IDP: Verify JWT signature
    IDP->>IDP: Create identity (pending)
    IDP-->>R: 200 OK

    IDP->>O: Send activation email/link
    O->>IDP: POST /api/idp/activate {refId}
    IDP->>IDP: Activate identity
    IDP-->>O: 200 OK {identity}

Community Ownership

When owner_id_tag is provided:

  1. The identity is owned by the specified community/user
  2. The registrar only has control while status is pending
  3. After activation, only the owner can manage the identity
  4. Email is optional (owner relationship is the primary identifier)

Security Considerations

Signature Verification

The receiving IDP verifies:

  1. JWT signature matches issuer’s public key
  2. Issuer is allowed to create identities on this IDP
  3. Requested id_tag domain matches IDP’s domain

Domain Validation

The id_tag in content must match the IDP’s domain:

  • Request: alice.cloudillo.net β†’ IDP: cloudillo.net βœ…
  • Request: alice.other.net β†’ IDP: cloudillo.net ❌ (rejected)

Registrar Control Limits

Registrar control is time-limited:

  • Full control while identity status is pending
  • No control after identity is activated
  • Prevents registrar from managing user’s active identity

See Also

Approval Token

This token represents an approval of another user’s content. When you approve someone’s action (e.g., a POST), it signals trust and enables federated fan-out to your followers.

The approval token must NOT contain a content (c) or attachments (a) field. The token must contain a subject (sub) field referencing the action being approved. For other constraints see the Action Tokens.

Content-Addressing

This token is content-addressed using SHA-256:

  • The entire JWT token (header + payload + signature) is hashed
  • Action ID format: a1~{base64_hash}
  • Changing any field invalidates the action_id
  • See Content-Addressing & Merkle Trees for details

Immutability: Once created, an APRV token cannot be modified without changing its action ID.

Purpose

APRV tokens serve several important purposes in the federated network:

  1. Trust Signal: Indicates that you endorse the referenced content
  2. Federated Fan-Out: When you approve a POST, it gets broadcast to your followers along with the approved content
  3. Content Discovery: Helps content spread across the network through trusted connections
  4. Status Update: Updates the original action’s status to ‘Active’ (A) on the original author’s instance

Required Fields

Field Required Description
iss Yes Your identity (the approver)
aud Yes The issuer of the approved action (content creator)
sub Yes The action_id being approved (a1~...)
t Yes “APRV”
c Forbidden Content field is not allowed
a Forbidden Attachments field is not allowed

Subject Reference

The sub (subject) field references the action being approved:

  • Contains the target action’s action_id (a1~...)
  • Target action must exist and be verifiable
  • Creates a non-hierarchical reference

Why Subject Instead of Parent:

  • APRV doesn’t create a visible hierarchy (unlike comments)
  • It references the action without threading
  • The semantic is “this action is about that action”

Broadcast Behavior

APRV tokens have broadcast=true behavior, meaning:

  • When you create an APRV, it’s sent to all your followers
  • The approved action (e.g., the POST) is bundled with the APRV delivery
  • Recipients receive both the APRV and the related action in a single delivery

This is how content spreads across the federated network through trust relationships.

Federation Flow

Creating an Approval

User approves remote content
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Create APRV token           β”‚
β”‚ - aud = content creator     β”‚
β”‚ - sub = content action_id   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_delivery()         β”‚
β”‚ - Check subject broadcast   β”‚
β”‚ - Schedule broadcast fanout β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_broadcast_delivery β”‚
β”‚ - Get all followers         β”‚
β”‚ - Include original author   β”‚
β”‚ - Bundle approved action    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    Delivered to all followers
    with approved POST bundled

Receiving an Approval

When you receive an APRV for your content:

APRV arrives at /inbox
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ ActionVerifierTask          β”‚
β”‚ - Verify JWT signature      β”‚
β”‚ - Check permissions         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ on_receive hook             β”‚
β”‚ - Find subject action       β”‚
β”‚ - Update status to 'A'      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    Your content is now approved

Auto-Approval

The system can automatically create APRV tokens for content from trusted connections. See Auto-Approval for details.

Auto-approval conditions:

  1. Action type must be approvable (POST, MSG, REPOST)
  2. Action is addressed to you (audience = your id_tag)
  3. Sender is different from you
  4. federation.auto_approve setting is enabled
  5. Sender is connected (bidirectional connection established)

When an APRV is delivered, the approved action is included:

{
  "token": "eyJhbGciOi...APRV_TOKEN",
  "related": ["eyJhbGciOi...APPROVED_POST_TOKEN"]
}

Recipients receive both tokens in a single request. The related action is processed after the APRV, with permission checks skipped (pre-approved by the APRV issuer’s trust).

Example

User @alice.cloudillo.net approves a post from @bob.cloudillo.net:

Field Value
iss alice.cloudillo.net
aud bob.cloudillo.net
iat 2024-04-13T00:01:10.000Z
k 20240301
t APRV
sub a1~NAado5PS4j5+abYtRpBELU0e5OQ+zGf/tuuWvUwQ6PA=

Flow:

  1. Alice approves Bob’s post
  2. APRV is sent to Bob (notification)
  3. APRV + Bob’s POST are broadcast to Alice’s followers
  4. Bob’s post status is updated to ‘A’ (Active/Approved)

See Also

Presence Token

This token represents an ephemeral presence indication, such as typing status or online presence. Unlike other action types, presence tokens are NOT persisted to the database - they are only forwarded via WebSocket in real-time.

The presence token must contain a subject (sub) field referencing the context (e.g., a conversation). For other constraints see the Action Tokens.

Ephemeral Nature

IMPORTANT: PRES tokens are fundamentally different from other action types:

  • NOT persisted: Not stored in the database
  • NOT content-addressed: No action_id is generated
  • Real-time only: Forwarded via WebSocket immediately
  • Short TTL: Default time-to-live of 30 seconds
  • No delivery tasks: No scheduling or retry logic

This makes PRES tokens ideal for real-time status updates that don’t need to be permanent.

Purpose

PRES tokens serve several important purposes:

  1. Typing Indicators: Show when someone is typing in a conversation
  2. Online Status: Indicate when users are online/active
  3. Activity Updates: Real-time presence in collaborative contexts
  4. Low-Overhead Updates: Frequent status changes without database writes

Fields

Field Required Description
iss Yes The user’s identity
sub Yes Context (e.g., CONV action_id)
aud Optional Specific target user (if omitted, broadcast to context)
t Yes “PRES:TYPING”, “PRES:ONLINE”, etc.
c Optional Additional metadata
a Forbidden Attachments are not allowed

Subtypes

Subtype Description
PRES:TYPING User is currently typing
PRES:ONLINE User is online/active
PRES:AWAY User is away/idle
PRES:OFFLINE User has gone offline

Additional subtypes can be defined for application-specific presence states.

Subject Reference

The sub (subject) field specifies the context for the presence update:

  • For typing indicators: the CONV action_id
  • For online status: could be a user’s profile context
  • Creates a scoped presence update

Why Subject Instead of Parent:

  • PRES doesn’t create hierarchy (unlike comments)
  • It’s contextual - “presence in this context”
  • Non-hierarchical reference semantics

Processing Flow

User starts typing
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Create PRES:TYPING token    β”‚
β”‚ - sub = CONV action_id      β”‚
β”‚ - aud = (optional recipient)β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Check: Is ephemeral?        β”‚
β”‚ - Yes! (PRES is ephemeral)  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Forward to WebSocket        β”‚
β”‚ - No database storage       β”‚
β”‚ - No action_id generated    β”‚
β”‚ - No delivery tasks         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Deliver to audience         β”‚
β”‚ - If aud set: single user   β”‚
β”‚ - If not: context subscribersβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Federation Flow

Local Presence (Same Instance)

User starts typing
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ WebSocket message           β”‚
β”‚ - Signed PRES token         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Broadcast to CONV           β”‚
β”‚ subscribers on same instanceβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Remote Presence (Cross-Instance)

User starts typing (instance A)
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Forward to CONV owner       β”‚
β”‚ (instance B)                β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό (HTTP POST to /inbox)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Instance B receives PRES    β”‚
β”‚ - Verifies signature        β”‚
β”‚ - Does NOT store            β”‚
β”‚ - Forwards to WebSocket     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Broadcast to other          β”‚
β”‚ subscribers on instance B   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Time-To-Live (TTL)

PRES tokens have an implicit TTL:

  • Default: 30 seconds
  • After TTL expires, presence is considered stale
  • Clients should send periodic updates to maintain presence
  • No explicit expiration in the token (handled by clients)

Typing indicator flow:

  1. User starts typing β†’ PRES:TYPING sent
  2. User continues typing β†’ Periodic PRES:TYPING (e.g., every 5s)
  3. User stops typing β†’ No more updates
  4. After 30s β†’ Recipients consider typing stopped

Example

User @alice.cloudillo.net is typing in a conversation:

Field Value
iss alice.cloudillo.net
sub a1~NAado5PS4j5+abYtRpBELU0e5OQ+zGf/tuuWvUwQ6PA=
iat 2024-04-13T00:01:10.000Z
k 20240301
t PRES:TYPING

Properties:

  • No action_id generated (ephemeral)
  • Not stored in database
  • Forwarded immediately via WebSocket
  • Recipients see “Alice is typing…”
  • After 30s without update, typing indicator disappears

Security Considerations

Since PRES tokens are ephemeral:

  • Still require valid signature verification
  • Rate limiting applies to prevent spam
  • Context (subject) must be valid
  • User must have access to the context

Comparison with Persistent Actions

Aspect PRES (Ephemeral) Other Actions
Database storage No Yes
Action ID None SHA256 hash
Delivery tasks None Scheduled with retry
TTL 30 seconds Permanent (until deleted)
Use case Real-time status Permanent records

See Also

Federation Architecture

Cloudillo is designed as a federated system where independent instances communicate to share content, enable collaboration, and maintain user sovereignty. This architecture ensures users can self-host while remaining connected to the broader network.

Why Federation?

The email analogy: You don’t need a Gmail account to email someone on Gmailβ€”any email server can talk to any other. Cloudillo works the same way for social interactions and collaboration.

What federation enables:

  • Alice hosts her own server β†’ She controls her data, privacy settings, and uptime
  • Bob uses a community server β†’ He gets the convenience of managed hosting
  • Alice and Bob collaborate β†’ Their servers communicate directly, no middleman

Comparison with centralized platforms:

Aspect Centralized (Twitter, Slack) Federated (Cloudillo, Email)
Data location Company servers Your chosen server
Account survival Company decides You control
Privacy policy Take it or leave it Server-by-server
Feature changes Company decides Choose your instance
Network effects All users on one platform All users, many servers

Why not fully peer-to-peer? Pure P2P requires both parties online simultaneously. Federation gives you the benefits of decentralization while allowing offline message delivery, persistent hosting, and simpler client applications.

Core Principles

Decentralization

  • No central authority: No single server controls the network
  • User sovereignty: Users choose where their data lives
  • Instance independence: Each instance operates autonomously

Privacy-First

  • Explicit consent: Users control what they share
  • Relationship-based: Only share with connected/following users
  • End-to-end verification: Cryptographic signatures prove authenticity
  • No data harvesting: Federation for communication, not surveillance

Interoperability

  • Standard protocols: HTTP/HTTPS, WebSocket, JWT
  • Content addressing: SHA256 ensures integrity across instances
  • Action tokens: Portable, verifiable activity records
  • DNS-based identity: Discoverable profiles via domain names

Inter-Instance Communication

Request Module

The request module provides HTTP client functionality for federation:

Request Structure:

  • client: HTTP client (reqwest) for making requests
  • timeout: Request timeout duration
  • retry_policy: Configuration for automatic retries

Request Methods:

  • get(url) - Generic HTTP GET
  • post(url, body) - Generic HTTP POST with JSON body
  • fetch_profile(id_tag) - GET /api/me from remote instance
  • fetch_action(id_tag, action_id) - GET /api/action/:id from remote
  • post_inbox(id_tag, token) - POST /api/inbox with action token
  • fetch_file(id_tag, file_id) - GET /api/file/:id from remote

Federation Flow

When Alice (on instance A) follows Bob (on instance B):

Instance A                          Instance B
  |                                   |
  |--- GET /api/me ------------------>| (Fetch Bob's profile)
  |<-- 200 OK {profile} --------------|
  |                                   |
  |--- Create FLLW action token ---   |
  |                                   |
  |--- POST /api/inbox -------------->| (Send follow action)
  |    {token: "eyJhbGc..."}         |
  |                                   |--- Verify signature
  |                                   |--- Check permissions
  |                                   |--- Store action
  |<-- 202 Accepted ------------------|

See Also

Subsections of Federation Architecture

Action Delivery

How actions are distributed and received across the federated Cloudillo network.

Outbound Actions

When a user creates an action, it’s distributed to relevant recipients:

Algorithm: Distribute Action

Input: action_token, action_type
Output: Result<()>

1. Determine recipients based on action_type:
   a. "POST" β†’ All followers of the user
   b. "CMNT" / "REACT" β†’ Owner of parent post
   c. "CONN" / "FLLW" β†’ Target from action audience (aud claim)
   d. "MSG" β†’ All conversation participants
   e. Other β†’ No recipients (skip)

2. For each recipient instance:
   a. Construct URL: https://cl-o.{recipient_id_tag}/api/inbox
   b. POST action token as JSON: {"token": "..."}
   c. Continue on error (best-effort delivery)

3. Return success

Inbound Actions

Cloudillo provides two endpoints for receiving federated actions, optimized for different use cases.

Async Inbox (/api/inbox)

The standard endpoint for most federated actions:

HTTP Endpoint: POST /api/inbox

Handler Algorithm:
1. Extract action token from JSON payload
2. No authentication required (token is self-authenticating)
3. Create ActionVerifierTask with token
4. Schedule task in task scheduler (for async processing)
5. Return HTTP 202 (Accepted) immediately
   - Verification and processing happen asynchronously
   - Prevents network timeouts on slow verification

Sync Inbox (/api/inbox/sync)

A synchronous endpoint for actions requiring immediate confirmation:

HTTP Endpoint: POST /api/inbox/sync

Handler Algorithm:
1. Extract action token from JSON payload
2. No authentication required (token is self-authenticating)
3. Process action SYNCHRONOUSLY (not queued)
4. Verify signature, check permissions, store action
5. Return HTTP 200 (OK) with result, or error response

Use Cases:
- Identity registration tokens (IDP:REG)
- Actions where sender needs immediate confirmation
- Critical federation handshakes

When to use each endpoint:

  • /api/inbox: Default for most actions (POST, CMNT, REACT, MSG, etc.)
  • /api/inbox/sync: Only for critical operations requiring immediate feedback

Delivery Guarantees

Persistent Delivery with Retry

  • Delivery tasks are persisted to survive server restarts
  • Automatic retry with exponential backoff
  • Comprehensive failure handling

Retry Policy

Cloudillo uses exponential backoff for delivery retries:

RetryPolicy Configuration:
  initial_wait:  10 seconds    # First retry delay
  max_wait:      12 hours      # Maximum delay between retries
  backoff_step:  50 seconds    # Backoff increment per retry

Retry Schedule Example:

Attempt 1: Immediate
  ↓ fails
Attempt 2: 10s later (initial_wait)
  ↓ fails
Attempt 3: 60s later (10s + 50s backoff)
  ↓ fails
Attempt 4: 110s later (60s + 50s backoff)
  ↓ fails
...continues with 50s backoff steps...
  ↓
Maximum wait capped at 12 hours between attempts
  • Task deduplication: Key pattern delivery:{action_id}:{recipient} prevents duplicate deliveries
  • Persistent delivery: Tasks survive server restarts via MetaAdapter storage

Failure Handling

On POST /api/inbox result:

1. Success (2xx):
   - Remove from delivery queue
   - Log successful delivery

2. Temporary Error (network timeout, 5xx, connection refused):
   - Keep in queue
   - Schedule retry with backoff
   - Continue retrying until max attempts

3. Permanent Error (4xx status, validation error):
   - Log error with context
   - Mark as undeliverable
   - Remove from queue (don't waste resources)

When certain actions are approved, they trigger delivery of related actions to additional recipients:

APRV (Approval) Fan-out

When a user approves another user’s content (e.g., repost), the system distributes the original content to the approver’s followers:

APRV Fan-out Algorithm:

1. User A creates POST action
2. User B reposts (creates APRV referencing POST)
3. On APRV creation:
   a. Fetch User B's followers
   b. For each follower:
      - Create delivery task for original POST
      - Include APRV context (who approved)
   c. Followers receive both POST and APRV

Use Cases:

  • Repost/Share: Original post reaches approver’s audience
  • Endorsement: Endorsed content gets wider distribution
  • Curated feeds: Content discovery through trusted connections
Trigger Action Related Actions Delivered Recipients
APRV (Approval) Original referenced action Approver’s followers
CONV (Conversation) Participant subscriptions All participants
INVT (Invitation) Conversation context Invited user

See Also

Key Verification & Caching

When receiving federated actions, the server must verify the JWT signature using the issuer’s public key. This involves a 3-tier caching strategy to balance security with performance.

3-Tier Caching Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                     Verification Request                        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                β”‚
                                β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Tier 1: In-Memory Failure Cache                                β”‚
β”‚  β”œβ”€ Purpose: Prevent repeated requests to unreachable instances β”‚
β”‚  β”œβ”€ TTL (network errors): 5 minutes                             β”‚
β”‚  β”œβ”€ TTL (persistent errors): 1 hour                             β”‚
β”‚  └─ LRU eviction when capacity exceeded                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                β”‚ (cache miss or expired)
                                β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Tier 2: SQLite Key Cache                                       β”‚
β”‚  β”œβ”€ Purpose: Persistent cache of successful key fetches         β”‚
β”‚  β”œβ”€ Key: (issuer, key_id)                                       β”‚
β”‚  β”œβ”€ Stores: public key + expiration timestamp                   β”‚
β”‚  └─ If valid & not expired: verify signature immediately        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                β”‚ (cache miss or expired)
                                β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Tier 3: HTTP Fetch from Remote                                 β”‚
β”‚  β”œβ”€ Endpoint: GET https://cl-o.{issuer}/api/me                  β”‚
β”‚  β”œβ”€ Find matching key by key_id                                 β”‚
β”‚  β”œβ”€ On success: cache in SQLite, clear failure cache            β”‚
β”‚  └─ On failure: record in failure cache                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Failure Types and TTLs

Error Type TTL Reason
Network timeout 5 minutes May recover quickly
Connection refused 5 minutes Server may restart
404 Not Found 1 hour Key doesn’t exist
403 Forbidden 1 hour Permission denied
Parse error 1 hour Invalid response format

Verification Flow

verify_action_token(token):
    1. Decode JWT without verifying (extract issuer, key_id)

    2. Check failure cache:
       if failure_cache.has(issuer, key_id) and not expired:
           return Error(CachedFailure)

    3. Check SQLite cache:
       if key_cache.has(issuer, key_id):
           key = key_cache.get(issuer, key_id)
           if key.expires_at > now:
               return verify_signature(token, key.public_key)

    4. Fetch from remote:
       response = HTTP GET https://cl-o.{issuer}/api/me
       if error:
           failure_cache.record(issuer, key_id, error_type)
           return Error(KeyFetchFailed)

       public_key = find_key_by_id(response.keys, key_id)
       if not found:
           failure_cache.record(issuer, key_id, NotFound)
           return Error(KeyNotFound)

       key_cache.store(issuer, key_id, public_key)
       failure_cache.clear(issuer, key_id)

       return verify_signature(token, public_key)

See Also

Trust & Content Distribution

How trust relationships enable automatic content approval and efficient message distribution across the federated network.

Auto-Approval for Trusted Connections

When receiving actions from connected users, the system can automatically create approval (APRV) tokens, enabling content to spread through the network via trusted relationships.

Auto-Approval Conditions

For an action to be auto-approved, ALL of the following must be true:

  1. Approvable action type: Action type has approvable=true behavior
    • Applies to: POST, MSG, REPOST
  2. Addressed to us: action.audience == our_id_tag
  3. From different user: action.issuer != our_id_tag
  4. Setting enabled: federation.auto_approve = true in settings
  5. Connection established: Issuer has bidirectional connection with us

Auto-Approval Flow

Inbound action received
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Check auto-approval         β”‚
β”‚ conditions (all 5)          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”
    β”‚               β”‚
    β–Ό               β–Ό
Conditions      Conditions
met             not met
    β”‚               β”‚
    β–Ό               β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  No action
β”‚ Update      β”‚  (standard
β”‚ action      β”‚  processing)
β”‚ status='A'  β”‚
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
       β”‚
       β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Create APRV token           β”‚
β”‚ - aud = original issuer     β”‚
β”‚ - sub = action_id           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Deliver APRV to issuer      β”‚
β”‚ (notification)              β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Broadcast APRV + action     β”‚
β”‚ to our followers            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Trust Indicator

Trust = Bidirectional Connection

The system checks issuer_profile.connected.is_connected() to determine trust. A connection is established when both parties have sent CONN tokens to each other.

Subscriber Fan-Out

For subscribable actions (like CONV), messages need to be delivered to all subscribers. The fan-out mechanism ensures efficient message distribution across the federated network.

Subscribable Actions

Actions with subscribable=true behavior can have subscribers:

  • CONV (Conversation) - subscribers receive messages in the group

Fan-Out Algorithm

schedule_subscriber_fanout(action_id, parent_id, issuer):
    1. Walk parent chain to find subscribable root:
       current = parent_id
       while current:
           parent_action = fetch_action(current)
           if is_subscribable(parent_action.type):
               subscribable_root = parent_action
               break
           current = parent_action.parent_id

    2. Check if we own the subscribable root:
       is_local = (subscribable_root.audience == null
                   && subscribable_root.issuer == our_id_tag)
                  || subscribable_root.audience == our_id_tag

       if not is_local:
           return  // Remote owner handles fan-out

    3. Get subscribers:
       subscribers = query_subscriptions(subscribable_root.action_id)
                     .filter(status = 'A')  // Active only

    4. Create delivery tasks:
       for subscriber in subscribers:
           if subscriber != our_id_tag
              && subscriber != issuer:  // Exclude self and sender
               schedule_delivery_task(
                   action_id,
                   subscriber,
                   key = "fanout:{action_id}:{subscriber}"
               )

Local vs Remote Fan-Out

Scenario 1: Local CONV Owner Sends Message

Alice (CONV owner) sends MSG
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ ActionCreatorTask           β”‚
β”‚ - Creates MSG               β”‚
β”‚ - Calls schedule_delivery() β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_subscriber_fanout()β”‚
β”‚ - Finds CONV (subscribable) β”‚
β”‚ - CONV is local (we own it) β”‚
β”‚ - Gets all SUBS             β”‚
β”‚ - Schedules delivery to eachβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    Bob, Charlie receive MSG
    (all other subscribers)

Scenario 2: Remote Subscriber Sends Message

Bob (subscriber, not owner) sends MSG
on his instance
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_subscriber_fanout()β”‚
β”‚ - Finds CONV (subscribable) β”‚
β”‚ - CONV is NOT local         β”‚
β”‚ - No local fan-out          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_delivery()         β”‚
β”‚ - Deliver to CONV owner     β”‚
β”‚   (Alice)                   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό (federation)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Alice's instance receives   β”‚
β”‚ MSG at /inbox               β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_subscriber_fanout()β”‚
β”‚ - CONV is local to Alice    β”‚
β”‚ - Fan out to all except Bob β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
    Charlie, others receive MSG

When delivering certain actions (like APRV), related actions are bundled together in a single delivery to provide context.

Bundling Use Cases

  1. APRV + Approved POST: When broadcasting an approval, include the approved content
  2. INVT + Subject CONV: When inviting, include the conversation details

Delivery Payload Structure

{
  "token": "eyJhbGciOi...MAIN_ACTION_TOKEN",
  "related": [
    "eyJhbGciOi...RELATED_ACTION_TOKEN_1",
    "eyJhbGciOi...RELATED_ACTION_TOKEN_2"
  ]
}
Inbox receives action with related tokens
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Store related tokens        β”‚
β”‚ - Status = 'W' (waiting)    β”‚
β”‚ - ack_token = main action   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Process main action         β”‚
β”‚ - Verify signature          β”‚
β”‚ - Check permissions         β”‚
β”‚ - Store action              β”‚
β”‚ - Execute on_receive hook   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ process_related_actions()   β”‚
β”‚ - Get tokens with           β”‚
β”‚   ack_token = main action   β”‚
β”‚ - For each:                 β”‚
β”‚   - Skip permission check   β”‚
β”‚   - Verify signature        β”‚
β”‚   - Store action            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Key point: Related actions skip permission checks because they are pre-approved by the main action issuer. The trust flows from the APRV issuer to the related content.

APRV Broadcast Example

Alice approves Bob's POST
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Create APRV                 β”‚
β”‚ - sub = Bob's post action_idβ”‚
β”‚ - Check: subject.broadcast  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ schedule_broadcast_delivery β”‚
β”‚ - Get Alice's followers     β”‚
β”‚ - Include Bob's POST token  β”‚
β”‚   as related action         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚
            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ For each follower:          β”‚
β”‚ POST /inbox                 β”‚
β”‚ {                           β”‚
β”‚   token: APRV_token,        β”‚
β”‚   related: [POST_token]     β”‚
β”‚ }                           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

See Also

ProxyToken Authentication

For cross-instance requests, ProxyTokens authenticate the requesting instance.

ProxyToken Structure

{
  "iss": "alice.example.com",      // Requesting instance
  "aud": "bob.example.com",         // Target instance
  "sub": "alice.example.com",       // User identity
  "exp": 1738400000,                // Short-lived (5-60 min)
  "iat": 1738396800,
  "action": "read_file",            // Requested operation
  "resource": "f1~abc123...",       // Resource identifier
  "k": "20250205"                   // Signing key ID
}

Creation

Algorithm: Create ProxyToken

Input: requester id_tag, target_instance, action, resource
Output: JWT token string

1. Retrieve latest signing key:
   - Query latest key_id for tenant
   - Load private key from AuthAdapter

2. Build JWT claims:
   - iss: Requester's id_tag
   - aud: Target instance domain
   - sub: Requester's id_tag
   - exp: Current time + 30 minutes
   - iat: Current time
   - action: Requested operation
   - resource: Resource identifier
   - k: Key ID used for signing

3. Sign JWT using ES384 algorithm:
   - Use private key
   - Standard JWT encoding

4. Return base64-encoded JWT

Validation

Algorithm: Validate ProxyToken

Input: JWT token string
Output: Result<ProxyTokenClaims>

1. Decode JWT without signature verification (read claims)
2. Extract issuer and key_id from unverified claims
3. Fetch issuer's profile from remote instance
4. Look up public key by key_id in profile
5. Verify JWT signature using issuer's public key (ES384)
6. Check expiration timestamp:
   - If exp < current_time: Return TokenExpired error
7. Check audience claim:
   - If aud != this_instance.base_id_tag: Return InvalidAudience error
8. Return verified claims

Validation ensures:
- Token signed by claimed issuer
- Token not expired
- Token intended for this instance

Use Cases

ProxyTokens are used for:

  • File fetching: Downloading attachments from remote instances
  • Profile queries: Accessing extended profile information
  • Database sync: Read access to federated databases

See Also

Data Synchronization

How files, profiles, and databases are synchronized across federated instances.

File Synchronization

Attachment Fetching

When receiving an action with file attachments:

Algorithm: Sync Attachments

Input: attachment_ids from remote action
Output: Result<()>

For each attachment_id:
1. Check if already exists locally
   - If exists: Skip to next (already synced)

2. Construct remote URL:
   https://cl-o.{issuer_id_tag}/api/file/{attachment_id}

3. Download file from remote instance

4. Verify content integrity:
   - Compute SHA256 hash of downloaded data
   - Compare hash with attachment_id
   - If mismatch: Return FileIntegrityCheckFailed error

5. Store file data in blob adapter

6. Extract and store metadata:
   - Read X-File-Metadata header (if present)
   - Parse as JSON
   - Store in metadata adapter

7. Continue to next attachment

This ensures:
- Content-addressed files (hash = ID)
- No duplicate downloads
- Cryptographic integrity verification

Lazy Loading

Files are fetched on-demand rather than proactively:

User views post with image attachment
  ↓
Check if image exists locally
  ↓
If not, fetch from remote instance
  ↓
Verify content hash
  ↓
Store locally
  ↓
Serve to user

Profile Synchronization

Remote Profile Caching

Cache remote profiles locally for performance:

Algorithm: Sync Profile with Caching

Input: id_tag (remote user identifier)
Output: Result<Profile>

1. Check local cache for profile:
   - If cached AND cache_age < 24 hours: Return cached profile
   - If cache_age >= 24 hours: Continue to step 2

2. Fetch profile from remote instance:
   - GET https://cl-o.{id_tag}/api/me

3. Update local cache:
   - Store profile with current timestamp

4. Return profile

Benefits:
- Reduces network requests (24h TTL)
- Improves performance for repeated access
- Staleness acceptable for user profiles

Profile Updates

Profiles don’t push updates; instances pull when needed:

Need to display Alice's profile
  ↓
Check cache (last updated < 24h?)
  ↓
If fresh: use cache
If stale: fetch from Alice's instance
  ↓
Update cache
  ↓
Display profile

Database Federation

Read-Only Replication

Subscribe to remote database updates:

FederatedDatabase Structure:

  • origin_instance: Source instance domain (e.g., alice.example.com)
  • local_replica: Whether to maintain local copy for fast access
  • sync_mode: Synchronization mode (see below)

SyncMode Enum:

  • ReadOnly: Subscribe to updates from remote, no local edits
  • ReadWrite: Bidirectional synchronization
  • Periodic(Duration): Full sync every N seconds (fallback for network issues)

Sync Protocol

Using action/inbox mechanism:

DatabaseSyncAction Structure:

  • db_file_id: SHA256 identifier of database file
  • updates: Binary update payload (Yrs CRDT or redb operations)
  • state_vector: Current state hash for conflict detection
  • timestamp: Unix timestamp of update creation

Database Update Distribution Algorithm:

For each subscriber instance:

  1. Create DatabaseSyncAction with:

    • Database file ID
    • Binary updates (from CRDT or redb)
    • Computed state vector
    • Current timestamp
  2. POST to subscriber’s inbox:

    • Endpoint: https://cl-o.{subscriber_id_tag}/api/inbox
    • Send DatabaseSyncAction as JSON
  3. Subscriber’s ActionVerifierTask processes:

    • Extracts binary updates
    • Applies to local replica
    • Merges with any local changes

This pattern allows:

  • Real-time database synchronization
  • Conflict resolution via CRDTs
  • Federation of collaborative databases

See Also

Relationship Management

How following and connection relationships are established and managed across federated instances.

Following

When Alice follows Bob:

Algorithm: Follow User

Input: follower_id_tag, target_id_tag
Output: Result<()>

1. Create FLLW action token:
   - issuer: follower's id_tag
   - subject: target's id_tag
   - action type: "FLLW"
   - Sign with follower's private key (ES384)

2. Store action locally:
   - Record in metadata adapter
   - Marks that follower follows target

3. Send to target instance:
   - POST https://cl-o.{target_id_tag}/api/inbox
   - Include signed action token

4. Return success

Connection Establishment

Connections require mutual agreement:

Alice sends CONN to Bob
  ↓
Bob receives, stores CONN
  ↓
Bob sends CONN to Alice
  ↓
Alice receives, detects mutual CONN
  ↓
Connection established (both sides)

Connection vs Following

Aspect Following Connection
Direction One-way Bidirectional
Consent None required Mutual agreement
Trust level Low High (auto-approval)
Use case Content subscription Direct messaging, trusted sharing

Unfollowing

Unfollowing creates a new action that supersedes the previous follow:

Algorithm: Unfollow User

Input: follower_id_tag, target_id_tag
Output: Result<()>

1. Create FLLW action token with removed=true:
   - issuer: follower's id_tag
   - subject: target's id_tag
   - action type: "FLLW"
   - removed: true (indicates unfollow)
   - Sign with follower's private key (ES384)

2. Store action locally:
   - New FLLW action with removed=true
   - Supersedes previous FLLW action
   - Marks unfollow event

3. Send to target instance:
   - POST https://cl-o.{target_id_tag}/api/inbox
   - Include signed action token

4. Return success

The removed=true flag indicates this action cancels the previous follow.

Disconnecting

Similar to unfollowing, disconnection uses the removed=true flag:

Algorithm: Disconnect

Input: user_id_tag, target_id_tag
Output: Result<()>

1. Create CONN action token with removed=true
2. Store locally (marks disconnection)
3. Send to target instance
4. Target removes connection status

See Also

Security Considerations

Security model, spam prevention, and protection mechanisms for federated communication.

Trust Model

DNS-Based Trust

  • Domain ownership proves identity
  • TLS certificates prove server authenticity
  • Action signatures prove content authenticity

Progressive Trust

  • Initial federation is cautious
  • Trust builds through successful interactions
  • Users can block instances/users

Spam Prevention

Relationship-Based Action Acceptance

Algorithm: Should Accept Action

Input: action_type, issuer, local_user
Output: bool (accept or reject)

Decision logic by action_type:

1. "POST" β†’ Only from followed or connected users
   - Check relationship between issuer and local_user
   - Accept if: relationship.following OR relationship.connected
   - Reject if: stranger with no relationship

2. "CMNT" / "REACT" β†’ Always accept
   - Verification phase checks ownership of target content
   - Accept all, let verification handle validation

3. "CONN" / "FLLW" β†’ Always accept
   - Relationship requests always received
   - User can block after receiving

4. Other action types β†’ Reject

Federation Rate Limits

Default rate limits (hardcoded):

Limit Value Description
max_actions_per_instance_per_hour 1000 Per federated instance
max_actions_per_user_per_hour 100 Per remote user
max_concurrent_connections 100 Simultaneous federation connections
max_file_requests_per_hour 500 File sync requests

Blocklisting

Users can block instances or specific users:

Algorithm: Block Instance

Input: blocked_instance domain
Output: Result<()>

1. Add instance domain to user's blocklist
2. Store in metadata adapter
3. All future actions from this instance are rejected
4. Return success

User can later unblock by removing from blocklist.

Blocking Users

Individual users can be blocked without blocking the entire instance:

Algorithm: Block User

Input: blocked_user id_tag
Output: Result<()>

1. Add user id_tag to user's blocklist
2. Store in metadata adapter
3. All future actions from this user are rejected
4. Return success

Signature Verification

All federated actions must pass signature verification:

  1. JWT signature - Proves token signed by claimed issuer
  2. Key fetch - Public key retrieved from issuer’s instance
  3. Expiration check - Token not expired
  4. Audience check - Token intended for this instance

See Key Verification for details.

See Also

Operations Guide

Monitoring, best practices, and troubleshooting for federation.

Monitoring & Observability

Metrics

FederationMetrics Structure:

  • outbound_actions_sent (AtomicU64) - Successfully sent federation actions
  • outbound_actions_failed (AtomicU64) - Failed outbound deliveries
  • inbound_actions_received (AtomicU64) - Received federated actions
  • inbound_actions_rejected (AtomicU64) - Rejected actions (spam, invalid, blocked)
  • profiles_synced (AtomicU64) - Remote profiles cached/updated
  • files_synced (AtomicU64) - Attachment files downloaded
  • active_federation_connections (AtomicUsize) - Open federation connections

Logging

Sent Action Log:

  • instance: Target instance domain
  • action_type: Type of action (POST, FLLW, etc.)
  • action_id: Unique action identifier

Rejected Action Log:

  • instance: Source instance domain
  • action_type: Type of action
  • reason: Why rejected (spam, invalid sig, blocked, etc.)

Best Practices

For Instance Operators

βœ… Enable HTTPS: Always use TLS for federation βœ… Monitor logs: Watch for spam or abuse βœ… Set rate limits: Protect against DoS βœ… Backup regularly: Federation doesn’t replace backups βœ… Update promptly: Security patches are critical

For Developers

βœ… Verify signatures: Never trust unverified content βœ… Check relationships: Enforce connection requirements βœ… Handle failures: Network is unreliable βœ… Cache wisely: Balance freshness vs performance βœ… Test federation: Use multiple instances for testing

Troubleshooting

Actions Not Federating

Symptoms: Actions created locally but not received on remote instances

Checks:

  1. DNS resolution of target instance
  2. TLS certificate validity
  3. Firewall rules (port 443)
  4. Federation logs for errors

Common causes:

  • Target instance unreachable
  • Certificate expired
  • Rate limited

Signature Verification Failures

Symptoms: Inbound actions rejected with signature errors

Checks:

  1. Issuer’s public key is fetchable (GET /api/me)
  2. Key expiration
  3. Algorithm matches (ES384)

Common causes:

  • Key rotated without propagation
  • Clock skew between instances
  • Corrupted token in transit

File Sync Failures

Symptoms: Attachments not displaying, download errors

Checks:

  1. Content hash computation
  2. Blob adapter permissions
  3. Sufficient storage space

Common causes:

  • Hash mismatch (corrupted file)
  • Disk full
  • Permission denied

Profile Sync Issues

Symptoms: Stale profile data, missing avatars

Checks:

  1. Cache TTL (24h default)
  2. Remote instance reachable
  3. Profile endpoint returning valid data

Common causes:

  • Cache not invalidating
  • Remote instance down
  • Profile changed but not refreshed

See Also

Runtime Systems

Cloudillo’s internal runtime systems that coordinate task execution, manage concurrent processing, and enable real-time communication.

Core Systems

  • Task Scheduler - Asynchronous task scheduling and dependency management
  • Worker Pool - Concurrent task execution with pooled workers
  • WebSocket Bus - Real-time bidirectional communication infrastructure

Subsections of Runtime Systems

Task Scheduler System

Cloudillo’s Task Scheduler is a sophisticated persistent task execution system that enables reliable, async background processing with dependency management, automatic retries, and cron-style scheduling. This system is critical for federation, file processing, and any operations that need to survive server restarts.

Why Task-Based Processing?

Traditional async tasks (like tokio::spawn) have significant limitations for production systems:

Problems with Simple Async:

  • ❌ Lost on restart: Server restarts lose all pending tasks
  • ❌ No retry logic: Failures require manual handling
  • ❌ No dependencies: Can’t wait for other tasks to complete
  • ❌ No scheduling: Can’t run tasks at specific times
  • ❌ No observability: Hard to track task progress/failures

Task Scheduler Solutions:

  • βœ… Persistent: Tasks survive server restarts via MetaAdapter
  • βœ… Automatic Retry: Exponential backoff with configurable limits
  • βœ… Dependency Tracking: Tasks wait for dependencies (DAG)
  • βœ… Cron Scheduling: Recurring tasks (daily, hourly, etc.)
  • βœ… Observable: Track task status, retries, and failures
  • βœ… Priority Queues: High/medium/low priority execution

Architecture Overview

Components

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Task Scheduler                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Task Registry                        β”‚  β”‚
β”‚  β”‚  - TaskBuilder functions             β”‚  β”‚
β”‚  β”‚  - Task type mapping                 β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Task Queue                           β”‚  β”‚
β”‚  β”‚  - Pending tasks                     β”‚  β”‚
β”‚  β”‚  - Running tasks                     β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Dependency Tracker                   β”‚  β”‚
β”‚  β”‚  - DAG (Directed Acyclic Graph)      β”‚  β”‚
β”‚  β”‚  - Waiting tasks                     β”‚  β”‚
β”‚  β”‚  - Completion notifications          β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Retry Manager                        β”‚  β”‚
β”‚  β”‚  - Exponential backoff               β”‚  β”‚
β”‚  β”‚  - Retry counters                    β”‚  β”‚
β”‚  β”‚  - Backoff timers                    β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Cron Scheduler                       β”‚  β”‚
β”‚  β”‚  - Cron expressions                  β”‚  β”‚
β”‚  β”‚  - Next execution time               β”‚  β”‚
β”‚  β”‚  - Recurring task tracking           β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                    ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ MetaAdapter (Persistence)                  β”‚
β”‚  - Task metadata storage                   β”‚
β”‚  - Task status tracking                    β”‚
β”‚  - Dependency relationships                β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                    ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Worker Pool (Execution)                    β”‚
β”‚  - High priority workers                   β”‚
β”‚  - Medium priority workers                 β”‚
β”‚  - Low priority workers                    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Execution Flow

1. Task Created
   ↓
2. Store in MetaAdapter (persistence)
   ↓
3. Check Dependencies / scheduling
   β”œβ”€ Has dependencies? β†’ Wait
   └─ No dependencies? β†’ Queue
   ↓
4. Add to Priority Queue
   ↓
5. Execute Task
   β”œβ”€ Success β†’ Mark complete, notify dependents
   β”œβ”€ Failure (transient) β†’ Schedule retry with backoff
   └─ Failure (permanent) β†’ Mark failed, notify dependents
   ↓
6. Trigger Dependent Tasks

Task Trait System

All tasks implement the Task<S> trait:

#[async_trait]
pub trait Task<S: Clone>: Send + Sync + Debug {
    /// Task type identifier (e.g., "ActionCreator", "ImageResizer")
    fn kind() -> &'static str where Self: Sized;

    /// Build task from serialized context
    fn build(id: TaskId, context: &str) -> ClResult<Arc<dyn Task<S>>>
    where Self: Sized;

    /// Serialize task for persistence
    fn serialize(&self) -> String;

    /// Execute the task
    async fn run(&self, state: &S) -> ClResult<()>;

    /// Get task type of this instance
    fn kind_of(&self) -> &'static str;
}

Generic State Parameter

The S parameter is the application state (typically Arc<AppState>):

// Task receives app state for accessing adapters
async fn run(&self, state: &Arc<AppState>) -> ClResult<()> {
    // Access adapters through state
    state.meta_adapter.create_action(...).await?;
    state.blob_adapter.create_blob(...).await?;
    Ok(())
}

Task Serialization

Tasks must serialize to strings for persistence:

MyTask.serialize():
    return JSON.stringify(self)

MyTask.build(id, context):
    task = JSON.parse(context)
    return Arc(task)

Key Features

1. Dependency Tracking

Tasks can depend on other tasks forming a Directed Acyclic Graph (DAG):

# Create file processing task
file_task_id = scheduler
    .task(FileIdGeneratorTask(temp_path))
    .schedule()

# Create image resizing task that depends on file task
resize_task_id = scheduler
    .task(ImageResizerTask(file_id, "hd"))
    .depend_on([file_task_id])    # Wait for file task
    .schedule()

# Create action that depends on both
action_task_id = scheduler
    .task(ActionCreatorTask(action_data))
    .depend_on([file_task_id, resize_task_id])  # Wait for both
    .schedule()

Dependency Resolution:

FileIdGeneratorTask (no dependencies)
  ↓ completes
ImageResizerTask (depends on file task)
  ↓ completes
ActionCreatorTask (depends on both)
  ↓ executes

Cycle Detection: The scheduler detects and rejects circular dependencies:

# This would be rejected
task_a.depend_on([task_b])
task_b.depend_on([task_a])  # Error: cycle detected

2. Exponential Backoff Retry

Tasks can automatically retry on failure with increasing delays:

retry_policy = RetryPolicy(
    min_wait: 10s
    max_wait: 3600s (1 hour)
    max_attempts: 5
)

scheduler
    .task(ActionDeliveryTask(action_token, recipient))
    .with_retry(retry_policy)
    .schedule()

Retry Schedule:

Attempt 1: Immediate
  ↓ fails
Attempt 2: 10s later (min wait)
  ↓ fails
Attempt 3: 20s later (2x backoff)
  ↓ fails
Attempt 4: 40s later (2x backoff)
  ↓ fails
Attempt 5: 80s later (capped at max)
  ↓ fails
Attempt 6: Not attempted (max retries reached)
  ↓
Task marked as failed

Retry Logic:

struct RetryPolicy {
    min_wait_secs: u64      # Minimum wait between retries
    max_wait_secs: u64      # Maximum wait between retries
    max_attempts: u32       # Maximum retry attempts
}

Wait calculation:

calculate_wait(attempt):
    wait = min_wait_secs * 2^(attempt - 1)
    wait = min(wait, max_wait_secs)
    return wait

3. Cron Scheduling

Tasks can run on a schedule using cron expressions:

# Every day at 2:30 AM
scheduler
    .task(CleanupTask(temp_dir))
    .daily_at(2, 30)
    .schedule()

# Every hour at :00
scheduler
    .task(BackupTask(db_path))
    .hourly_at(0)
    .schedule()

# Every Monday at 9:00 AM
scheduler
    .task(WeeklyReportTask())
    .weekly_on(Weekday::Monday, 9, 0)
    .schedule()

# 1st of every month at midnight
scheduler
    .task(MonthlyTask())
    .monthly_on(1, 0, 0)
    .schedule()

Cron Expression Format:

minute hour day month weekday

* * * * *  (every minute)
0 * * * *  (every hour at :00)
0 0 * * *  (every day at midnight)
0 0 * * 1  (every Monday at midnight)
0 0 1 * *  (1st of month at midnight)

Fields:

  • minute: 0-59
  • hour: 0-23
  • day: 1-31
  • month: 1-12
  • weekday: 0-6 (0 = Sunday)

4. Persistence

Tasks are persisted to MetaAdapter and survive server restarts:

start_scheduler(app):
    scheduler = Scheduler(app)

    # Load unfinished tasks from database
    pending_tasks = app.meta_adapter.list_pending_tasks()

    for task_meta in pending_tasks:
        # Rebuild task from serialized context
        task = TaskRegistry.build(
            task_meta.kind,
            task_meta.id,
            task_meta.context
        )

        # Re-queue task
        scheduler.enqueue(task_meta.id, task)

    # Start processing
    scheduler.start()

Task States:

TaskStatus:
    Pending     β†’ Created, not yet run
    Completed   β†’ Successfully finished
    Failed      β†’ Permanently failed

Task Store Implementations

The scheduler supports pluggable storage backends via the TaskStore trait. Two implementations are provided:

InMemoryTaskStore

A non-persistent store for testing and development:

struct InMemoryTaskStore {
    tasks: HashMap<TaskId, TaskMetadata>,
    pending_queue: VecDeque<TaskId>,
}

impl TaskStore for InMemoryTaskStore {
    fn store_task(&mut self, task: TaskMetadata) -> Result<()>;
    fn load_task(&self, id: TaskId) -> Option<TaskMetadata>;
    fn update_status(&mut self, id: TaskId, status: TaskStatus) -> Result<()>;
    fn list_pending(&self) -> Vec<TaskMetadata>;
    fn remove_task(&mut self, id: TaskId) -> Result<()>;
}

Characteristics:

  • Fast: No I/O overhead
  • Non-persistent: Lost on restart
  • Use case: Unit tests, integration tests, development

MetaAdapterTaskStore

The production store using the MetaAdapter for persistence:

struct MetaAdapterTaskStore {
    meta_adapter: Arc<MetaAdapter>,
}

impl TaskStore for MetaAdapterTaskStore {
    fn store_task(&mut self, task: TaskMetadata) -> Result<()> {
        // INSERT INTO tasks (id, kind, context, status, ...)
        self.meta_adapter.create_task(task).await
    }

    fn load_task(&self, id: TaskId) -> Option<TaskMetadata> {
        // SELECT * FROM tasks WHERE id = ?
        self.meta_adapter.read_task(id).await
    }

    fn update_status(&mut self, id: TaskId, status: TaskStatus) -> Result<()> {
        // UPDATE tasks SET status = ? WHERE id = ?
        self.meta_adapter.update_task_status(id, status).await
    }

    fn list_pending(&self) -> Vec<TaskMetadata> {
        // SELECT * FROM tasks WHERE status = 'pending' ORDER BY priority, created_at
        self.meta_adapter.list_pending_tasks().await
    }

    fn remove_task(&mut self, id: TaskId) -> Result<()> {
        // DELETE FROM tasks WHERE id = ?
        self.meta_adapter.delete_task(id).await
    }
}

Characteristics:

  • Persistent: Survives server restarts
  • Transactional: ACID guarantees via SQLite
  • Indexed: Fast queries on status, priority, scheduled time
  • Use case: Production deployments

Choosing a Store

// Development/Testing
let store = InMemoryTaskStore::new();
let scheduler = Scheduler::new(store, app_state);

// Production
let store = MetaAdapterTaskStore::new(meta_adapter.clone());
let scheduler = Scheduler::new(store, app_state);

The scheduler is store-agnostic, allowing easy switching between implementations.


Built-in Task Types

ActionCreatorTask

Creates and signs action tokens for federation.

Purpose: Generate cryptographically signed action tokens

Usage:

task = ActionCreatorTask {
    tn_id: alice_tn_id
    action_type: "POST"
    content: post_data
    attachments: ["f1~abc123"]
    audience: null
}

task_id = scheduler
    .task(task)
    .depend_on([file_task_id])  # Wait for file upload
    .schedule()

What it does:

  1. Loads private signing key from AuthAdapter
  2. Builds JWT claims (iss, iat, t, c, p, a, etc.)
  3. Signs JWT with ES384 (P384 key)
  4. Computes action ID (SHA256 hash of token)
  5. Stores action in MetaAdapter
  6. Returns action ID

Location: server/src/action/action.rs


ActionVerifierTask

Validates incoming federated action tokens.

Purpose: Verify action tokens from remote instances

Usage:

# Triggered when receiving action at /api/inbox
task = ActionVerifierTask {
    token: incoming_token
}

scheduler
    .task(task)
    .schedule()

What it does:

  1. Decodes JWT without verification
  2. Fetches issuer’s public key from their /api/me/keys
  3. Verifies ES384 signature
  4. Checks expiration (if present)
  5. Validates permissions (following/connected status)
  6. Downloads missing file attachments
  7. Stores verified action in MetaAdapter

Location: server/src/action/action.rs


ActionDeliveryTask

Delivers action tokens to remote instances with retry.

Purpose: Federate actions to other Cloudillo instances

Usage:

task = ActionDeliveryTask {
    action_token: signed_token
    recipient: "bob.example.com"
}

retry_policy = RetryPolicy(min: 10s, max: 3600s, max_attempts: 5)

scheduler
    .task(task)
    .with_retry(retry_policy)
    .schedule()

What it does:

  1. POSTs action token to https://cl-o.{recipient}/api/inbox
  2. On success: Mark as delivered
  3. On temporary failure: Schedule retry with exponential backoff
  4. On permanent failure: Log and mark as failed

Retry behavior:

  • Network errors β†’ Retry
  • 5xx errors β†’ Retry
  • 4xx errors β†’ Don’t retry (permanent failure)
  • Timeout β†’ Retry

Location: server/src/action/action.rs


FileIdGeneratorTask

Generates content-addressed file IDs using SHA256.

Purpose: Create immutable file IDs for blob storage

Usage:

task = FileIdGeneratorTask {
    tn_id: alice_tn_id
    temp_file_path: "/tmp/upload-xyz"
    original_filename: "photo.jpg"
}

task_id = scheduler
    .task(task)
    .schedule()

What it does:

  1. Opens file from temp path
  2. Computes SHA256 hash (streaming for large files)
  3. Encodes hash as base64url
  4. Formats as f1~{hash}
  5. Moves file to permanent BlobAdapter storage
  6. Stores file metadata in MetaAdapter

Location: server/src/file/file.rs


ImageResizerTask

Generates image variants (thumbnails, different sizes).

Purpose: Create optimized image variants for different use cases

Usage:

task = ImageResizerTask {
    tn_id: alice_tn_id
    source_file_id: "f1~abc123"
    variant: "hd"  # tn, sd, md, hd, xd
    target_width: 1920
    target_height: 1080
    format: "avif"
    quality: 85
}

scheduler
    .task(task)
    .depend_on([file_id_task])  # Wait for original upload
    .priority(Priority::Low)    # CPU-intensive, low priority
    .schedule()

What it does:

  1. Loads source image from BlobAdapter
  2. Resizes with Lanczos3 filter (high quality)
  3. Encodes to target format (AVIF/WebP/JPEG)
  4. Uses worker pool for CPU-intensive work
  5. Computes variant ID (SHA256)
  6. Stores variant in BlobAdapter
  7. Updates file metadata with variant info

Location: server/src/file/image.rs


VideoTranscoderTask

Transcodes video files to web-optimized formats.

Purpose: Create streaming-ready video variants (WebM, HLS, DASH)

Usage:

task = VideoTranscoderTask {
    tn_id: alice_tn_id
    source_file_id: "f1~video123"
    target_format: "webm"
    resolution: "720p"
    bitrate: "2M"
}

scheduler
    .task(task)
    .depend_on([file_id_task])
    .priority(Priority::Low)
    .schedule()

What it does:

  1. Loads source video from BlobAdapter
  2. Spawns ffmpeg process for transcoding
  3. Creates HLS/DASH segments if requested
  4. Computes variant IDs
  5. Stores variants in BlobAdapter
  6. Updates file metadata with variants

PdfProcessorTask

Extracts text and metadata from PDF files.

Purpose: Enable PDF search and preview generation

What it does:

  1. Extracts text content for full-text search
  2. Generates page thumbnails
  3. Extracts metadata (author, title, page count)
  4. Stores extracted text in MetaAdapter

AudioExtractorTask

Extracts metadata from audio files.

Purpose: Extract audio metadata for media library

What it does:

  1. Extracts ID3 tags (artist, album, track)
  2. Computes audio duration
  3. Generates waveform preview
  4. Stores metadata in MetaAdapter

Location: server/src/file/audio.rs


EmailSenderTask

Sends emails asynchronously with retry support.

Purpose: Reliable email delivery for notifications, verifications, etc.

What it does:

  1. Loads email template
  2. Renders email content
  3. Sends via SMTP (lettre crate)
  4. Handles delivery failures with retry

Location: server/src/email/task.rs


CertRenewalTask

Handles automatic TLS certificate renewal via ACME.

Purpose: Ensure TLS certificates stay valid

Usage:

# Scheduled daily to check certificates
scheduler
    .task(CertRenewalTask())
    .daily_at(2, 0)  # 2:00 AM
    .schedule()

What it does:

  1. Checks certificate expiration dates
  2. Identifies certificates needing renewal (within 30 days)
  3. Initiates ACME challenges (HTTP-01)
  4. Stores renewed certificates in AuthAdapter

Location: server/src/core/acme.rs


ProfileRefreshBatchTask

Batch-refreshes profiles from remote instances.

Purpose: Keep cached remote profile data up-to-date

What it does:

  1. Lists profiles due for refresh
  2. Fetches updated profile data from remote instances
  3. Updates local profile cache
  4. Handles federation errors gracefully

Location: server/src/profile/sync.rs


TenantImageUpdaterTask

Updates tenant avatar and banner images.

Purpose: Process and store tenant profile images

What it does:

  1. Loads uploaded image
  2. Generates required variants (avatar sizes)
  3. Stores variants in BlobAdapter
  4. Updates tenant metadata

Location: server/src/profile/media.rs


Builder Pattern API

The scheduler uses a fluent builder API for task configuration:

Basic Usage

scheduler
    .task(MyTask(data))
    .schedule()

With Key (Idempotency)

scheduler
    .task(MyTask(data))
    .key("unique-key")  # For task identification
    .schedule()

If a task with the same key already exists, scheduling is skipped.

With Dependencies

scheduler
    .task(MyTask(data))
    .depend_on([task_id_1, task_id_2])
    .schedule()

With Retry Policy

retry = RetryPolicy(min: 10s, max: 3600s, max_attempts: 5)

scheduler
    .task(MyTask(data))
    .with_retry(retry)
    .schedule()

With Cron Schedule

# Daily at 2:30 AM
scheduler.task(MyTask(data)).daily_at(2, 30).schedule()

# Hourly at :00
scheduler.task(MyTask(data)).hourly_at(0).schedule()

# Weekly on Monday at 9:00 AM
scheduler.task(MyTask(data)).weekly_on(Weekday::Monday, 9, 0).schedule()

# Monthly on 1st at midnight
scheduler.task(MyTask(data)).monthly_on(1, 0, 0).schedule()

# Custom cron expression
scheduler.task(MyTask(data)).cron("0 0 * * 1").schedule()  # Monday midnight

Combining Multiple Options

retry = RetryPolicy(min: 10s, max: 3600s, max_attempts: 5)

scheduler
    .task(ActionDeliveryTask(token, recipient))
    .key(format("deliver-{}-{}", action_id, recipient))
    .depend_on([action_creator_task])
    .with_retry(retry)
    .priority(Priority::Medium)
    .schedule()

Creating Custom Task Types

Step 1: Define Task Struct

struct EmailTask {
    to: String
    subject: String
    body: String
}

impl EmailTask {
    new(to, subject, body):
        return EmailTask { to, subject, body }
}

Step 2: Implement Task Trait

impl Task<App> for EmailTask:
    kind() β†’ "EmailTask"
    kind_of() β†’ "EmailTask"
    serialize() β†’ JSON.stringify(self)
    build(id, context) β†’ JSON.parse(context)

    run(state):
        client = new HttpClient()
        client.post("https://api.emailservice.com/send")
              .json({"to": self.to, "subject": self.subject, "body": self.body})
              .send()
        log("Email sent to {}", self.to)

Step 3: Register Task Type

# In scheduler initialization
scheduler.register(EmailTask, "EmailTask", EmailTask.build)

Step 4: Use the Task

task = EmailTask(
    to: "user@example.com"
    subject: "Welcome!"
    body: "Welcome to Cloudillo!"
)

retry = RetryPolicy(min: 30s, max: 3600s, max_attempts: 3)

scheduler
    .task(task)
    .with_retry(retry)
    .schedule()

Integration with Worker Pool

CPU-intensive tasks should use the worker pool:

impl Task<App> for ImageResizerTask:
    run(state):
        # Load image (async I/O)
        image_data = state.blob_adapter.read_blob_buf(
            self.tn_id, self.source_file_id
        )

        # Resize image (CPU-intensive - use worker pool)
        resized = state.worker.run(closure:
            img = image.load_from_memory(image_data)
            resized = img.resize(
                self.target_width,
                self.target_height,
                FilterType::Lanczos3
            )
            # Encode to format
            buffer = resized.encode_to(self.format)
            return buffer
        )

        # Store variant (async I/O)
        variant_id = compute_file_id(resized)
        state.blob_adapter.create_blob_buf(
            self.tn_id, variant_id, resized
        )

When to use worker pool:

  • Image/video processing
  • Compression/decompression
  • Cryptographic operations
  • Complex computations
  • Any blocking I/O

When NOT to use worker pool:

  • Simple async I/O (database, network)
  • Quick operations (<1ms)

Examples

Example 1: Simple Task

# Create a simple cleanup task
task = TempFileCleanupTask {
    directory: "/tmp/uploads"
    older_than_hours: 24
}

scheduler.task(task).schedule()

Example 2: Task with Dependencies

# Upload file
file_task = FileIdGeneratorTask(temp_path)
file_task_id = scheduler.task(file_task).schedule()

# Generate thumbnails (depends on file upload)
thumb_task = ImageResizerTask(file_id, "tn")
scheduler
    .task(thumb_task)
    .depend_on([file_task_id])
    .schedule()

Example 3: Task with Retry

# Federation delivery with retry
task = ActionDeliveryTask {
    action_token: signed_token
    recipient: "remote.example.com"
}

retry = RetryPolicy(min: 10s, max: 3600s, max_attempts: 5)

scheduler
    .task(task)
    .with_retry(retry)
    .schedule()

Example 4: Scheduled Task

# Daily backup at 2:00 AM
task = BackupTask {
    source_dir: "/data"
    dest_dir: "/backups"
}

scheduler.task(task).daily_at(2, 0).schedule()

Example 5: Complex Workflow

# 1. Upload file
upload_task = FileIdGeneratorTask(temp_path)
upload_id = scheduler.task(upload_task).schedule()

# 2. Generate variants (parallel, all depend on upload)
variant_ids = []
for variant in ["tn", "sd", "md", "hd"]:
    task = ImageResizerTask(file_id, variant)
    id = scheduler
        .task(task)
        .depend_on([upload_id])
        .priority(Priority::Low)
        .schedule()
    variant_ids.push(id)

# 3. Create action (depends on all variants)
action_task = ActionCreatorTask(post_data)
scheduler
    .task(action_task)
    .depend_on(variant_ids)
    .schedule()

Monitoring & Debugging

Task Status Queries

status = scheduler.get_task_status(task_id)

switch status:
    case Pending: print("Task waiting to run")
    case Running: print("Task currently executing")
    case WaitingDeps: print("Task waiting for dependencies")
    case Retrying: print("Task failed, will retry")
    case Completed: print("Task finished successfully")
    case Failed: print("Task permanently failed")

List Tasks

# List all pending tasks
pending = scheduler.list_tasks(TaskStatus::Pending)

for task in pending:
    print("{}: {} ({})", task.id, task.kind, task.created_at)

Task Logs

# Enable task logging
RUST_LOG=cloudillo::scheduler=debug cargo run

# Logs show:
# [DEBUG] Scheduling task TaskId(123) of type ActionCreatorTask
# [DEBUG] Task TaskId(123) waiting for dependencies: [TaskId(122)]
# [INFO]  Task TaskId(122) completed successfully
# [DEBUG] Task TaskId(123) dependencies satisfied, queueing
# [INFO]  Executing task TaskId(123) (ActionCreatorTask)
# [INFO]  Task TaskId(123) completed in 42ms

Error Handling

Transient vs Permanent Failures

Tasks should distinguish between transient (retry) and permanent (fail) errors:

run(state):
    result = deliver_action(self.recipient)

    if result == Ok:
        return Ok

    # Transient errors - will retry
    if is_network_error(result):
        return error (will retry)
    if is_timeout(result):
        return error (will retry)
    if is_5xx_error(result):
        return error (will retry)

    # Permanent errors - won't retry
    if is_4xx_error(result):
        log_warn("Permanent delivery failure: {}", result)
        return PermanentFailure(result)

    return error

Handling Dependencies Failure

When a dependency fails, dependent tasks are notified:

# Task A fails
# Tasks B, C, D depend on A

# B can handle failure:
run(state):
    dep_error = check_dep_failures()

    if dep_error exists:
        # Use fallback
        return run_with_fallback(state)

    return run_normal(state)

# Or task can fail immediately when dependency fails

Performance Considerations

Task Granularity

Too Fine-Grained (Bad):

# Creating 1000 tiny tasks
for i in 0..1000:
    scheduler.task(ProcessItemTask(item_id: i)).schedule()
# Overhead: 1000 DB writes, 1000 queue operations

Appropriate Granularity (Good):

# One task processes batch
scheduler.task(ProcessBatchTask(item_ids: 0..1000)).schedule()
# Overhead: 1 DB write, 1 queue operation

Dependency Depth

Avoid deep dependency chains:

// Bad: Deep chain (slow to start)
A β†’ B β†’ C β†’ D β†’ E β†’ F

// Good: Wide graph (parallel execution)
    β”Œβ”€ B
A ──┼─ C
    β”œβ”€ D
    └─ E β†’ F

Priority Usage

Use priorities appropriately:

  • High: User-facing operations (<100/second)
  • Medium: Normal operations (default)
  • Low: Background tasks, cleanups

Too many high-priority tasks defeat the purpose.


Troubleshooting

Task Stuck in “Pending”

Possible causes:

  1. Waiting for dependencies

    deps = scheduler.get_task_dependencies(task_id)
    for dep in deps:
        print("Waiting for: {} ({})", dep.id, dep.status)
  2. Circular dependency (should be caught, but check)

    scheduler.check_for_cycles(task_id)
  3. Scheduler not running

    # Check if scheduler started
    scheduler.is_running()

Task Keeps Retrying

Causes:

  • Transient error not resolving
  • Wrong error classification (should be permanent)
  • Retry policy too aggressive

Solutions:

# 1. Check retry count
retry_count = scheduler.get_retry_count(task_id)

# 2. Check last error
last_error = scheduler.get_last_error(task_id)

# 3. Cancel task if stuck
scheduler.cancel_task(task_id)

# 4. Adjust retry policy
new_retry = RetryPolicy(min: 60s, max: 3600s, max_attempts: 3)

High Memory Usage

Causes:

  • Too many tasks in memory
  • Large task payloads
  • Not cleaning completed tasks

Solutions:

# 1. Limit concurrent tasks
MAX_CONCURRENT_TASKS = 100

# 2. Clean old completed tasks
scheduler.cleanup_completed_tasks(older_than_hours: 24)

# 3. Use pagination for large data
# Instead of embedding 10MB in task:
ProcessFileTask:
    file_id: String  # Reference, not data

Best Practices

1. Make Tasks Idempotent

Tasks may run multiple times (retries, restarts):

# Bad: Not idempotent
run(state):
    state.counter.increment()  # Runs twice = double increment

# Good: Idempotent
run(state):
    state.counter.set(42)  # Runs twice = same result

# Good: Check before acting
run(state):
    if NOT state.already_processed(self.id):
        state.counter.increment()
        state.mark_processed(self.id)

2. Use Task Keys for Deduplication

# Prevent duplicate tasks
key = format("deliver-{}-{}", action_id, recipient)

scheduler
    .task(ActionDeliveryTask(token, recipient))
    .key(key)  # Won't schedule if already exists
    .schedule()

3. Keep Task Payloads Small

# Bad: Large payload stored in DB
ProcessDataTask:
    data: Vec<u8>  # 10 MB!

# Good: Reference to data
ProcessDataTask:
    file_id: String  # Load from blob when needed

run(state):
    data = state.blob_adapter.read_blob(self.file_id)
    # Process data...

4. Clean Up Temporary Data

run(state):
    # Create temp file
    temp_path = create_temp_file()

    # Use temp file
    result = process_file(temp_path)

    # Always clean up (even on error)
    remove_file(temp_path)  # Safe even if fails

    return result

5. Log Progress for Long Tasks

run(state):
    items = load_items()
    total = items.length

    for (i, item) in enumerate(items):
        process_item(item)

        if i % 100 == 0:
            log("Progress: {}/{} ({}%)", i, total, i * 100 / total)

See Also

Worker Pool Architecture

Cloudillo’s Worker Pool provides a three-tier priority thread pool for executing CPU-intensive and blocking operations. This system complements the async runtime by handling work that shouldn’t block async tasks, ensuring responsive performance even under heavy computational load.

Why a Separate Worker Pool?

The async runtime (Tokio) is optimized for I/O-bound tasks, not CPU-bound work:

Problems with CPU Work on Async Runtime:

  • ❌ Blocks other tasks: Heavy CPU work prevents other async tasks from making progress
  • ❌ Thread starvation: Can exhaust async thread pool
  • ❌ Poor latency: User-facing requests wait for CPU work to complete
  • ❌ No prioritization: Can’t prioritize urgent CPU work over background processing

Worker Pool Solutions:

  • βœ… Dedicated threads: Separate from async runtime
  • βœ… Priority-based: Three tiers (High/Medium/Low) for different urgency levels
  • βœ… Work stealing: Efficient load balancing across workers
  • βœ… Future-based API: Async-friendly interface
  • βœ… Backpressure: Prevents overwhelming the system

Architecture Overview

Three-Tier Priority System

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ High Priority Queue (1 dedicated thread)  β”‚
β”‚  - User-facing operations                 β”‚
β”‚  - Time-sensitive tasks                   β”‚
β”‚  - Cryptographic signing                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Medium Priority Queue (2 threads)         β”‚
β”‚  - Processes: High + Medium               β”‚
β”‚  - Image resizing for posts               β”‚
β”‚  - File compression                       β”‚
β”‚  - Normal background work                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Low Priority Queue (1 thread)             β”‚
β”‚  - Processes: High + Medium + Low         β”‚
β”‚  - Batch operations                       β”‚
β”‚  - Cleanup tasks                          β”‚
β”‚  - Non-urgent processing                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Default Configuration:

  • High priority: 1 dedicated thread
  • Medium priority: 2 threads (also process High)
  • Low priority: 1 thread (also processes High + Medium)
  • Total threads: 4

Priority Cascading

Higher priority workers process lower priority work when idle:

Thread 1 (High):     High tasks only
Threads 2-3 (Med):   High β†’ Medium (if no High)
Thread 4 (Low):      High β†’ Medium β†’ Low (if none above)

Benefits:

  • High-priority work always has dedicated capacity
  • Lower-priority work still gets done when system is idle
  • Automatic load balancing

Core Components

WorkerPool Structure

pub struct WorkerPool {
    high_tx: flume::Sender<Job>,
    medium_tx: flume::Sender<Job>,
    low_tx: flume::Sender<Job>,
    handles: Vec<JoinHandle<()>>,
}

type Job = Box<dyn FnOnce() -> ClResult<Box<dyn Any + Send>> + Send>;

Initialization

Algorithm: Initialize Worker Pool

1. Create three unbounded message channels (high, medium, low)
2. Create thread handles vector
3. For each high-priority worker:
   - Clone high-priority receiver
   - Spawn worker thread (processes high priority only)
   - Store thread handle
4. For each medium-priority worker:
   - Clone high and medium receivers
   - Spawn cascading worker (processes high first, then medium)
   - Store thread handle
5. For each low-priority worker:
   - Clone all three receivers
   - Spawn cascading worker (processes high β†’ medium β†’ low)
   - Store thread handle
6. Return WorkerPool struct with senders and handles

Worker Thread

Basic Worker Spawning:

Algorithm: Spawn Single-Priority Worker

1. Spawn OS thread with receiver
2. Loop:
   a. Receive job from channel (blocking)
   b. Execute job
   c. Log success or error
   d. If channel disconnected, exit thread

Cascading Worker Spawning:

Algorithm: Spawn Multi-Priority Cascading Worker

1. Spawn OS thread with multiple receivers (high β†’ medium β†’ low)
2. Continuous loop:
   a. For each receiver in priority order:
      - Try to receive job (non-blocking)
      - If job available: execute, log result, restart loop
      - If empty: try next priority
      - If disconnected: exit thread
   b. If all queues empty: sleep 10ms and retry

This ensures high priority work is processed first, then medium, then low.

API Usage

Execute Method

Algorithm: Execute Work on Worker Pool

Input: priority level, CPU-bound function F
Output: Result<T>

1. Create async oneshot channel (tx, rx)
2. Wrap function F:
   - Execute F on worker thread
   - Send result through tx
   - Return success boxed as Any
3. Send wrapped job to appropriate priority queue:
   - High priority β†’ high_tx
   - Medium priority β†’ medium_tx
   - Low priority β†’ low_tx
4. Await result through rx (async, doesn't block executor)
5. Return result to caller

This pattern allows async code to offload CPU work without blocking the async runtime.

Priority Enum

pub enum Priority {
    High,    // User-facing, time-sensitive
    Medium,  // Normal operations
    Low,     // Background, non-urgent
}

When to Use Each Priority

High Priority (Dedicated Resources)

Use for:

  • βœ… Cryptographic operations during login
  • βœ… Image processing for profile picture uploads (user waiting)
  • βœ… Real-time compression/decompression
  • βœ… Time-sensitive computations

Characteristics:

  • User is actively waiting
  • Quick feedback required (<1 second)
  • Affects UX directly
  • Should be fast operations (<100ms typical)

Example Pattern:

// User uploading profile picture (waiting)
compressed = await worker.execute(Priority::High, || {
    compress_image(image_data, quality)
})

Guidelines:

  • Target volume: <100 jobs/second
  • Typical duration: <100ms
  • When not to use: Background processing, batch operations

Medium Priority (Default)

Use for:

  • βœ… Image processing for posts (async to user)
  • βœ… File compression for uploads
  • βœ… Data transformations
  • βœ… Most CPU-intensive work

Characteristics:

  • User submitted request but isn’t waiting
  • Processing happens in background
  • Results needed “soon” (seconds to minutes)
  • Default choice for most CPU work

Example Pattern:

// Generating image variants for a post
thumbnail = await worker.execute(Priority::Medium, || {
    resize_image(image_data, 200, 200)
})

Guidelines:

  • Target volume: <1000 jobs/second
  • Typical duration: 100ms - 10 seconds
  • When not to use: User actively waiting, or truly non-urgent

Low Priority (Background)

Use for:

  • βœ… Batch processing
  • βœ… Cleanup operations
  • βœ… Pre-generation of assets
  • βœ… Non-urgent optimization tasks

Characteristics:

  • No user waiting
  • Can take as long as needed
  • Won’t impact user-facing operations
  • Can be delayed indefinitely

Example Pattern:

// Batch thumbnail generation
for image in large_image_set:
    await worker.execute(Priority::Low, || {
        generate_all_variants(image)
    })

Guidelines:

  • Target volume: Variable (throttle if needed)
  • Typical duration: Seconds to minutes
  • When not to use: Anything time-sensitive

Integration Patterns

With Task Scheduler

Tasks often use the worker pool for CPU-intensive work following this pattern:

Algorithm: Task with Worker Pool Integration

1. Async I/O: Load data from blob storage
2. CPU work: Execute on worker pool:
   - Load image from memory
   - Resize using high-quality filter (Lanczos3)
   - Encode to desired format
   - Return resized buffer
3. Async I/O: Store result to blob storage

This separates I/O (async) from CPU (worker thread), allowing:
- The async runtime to continue processing other tasks during image resizing
- Background tasks (Low priority) not to starve user-facing operations
- Efficient resource utilization

With HTTP Handlers

Handlers can offload CPU work using the same async/worker/async pattern:

Algorithm: HTTP Handler with Worker Pool

1. Async I/O: Load file from blob storage
2. CPU work (Priority::High): Compress data with zstd encoder:
   - Create encoder with compression level
   - Write file data to encoder
   - Finish and return compressed buffer
3. Return compressed data as HTTP response

User is waiting (blocking on HTTP request), so High priority ensures
quick response time.

Parallel Processing

Process multiple items in parallel:

Algorithm: Parallel Image Processing

1. For each image in images:
   - Submit job to worker pool with Priority::Medium
   - Collect resulting futures
2. Await all futures in parallel using try_join_all
3. Return all processed images

This allows multiple images to be processed concurrently across
available worker pool threads while the async runtime continues
handling other tasks.

Configuration

Sizing the Worker Pool

Determining thread counts:

Configuration Options:

Conservative (leave cores for async runtime):
  high = 1
  medium = max(num_cpus / 4, 1)
  low = max(num_cpus / 4, 1)
  β†’ Reserve 50% of cores for async runtime

Aggressive (for CPU-heavy workloads):
  high = 2
  medium = max(num_cpus / 2, 2)
  low = max(num_cpus / 2, 1)
  β†’ Use most cores for worker threads

Default (balanced, recommended):
  high = 1
  medium = 2
  low = 1
  β†’ Suitable for mixed I/O and CPU workloads

Factors to consider:

  • CPU cores: More cores β†’ more workers
  • Workload type: Heavy CPU β†’ more workers
  • Memory: Each thread has stack (usually 2-8 MB)
  • Async runtime: Leave cores for Tokio

Environment Configuration

Configuration loading pattern:

high_workers = parse_env("WORKER_POOL_HIGH", default: 1)
medium_workers = parse_env("WORKER_POOL_MEDIUM", default: 2)
low_workers = parse_env("WORKER_POOL_LOW", default: 1)
worker_pool = WorkerPool::new(high_workers, medium_workers, low_workers)

Environment variables:

WORKER_POOL_HIGH=2    # 2 high-priority threads
WORKER_POOL_MEDIUM=4  # 4 medium-priority threads
WORKER_POOL_LOW=2     # 2 low-priority threads

Performance Characteristics

Throughput

High priority:

  • Theoretical max: ~100 jobs/second (assuming 10ms each)
  • Practical: 50-100 jobs/second
  • Bottleneck: Single dedicated thread

Medium priority:

  • Theoretical max: ~200 jobs/second (2 threads Γ— 100 jobs)
  • Practical: 100-200 jobs/second
  • Bottleneck: Thread count Γ— job duration

Low priority:

  • Throughput: Variable (depends on high/medium load)
  • Practical: 10-50 jobs/second when system busy

Latency

High priority:

  • Best case: <1ms (if queue empty)
  • Typical: 1-10ms
  • Worst case: <100ms (queued behind one job)

Medium priority:

  • Best case: <1ms (if queue empty)
  • Typical: 10-100ms
  • Worst case: Seconds (if many jobs queued)

Low priority:

  • Best case: <1ms (if all queues empty)
  • Typical: 100ms - 1s
  • Worst case: Minutes (if system busy)

Examples

Example 1: Image Processing

  • Load image from memory
  • Resize to 800Γ—600 using Lanczos3 filter (high quality)
  • Encode to PNG format
  • Use Priority::Medium (user submitted but not actively waiting)

Example 2: Compression

  • Compress data buffer using zstd compression level 3
  • Use Priority::Low (non-urgent background task)
  • Returns compressed buffer

Example 3: Cryptographic Operations

  • Sign JWT token with ES384 algorithm
  • Use Priority::High (user waiting on login)
  • User is blocked waiting for response

Example 4: Batch Processing

  • Process 1000+ items in chunks of 100
  • Use Priority::Low (background task)
  • Ensures individual chunks don’t block higher priority work
  • Each chunk: iterate items and process

Example 5: Parallel Execution

  • Submit multiple image processing jobs to worker pool
  • Collect futures from all jobs
  • Await all futures in parallel
  • Use Priority::Medium for concurrent image resizing
  • All images process in parallel across available worker threads

Best Practices

1. Choose the Right Priority

  • βœ… User waiting β†’ High: Direct user request, waiting for result
  • ❌ Background task β†’ High: Wastes dedicated capacity, delays urgent work
  • βœ… Background β†’ Low: Non-urgent work processed when queue is empty

2. Keep Work Units Small

  • ❌ Bad: One job processing 1,000,000 items (blocks that worker thread)
  • βœ… Good: Split into chunks of 1,000 items (allows interleaving with other jobs)
  • Benefit: Prevents starvation of higher priority work

3. Don’t Block on I/O in Worker

  • ❌ Bad: Blocking file I/O inside worker (blocks OS thread)
  • βœ… Good: Async I/O outside worker, pass result to worker
  • Pattern: async_io().await β†’ worker.execute() β†’ cpu_work

4. Handle Errors Appropriately

  • Wrap risky operations in match/try blocks
  • Log failures with context (which job, what error)
  • Match result and handle errors:
    • Transient errors: Retry
    • Permanent errors: Log and alert
    • Propagate critical errors up

5. Monitor Queue Depth

  • Periodically check queue depths for all priorities
  • Alert if High priority queue > 100 jobs (indicates bottleneck)
  • Use metrics to detect:
    • Capacity issues (need more workers)
    • Workload classification issues (wrong priorities)

Monitoring

Queue Metrics

WorkerPoolMetrics tracks:

  • high_queue_depth (AtomicUsize) - Current jobs in high priority queue
  • medium_queue_depth (AtomicUsize) - Current jobs in medium priority queue
  • low_queue_depth (AtomicUsize) - Current jobs in low priority queue
  • jobs_completed (AtomicU64) - Total successfully completed jobs
  • jobs_failed (AtomicU64) - Total failed jobs
  • total_execution_time (AtomicU64) - Cumulative job execution time in microseconds

Call metrics() to retrieve current state.

Logging

Enable debug logging:

RUST_LOG=cloudillo::worker=debug cargo run

Expected log output:

[DEBUG] Submitting job to high priority queue
[DEBUG] High priority worker executing job
[DEBUG] Job completed in 42ms
[WARN]  Medium priority queue depth: 150 jobs

Key log events:

  • Job submission to queue (which priority)
  • Worker thread starting job execution
  • Job completion with duration
  • Queue depth warnings when backed up

Troubleshooting

High Priority Queue Backed Up

Symptoms:

  • High priority jobs taking too long
  • User-facing operations slow

Causes:

  1. Too many high priority jobs being submitted
  2. Individual jobs taking >100ms (should be <100ms)
  3. Insufficient high priority workers

Solutions:

  1. Audit usage: Review code submitting High priority jobs, downgrade non-urgent work
  2. Increase workers: Set WORKER_POOL_HIGH=2
  3. Optimize duration: Profile and optimize slow operations

Worker Thread Panicked

Symptoms:

  • Worker thread count decreases
  • Jobs stop being processed

Causes:

  • Panic in job code (unwrap/expect on None/Err)
  • Out of memory condition
  • Stack overflow in recursive code

Solutions:

  1. Catch panics: Use std::panic::catch_unwind() to handle panics gracefully
  2. Panic hooks: Install panic hook to log worker panics and optionally respawn
  3. Increase stack: Use std::thread::Builder::new().stack_size(8 * 1024 * 1024)

Memory Issues

Symptoms:

  • OOM (Out of Memory) errors
  • Increasing memory usage over time

Causes:

  • Large job payloads being copied into closures
  • Memory leaks in job code (unreleased buffers)
  • Too many worker threads (each has 2-8 MB stack)

Solutions:

  1. Reduce worker count: Lower WORKER_POOL_MEDIUM and WORKER_POOL_LOW
  2. Stream instead of loading: Process data in chunks rather than loading entire file
  3. Profile memory: Use valgrind or heaptrack to identify leaks

Comparison: Worker Pool vs Async Runtime

Feature Worker Pool Async Runtime (Tokio)
Best for CPU-intensive work I/O-bound operations
Threading OS threads Green threads (tasks)
Blocking OK to block Must not block
Overhead Higher (context switch) Lower (cooperative)
Priorities 3 tiers No built-in priorities
Use case Image processing, crypto Network, file I/O

When to use Worker Pool:

  • βœ… CPU-heavy operations (>10ms of CPU time)
  • βœ… Blocking operations (unavoidable blocking)
  • βœ… Need priority control
  • βœ… Parallel CPU work

When to use Async Runtime:

  • βœ… I/O operations (network, disk)
  • βœ… Many concurrent operations
  • βœ… Quick operations (<10ms)
  • βœ… High concurrency needed (1000s of tasks)

See Also

WebSocket Bus

Cloudillo’s WebSocket Bus provides real-time notifications and presence tracking for connected clients. This is separate from the RTDB and CRDT WebSocket protocols, serving as a general-purpose notification system.

WebSocket Endpoints Overview

Cloudillo provides three WebSocket endpoints for different real-time use cases:

Endpoint Purpose Protocol Documentation
/ws/bus Notifications, presence, typing indicators JSON messages This page
/ws/rtdb/{file_id} Real-time database subscriptions Binary protocol RTDB Protocol
/ws/crdt/{doc_id} Collaborative document editing (Yjs) Binary sync protocol CRDT Protocol

When to Use Each Endpoint

  • Bus (/ws/bus): General notifications, presence, typingβ€”use when you need to know about events happening across the platform (new posts, messages, connection requests).
  • RTDB (/ws/rtdb/{file_id}): Real-time app stateβ€”use when your application needs live-updating data (todo lists, dashboards, game state).
  • CRDT (/ws/crdt/{doc_id}): Collaborative editingβ€”use when multiple users edit the same document simultaneously (text documents, whiteboards, spreadsheets).

Overview

The WebSocket Bus enables:

  • βœ… Real-time notifications - Action events, messages, updates
  • βœ… Presence tracking - Online/offline status of users
  • βœ… Typing indicators - Show when users are typing
  • βœ… Action broadcasts - New posts, comments, reactions
  • βœ… Live updates - Content changes without polling

WebSocket Endpoint

Connection

URL: wss://cl-o.{domain}/ws/bus

Authentication: Required via query parameter or header

// Connect with access token
const token = 'your_access_token';
const ws = new WebSocket(`wss://cl-o.example.com/ws/bus?token=${token}`);

// Or via Authorization header (if supported by client)
const ws = new WebSocket('wss://cl-o.example.com/ws/bus');
// Set Authorization header before connecting

Connection Flow

Client                          Server
  |                               |
  |--- GET /ws/bus?token=... ---->|
  |                               |--- Validate token
  |                               |--- Create session
  |                               |
  |<-- 101 Switching Protocols ---|
  |                               |
  |<===== WebSocket Open ========>|
  |                               |
  |<-- welcome message ----------|
  |                               |
  |--- subscribe message -------->|
  |                               |
  |<===== Event Stream ==========>|

Message Protocol

All messages use JSON format (not binary like RTDB/CRDT).

Message Structure

{
  "type": "message_type",
  "data": { ... },
  "timestamp": 1738483200
}

Client β†’ Server Messages

Subscribe to Events

Subscribe to specific event types:

{
  "type": "subscribe",
  "events": ["action", "presence", "typing"]
}

Event Types:

  • action - New actions (posts, comments, reactions)
  • presence - User online/offline status
  • typing - Typing indicators
  • notification - General notifications
  • message - Direct messages

Unsubscribe from Events

{
  "type": "unsubscribe",
  "events": ["typing"]
}

Send Presence Update

Update your online status:

{
  "type": "presence",
  "status": "online",
  "activity": "browsing"
}

Status values:

  • online - Active and available
  • away - Inactive but connected
  • busy - Do not disturb
  • offline - Explicitly offline

Send Typing Indicator

Notify others you’re typing:

{
  "type": "typing",
  "recipient": "bob.example.com",
  "conversation_id": "conv_123"
}

Auto-timeout: Typing indicators automatically expire after 5 seconds.

Heartbeat/Ping

Keep connection alive:

{
  "type": "ping"
}

Server responds with:

{
  "type": "pong",
  "timestamp": 1738483200
}

Server β†’ Client Messages

Welcome Message

Sent immediately after connection:

{
  "type": "welcome",
  "session_id": "sess_abc123",
  "user": "alice.example.com",
  "timestamp": 1738483200
}

Action Notification

Notify about new actions:

{
  "type": "action",
  "action_type": "POST",
  "action_id": "a1~xyz789...",
  "issuer": "bob.example.com",
  "content": "Check out this cool feature!",
  "timestamp": 1738483200
}

Action types: POST, REPOST, CMNT, REACT, MSG, CONN, FLLW, etc.

Presence Update

Notify about user status changes:

{
  "type": "presence",
  "user": "bob.example.com",
  "status": "online",
  "activity": "browsing",
  "timestamp": 1738483200
}

Sent when:

  • User connects/disconnects
  • User explicitly updates status
  • User goes idle (after 5 minutes of inactivity)

Typing Indicator

Notify that someone is typing:

{
  "type": "typing",
  "user": "bob.example.com",
  "conversation_id": "conv_123",
  "timestamp": 1738483200
}

Auto-clear: Client should clear typing indicator after 5 seconds if no update received.

Direct Message Notification

Notify about new direct messages:

{
  "type": "message",
  "message_id": "a1~msg789...",
  "sender": "bob.example.com",
  "preview": "Hey, are you available for a call?",
  "timestamp": 1738483200
}

Preview: First ~100 characters of message content.

General Notification

System notifications:

{
  "type": "notification",
  "notification_type": "connection_request",
  "from": "charlie.example.com",
  "message": "Charlie wants to connect with you",
  "action_url": "/connections/pending",
  "timestamp": 1738483200
}

Error Message

Error responses:

{
  "type": "error",
  "code": "E-WS-AUTH",
  "message": "Invalid or expired token",
  "timestamp": 1738483200
}

Common error codes:

  • E-WS-AUTH - Authentication failed
  • E-WS-INVALID - Invalid message format
  • E-WS-RATE - Rate limit exceeded
  • E-WS-PERMISSION - Permission denied

Client Implementation

JavaScript/Browser

class CloudilloBus {
  constructor(domain, token) {
    this.domain = domain;
    this.token = token;
    this.ws = null;
    this.handlers = new Map();
  }

  connect() {
    this.ws = new WebSocket(`wss://cl-o.${this.domain}/ws/bus?token=${this.token}`);

    this.ws.onopen = () => {
      console.log('Connected to WebSocket Bus');

      // Subscribe to events
      this.send({
        type: 'subscribe',
        events: ['action', 'presence', 'typing', 'message']
      });
    };

    this.ws.onmessage = (event) => {
      const message = JSON.parse(event.data);
      this.handleMessage(message);
    };

    this.ws.onerror = (error) => {
      console.error('WebSocket error:', error);
    };

    this.ws.onclose = () => {
      console.log('WebSocket closed');
      // Reconnect after 5 seconds
      setTimeout(() => this.connect(), 5000);
    };

    // Send ping every 30 seconds
    setInterval(() => {
      if (this.ws.readyState === WebSocket.OPEN) {
        this.send({ type: 'ping' });
      }
    }, 30000);
  }

  send(message) {
    if (this.ws.readyState === WebSocket.OPEN) {
      this.ws.send(JSON.stringify(message));
    }
  }

  on(eventType, handler) {
    if (!this.handlers.has(eventType)) {
      this.handlers.set(eventType, []);
    }
    this.handlers.get(eventType).push(handler);
  }

  handleMessage(message) {
    const handlers = this.handlers.get(message.type) || [];
    handlers.forEach(handler => handler(message.data || message));
  }

  updatePresence(status, activity) {
    this.send({
      type: 'presence',
      status,
      activity
    });
  }

  sendTyping(recipient, conversationId) {
    this.send({
      type: 'typing',
      recipient,
      conversation_id: conversationId
    });
  }

  disconnect() {
    if (this.ws) {
      this.ws.close();
    }
  }
}

// Usage
const bus = new CloudilloBus('example.com', accessToken);

bus.on('action', (data) => {
  console.log('New action:', data);
  // Update UI with new post/comment/etc
});

bus.on('presence', (data) => {
  console.log('Presence update:', data);
  // Update online status indicators
});

bus.on('typing', (data) => {
  console.log('User typing:', data);
  // Show typing indicator
});

bus.on('message', (data) => {
  console.log('New message:', data);
  // Show notification, update message list
});

bus.connect();

React Hook

import { useEffect, useState, useRef } from 'react';

export function useCloudilloBus(domain, token) {
  const [connected, setConnected] = useState(false);
  const [events, setEvents] = useState([]);
  const busRef = useRef(null);

  useEffect(() => {
    if (!token) return;

    const bus = new CloudilloBus(domain, token);
    busRef.current = bus;

    bus.on('welcome', () => setConnected(true));
    bus.on('action', (data) => {
      setEvents(prev => [...prev, { type: 'action', data }]);
    });
    bus.on('message', (data) => {
      setEvents(prev => [...prev, { type: 'message', data }]);
    });

    bus.connect();

    return () => {
      bus.disconnect();
      setConnected(false);
    };
  }, [domain, token]);

  return {
    connected,
    events,
    bus: busRef.current
  };
}

// Usage in component
function MyComponent() {
  const { connected, events, bus } = useCloudilloBus('example.com', accessToken);

  const handleTyping = () => {
    bus?.sendTyping('bob.example.com', 'conv_123');
  };

  return (
    <div>
      <p>Status: {connected ? 'Connected' : 'Disconnected'}</p>
      <ul>
        {events.map((event, i) => (
          <li key={i}>{event.type}: {JSON.stringify(event.data)}</li>
        ))}
      </ul>
    </div>
  );
}

Server Implementation

Connection Handler

use axum::{
  extract::{Query, State, WebSocketUpgrade},
  response::Response,
};
use serde::{Deserialize, Serialize};

#[derive(Deserialize)]
struct BusQuery {
  token: String,
}

pub async fn ws_bus_handler(
  ws: WebSocketUpgrade,
  Query(query): Query<BusQuery>,
  State(app): State<Arc<App>>,
) -> Response {
  // Validate access token
  let auth = match validate_access_token(&app, &query.token).await {
    Ok(auth) => auth,
    Err(_) => return (StatusCode::UNAUTHORIZED).into_response(),
  };

  // Upgrade to WebSocket
  ws.on_upgrade(move |socket| handle_bus_socket(socket, auth, app))
}

async fn handle_bus_socket(
  mut socket: WebSocket,
  auth: Auth,
  app: Arc<App>,
) {
  // Generate session ID
  let session_id = generate_session_id();

  // Send welcome message
  let welcome = json!({
    "type": "welcome",
    "session_id": session_id,
    "user": auth.id_tag,
    "timestamp": Utc::now().timestamp(),
  });

  if socket.send(Message::Text(welcome.to_string())).await.is_err() {
    return;
  }

  // Register session in bus
  app.bus.add_session(session_id.clone(), auth.id_tag.clone()).await;

  // Handle incoming messages
  while let Some(msg) = socket.recv().await {
    match msg {
      Ok(Message::Text(text)) => {
        if let Ok(message) = serde_json::from_str::<BusMessage>(&text) {
          handle_bus_message(message, &auth, &app, &mut socket).await;
        }
      }
      Ok(Message::Close(_)) => break,
      _ => {}
    }
  }

  // Cleanup session
  app.bus.remove_session(&session_id).await;
}

Message Handling

#[derive(Deserialize)]
#[serde(tag = "type")]
enum BusMessage {
  Subscribe { events: Vec<String> },
  Unsubscribe { events: Vec<String> },
  Presence { status: String, activity: Option<String> },
  Typing { recipient: String, conversation_id: String },
  Ping,
}

async fn handle_bus_message(
  message: BusMessage,
  auth: &Auth,
  app: &App,
  socket: &mut WebSocket,
) {
  match message {
    BusMessage::Subscribe { events } => {
      // Subscribe to events
      app.bus.subscribe(&auth.id_tag, events).await;
    }
    BusMessage::Presence { status, activity } => {
      // Broadcast presence update
      let update = json!({
        "type": "presence",
        "user": auth.id_tag,
        "status": status,
        "activity": activity,
        "timestamp": Utc::now().timestamp(),
      });

      app.bus.broadcast(&auth.id_tag, update).await;
    }
    BusMessage::Typing { recipient, conversation_id } => {
      // Send typing indicator to recipient
      let typing = json!({
        "type": "typing",
        "user": auth.id_tag,
        "conversation_id": conversation_id,
        "timestamp": Utc::now().timestamp(),
      });

      app.bus.send_to(&recipient, typing).await;
    }
    BusMessage::Ping => {
      // Respond with pong
      let pong = json!({
        "type": "pong",
        "timestamp": Utc::now().timestamp(),
      });

      socket.send(Message::Text(pong.to_string())).await.ok();
    }
    _ => {}
  }
}

Use Cases

Real-time Feed Updates

Show new posts as they’re created:

bus.on('action', (data) => {
  if (data.action_type === 'POST' && data.issuer !== currentUser) {
    showNotification(`New post from ${data.issuer}`);
    prependToFeed(data);
  }
});

Online Status Indicators

Track who’s online:

const onlineUsers = new Set();

bus.on('presence', (data) => {
  if (data.status === 'online') {
    onlineUsers.add(data.user);
  } else {
    onlineUsers.delete(data.user);
  }

  updateOnlineIndicators();
});

Typing Indicators in Chat

Show when someone is typing:

const typingUsers = new Map();

bus.on('typing', (data) => {
  typingUsers.set(data.user, Date.now());
  showTypingIndicator(data.user, data.conversation_id);

  // Clear after 5 seconds
  setTimeout(() => {
    if (Date.now() - typingUsers.get(data.user) >= 5000) {
      hideTypingIndicator(data.user);
      typingUsers.delete(data.user);
    }
  }, 5000);
});

// Send typing indicator when user types
messageInput.addEventListener('input', () => {
  bus.sendTyping(recipientId, conversationId);
});

Message Notifications

Instant message delivery:

bus.on('message', (data) => {
  playNotificationSound();
  showNotification(`New message from ${data.sender}`, data.preview);

  if (currentConversation === data.sender) {
    fetchAndDisplayMessage(data.message_id);
  } else {
    incrementUnreadCount(data.sender);
  }
});

Real-time Action Forwarding

The WebSocket Bus integrates with the action processing system to provide real-time updates when actions are created or received.

Outbound Forwarding

When a user creates an action locally, it’s forwarded to their connected WebSocket clients:

Outbound Action Forwarding:

1. User creates action via POST /api/actions
2. Action is processed and stored
3. ActionCreatorTask completes
4. Forward to WebSocket Bus:
   a. Find all WebSocket sessions for user's tenant
   b. Send action notification to each session
5. Client receives real-time update

Audience Targeting:
- POST: Broadcast to creator's sessions
- MSG: Forward to all conversation participant sessions
- CMNT/REACT: Forward to parent action owner's sessions

Outbound Message Format:

{
  "type": "action",
  "action_type": "POST",
  "action_id": "a1~xyz789...",
  "issuer": "alice.example.com",
  "content": "New post content",
  "timestamp": 1738483200,
  "source": "local"
}

Inbound Forwarding

When a federated action is received from a remote instance:

Inbound Action Forwarding:

1. Remote action arrives at POST /api/inbox
2. ActionVerifierTask verifies and stores action
3. After successful processing:
   a. Identify target tenant from action audience
   b. Find all WebSocket sessions for that tenant
   c. Broadcast action to all sessions
4. All connected clients see the new action

Broadcast Scope:
- All clients for the target tenant receive notification
- Client-side filtering determines what to display
- Enables real-time feed updates without polling

Inbound Message Format:

{
  "type": "action",
  "action_type": "POST",
  "action_id": "a1~abc123...",
  "issuer": "bob.remote.com",
  "content": "Federated post",
  "timestamp": 1738483200,
  "source": "federated"
}

Forwarding Flow Diagram

Local Action Creation:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Client  │───►│ API      │───►│ Task        │───►│ WebSocket    β”‚
β”‚ (POST)  β”‚    β”‚ Handler  β”‚    β”‚ Scheduler   β”‚    β”‚ Bus          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
                                                         β”‚
                                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                                    β–Ό                    β–Ό                    β–Ό
                              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                              β”‚ Session 1β”‚        β”‚ Session 2β”‚        β”‚ Session 3β”‚
                              β”‚ (Tab 1)  β”‚        β”‚ (Tab 2)  β”‚        β”‚ (Mobile) β”‚
                              β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Federated Action Reception:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Remote      │───►│ /api/    │───►│ Verifier    │───►│ WebSocket    β”‚
β”‚ Instance    β”‚    β”‚ inbox    β”‚    β”‚ Task        β”‚    β”‚ Bus          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
                                                              β”‚
                                                    Broadcast to all
                                                    tenant sessions

Push Notification Integration

For users who are offline (no active WebSocket connection), the system can send push notifications for important actions.

Push Notification Decision Tree

Push Notification Decision:

1. Action is processed successfully
   β”‚
2. Check if forwarded to WebSocket
   β”œβ”€ Yes β†’ User is online, skip push notification
   └─ No  β†’ User is offline, continue...
         β”‚
3. Check action type eligibility
   β”œβ”€ MSG (Message) β†’ Eligible
   β”œβ”€ CONN (Connection request) β†’ Eligible
   β”œβ”€ FSHR (File share) β†’ Eligible
   β”œβ”€ CMNT (Comment on user's content) β†’ Eligible
   └─ Other β†’ Not eligible, skip
         β”‚
4. Check user notification preferences
   β”œβ”€ Notifications disabled β†’ Skip
   └─ Notifications enabled β†’ Continue...
         β”‚
5. Check notification throttling
   β”œβ”€ Too many recent notifications β†’ Skip (prevent spam)
   └─ Within limits β†’ Continue...
         β”‚
6. Schedule push notification task

Push Notification Types

Action Type Push Notification Content
MSG Direct message “{sender} sent you a message”
CONN Connection request “{sender} wants to connect”
FSHR File share “{sender} shared a file with you”
CMNT Comment “{sender} commented on your post”
REACT Reaction Grouped, not individual pushes
FLLW Follow “{sender} started following you”

Push Task Scheduling

schedule_push_notification(action, recipient):
    # Check eligibility
    if NOT is_push_eligible(action.type):
        return

    # Check if user is online
    if websocket_bus.has_active_session(recipient):
        return  # User will see via WebSocket

    # Check user preferences
    prefs = load_notification_preferences(recipient)
    if NOT prefs.push_enabled:
        return

    if NOT prefs.allows_type(action.type):
        return

    # Check throttling
    recent_count = count_recent_pushes(recipient, window: 1 hour)
    if recent_count > MAX_PUSHES_PER_HOUR:
        return

    # Schedule the push
    task = PushNotificationTask {
        recipient: recipient,
        action_type: action.type,
        sender: action.issuer,
        preview: generate_preview(action),
        action_url: generate_deep_link(action),
    }

    scheduler.task(task).schedule()

Client Registration

Clients register for push notifications during WebSocket connection:

// During WebSocket setup
bus.send({
  type: 'register_push',
  token: pushNotificationToken,  // From FCM/APNs
  platform: 'android'  // or 'ios', 'web'
});

// Server stores token for offline delivery

Performance Considerations

Connection Limits

Per server instance:

  • Default: 10,000 concurrent connections
  • Configurable via MAX_WS_CONNECTIONS

Per user:

  • Default: 5 concurrent connections (multiple tabs/devices)
  • Configurable via MAX_WS_CONNECTIONS_PER_USER

Message Rate Limiting

Client β†’ Server:

  • 100 messages per minute per connection
  • Bursts of 20 messages allowed

Server β†’ Client:

  • No hard limit (trust server)
  • Throttling applied for presence updates (max 1 per second per user)

Reconnection Strategy

Client should implement:

  1. Exponential backoff (start at 1s, max 60s)
  2. Random jitter to prevent thundering herd
  3. Token refresh if expired
  4. Event replay from last known timestamp

Security Considerations

Authentication

  • Token required for all connections
  • Token validated on initial connection
  • Invalid token = immediate disconnect
  • Token expiration checked periodically

Authorization

  • Users only receive events they have permission to see
  • Presence updates only for connections/followers
  • Message notifications only for actual recipients
  • Action notifications respect ABAC visibility

Rate Limiting

  • Per-connection message rate limiting
  • Per-user connection limits
  • Typing indicator throttling
  • Presence update throttling

See Also

Rate Limiting

Overview

Cloudillo implements a sophisticated rate limiting system to protect against abuse, DDoS attacks, and credential stuffing. The system uses the Generic Cell Rate Algorithm (GCRA) with hierarchical address grouping and dual-tier limits.

Algorithm: GCRA

The rate limiter uses GCRA (Generic Cell Rate Algorithm), also known as the “leaky bucket as a meter” algorithm. GCRA provides:

  • Smooth rate limiting: Distributes requests evenly over time
  • Burst tolerance: Allows short bursts within limits
  • Memory efficiency: O(1) per key, no per-request storage
  • Precision: Nanosecond-level timing accuracy

Hierarchical Address Levels

Requests are rate-limited at multiple network levels simultaneously. All levels must pass for a request to succeed.

IPv4 Levels

Level Mask Description Example
Individual /32 Single IP address 192.168.1.100
Network /24 Class C network (256 IPs) 192.168.1.0/24

IPv6 Levels

Level Mask Description Example
Subnet /64 Single subnet 2001:db8:1234:5678::/64
Provider /48 ISP allocation 2001:db8:1234::/48
Multi-Level Protection

A single abusive IP can’t overwhelm the system, but neither can a botnet spread across a /24 network. Both individual and network-level limits apply.

Dual-Tier Limits

Each address level has two tiers of limits:

Tier Period Purpose
Short-term Per-second Burst protection
Long-term Per-hour Sustained abuse protection

Both tiers must pass. This prevents:

  • Burst attacks: Limited by short-term tier
  • Slow-and-steady attacks: Limited by long-term tier

Endpoint Categories

Different endpoints have different rate limit profiles:

Auth Category

Strict limits for authentication endpoints to prevent credential stuffing.

Endpoint Description
POST /api/auth/login User login
POST /api/profile/register New user registration
POST /api/auth/password Password reset

Default limits (per individual IPv4):

  • Short-term: 2 req/s, burst 5
  • Long-term: 30 req/h, burst 30

Federation Category

Moderate limits for inter-instance communication.

Endpoint Description
POST /api/inbox Receive federated actions
POST /api/inbox/sync Sync federated actions

Default limits (per individual IPv4):

  • Short-term: 5 req/s, burst 15
  • Long-term: 1000 req/h, burst 100

General Category

Relaxed limits for normal browsing and API usage.

Endpoint Description
GET /api/profiles/* Profile viewing
GET /api/actions/* Action listing
GET /api/files/* File access

Default limits (per individual IPv4):

  • Short-term: 20 req/s, burst 50
  • Long-term: 5000 req/h, burst 500

WebSocket Category

Moderate limits for WebSocket connections.

Endpoint Description
/ws/bus Notification bus
/ws/crdt/* Collaborative editing
/ws/rtdb/* Real-time database

Default limits (per individual IPv4):

  • Short-term: 10 req/s, burst 20
  • Long-term: 200 req/h, burst 100

Default Rate Limits Table

Category IPv4 Individual IPv4 Network IPv6 Subnet IPv6 Provider
Auth 2/s, 30/h 10/s, 300/h 20/s, 600/h 50/s, 3000/h
Federation 5/s, 1000/h 50/s, 5000/h 10/s, 5000/h 200/s, 20000/h
General 20/s, 5000/h 100/s, 50000/h 100/s, 50000/h 500/s, 200000/h
WebSocket 10/s, 200/h 50/s, 1000/h 50/s, 1000/h 200/s, 5000/h

Proof-of-Work Protection

CONN (connection request) actions require proof-of-work when suspicious behavior is detected from an IP address.

How It Works

  1. Violation detected: Failed signature, duplicate pending, or rejected CONN increments counter
  2. Counter determines requirement: Token must end with N ‘A’ characters (where N = counter)
  3. Two-level tracking: Both individual IP and network range are tracked
  4. Automatic decay: Counter decreases by 1 every hour without new violations

Violation Types

Reason Level Description
ConnSignatureFailure Individual + Network CONN action failed signature verification
ConnDuplicatePending Individual + Network Duplicate CONN while another is pending
ConnRejected Individual only CONN rejected by user or policy
ConnPowCheckFailed Network only Failed PoW verification

PoW Requirement

The PoW requirement is a simple suffix check on the action token:

Counter = 0: No requirement
Counter = 1: Token must end with "A"
Counter = 2: Token must end with "AA"
Counter = 3: Token must end with "AAA"
...
Counter = 10: Token must end with "AAAAAAAAAA"

When PoW is required, the server responds with HTTP 428 (Precondition Required) and the required suffix.

Verification Flow

CONN action received at /api/inbox
    ↓
Check PoW requirement for source IP
    ↓
Requirement > 0?
  β”œβ”€ Yes β†’ Check token.ends_with("A" Γ— N)
  β”‚         β”œβ”€ Pass β†’ Process action
  β”‚         └─ Fail β†’ 428 Precondition Required
  └─ No β†’ Process action
    ↓
Violation during processing?
  β”œβ”€ Yes β†’ Increment counter for IP/network
  └─ No β†’ Continue

Configuration

Parameter Default Description
max_counter 10 Maximum requirement (10 ‘A’ characters)
decay_interval_secs 3600 Counter decay interval (1 hour)
max_individual_entries 50,000 LRU cache size for individual IPs
max_network_entries 10,000 LRU cache size for network ranges

Address Tracking

Level IPv4 IPv6 Purpose
Individual /32 /64 Single source penalties
Network /24 /64 Distributed attack protection

The effective requirement is the maximum of individual and network counters.

HTTP Response Headers

Rate-limited responses include informative headers:

HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: 30
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1735003600
X-RateLimit-Level: ipv4_individual
Header Description
Retry-After Seconds until request can be retried
X-RateLimit-Limit Maximum requests per period
X-RateLimit-Remaining Requests remaining in period
X-RateLimit-Reset Unix timestamp when limit resets
X-RateLimit-Level Which address level triggered the limit

Configuration

Rate limits can be configured per-tenant via settings:

{
  "rateLimit.auth.ipv4Individual.shortRps": 2,
  "rateLimit.auth.ipv4Individual.shortBurst": 5,
  "rateLimit.auth.ipv4Individual.longRph": 30,
  "rateLimit.auth.ipv4Individual.longBurst": 30
}

Monitoring

The rate limiter exposes statistics for monitoring:

Metric Description
total_limited Total requests rate-limited
total_bans Total IPs banned
active_bans Current active ban count
cache_size Current tracked IP count

Best Practices

For API Clients

  1. Implement exponential backoff: On 429 responses, wait and retry
  2. Respect Retry-After: Don’t retry before the indicated time
  3. Use authentication: Authenticated requests may have higher limits
  4. Batch requests: Combine multiple operations where possible

For Administrators

  1. Monitor rate limit metrics: Watch for patterns indicating attacks
  2. Adjust limits per workload: Increase limits for known good IPs
  3. Review ban lists: Periodically check for false positives
  4. Enable PoW for spam-prone endpoints: Add protection to vulnerable actions

See Also

Push Notifications

Overview

Cloudillo implements Web Push notifications using the VAPID (Voluntary Application Server Identification) protocol. Push notifications are sent when users receive actions while offline or not connected via WebSocket.

Architecture

User Action Created
       ↓
Action Forwarding Decision
       ↓
Is recipient connected via WebSocket?
  β”œβ”€ Yes β†’ Send via WebSocket (real-time)
  └─ No β†’ Send Push Notification
              ↓
         Lookup user's push subscriptions
              ↓
         Encrypt notification payload
              ↓
         POST to push service endpoint
              ↓
         Browser receives notification

Web Push Standards

The implementation follows these RFCs:

RFC Title Purpose
RFC 8292 VAPID for Web Push Server identification
RFC 8188 Encrypted Content-Encoding for HTTP Payload encryption
RFC 8291 Message Encryption for Web Push End-to-end encryption

VAPID Keys

Each tenant has a VAPID key pair for authenticating with push services:

  • Private key: Stored securely in the database
  • Public key: Shared with clients for subscription

VAPID keys are automatically generated on first request if they don’t exist.

Key Management

Client requests VAPID public key
       ↓
Server checks for existing key
       ↓
Key exists? β†’ Return public key
       ↓
No key? β†’ Generate new P-256 key pair
          Store in database
          Return public key

Subscription Flow

sequenceDiagram
    participant C as Client
    participant SW as Service Worker
    participant S as Cloudillo Server
    participant PS as Push Service

    C->>S: GET /api/notification/vapid-public-key
    S-->>C: {vapidPublicKey: "BM5..."}

    C->>SW: pushManager.subscribe({userVisibleOnly: true, applicationServerKey})
    SW->>PS: Subscribe request
    PS-->>SW: PushSubscription
    SW-->>C: PushSubscription

    C->>S: POST /api/notification/subscription {subscription}
    S-->>C: {id: 12345}

Push Delivery

When an action is received for an offline user:

sequenceDiagram
    participant A as Action Sender
    participant S as Cloudillo Server
    participant PS as Push Service
    participant B as User's Browser

    A->>S: POST /api/inbox {action}
    S->>S: Process action
    S->>S: Check if recipient is online

    alt User is online (WebSocket connected)
        S->>B: WebSocket message
    else User is offline
        S->>S: Load push subscriptions
        S->>S: Encrypt payload with user's public key
        S->>PS: POST {encrypted payload}
        PS->>B: Push notification
        B->>B: Display notification
    end

Notification Types

Users can configure which notification types they receive:

Setting Default Description
notify.push.message true New direct messages
notify.push.mention true Mentioned in content
notify.push.reaction false Reactions to your content
notify.push.connection true Connection requests
notify.push.follow true New followers

Settings are stored per-user and checked before sending notifications.

Encryption

Push notification payloads are encrypted end-to-end:

  1. Client generates keys: P-256 key pair on subscription
  2. Server encrypts: Using client’s public key + shared secret
  3. Push service cannot read: Only forwards encrypted data
  4. Client decrypts: Using private key in browser

Encryption Parameters

Parameter Description
p256dh Client’s P-256 public key
auth 16-byte authentication secret
content-encoding aes128gcm

Payload Structure

{
  "type": "action",
  "actionType": "MSG",
  "issuer": "alice@example.com",
  "preview": "New message from Alice",
  "actionId": "a1~xyz789..."
}
Field Description
type Notification type (action, system)
actionType Action token type (MSG, CONN, FLLW)
issuer Action issuer identity
preview Short preview text
actionId Action ID for deep linking

Service Worker

The client service worker handles incoming push events:

self.addEventListener('push', function(event) {
  const data = event.data.json()

  const options = {
    body: data.preview,
    icon: '/icons/notification.png',
    badge: '/icons/badge.png',
    data: { url: `/action/${data.actionId}` }
  }

  event.waitUntil(
    self.registration.showNotification('Cloudillo', options)
  )
})

self.addEventListener('notificationclick', function(event) {
  event.notification.close()
  event.waitUntil(
    clients.openWindow(event.notification.data.url)
  )
})

Subscription Management

Multiple Devices

Users can have multiple push subscriptions (one per device/browser):

  • Each subscription has a unique ID
  • All subscriptions receive notifications
  • Expired subscriptions are automatically cleaned up

Subscription Expiration

Push subscriptions can expire:

  1. Browser reports expiration: Via expirationTime field
  2. Push service rejects: HTTP 410 Gone response
  3. User unsubscribes: Manual deletion

Expired subscriptions are removed automatically.

Error Handling

Push Service Response Action
201 Created Success
400 Bad Request Log error, don’t retry
410 Gone Remove subscription
429 Too Many Requests Retry with backoff
5xx Server Error Retry with backoff

See Also

API Documentation

Welcome to the Cloudillo API documentation for application developers. This guide will help you build applications on top of the Cloudillo decentralized collaboration platform.

What is Cloudillo?

Cloudillo is an open-source, decentralized collaboration platform that enables users to maintain control over their data while seamlessly collaborating with others. Built on DNS-based identity and cryptographically signed action tokens, Cloudillo allows users to self-host, use community servers, or choose third-party providers without vendor lock-in.

For Application Developers

Cloudillo provides a comprehensive set of APIs and client libraries for building:

  • Collaborative applications with real-time synchronization
  • Social features using action tokens (posts, comments, reactions)
  • Rich content editors with CRDT-based conflict-free editing
  • File management with automatic image variants
  • Real-time databases with Firebase-like APIs
  • Microfrontend applications that integrate with the Cloudillo shell

API Overview

Client Libraries (TypeScript/JavaScript)

REST API

The Cloudillo server provides a comprehensive REST API for:

  • Authentication - Login, registration, token management
  • Profiles - User and community profiles
  • Actions - Social interactions (posts, comments, reactions)
  • Files - File upload, download, and management
  • Settings - User preferences and configuration
  • References - Bookmarks and shortcuts

WebSocket API

Real-time features are provided via WebSocket connections:

  • Message Bus - Pub/sub notifications and presence
  • RTDB - Real-time database synchronization
  • CRDT - Collaborative document editing

Getting Started

New to Cloudillo development? Start here:

  1. Getting Started Guide - Create your first Cloudillo app
  2. Authentication Guide - Understand token-based auth
  3. Microfrontend Integration - Build apps for the Cloudillo shell

Developer Guides

Practical guides to help you build Cloudillo applications:

Key Concepts

DNS-Based Identity

Every user in Cloudillo has an identity tag (idTag) based on a domain name (e.g., alice@example.com). This decouples identity from storage location, allowing users to migrate their data between providers while maintaining their identity.

Action Tokens

Actions are cryptographically signed events that represent user activities:

  • POST - Creating posts and content
  • CMNT - Adding comments
  • REACT - Reactions (e.g., LOVE)
  • FLLW - Following users
  • CONN - User connections
  • FSHR - File sharing

See Actions API for details.

Multi-Tenancy

Cloudillo is designed for multi-tenant deployments. Every request includes a tenant ID (tnId) that isolates data between tenants. Application developers typically don’t need to manage this directly - it’s handled by the client libraries.

Real-Time Collaboration

Cloudillo provides three levels of real-time collaboration:

  1. CRDT - Conflict-free collaborative editing using Yjs
  2. RTDB - Real-time database with structured queries
  3. Message Bus - Pub/sub for notifications and presence

Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚         Your Application                β”‚
β”‚  (React, Vue, vanilla JS, etc.)         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚     Cloudillo Client Libraries          β”‚
β”‚  @cloudillo/core, @cloudillo/react      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚         REST + WebSocket APIs            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚       Cloudillo Server (Rust)            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚    Pluggable Storage Adapters            β”‚
β”‚  (Database, Blob, Auth, CRDT, etc.)     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Example: Your First App

import { getAppBus, createApiClient, openYDoc } from '@cloudillo/core'
import * as Y from 'yjs'

// Get message bus singleton
const bus = getAppBus()

// Initialize your app
await bus.init('my-app')

// Access state via bus properties
console.log('User:', bus.idTag)
console.log('Tenant:', bus.tnId)
console.log('Roles:', bus.roles)

// Create an API client
const api = createApiClient({
  idTag: bus.idTag!,
  authToken: bus.accessToken
})

// Fetch the user's profile
const profile = await api.profiles.getOwn()

// Open a collaborative document
const yDoc = new Y.Doc()
const { provider } = await openYDoc(yDoc, 'owner:my-document-id')

// Use the CRDT
const yText = yDoc.getText('content')
yText.insert(0, 'Hello, Cloudillo!')

Client Libraries

REST API

Developer Guides

Advanced Topics

Support

License

Cloudillo is open source software licensed under the MIT License.

Subsections of API Documentation

Quick Start

Get up and running with the Cloudillo API in minutes. This guide shows you how to accomplish common tasks.

Prerequisites

  • Node.js 16+ installed
  • A Cloudillo server instance (local or hosted)
  • Basic JavaScript/TypeScript knowledge

Installation

npm install @cloudillo/core

1. Initialize and Authenticate

Register a New Account

import * as cloudillo from '@cloudillo/core'

// Initialize the library
await cloudillo.init('my-app', {
  serverUrl: 'https://your-server.com'
})

// Register new user
const registerResponse = await fetch('/api/auth/register', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    idTag: 'alice@example.com',
    password: 'secure-password-123',
    name: 'Alice Johnson'
  })
})

const { data } = await registerResponse.json()
console.log('Registered! Token:', data.token)

// Store token for future requests
localStorage.setItem('cloudillo_token', data.token)

Login to Existing Account

const loginResponse = await fetch('/api/auth/login', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    idTag: 'alice@example.com',
    password: 'secure-password-123'
  })
})

const { data } = await loginResponse.json()
localStorage.setItem('cloudillo_token', data.token)

// Create authenticated API client
const api = cloudillo.createApiClient({
  token: data.token
})

2. Get User Profile

// Get your own profile
const myProfile = await api.profiles.getOwn()

console.log('My profile:', myProfile.data)
// {
//   tnId: 12345,
//   idTag: 'alice@example.com',
//   name: 'Alice Johnson',
//   profilePic: '/api/file/b1~abc123'
// }

3. Create a Post

// Create a simple text post
const post = await api.actions.create({
  type: 'POST',
  content: {
    text: 'Hello, Cloudillo! This is my first post.',
    title: 'My First Post'
  }
})

console.log('Post created:', post.data.actionId)

Create a Post with Images

// First, upload an image
const fileInput = document.querySelector('input[type="file"]')
const imageFile = fileInput.files[0]

const uploadResponse = await fetch(`/api/file/default/${encodeURIComponent(imageFile.name)}?tags=post,photo`, {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${token}`,
    'Content-Type': imageFile.type
  },
  body: imageFile
})

const { data: fileData } = await uploadResponse.json()
const fileId = fileData.fileId

// Create post with attachment
const postWithImage = await api.actions.create({
  type: 'POST',
  content: {
    text: 'Check out this photo!',
    title: 'Beach Sunset'
  },
  attachments: [fileId]
})

console.log('Post with image:', postWithImage.data.actionId)

4. Get Recent Posts

// Get recent posts from everyone
const posts = await api.actions.list({
  type: 'POST',
  status: 'A',
  limit: 20,
  sort: 'created',
  sortDir: 'desc'
})

posts.data.forEach(post => {
  console.log(`${post.issuer.name}: ${post.content.text}`)
})

// Example output:
// Alice Johnson: Hello, Cloudillo!
// Bob Smith: Just joined!
// Carol Davis: Having a great day!

5. Comment on a Post

// Add a comment to a post
const comment = await api.actions.create({
  type: 'CMNT',
  parentId: 'act_post123',
  content: {
    text: 'Great post! Thanks for sharing.'
  }
})

console.log('Comment added:', comment.data.actionId)

// Get all comments on a post
const comments = await api.actions.list({
  type: 'CMNT',
  parentId: 'act_post123',
  sort: 'created',
  sortDir: 'asc'
})

console.log(`${comments.data.length} comments`)

6. React to Content

// Add a reaction (like, love, etc.)
const reaction = await api.actions.addReaction('act_post123', {
  type: 'LOVE'
})

console.log('Reaction added:', reaction.data.actionId)

// Get actions with statistics
const post = await api.actions.get('act_post123')

console.log('Statistics:', post.data.stat)
// {
//   reactions: 15,
//   comments: 8,
//   ownReaction: 'LOVE'
// }

7. Follow a User

// Follow another user
const follow = await api.actions.create({
  type: 'FLLW',
  subject: 'bob@example.com'
})

console.log('Now following bob@example.com')

// Get list of people you follow
const following = await api.actions.list({
  type: 'FLLW',
  issuer: cloudillo.idTag,
  status: 'A'
})

console.log(`Following ${following.data.length} users`)

8. Upload and Manage Files

Upload Profile Picture

const imageInput = document.querySelector('input[type="file"]')
const profileImage = imageInput.files[0]

const response = await fetch('/api/me/image', {
  method: 'PUT',
  headers: {
    'Authorization': `Bearer ${token}`,
    'Content-Type': profileImage.type
  },
  body: profileImage
})

const { data } = await response.json()
console.log('Profile picture updated:', data.profilePic)

Upload File with Progress

async function uploadFileWithProgress(file, onProgress) {
  const xhr = new XMLHttpRequest()

  return new Promise((resolve, reject) => {
    xhr.upload.addEventListener('progress', (e) => {
      if (e.lengthComputable) {
        const percent = (e.loaded / e.total) * 100
        onProgress(percent)
      }
    })

    xhr.addEventListener('load', () => {
      if (xhr.status >= 200 && xhr.status < 300) {
        resolve(JSON.parse(xhr.responseText))
      } else {
        reject(new Error(xhr.statusText))
      }
    })

    xhr.addEventListener('error', () => reject(new Error('Upload failed')))

    xhr.open('POST', `/api/file/default/${encodeURIComponent(file.name)}`)
    xhr.setRequestHeader('Authorization', `Bearer ${token}`)
    xhr.setRequestHeader('Content-Type', file.type)
    xhr.send(file)
  })
}

// Usage
const result = await uploadFileWithProgress(file, (percent) => {
  console.log(`Upload progress: ${percent.toFixed(1)}%`)
  updateProgressBar(percent)
})

console.log('Upload complete:', result.data.fileId)

List Your Files

// Get all your files
const files = await api.files.list({
  fileTp: 'BLOB',
  limit: 50,
  sort: 'created',
  sortDir: 'desc'
})

files.data.forEach(file => {
  console.log(`${file.fileName} - ${file.contentType}`)
})

// Filter by type
const images = await api.files.list({
  fileTp: 'BLOB',
  contentType: 'image/*'
})

console.log(`You have ${images.data.length} images`)

9. Real-time Updates (WebSocket)

// Connect to WebSocket for real-time updates
const ws = new WebSocket(`wss://your-server.com/ws/bus`)

ws.onopen = () => {
  console.log('Connected to real-time bus')

  // Subscribe to events
  ws.send(JSON.stringify({
    type: 'subscribe',
    channels: ['actions', 'messages']
  }))
}

ws.onmessage = (event) => {
  const data = JSON.parse(event.data)

  switch (data.type) {
    case 'action':
      console.log('New action:', data.action)
      handleNewAction(data.action)
      break

    case 'message':
      console.log('New message:', data.message)
      showNotification(data.message)
      break
  }
}

ws.onerror = (error) => {
  console.error('WebSocket error:', error)
}

ws.onclose = () => {
  console.log('Disconnected from real-time bus')
  // Reconnect after delay
  setTimeout(() => connectWebSocket(), 5000)
}

10. Private Messaging

// Send a private message
const message = await api.actions.create({
  type: 'MSG',
  subject: 'bob@example.com',
  content: {
    text: 'Hey Bob, how are you?',
    subject: 'Checking in'
  }
})

console.log('Message sent:', message.data.actionId)

// Get your messages
const messages = await api.actions.list({
  type: 'MSG',
  involved: cloudillo.idTag,
  sort: 'created',
  sortDir: 'desc',
  limit: 50
})

console.log(`You have ${messages.data.length} messages`)

// Display messages
messages.data.forEach(msg => {
  const from = msg.issuerTag === cloudillo.idTag ? 'You' : msg.issuer.name
  const to = msg.subject === cloudillo.idTag ? 'You' : msg.subject
  console.log(`${from} β†’ ${to}: ${msg.content.text}`)
})

11. Search and Filter

Search Posts by Text

// Get posts from the last week
const lastWeek = Math.floor(Date.now() / 1000) - 7 * 24 * 60 * 60

const recentPosts = await api.actions.list({
  type: 'POST',
  status: 'A',
  createdAfter: lastWeek,
  limit: 100
})

// Filter in client (for complex searches)
const searchTerm = 'cloudillo'
const matchingPosts = recentPosts.data.filter(post =>
  post.content.text?.toLowerCase().includes(searchTerm)
)

console.log(`Found ${matchingPosts.length} posts mentioning "${searchTerm}"`)

Get User’s Activity

// Get everything involving a specific user
const userActivity = await api.actions.list({
  involved: 'bob@example.com',
  limit: 100,
  sort: 'created',
  sortDir: 'desc'
})

// Categorize by type
const byType = userActivity.data.reduce((acc, action) => {
  acc[action.type] = (acc[action.type] || 0) + 1
  return acc
}, {})

console.log('Activity breakdown:', byType)
// { POST: 15, CMNT: 32, REACT: 48, FLLW: 5 }

12. Share Files

// Share a file with another user
const share = await api.actions.create({
  type: 'FSHR',
  subject: 'bob@example.com',
  attachments: ['b1~abc123'],
  content: {
    permission: 'READ',
    message: 'Check out this document!'
  }
})

console.log('File shared:', share.data.actionId)

// Share with write permission
const shareWrite = await api.actions.create({
  type: 'FSHR',
  subject: 'carol@example.com',
  attachments: ['f1~xyz789'],
  content: {
    permission: 'WRITE',
    message: 'Let\'s collaborate on this!'
  }
})

13. Update Profile

// Update your profile information
const updated = await api.profiles.updateOwn({
  name: 'Alice Johnson-Smith',
  x: {
    bio: 'Software developer and Cloudillo enthusiast',
    location: 'San Francisco, CA',
    website: 'https://alice.example.com',
    twitter: '@alice'
  }
})

console.log('Profile updated:', updated.data)

14. Handle Errors Gracefully

async function createPostSafely(content) {
  try {
    const post = await api.actions.create({
      type: 'POST',
      content
    })

    console.log('βœ“ Post created:', post.data.actionId)
    return post.data

  } catch (error) {
    if (error.response) {
      const { code, message } = error.response.data.error

      switch (code) {
        case 'E-AUTH-EXPIRED':
          console.error('Session expired. Please login again.')
          window.location.href = '/login'
          break

        case 'E-PERM-DENIED':
          console.error('Permission denied:', message)
          alert('You don\'t have permission to post')
          break

        case 'E-RATE-LIMIT':
          console.error('Rate limited. Please slow down.')
          alert('Too many posts. Please wait a moment.')
          break

        default:
          console.error('Error creating post:', message)
          alert('Failed to create post. Please try again.')
      }
    } else {
      console.error('Network error:', error.message)
      alert('Network error. Check your connection.')
    }

    throw error
  }
}

// Usage
await createPostSafely({
  text: 'Hello, world!',
  title: 'My Post'
})

15. Pagination

async function getAllPosts() {
  const allPosts = []
  let cursor = undefined
  const limit = 50

  while (true) {
    const response = await api.actions.list({
      type: 'POST',
      status: 'A',
      limit: limit,
      cursor
    })

    allPosts.push(...response.data)

    console.log(`Loaded ${allPosts.length} posts`)

    if (!response.cursorPagination?.hasMore) {
      break
    }

    cursor = response.cursorPagination.nextCursor
  }

  return allPosts
}

// Load all posts (be careful with large datasets!)
const posts = await getAllPosts()
console.log(`Total posts loaded: ${posts.length}`)

Complete Example: Simple Social Feed

import * as cloudillo from '@cloudillo/core'

class SocialFeed {
  constructor(token) {
    this.api = cloudillo.createApiClient({ token })
  }

  async initialize() {
    // Get user profile
    const profile = await this.api.profiles.getOwn()
    console.log('Logged in as:', profile.data.name)
  }

  async loadFeed(limit = 20) {
    // Get recent posts with full profiles
    const posts = await this.api.actions.list({
      type: 'POST',
      status: 'A',
      limit: limit,
      sort: 'created',
      sortDir: 'desc'
    })

    return posts.data
  }

  async createPost(text, title) {
    const post = await this.api.actions.create({
      type: 'POST',
      content: { text, title }
    })

    console.log('βœ“ Post created')
    return post.data
  }

  async addComment(postId, text) {
    const comment = await this.api.actions.create({
      type: 'CMNT',
      parentId: postId,
      content: { text }
    })

    console.log('βœ“ Comment added')
    return comment.data
  }

  async addReaction(actionId, reactionType = 'LOVE') {
    const reaction = await this.api.actions.addReaction(actionId, {
      type: reactionType
    })

    console.log('βœ“ Reaction added')
    return reaction.data
  }

  async getComments(postId) {
    const comments = await this.api.actions.list({
      type: 'CMNT',
      parentId: postId,
      sort: 'created',
      sortDir: 'asc'
    })

    return comments.data
  }

  displayPost(post) {
    console.log('\n' + '='.repeat(60))
    console.log(`${post.issuer.name} (@${post.issuerTag})`)
    console.log(`${post.content.title}`)
    console.log(post.content.text)
    console.log(`❀️ ${post.stat?.reactions || 0} | πŸ’¬ ${post.stat?.comments || 0}`)
    console.log('='.repeat(60))
  }
}

// Usage
const feed = new SocialFeed(token)
await feed.initialize()

// Load and display feed
const posts = await feed.loadFeed(10)
posts.forEach(post => feed.displayPost(post))

// Create a new post
await feed.createPost(
  'Just discovered Cloudillo - it\'s amazing!',
  'First Impressions'
)

// Interact with a post
await feed.addReaction('act_post123', 'LOVE')
await feed.addComment('act_post123', 'Totally agree!')

// Get comments
const comments = await feed.getComments('act_post123')
console.log(`\nComments (${comments.length}):`)
comments.forEach(c => {
  console.log(`  ${c.issuer.name}: ${c.content.text}`)
})

Next Steps

Now that you know the basics, explore:

Tips for Success

  1. Always handle errors - Network issues happen, be prepared
  2. Use pagination - Don’t load thousands of items at once
  3. Expand wisely - Only expand what you need to reduce payload size
  4. Cache when possible - Store user profiles, settings locally
  5. Test incrementally - Start small, add features gradually
  6. Monitor rate limits - Respect the API limits
  7. Validate input - Check data before sending to API
  8. Log requests - Keep request IDs for debugging

Common Gotchas

❌ Forgetting to add /api/ prefix

fetch('/action')  // Wrong!
fetch('/api/action')  // Correct

❌ Not handling authentication expiry

// Always check for auth errors and redirect to login

❌ Using wrong file upload endpoint

// Wrong: POST /file/upload
// Correct: POST /api/file/{preset}/{file_name}

❌ Not expanding relations

// Inefficient - requires N+1 queries
const actions = await api.actions.list({ type: 'POST' })
for (const action of actions.data) {
  const issuer = await api.profiles.get(action.issuerTag)
}

// Efficient - actions include issuer data
const actions = await api.actions.list({
  type: 'POST'
})

Happy coding! πŸš€

Getting Started

This guide will walk you through creating your first Cloudillo application.

Prerequisites

  • Node.js 18+ and pnpm installed
  • Basic knowledge of TypeScript/JavaScript
  • Familiarity with React (optional, for UI apps)

Installation

For Standalone Apps

pnpm add @cloudillo/core

For React Apps

pnpm add @cloudillo/core @cloudillo/react

For Real-Time Database

pnpm add @cloudillo/rtdb

For Collaborative Editing

pnpm add @cloudillo/core yjs y-websocket

Your First App: Hello Cloudillo

Let’s create a simple app that displays the current user’s profile.

Step 1: Initialize the App

Create src/index.ts:

import { getAppBus } from '@cloudillo/core'

async function main() {
  // Get the singleton message bus
  const bus = getAppBus()

  // Initialize with your app name
  await bus.init('hello-cloudillo')

  console.log('Initialized successfully!')
  console.log('Access Token:', bus.accessToken)
  console.log('User ID:', bus.idTag)
  console.log('Tenant ID:', bus.tnId)
  console.log('Roles:', bus.roles)
}

main().catch(console.error)

Step 2: Fetch User Profile

import { getAppBus, createApiClient } from '@cloudillo/core'

async function main() {
  const bus = getAppBus()
  await bus.init('hello-cloudillo')

  // Create an API client
  const api = createApiClient({
    idTag: bus.idTag!,
    authToken: bus.accessToken
  })

  // Fetch the current user's profile
  const profile = await api.profiles.getOwn()

  console.log('Profile:', profile)
  console.log('Name:', profile.name)
  console.log('ID Tag:', profile.idTag)
  console.log('Profile Picture:', profile.profilePic)
}

main().catch(console.error)

Step 3: Create a Post

import { getAppBus, createApiClient } from '@cloudillo/core'

async function main() {
  const bus = getAppBus()
  await bus.init('hello-cloudillo')

  const api = createApiClient({
    idTag: bus.idTag!,
    authToken: bus.accessToken
  })

  // Create a new post
  const newPost = await api.actions.create({
    type: 'POST',
    content: {
      text: 'Hello from my first Cloudillo app!',
      title: 'My First Post'
    }
  })

  console.log('Post created:', newPost)
}

main().catch(console.error)

React Example

For React applications, use the provided hooks:

import React from 'react'
import { useCloudillo, useAuth, useApi } from '@cloudillo/react'

function App() {
  // useCloudillo handles initialization
  const { token } = useCloudillo('hello-cloudillo')

  if (!token) return <div>Loading...</div>

  return <Profile />
}

function Profile() {
  const [auth] = useAuth()  // Returns tuple [auth, setAuth]
  const { api } = useApi()  // Returns { api, authenticated, setIdTag }
  const [profile, setProfile] = React.useState(null)

  React.useEffect(() => {
    if (!api) return
    api.profiles.getOwn().then(setProfile)
  }, [api])

  if (!api) return <div>No API client</div>
  if (!profile) return <div>Loading...</div>

  return (
    <div>
      <h1>Welcome, {profile.name}!</h1>
      <p>ID: {auth?.idTag}</p>
      {profile.profilePic && (
        <img src={profile.profilePic} alt="Profile" />
      )}
    </div>
  )
}

export default App

Real-Time Database Example

Here’s how to use the real-time database:

import { getAppBus, getRtdbUrl } from '@cloudillo/core'
import { RtdbClient } from '@cloudillo/rtdb'

async function main() {
  const bus = getAppBus()
  await bus.init('rtdb-example')

  // Create RTDB client
  const rtdb = new RtdbClient({
    dbId: 'my-database-file-id',
    auth: { getToken: () => bus.accessToken },
    serverUrl: getRtdbUrl(bus.idTag!, 'my-database-file-id', bus.accessToken!)
  })

  // Connect to the database
  await rtdb.connect()

  // Get a collection reference
  const todos = rtdb.collection('todos')

  // Subscribe to real-time updates
  todos.onSnapshot((snapshot) => {
    console.log('Todos updated:', snapshot.docs.map(doc => doc.data()))
  })

  // Create a document using batch
  const batch = rtdb.batch()
  batch.create(todos, {
    title: 'Learn Cloudillo',
    completed: false,
    createdAt: Date.now()
  })
  await batch.commit()

  // Query documents
  const incompleteTodos = await todos.query({
    filter: { equals: { completed: false } },
    sort: [{ field: 'createdAt', ascending: false }]
  })

  console.log('Incomplete todos:', incompleteTodos)
}

main().catch(console.error)

Collaborative Editing Example

Create a collaborative text editor:

import { getAppBus, openYDoc } from '@cloudillo/core'
import * as Y from 'yjs'

async function main() {
  const bus = getAppBus()
  await bus.init('collab-editor')

  // Create a Yjs document
  const yDoc = new Y.Doc()

  // Open collaborative document (format: ownerTag:documentId)
  const { provider } = await openYDoc(yDoc, 'owner.cloudillo.net:my-doc-id')

  // Get shared text
  const yText = yDoc.getText('content')

  // Listen for changes
  yText.observe(() => {
    console.log('Text changed:', yText.toString())
  })

  // Insert text
  yText.insert(0, 'Hello, collaborative world!')

  // See awareness (other users' cursors/selections)
  provider.awareness.on('change', () => {
    const states = provider.awareness.getStates()
    console.log('Connected users:', states.size)
  })
}

main().catch(console.error)

Microfrontend Integration

If you’re building an app to run inside the Cloudillo shell:

import { getAppBus, createApiClient } from '@cloudillo/core'

async function main() {
  // Get the singleton message bus
  const bus = getAppBus()

  // init() automatically handles the shell protocol
  await bus.init('my-microfrontend')

  // Create an API client
  const api = createApiClient({
    idTag: bus.idTag!,
    authToken: bus.accessToken
  })

  // The shell provides via bus properties:
  // - bus.idTag (user's identity)
  // - bus.tnId (tenant ID)
  // - bus.roles (user roles)
  // - bus.darkMode (theme preference)
  // - bus.access ('read' or 'write')

  // Your app logic here...
}

main().catch(console.error)

Error Handling

All API calls can throw errors. Handle them appropriately:

import { getAppBus, createApiClient } from '@cloudillo/core'

const bus = getAppBus()
await bus.init('my-app')

try {
  const api = createApiClient({
    idTag: bus.idTag!,
    authToken: bus.accessToken
  })
  const profile = await api.profiles.getOwn()
} catch (error) {
  if (error instanceof Error) {
    console.error('Error:', error.message)
  }
}

Next Steps

Now that you’ve created your first Cloudillo app, explore more features:

Common Patterns

Handling Dark Mode

import { getAppBus } from '@cloudillo/core'

const bus = getAppBus()
await bus.init('my-app')

// Check dark mode preference
if (bus.darkMode) {
  document.body.classList.add('dark-theme')
}

Using Query Parameters

import { getAppBus, createApiClient } from '@cloudillo/core'

const bus = getAppBus()
await bus.init('my-app')

const api = createApiClient({
  idTag: bus.idTag!,
  authToken: bus.accessToken
})

// List actions with filters
const posts = await api.actions.list({
  type: 'POST',
  status: 'A', // Active
  limit: 20
})

Uploading Files

import { getAppBus, createApiClient } from '@cloudillo/core'

const bus = getAppBus()
await bus.init('my-app')

const api = createApiClient({
  idTag: bus.idTag!,
  authToken: bus.accessToken
})

// Upload a file using the uploadBlob helper
const result = await api.files.uploadBlob(
  'gallery',      // preset
  'image.png',    // fileName
  imageBlob,      // file data
  'image/png'     // contentType
)

console.log('Uploaded file:', result.fileId)

Build and Deploy

Development

Most apps run as microfrontends inside the Cloudillo shell. Use your preferred build tool (Rollup, Webpack, Vite):

# Using Rollup (like the example apps)
pnpm build

# Using Vite
vite build

Production

Deploy your built app to any static hosting:

# The built output goes to the shell's apps directory
cp -r dist /path/to/cloudillo/shell/public/apps/my-app

Troubleshooting

“Failed to initialize”

Make sure you’re either:

  1. Running inside the Cloudillo shell (as a microfrontend), or
  2. Providing authentication manually for standalone apps

“CORS errors”

Ensure your Cloudillo server is configured to allow requests from your app’s origin.

“WebSocket connection failed”

Check that:

  1. The WebSocket URL is correct (wss:// for production)
  2. The server is running and accessible
  3. Your authentication token is valid

Example Apps

Check out the example apps in the Cloudillo repository:

  • Quillo - Rich text editor with Quill
  • Prello - Presentation tool
  • Sheello - Spreadsheet application
  • Formillo - Form builder
  • Todollo - Task management

All use the same patterns described in this guide.

Authentication

Cloudillo uses JWT-based authentication with three types of tokens for different use cases. This guide explains how authentication works and how to use it in your applications.

Token Types

1. Access Token (Session Token)

The access token is your primary authentication credential for API requests.

Characteristics:

  • JWT signed with ES384 elliptic curve algorithm
  • Contains: tnId (tenant ID), idTag (user identity), roles, iat (issued at), exp (expiration)
  • Used for all authenticated API requests
  • Typically valid for the duration of a session

Usage:

// The access token is automatically managed by @cloudillo/core
import * as cloudillo from '@cloudillo/core'

const token = await cloudillo.init('my-app')
// Token is now stored and used automatically for all API calls

// Or access it directly
console.log(cloudillo.accessToken)

2. Action Token (Federation Token)

Action tokens are cryptographically signed events used for federation between Cloudillo instances.

Characteristics:

  • Represents a specific action (POST, CMNT, REACT, etc.)
  • Signed by the issuer’s private key
  • Can be verified by anyone with the issuer’s public key
  • Enables trust-free federation

Usage:

// Action tokens are created automatically when you post actions
const api = cloudillo.createApiClient()

const action = await api.actions.create({
  type: 'POST',
  content: { text: 'Hello, world!' }
})

// The server automatically signs the action with your key
// Other instances can verify it without trusting your server

3. Proxy Token (Cross-Instance Token)

Proxy tokens enable accessing resources on remote Cloudillo instances.

Characteristics:

  • Short-lived (typically 5 minutes)
  • Grants read access to specific resources
  • Used for federation scenarios

Usage:

const api = cloudillo.createApiClient()

// Get a proxy token for accessing a remote instance
const proxyToken = await api.auth.proxyToken.get()

// Use it to fetch resources from another instance
// (typically handled automatically by the client)

Authentication Flow

For Microfrontend Apps

When running inside the Cloudillo shell, authentication is handled automatically:

import * as cloudillo from '@cloudillo/core'

// The init() function receives the token from the shell via postMessage
const token = await cloudillo.init('my-app')

// All API calls now use this token automatically
const api = cloudillo.createApiClient()
const profile = await api.profiles.getOwn() // Authenticated request

For Standalone Apps

For standalone applications, you need to handle authentication manually:

import * as cloudillo from '@cloudillo/core'

// Option 1: Manual token management
cloudillo.accessToken = 'your-jwt-token-here'
const api = cloudillo.createApiClient()

// Option 2: Login flow
const response = await fetch('https://your-server.com/auth/login', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    idTag: 'alice@example.com',
    password: 'secret123'
  })
})

const { token, tnId, idTag, name, roles } = await response.json()

cloudillo.accessToken = token
cloudillo.tnId = tnId
cloudillo.idTag = idTag
cloudillo.roles = roles

Authentication Endpoints

POST /auth/register

Register a new user account.

Request:

{
  "idTag": "alice@example.com",
  "password": "secure-password",
  "name": "Alice Johnson",
  "profilePic": "https://example.com/alice.jpg"
}

Response:

{
  "data": {
    "tnId": 12345,
    "idTag": "alice@example.com",
    "name": "Alice Johnson",
    "token": "eyJhbGc..."
  }
}

POST /auth/login

Authenticate and receive an access token.

Request:

{
  "idTag": "alice@example.com",
  "password": "secure-password"
}

Response:

{
  "data": {
    "tnId": 12345,
    "idTag": "alice@example.com",
    "name": "Alice Johnson",
    "profilePic": "/file/b1~abcd1234",
    "roles": ["user", "admin"],
    "token": "eyJhbGc...",
    "settings": [
      ["theme", "dark"],
      ["language", "en"]
    ]
  }
}

POST /auth/logout

Invalidate the current session.

Request:

POST /auth/logout
Authorization: Bearer eyJhbGc...

Response:

{
  "data": "ok"
}

GET /auth/access-token

Exchange credentials for a scoped access token.

Query Parameters:

  • idTag - User identity
  • password - User password
  • roles - Optional: Requested roles (comma-separated)
  • ttl - Optional: Token lifetime in seconds

Response:

{
  "data": {
    "token": "eyJhbGc...",
    "expiresAt": 1735555555
  }
}

GET /auth/proxy-token

Get a proxy token for accessing remote resources.

Request:

GET /auth/proxy-token?target=bob@remote.com
Authorization: Bearer eyJhbGc...

Response:

{
  "data": {
    "token": "eyJhbGc...",
    "expiresAt": 1735555555
  }
}

Role-Based Access Control

Cloudillo supports role-based access control (RBAC) for fine-grained permissions.

Default Roles

  • user - Standard user permissions (read/write own data)
  • admin - Administrative permissions (manage server, users)
  • read - Read-only access
  • write - Write access to resources

Checking Roles

import * as cloudillo from '@cloudillo/core'

await cloudillo.init('my-app')

// Check if user has a specific role
if (cloudillo.roles?.includes('admin')) {
  console.log('User is an admin')
}

// Enable/disable features based on roles
const canModerate = cloudillo.roles?.includes('admin') ||
                    cloudillo.roles?.includes('moderator')

Requesting Specific Roles

// Request an access token with specific roles
const response = await fetch(
  '/auth/access-token?idTag=alice@example.com&password=secret&roles=user,admin'
)

Token Validation

All tokens are validated on the server for:

  1. Signature verification - Using ES384 algorithm
  2. Expiration check - Tokens expire after a set period
  3. Tenant isolation - Tokens are tied to specific tenants
  4. Role validation - Roles must be granted by the server

Security Best Practices

1. Token Storage

For web apps:

// Don't store tokens in localStorage (XSS vulnerable)
// ❌ localStorage.setItem('token', token)

// Use memory storage (managed by @cloudillo/core)
// βœ… cloudillo.accessToken = token

// Or use httpOnly cookies (server-side)

2. Token Renewal

// Implement token renewal before expiration
async function renewToken() {
  const api = cloudillo.createApiClient()

  try {
    const newToken = await api.auth.loginToken.get()
    cloudillo.accessToken = newToken.token
  } catch (error) {
    // Token expired, redirect to login
    window.location.href = '/login'
  }
}

// Renew every 50 minutes (if token lasts 60 minutes)
setInterval(renewToken, 50 * 60 * 1000)

3. Error Handling

import { FetchError } from '@cloudillo/core'

try {
  const api = cloudillo.createApiClient()
  const data = await api.profiles.getOwn()
} catch (error) {
  if (error instanceof FetchError) {
    if (error.code === 'E-AUTH-UNAUTH') {
      // Unauthorized - token expired or invalid
      window.location.href = '/login'
    } else if (error.code === 'E-AUTH-FORBID') {
      // Forbidden - insufficient permissions
      alert('You do not have permission to access this resource')
    }
  }
}

4. HTTPS Only

Always use HTTPS in production:

// βœ… Good
const api = cloudillo.createApiClient({
  baseUrl: 'https://api.cloudillo.com'
})

// ❌ Bad (only for local development)
const api = cloudillo.createApiClient({
  baseUrl: 'http://localhost:3000'
})

WebAuthn Support

Cloudillo supports WebAuthn for passwordless authentication.

Registration Flow

// 1. Get registration options from server
const optionsResponse = await fetch('/auth/webauthn/register/options', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ idTag: 'alice@example.com' })
})

const options = await optionsResponse.json()

// 2. Create credential with browser WebAuthn API
const credential = await navigator.credentials.create({
  publicKey: options
})

// 3. Verify credential with server
const verifyResponse = await fetch('/auth/webauthn/register/verify', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    idTag: 'alice@example.com',
    credential: {
      id: credential.id,
      rawId: Array.from(new Uint8Array(credential.rawId)),
      response: {
        clientDataJSON: Array.from(new Uint8Array(credential.response.clientDataJSON)),
        attestationObject: Array.from(new Uint8Array(credential.response.attestationObject))
      },
      type: credential.type
    }
  })
})

Authentication Flow

// 1. Get authentication options
const optionsResponse = await fetch('/auth/webauthn/login/options', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ idTag: 'alice@example.com' })
})

const options = await optionsResponse.json()

// 2. Get assertion with browser WebAuthn API
const assertion = await navigator.credentials.get({
  publicKey: options
})

// 3. Verify assertion with server
const verifyResponse = await fetch('/auth/webauthn/login/verify', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    idTag: 'alice@example.com',
    credential: {
      id: assertion.id,
      rawId: Array.from(new Uint8Array(assertion.rawId)),
      response: {
        clientDataJSON: Array.from(new Uint8Array(assertion.response.clientDataJSON)),
        authenticatorData: Array.from(new Uint8Array(assertion.response.authenticatorData)),
        signature: Array.from(new Uint8Array(assertion.response.signature)),
        userHandle: assertion.response.userHandle ?
          Array.from(new Uint8Array(assertion.response.userHandle)) : null
      },
      type: assertion.type
    }
  })
})

const { token } = await verifyResponse.json()
cloudillo.accessToken = token

Multi-Tenant Considerations

Every request in Cloudillo is scoped to a tenant:

// The tnId is automatically included in all requests
console.log('Tenant ID:', cloudillo.tnId)

// Tokens are tenant-specific and cannot access other tenants' data
// This is enforced at the database level for security

Common Authentication Scenarios

Scenario 1: User Login

async function login(idTag: string, password: string) {
  const response = await fetch('/auth/login', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ idTag, password })
  })

  if (!response.ok) {
    throw new Error('Login failed')
  }

  const data = await response.json()

  cloudillo.accessToken = data.data.token
  cloudillo.tnId = data.data.tnId
  cloudillo.idTag = data.data.idTag
  cloudillo.roles = data.data.roles

  return data.data
}

Scenario 2: Automatic Token Management

// @cloudillo/react handles this automatically
import { CloudilloProvider } from '@cloudillo/react'

function App() {
  return (
    <CloudilloProvider appName="my-app">
      {/* Token is managed automatically */}
      <YourApp />
    </CloudilloProvider>
  )
}

Scenario 3: Token Refresh

// Check token expiration and refresh
async function ensureAuthenticated() {
  const api = cloudillo.createApiClient()

  try {
    // Try to use the current token
    await api.profiles.getOwn()
  } catch (error) {
    if (error.code === 'E-AUTH-UNAUTH') {
      // Token expired, get a new one
      const { token } = await api.auth.loginToken.get()
      cloudillo.accessToken = token
    }
  }
}

Next Steps

Client Libraries

Overview

Cloudillo provides a comprehensive set of TypeScript/JavaScript client libraries for building applications. These libraries handle authentication, API communication, real-time synchronization, and React integration.

Available Libraries

@cloudillo/core

Core SDK for initialization and API access. This is the foundation for all Cloudillo applications.

Key Features:

  • App initialization via message bus (getAppBus())
  • Type-safe REST API client
  • CRDT document opening
  • URL helper functions
  • Storage API for apps

Install:

pnpm add @cloudillo/core

@cloudillo/react

React hooks for Cloudillo integration.

Key Features:

  • useAuth() hook for authentication state (returns tuple)
  • useApi() hook for API client access (returns object)
  • useCloudillo() hook for microfrontend initialization
  • useCloudilloEditor() hook for CRDT editors
  • useInfiniteScroll() hook for pagination

Install:

pnpm add @cloudillo/react

@cloudillo/types

Shared TypeScript types with runtime validation using @symbion/runtype.

Key Features:

  • All data types (Profile, Action, File, etc.)
  • Runtime type validation
  • Compile-time type safety
  • Action type enums
  • Type guards and validators

Install:

pnpm add @cloudillo/types

@cloudillo/rtdb

Real-time database client with Firebase-like API.

Key Features:

  • Firebase-like API (familiar to developers)
  • Real-time subscriptions
  • Type-safe queries
  • Batch operations

Install:

pnpm add @cloudillo/rtdb

@cloudillo/canvas-tools

React components and hooks for interactive object manipulation in SVG canvas applications.

Key Features:

  • Transform gizmo with rotation, scaling, and positioning
  • Rotation and pivot handle components
  • Gradient picker with presets
  • Coordinate and geometry utilities

Install:

pnpm add @cloudillo/canvas-tools

@cloudillo/fonts

Font metadata and pairing suggestions for typography systems.

Key Features:

  • Curated metadata for 22 Google Fonts
  • Pre-defined font pairings (heading + body combinations)
  • Helper functions for filtering by category and role
  • Full TypeScript support

Install:

pnpm add @cloudillo/fonts

Quick Comparison

Library Purpose Use When
@cloudillo/core Core functionality Every app
@cloudillo/react React integration Building React apps
@cloudillo/types Type definitions TypeScript projects
@cloudillo/rtdb Real-time database Need structured real-time data
@cloudillo/canvas-tools SVG canvas manipulation Building drawing/design apps
@cloudillo/fonts Font metadata and pairings Typography selection UI

Installation

Minimal Setup (vanilla JS)

pnpm add @cloudillo/core

React Setup

pnpm add @cloudillo/core @cloudillo/react

Full Setup (with real-time features)

pnpm add @cloudillo/core @cloudillo/react @cloudillo/rtdb yjs y-websocket

Basic Usage

With @cloudillo/core

import { getAppBus, createApiClient } from '@cloudillo/core'

// Get the singleton message bus
const bus = getAppBus()

// Initialize (communicates with shell)
await bus.init('my-app')

// Access state
console.log(bus.idTag)       // User's identity
console.log(bus.accessToken) // JWT token

// Create API client
const api = createApiClient({
  idTag: bus.idTag!,
  authToken: bus.accessToken
})

// Make requests
const profile = await api.profiles.getOwn()

With @cloudillo/react

import { useCloudillo, useAuth, useApi } from '@cloudillo/react'

function App() {
  // useCloudillo handles initialization
  const { token, idTag, ownerTag, fileId } = useCloudillo('my-app')

  if (!token) return <div>Loading...</div>

  return <MyComponent />
}

function MyComponent() {
  const [auth] = useAuth()  // Returns tuple [auth, setAuth]
  const { api } = useApi()  // Returns { api, authenticated, setIdTag }

  if (!api) return <div>Loading...</div>

  // Use auth and api...
}

With @cloudillo/rtdb

import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

const rtdb = new RtdbClient({
  dbId: 'my-db-file-id',
  auth: { getToken: () => bus.accessToken },
  serverUrl: getRtdbUrl(bus.idTag!, 'my-db-file-id', bus.accessToken!)
})

await rtdb.connect()

const todos = rtdb.collection('todos')
todos.onSnapshot(snapshot => {
  console.log(snapshot.docs.map(doc => doc.data()))
})

Common Patterns

Pattern 1: Authentication Flow

import { getAppBus } from '@cloudillo/core'

// Get message bus singleton
const bus = getAppBus()

// Initialize (gets token from shell)
const state = await bus.init('my-app')

// Access auth state via bus properties
console.log(bus.idTag)       // User's identity
console.log(bus.tnId)        // Tenant ID
console.log(bus.roles)       // User roles
console.log(bus.accessToken) // JWT token
console.log(bus.access)      // 'read' or 'write'

Pattern 2: API Requests

import { getAppBus, createApiClient } from '@cloudillo/core'

const bus = getAppBus()
await bus.init('my-app')

const api = createApiClient({
  idTag: bus.idTag!,
  authToken: bus.accessToken
})

// GET requests
const profile = await api.profiles.getOwn()

// POST requests
const action = await api.actions.create({
  type: 'POST',
  content: { text: 'Hello!' }
})

// Query with parameters
const actions = await api.actions.list({
  type: 'POST',
  limit: 20
})

Pattern 3: Real-Time CRDT

import { getAppBus, openYDoc } from '@cloudillo/core'
import * as Y from 'yjs'

const bus = getAppBus()
await bus.init('my-app')

// Open CRDT document
const yDoc = new Y.Doc()
const { provider } = await openYDoc(yDoc, 'alice.cloudillo.net:doc-id')

// Use shared types
const yText = yDoc.getText('content')
yText.insert(0, 'Hello!')

// Listen for changes
yText.observe(() => {
  console.log('Text updated:', yText.toString())
})

Pattern 4: React Integration

import { useCloudillo, useApi, useAuth } from '@cloudillo/react'
import { useEffect, useState } from 'react'

function App() {
  const { token } = useCloudillo('my-app')

  if (!token) return <div>Initializing...</div>

  return <PostsList />
}

function PostsList() {
  const { api } = useApi()
  const [posts, setPosts] = useState([])

  useEffect(() => {
    if (!api) return

    api.actions.list({ type: 'POST', limit: 20 })
      .then(setPosts)
  }, [api])

  if (!api) return <div>Loading...</div>

  return (
    <div>
      {posts.map(post => (
        <div key={post.actionId}>{post.content?.text}</div>
      ))}
    </div>
  )
}

TypeScript Support

All libraries are written in TypeScript and provide full type definitions.

import type { Profile, Action, NewAction } from '@cloudillo/types'
import { createApiClient } from '@cloudillo/core'

// Types are automatically inferred
const api = createApiClient({ idTag: 'alice.cloudillo.net' })
const profile: Profile = await api.profiles.getOwn()

// Type-safe action creation
const newAction: NewAction = {
  type: 'POST',
  content: { text: 'Hello!' }
}

const created: Action = await api.actions.create(newAction)

Error Handling

API errors are thrown as standard errors with status codes:

import { createApiClient } from '@cloudillo/core'

try {
  const api = createApiClient({ idTag: 'alice.cloudillo.net' })
  const data = await api.profiles.getOwn()
} catch (error) {
  if (error instanceof Error) {
    console.error('Error:', error.message)
  }
}

Library Details

Explore each library in detail:

Next Steps

Subsections of Client Libraries

@cloudillo/core

Overview

The @cloudillo/core library is the core SDK for Cloudillo applications. It provides initialization via the message bus, API client creation, CRDT document support, and URL helpers.

Installation

pnpm add @cloudillo/core

Core Pattern: AppMessageBus

The main API is accessed through the getAppBus() singleton. This provides authentication state, storage, and shell communication for apps running in the Cloudillo shell.

import { getAppBus, openYDoc } from '@cloudillo/core'

// Get the singleton message bus
const bus = getAppBus()

// Initialize your app (communicates with shell)
const state = await bus.init('my-app')

// Access state via bus properties
console.log('Token:', bus.accessToken)
console.log('User ID:', bus.idTag)
console.log('Tenant ID:', bus.tnId)
console.log('Roles:', bus.roles)
console.log('Access level:', bus.access)
console.log('Dark mode:', bus.darkMode)

AppState Properties

After calling bus.init(), these properties are available on the bus:

Property Type Description
accessToken string | undefined Current JWT access token
idTag string | undefined User’s identity tag (e.g., “alice.cloudillo.net”)
tnId number | undefined Tenant ID
roles string[] | undefined User’s roles
access 'read' | 'write' Access level to current resource
darkMode boolean Dark mode preference
tokenLifetime number | undefined Token lifetime in seconds
displayName string | undefined Display name (for anonymous guests)

Storage API

The bus provides namespaced key-value storage:

const bus = getAppBus()
await bus.init('my-app')

// Store data
await bus.storage.set('my-app', 'settings', { theme: 'dark' })

// Retrieve data
const settings = await bus.storage.get<{ theme: string }>('my-app', 'settings')

// List keys
const keys = await bus.storage.list('my-app', 'user-')

// Delete data
await bus.storage.delete('my-app', 'settings')

// Clear namespace
await bus.storage.clear('my-app')

// Check quota
const quota = await bus.storage.quota('my-app')
console.log(`Used ${quota.used} of ${quota.limit} bytes`)

App Lifecycle Notifications

Notify the shell about your app’s loading progress:

const bus = getAppBus()
await bus.init('my-app')

// After auth init (called automatically by init())
bus.notifyReady('auth')

// After CRDT sync complete
bus.notifyReady('synced')

// When fully interactive
bus.notifyReady('ready')

Token Refresh

Request a fresh token when needed:

const bus = getAppBus()

// Manually refresh token
const newToken = await bus.refreshToken()

// Listen for token updates pushed from shell
bus.on('auth:token.push', (msg) => {
  console.log('Token updated:', bus.accessToken)
})

API Client

createApiClient(opts: ApiClientOpts): ApiClient

Create a type-safe REST API client.

import { createApiClient } from '@cloudillo/core'

const api = createApiClient({
  idTag: 'alice.cloudillo.net',  // Required: target tenant
  authToken: 'jwt-token'         // Optional: authentication token
})

// Use the API
const profile = await api.profiles.getOwn()
const posts = await api.actions.list({ type: 'POST', limit: 20 })

Options:

interface ApiClientOpts {
  idTag: string       // Required: identity tag of the tenant
  authToken?: string  // Optional: JWT token for authentication
}
idTag is Required

Unlike the old documentation, idTag is required to create an API client. This specifies which Cloudillo instance to connect to.

Using with AppMessageBus

import { getAppBus, createApiClient } from '@cloudillo/core'

const bus = getAppBus()
const state = await bus.init('my-app')

const api = createApiClient({
  idTag: bus.idTag!,
  authToken: bus.accessToken
})

const files = await api.files.list({ limit: 20 })

CRDT Document Functions

openYDoc(yDoc: Y.Doc, docId: string): Promise<DocConnection>

Open a Yjs document for collaborative editing with WebSocket synchronization.

import { getAppBus, openYDoc } from '@cloudillo/core'
import * as Y from 'yjs'

const bus = getAppBus()
await bus.init('my-app')

const yDoc = new Y.Doc()
const { yDoc: doc, provider } = await openYDoc(yDoc, 'alice.cloudillo.net:document-id')

// Use shared types
const yText = yDoc.getText('content')
yText.insert(0, 'Hello, collaborative world!')

// Listen for changes
yText.observe(() => {
  console.log('Text changed:', yText.toString())
})

// Access awareness (other users' cursors, selections)
provider.awareness.on('change', () => {
  const states = provider.awareness.getStates()
  console.log('Connected users:', states.size)
})

Parameters:

  • yDoc - The Yjs document to synchronize
  • docId - Document ID in format "targetTag:resourceId"

Returns:

interface DocConnection {
  yDoc: Y.Doc
  provider: WebsocketProvider
}

Error Handling:

  • Throws if no access token (must call init() first)
  • Throws if docId format is invalid
  • WebSocket close codes 4401/4403/4404 stop reconnection (auth/permission/not found errors)

URL Helper Functions

Build URLs for Cloudillo services:

getInstanceUrl(idTag: string): string

Build the base URL for a Cloudillo instance.

import { getInstanceUrl } from '@cloudillo/core'

const url = getInstanceUrl('alice.cloudillo.net')
// Returns: "https://cl-o.alice.cloudillo.net"

getApiUrl(idTag: string): string

Build the API base URL.

import { getApiUrl } from '@cloudillo/core'

const url = getApiUrl('alice.cloudillo.net')
// Returns: "https://cl-o.alice.cloudillo.net/api"

getFileUrl(idTag: string, fileId: string, variant?: string): string

Build URL for file access with optional variant.

import { getFileUrl } from '@cloudillo/core'

// Basic file URL
const url = getFileUrl('alice.cloudillo.net', 'file-123')
// Returns: "https://cl-o.alice.cloudillo.net/api/files/file-123"

// With variant
const thumbnailUrl = getFileUrl('alice.cloudillo.net', 'file-123', 'vis.tn')
// Returns: "https://cl-o.alice.cloudillo.net/api/files/file-123?variant=vis.tn"

getCrdtUrl(idTag: string): string

Build the CRDT WebSocket URL.

import { getCrdtUrl } from '@cloudillo/core'

const url = getCrdtUrl('alice.cloudillo.net')
// Returns: "wss://cl-o.alice.cloudillo.net/ws/crdt"

getRtdbUrl(idTag: string, fileId: string, token: string): string

Build the RTDB WebSocket URL with authentication.

import { getRtdbUrl } from '@cloudillo/core'

const url = getRtdbUrl('alice.cloudillo.net', 'db-file-id', 'jwt-token')
// Returns: "wss://cl-o.alice.cloudillo.net/ws/rtdb/db-file-id?token=jwt-token"

getMessageBusUrl(idTag: string, token: string): string

Build the Message Bus WebSocket URL.

import { getMessageBusUrl } from '@cloudillo/core'

const url = getMessageBusUrl('alice.cloudillo.net', 'jwt-token')
// Returns: "wss://cl-o.alice.cloudillo.net/ws/bus?token=jwt-token"

Image/Video Variant Helpers

getOptimalImageVariant(context, localVariants?): string

Get the optimal image variant for a display context.

import { getOptimalImageVariant } from '@cloudillo/core'

// For thumbnail display
const variant = getOptimalImageVariant('thumbnail')  // 'vis.tn'

// For preview
const variant = getOptimalImageVariant('preview')    // 'vis.sd'

// For fullscreen/lightbox
const variant = getOptimalImageVariant('fullscreen') // 'vis.hd'

// With available variants
const variant = getOptimalImageVariant('fullscreen', ['vis.sd', 'vis.hd', 'vis.xd'])
// Returns highest quality available: 'vis.xd'

Image variants: vis.tn (150px), vis.sd (640px), vis.md (1280px), vis.hd (1920px), vis.xd (original)

getOptimalVideoVariant(context, localVariants?): string

Get the optimal video variant for a display context.

import { getOptimalVideoVariant } from '@cloudillo/core'

const variant = getOptimalVideoVariant('preview')    // 'vid.sd'
const variant = getOptimalVideoVariant('fullscreen') // 'vid.hd'

Video variants: vid.sd, vid.md, vid.hd, vid.xd

getImageVariantForDisplaySize(width, height): string

Get optimal variant for a specific display size in pixels.

import { getImageVariantForDisplaySize } from '@cloudillo/core'

// For a 300x200 canvas at 2x zoom (600x400 screen pixels)
const variant = getImageVariantForDisplaySize(600, 400)
// Returns: 'vis.sd'

API Client Structure

The API client provides access to all REST endpoints organized by namespace:

const api = createApiClient({ idTag: 'alice.cloudillo.net', authToken: token })

// ============================================================================
// AUTH - Authentication & Authorization
// ============================================================================
await api.auth.login({ idTag, password })
await api.auth.logout()
await api.auth.getLoginToken()
await api.auth.getAccessToken({ scope, lifetime })
await api.auth.getAccessTokenByRef(refId)           // Guest access via reference
await api.auth.getAccessTokenByApiKey(apiKey)       // API key authentication
await api.auth.getProxyToken(targetIdTag)           // Federation proxy token
await api.auth.getVapidPublicKey()                  // Push notification key
await api.auth.changePassword({ oldPassword, newPassword })
await api.auth.setPassword({ refId, password })     // Set password via reset link
await api.auth.forgotPassword({ email })

// WebAuthn
await api.auth.listWebAuthnCredentials()
await api.auth.getWebAuthnRegChallenge()
await api.auth.registerWebAuthnCredential({ token, credential })
await api.auth.deleteWebAuthnCredential(credentialId)
await api.auth.getWebAuthnLoginChallenge()
await api.auth.webAuthnLogin({ token, credential })

// API Keys
await api.auth.listApiKeys()
await api.auth.createApiKey({ name, scopes })
await api.auth.deleteApiKey(keyId)

// ============================================================================
// PROFILE - Registration & Profile Creation
// ============================================================================
await api.profile.verify({ type, idTag, email })    // Check availability
await api.profile.register({ type, idTag, name, email, password })

// ============================================================================
// PROFILES - Profile Management
// ============================================================================
await api.profiles.getOwn()                         // GET /me
await api.profiles.getOwnFull()                     // GET /me/full
await api.profiles.updateOwn({ name, bio })         // PATCH /me
await api.profiles.list({ q, type, role })          // List/search profiles
await api.profiles.get(idTag)                       // Get profile by idTag
await api.profiles.updateConnection(idTag, data)    // Update relationship
await api.profiles.adminUpdate(idTag, { status, roles }) // Admin operations

// ============================================================================
// ACTIONS - Social Interactions
// ============================================================================
await api.actions.list({ type, status, limit })     // List actions
await api.actions.listPaginated({ cursor, limit })  // With cursor pagination
await api.actions.create({ type, content, ... })    // Create action
await api.actions.get(actionId)                     // Get single action
await api.actions.update(actionId, patch)           // Update draft
await api.actions.delete(actionId)                  // Delete action
await api.actions.accept(actionId)                  // Accept (e.g., connection)
await api.actions.reject(actionId)                  // Reject
await api.actions.updateStat(actionId, stat)        // Update statistics
await api.actions.addReaction(actionId, { type })   // Add reaction

// ============================================================================
// FILES - File Management
// ============================================================================
await api.files.list({ parentId, tag, limit })      // List files
await api.files.listPaginated({ cursor, limit })    // With cursor pagination
await api.files.create({ fileName, fileTp })        // Create metadata-only file
await api.files.uploadBlob(preset, name, data, contentType) // Upload file
await api.files.get(fileId)                         // Get file content
await api.files.getVariant(variantId)               // Get specific variant
await api.files.getDescriptor(fileId)               // Get file metadata
await api.files.update(fileId, { fileName })        // Update metadata
await api.files.delete(fileId)                      // Soft delete (to trash)
await api.files.permanentDelete(fileId)             // Permanent delete
await api.files.restore(fileId, parentId)           // Restore from trash
await api.files.updateUserData(fileId, { starred }) // Update user-specific data
await api.files.setStarred(fileId, true)            // Toggle starred
await api.files.setPinned(fileId, true)             // Toggle pinned
await api.files.addTag(fileId, 'tag')               // Add tag
await api.files.removeTag(fileId, 'tag')            // Remove tag
await api.files.favorite(fileId)                    // Add to favorites
await api.files.unfavorite(fileId)                  // Remove from favorites
await api.files.listFavorites({ limit })            // List favorites
await api.files.listRecent({ limit })               // List recent files

// ============================================================================
// TRASH - Trash Management
// ============================================================================
await api.trash.list({ limit })                     // List trashed files
await api.trash.empty()                             // Empty trash permanently

// ============================================================================
// COLLECTIONS - Favorites, Bookmarks, Pins
// ============================================================================
await api.collections.list('FAVR', { limit })       // List collection items
await api.collections.add('BKMK', itemId)           // Add to collection
await api.collections.remove('PIND', itemId)        // Remove from collection
// Collection types: 'FAVR' | 'RCNT' | 'BKMK' | 'PIND'

// ============================================================================
// TAGS - Tag Management
// ============================================================================
await api.tags.list({ prefix, withCounts })         // List tags

// ============================================================================
// SETTINGS - User Settings
// ============================================================================
await api.settings.list({ prefix })                 // List settings
await api.settings.get(name)                        // Get setting value
await api.settings.update(name, { value })          // Update setting

// ============================================================================
// NOTIFICATIONS - Push Notifications
// ============================================================================
await api.notifications.subscribe({ subscription }) // Subscribe to push

// ============================================================================
// REFS - Share Links & References
// ============================================================================
await api.refs.list({ type, resourceId })           // List references
await api.refs.get(refId)                           // Get reference
await api.refs.create({ type, resourceId, access }) // Create share link
await api.refs.delete(refId)                        // Delete reference

// ============================================================================
// IDP - Identity Provider (End User)
// ============================================================================
await api.idp.getInfo(providerDomain)               // Get provider info
await api.idp.activate({ refId })                   // Activate identity

// ============================================================================
// IDP MANAGEMENT - Identity Provider Admin
// ============================================================================
await api.idpManagement.listIdentities({ q, status })
await api.idpManagement.createIdentity({ idTag, email, createApiKey })
await api.idpManagement.getIdentity(idTag)
await api.idpManagement.updateIdentity(idTag, { dyndns })
await api.idpManagement.deleteIdentity(idTag)
await api.idpManagement.listApiKeys(idTag)
await api.idpManagement.createApiKey({ idTag, name })
await api.idpManagement.deleteApiKey(keyId, idTag)

// ============================================================================
// COMMUNITIES - Community Management
// ============================================================================
await api.communities.create(idTag, { type, name, ownerIdTag })
await api.communities.verify({ type, idTag })       // Deprecated: use profile.verify

// ============================================================================
// ADMIN - System Administration
// ============================================================================
await api.admin.listTenants({ q, status, limit })   // List all tenants
await api.admin.sendPasswordReset(idTag)            // Send password reset
await api.admin.sendTestEmail(to)                   // Test SMTP config

Helper Functions

delay(ms: number): Promise<void>

Delay execution for a specified time.

import { delay } from '@cloudillo/core'

await delay(1000) // Wait 1 second

See Also

@cloudillo/react

Overview

React hooks and components for integrating Cloudillo into React applications.

Installation

pnpm add @cloudillo/react @cloudillo/core

Hooks

useAuth()

Access authentication state using Jotai atoms. Returns a tuple [auth, setAuth] for reading and updating auth state.

import { useAuth } from '@cloudillo/react'

function UserInfo() {
  const [auth, setAuth] = useAuth()

  if (!auth?.idTag) {
    return <div>Not authenticated</div>
  }

  return (
    <div>
      <p>User: {auth.idTag}</p>
      <p>Name: {auth.name}</p>
      <p>Tenant: {auth.tnId}</p>
      <p>Roles: {auth.roles?.join(', ')}</p>
      {auth.profilePic && <img src={auth.profilePic} alt="Profile" />}
    </div>
  )
}

Returns: [AuthState | undefined, SetAtom<AuthState>]

interface AuthState {
  tnId: number          // Tenant ID
  idTag?: string        // Identity tag (e.g., "alice.cloudillo.net")
  name?: string         // Display name
  profilePic?: string   // Profile picture ID
  roles?: string[]      // Community roles
  token?: string        // JWT access token
}

Usage patterns:

// Destructure the tuple
const [auth, setAuth] = useAuth()

// Check if user is authenticated
if (!auth?.idTag) {
  return <LoginPrompt />
}

// Check for specific role
if (auth?.roles?.includes('admin')) {
  return <AdminPanel />
}

// Update auth state (typically done by useCloudillo)
setAuth({
  tnId: 123,
  idTag: 'alice.cloudillo.net',
  token: 'jwt-token'
})

useApi()

Get a type-safe API client with automatic caching per idTag/token combination.

import { useApi } from '@cloudillo/react'
import { useEffect, useState } from 'react'

function PostsList() {
  const { api, authenticated, setIdTag } = useApi()
  const [posts, setPosts] = useState([])
  const [loading, setLoading] = useState(true)

  useEffect(() => {
    if (!api) return

    api.actions.list({ type: 'POST', limit: 20 })
      .then(setPosts)
      .finally(() => setLoading(false))
  }, [api])

  if (!api) return <div>No API client (no idTag)</div>
  if (!authenticated) return <div>Please log in</div>
  if (loading) return <div>Loading...</div>

  return (
    <div>
      {posts.map(post => (
        <div key={post.actionId}>{post.content}</div>
      ))}
    </div>
  )
}

Returns:

interface ApiHook {
  api: ApiClient | null    // Type-safe API client (null if no idTag)
  authenticated: boolean   // Whether user has a token
  setIdTag: (idTag: string) => void // Set idTag for login flow
}

The API client is the same as returned by createApiClient() from @cloudillo/core.

api can be null

api is null until an idTag is available (either from auth state or set via setIdTag). Always check for null before using.

useCloudillo()

Unified hook for microfrontend app initialization. Handles shell communication, authentication, and document context.

import { useCloudillo } from '@cloudillo/react'

function MyApp() {
  const { token, ownerTag, fileId, idTag, roles, access, displayName } = useCloudillo('my-app')

  if (!token) return <div>Loading...</div>

  return (
    <div>
      <p>Document: {fileId}</p>
      <p>Owner: {ownerTag}</p>
      <p>Access: {access}</p>
      <p>User: {idTag}</p>
    </div>
  )
}

This hook:

  1. Calls getAppBus().init(appName) on mount
  2. Parses ownerTag and fileId from location.hash (format: #ownerTag:fileId)
  3. Updates auth state via useAuth()
  4. Returns combined state from bus and URL

Returns:

interface UseCloudillo {
  token?: string           // Access token
  ownerTag: string         // Owner of the current document (from URL hash)
  fileId?: string          // Current file/document ID (from URL hash)
  idTag?: string           // Current user's identity tag
  tnId?: number            // Tenant ID
  roles?: string[]         // User's roles
  access?: 'read' | 'write' // Access level to current resource
  displayName?: string     // User's display name (for anonymous guests)
}

useCloudilloEditor()

Extended hook for CRDT-based collaborative document editing. Returns everything from useCloudillo() plus Yjs document and sync state.

import { useCloudilloEditor } from '@cloudillo/react'

function CollaborativeEditor() {
  const { token, yDoc, provider, synced, ownerTag, fileId, access } = useCloudilloEditor('quillo')

  if (!synced) return <LoadingSpinner />

  // Use yDoc with your editor binding (Quill, TipTap, Monaco, etc.)
  const yText = yDoc.getText('content')

  return <QuillEditor yText={yText} provider={provider} />
}

This hook:

  1. Calls useCloudillo(appName) internally
  2. Opens CRDT connection via openYDoc() when token and docId are available
  3. Listens for sync events and notifies shell via bus.notifyReady('synced')
  4. Cleans up WebSocket provider on unmount

Returns:

interface UseCloudilloEditor extends UseCloudillo {
  yDoc: Y.Doc              // Yjs document instance
  provider?: WebsocketProvider // WebSocket sync provider
  synced: boolean          // Whether initial sync is complete
}

useInfiniteScroll()

Cursor-based infinite scroll pagination with IntersectionObserver.

import { useInfiniteScroll } from '@cloudillo/react'

function FileList() {
  const { api } = useApi()

  const { items, isLoading, hasMore, sentinelRef, prepend, reset } = useInfiniteScroll({
    fetchPage: async (cursor, limit) => {
      const result = await api.files.listPaginated({ cursor, limit })
      return {
        items: result.data,
        nextCursor: result.cursorPagination?.nextCursor ?? null,
        hasMore: result.cursorPagination?.hasMore ?? false
      }
    },
    pageSize: 30,
    deps: [folderId, sortField] // Reset when these change
  })

  return (
    <div>
      {items.map(file => <FileCard key={file.fileId} file={file} />)}
      {/* Sentinel triggers loadMore when visible */}
      <div ref={sentinelRef} />
      {isLoading && <LoadingSpinner />}
    </div>
  )
}

Options:

interface UseInfiniteScrollOptions<T> {
  fetchPage: (cursor: string | null, limit: number) => Promise<{
    items: T[]
    nextCursor: string | null
    hasMore: boolean
  }>
  pageSize?: number           // Default: 20
  deps?: React.DependencyList // Reset when these change
  enabled?: boolean           // Default: true
}

Returns:

interface UseInfiniteScrollReturn<T> {
  items: T[]                    // All loaded items
  isLoading: boolean            // Initial load in progress
  isLoadingMore: boolean        // Loading more pages
  error: Error | null           // Last fetch error
  hasMore: boolean              // More items available
  loadMore: () => void          // Manually load next page
  reset: () => void             // Reset and reload
  prepend: (items: T[]) => void // Add items to start (real-time)
  sentinelRef: RefObject<HTMLDivElement> // Attach to trigger element
}

Common Patterns

Pattern 1: Microfrontend App

The typical pattern for a Cloudillo microfrontend app:

import { useCloudillo, useApi } from '@cloudillo/react'
import { useEffect, useState } from 'react'

function App() {
  const { token, idTag, ownerTag, fileId, access } = useCloudillo('my-app')
  const { api } = useApi()
  const [data, setData] = useState(null)

  useEffect(() => {
    if (!api || !fileId) return

    api.files.getDescriptor(fileId)
      .then(setData)
      .catch(console.error)
  }, [api, fileId])

  if (!token) return <div>Initializing...</div>
  if (!data) return <div>Loading...</div>

  return (
    <div>
      <h1>{data.fileName}</h1>
      <p>Owner: {ownerTag}</p>
      <p>Access: {access}</p>
    </div>
  )
}

Pattern 2: Collaborative Editor

import { useCloudilloEditor } from '@cloudillo/react'
import { useEffect } from 'react'

function Editor() {
  const { yDoc, provider, synced, access } = useCloudilloEditor('my-editor')

  useEffect(() => {
    if (!synced) return

    const yText = yDoc.getText('content')

    // Set up your editor binding here
    // e.g., QuillBinding, TipTapExtension, etc.

    return () => {
      // Clean up binding
    }
  }, [yDoc, synced])

  if (!synced) return <div>Syncing document...</div>

  return (
    <div>
      {access === 'read' && <div className="read-only-banner">Read only</div>}
      <div id="editor-container" />
    </div>
  )
}

Pattern 3: Fetching Data with useApi

import { useApi } from '@cloudillo/react'
import { useEffect, useState } from 'react'

function Profile({ idTag }) {
  const { api } = useApi()
  const [profile, setProfile] = useState(null)
  const [error, setError] = useState(null)

  useEffect(() => {
    if (!api) return

    api.profiles.get(idTag)
      .then(setProfile)
      .catch(setError)
  }, [api, idTag])

  if (!api) return <div>No API client</div>
  if (error) return <div>Error: {error.message}</div>
  if (!profile) return <div>Loading...</div>

  return (
    <div>
      <h1>{profile.name}</h1>
      <p>{profile.idTag}</p>
    </div>
  )
}

Pattern 4: Creating Actions

import { useApi } from '@cloudillo/react'
import { useState } from 'react'

function CreatePost() {
  const { api, authenticated } = useApi()
  const [text, setText] = useState('')
  const [posting, setPosting] = useState(false)

  if (!authenticated) return <div>Please log in to post</div>

  const handleSubmit = async (e) => {
    e.preventDefault()
    if (!api || !text.trim()) return

    setPosting(true)

    try {
      await api.actions.create({
        type: 'POST',
        content: { text }
      })
      setText('')
    } catch (error) {
      console.error('Failed to create post:', error)
    } finally {
      setPosting(false)
    }
  }

  return (
    <form onSubmit={handleSubmit}>
      <textarea
        value={text}
        onChange={(e) => setText(e.target.value)}
        placeholder="What's on your mind?"
      />
      <button type="submit" disabled={posting || !text.trim()}>
        {posting ? 'Posting...' : 'Post'}
      </button>
    </form>
  )
}

Pattern 5: Role-Based Rendering

import { useAuth } from '@cloudillo/react'

function AdminPanel() {
  const [auth] = useAuth()

  if (!auth?.roles?.includes('admin')) {
    return <div>Access denied. Admin role required.</div>
  }

  return (
    <div>
      <h1>Admin Panel</h1>
      {/* Admin-only features */}
    </div>
  )
}

Pattern 6: File Upload

import { useApi } from '@cloudillo/react'
import { useState } from 'react'

function ImageUpload() {
  const { api } = useApi()
  const [uploading, setUploading] = useState(false)
  const [result, setResult] = useState(null)

  const handleFileChange = async (e) => {
    const file = e.target.files?.[0]
    if (!file || !api) return

    setUploading(true)

    try {
      // Upload the file
      const uploaded = await api.files.uploadBlob(
        'gallery',      // preset
        file.name,      // fileName
        file,           // file data
        file.type       // contentType
      )

      setResult(uploaded)
    } catch (error) {
      console.error('Upload failed:', error)
    } finally {
      setUploading(false)
    }
  }

  return (
    <div>
      <input type="file" accept="image/*" onChange={handleFileChange} />
      {uploading && <div>Uploading...</div>}
      {result && <div>Uploaded: {result.fileId}</div>}
    </div>
  )
}

Pattern 7: Real-Time Updates with RTDB

import { useApi, useAuth } from '@cloudillo/react'
import { useEffect, useState } from 'react'
import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

function TodoList({ dbFileId }) {
  const [auth] = useAuth()
  const [todos, setTodos] = useState([])

  useEffect(() => {
    if (!auth?.token || !auth?.idTag) return

    const rtdb = new RtdbClient({
      dbId: dbFileId,
      auth: { getToken: () => auth.token },
      serverUrl: getRtdbUrl(auth.idTag, dbFileId, auth.token)
    })

    const todosRef = rtdb.collection('todos')

    // Subscribe to real-time updates
    const unsubscribe = todosRef.onSnapshot((snapshot) => {
      setTodos(snapshot.docs.map(doc => ({ id: doc.id, ...doc.data() })))
    })

    // Cleanup on unmount
    return () => {
      unsubscribe()
      rtdb.disconnect()
    }
  }, [auth?.token, auth?.idTag, dbFileId])

  return (
    <ul>
      {todos.map(todo => (
        <li key={todo.id}>{todo.title}</li>
      ))}
    </ul>
  )
}

TypeScript Support

All hooks are fully typed:

import type { AuthState, ApiHook } from '@cloudillo/react'
import { useAuth, useApi } from '@cloudillo/react'

function MyComponent() {
  const [auth, setAuth] = useAuth()
  const { api, authenticated, setIdTag }: ApiHook = useApi()

  // TypeScript knows the types
  auth?.idTag    // string | undefined
  auth?.tnId     // number | undefined
  auth?.roles    // string[] | undefined

  api?.profiles.getOwn()  // Returns Promise<ProfileKeys>
}

See Also

React components reference

Complete reference for all 85+ components, 12+ hooks, and utility functions exported by @cloudillo/react.

Layout Components

Container

Centered max-width container for page content.

import { Container } from '@cloudillo/react'

<Container>
  <h1>Page Content</h1>
</Container>

HBox, VBox, Group

Flexbox layout primitives.

import { HBox, VBox, Group } from '@cloudillo/react'

// Horizontal layout
<HBox gap="md">
  <Button>Left</Button>
  <Button>Right</Button>
</HBox>

// Vertical layout
<VBox gap="sm">
  <Input label="Name" />
  <Input label="Email" />
</VBox>

// Grouped items with spacing
<Group>
  <Tag>React</Tag>
  <Tag>TypeScript</Tag>
</Group>

Props (shared):

  • gap?: 'xs' | 'sm' | 'md' | 'lg' | 'xl' - Spacing between children
  • align?: 'start' | 'center' | 'end' | 'stretch' - Alignment
  • justify?: 'start' | 'center' | 'end' | 'between' | 'around' - Justification

Panel

Bordered container with optional header.

import { Panel } from '@cloudillo/react'

<Panel title="Settings">
  <p>Panel content here</p>
</Panel>

Card

Elevated card container.

import { Card } from '@cloudillo/react'

<Card>
  <h3>Card Title</h3>
  <p>Card content</p>
</Card>

Fcd (Filter-Content-Details)

Three-column responsive layout pattern for list views.

import { Fcd, FcdContainer, FcdFilter, FcdContent, FcdDetails } from '@cloudillo/react'

<FcdContainer>
  <FcdFilter>
    <FilterBar />
  </FcdFilter>
  <FcdContent>
    <ItemList />
  </FcdContent>
  <FcdDetails>
    <ItemDetails />
  </FcdDetails>
</FcdContainer>

Collapsible sidebar with mobile support.

import {
  useSidebar,
  Sidebar,
  SidebarContent,
  SidebarHeader,
  SidebarFooter,
  SidebarNav,
  SidebarSection,
  SidebarToggle,
  SidebarBackdrop,
  SidebarResizeHandle
} from '@cloudillo/react'

function Layout() {
  const sidebar = useSidebar({ defaultOpen: true })

  return (
    <div>
      <SidebarBackdrop {...sidebar} />
      <Sidebar {...sidebar}>
        <SidebarHeader>
          <Logo />
        </SidebarHeader>
        <SidebarContent>
          <SidebarNav>
            <SidebarSection title="Main">
              <NavItem to="/home">Home</NavItem>
              <NavItem to="/files">Files</NavItem>
            </SidebarSection>
          </SidebarNav>
        </SidebarContent>
        <SidebarFooter>
          <ProfileCard />
        </SidebarFooter>
        <SidebarResizeHandle />
      </Sidebar>
      <SidebarToggle {...sidebar} />
    </div>
  )
}

useSidebar options:

interface UseSidebarOptions {
  defaultOpen?: boolean
  defaultWidth?: number
  minWidth?: number
  maxWidth?: number
  breakpoint?: number // Mobile breakpoint
}

Additional components:

  • SidebarBackdrop - Mobile overlay backdrop that closes sidebar when tapped
  • SidebarResizeHandle - Draggable handle for resizing sidebar width
  • SidebarContext / useSidebarContext - Context provider and hook for nested components

Navigation menu components.

import { Nav, NavGroup, NavItem, NavLink } from '@cloudillo/react'

<Nav>
  <NavGroup title="Main">
    <NavItem icon={<HomeIcon />}>
      <NavLink to="/home">Home</NavLink>
    </NavItem>
    <NavItem icon={<FilesIcon />}>
      <NavLink to="/files">Files</NavLink>
    </NavItem>
  </NavGroup>
</Nav>

Tabs, Tab

Tabbed interface.

import { Tabs, Tab } from '@cloudillo/react'

<Tabs defaultValue="tab1">
  <Tab value="tab1" label="General">
    <GeneralSettings />
  </Tab>
  <Tab value="tab2" label="Advanced">
    <AdvancedSettings />
  </Tab>
</Tabs>

Context: TabsContext is available for building custom tab implementations.


Form Components

Input

Text input with label and validation.

import { Input } from '@cloudillo/react'

<Input
  label="Email"
  type="email"
  placeholder="you@example.com"
  error="Invalid email address"
/>

Props:

  • label?: string - Input label
  • error?: string - Error message
  • hint?: string - Helper text
  • size?: 'sm' | 'md' | 'lg' - Input size
  • All standard <input> props

TextArea

Multi-line text input.

import { TextArea } from '@cloudillo/react'

<TextArea
  label="Description"
  rows={4}
  placeholder="Enter description..."
/>

Select

Custom dropdown select.

import { Select } from '@cloudillo/react'

<Select
  label="Country"
  options={[
    { value: 'us', label: 'United States' },
    { value: 'uk', label: 'United Kingdom' },
    { value: 'de', label: 'Germany' }
  ]}
  onChange={(value) => console.log(value)}
/>

NativeSelect

Native browser select element.

import { NativeSelect } from '@cloudillo/react'

<NativeSelect label="Priority">
  <option value="low">Low</option>
  <option value="medium">Medium</option>
  <option value="high">High</option>
</NativeSelect>

NumberInput

Numeric input with increment/decrement buttons.

import { NumberInput } from '@cloudillo/react'

<NumberInput
  label="Quantity"
  min={0}
  max={100}
  step={1}
  value={5}
  onChange={(value) => console.log(value)}
/>

ColorInput

Color picker input.

import { ColorInput } from '@cloudillo/react'

<ColorInput
  label="Theme Color"
  value="#3b82f6"
  onChange={(color) => console.log(color)}
/>

Toggle

On/off toggle switch.

import { Toggle } from '@cloudillo/react'

<Toggle
  label="Enable notifications"
  checked={enabled}
  onChange={setEnabled}
/>

Fieldset

Group related form fields.

import { Fieldset } from '@cloudillo/react'

<Fieldset legend="Contact Information">
  <Input label="Phone" />
  <Input label="Address" />
</Fieldset>

InputGroup

Group input with addons (prefix/suffix).

import { InputGroup } from '@cloudillo/react'

<InputGroup>
  <InputGroup.Addon>https://</InputGroup.Addon>
  <Input placeholder="example.com" />
</InputGroup>

TagInput

Multi-value tag input with autocomplete.

import { TagInput } from '@cloudillo/react'

<TagInput
  label="Tags"
  value={['react', 'typescript']}
  onChange={setTags}
  suggestions={['react', 'vue', 'angular', 'typescript']}
  placeholder="Add tags..."
/>

InlineEditForm

Inline editable text field.

import { InlineEditForm } from '@cloudillo/react'

<InlineEditForm
  value={title}
  onSave={(newValue) => updateTitle(newValue)}
  placeholder="Click to edit..."
/>

Button Components

Button

Primary button component.

import { Button } from '@cloudillo/react'

<Button variant="primary" onClick={handleClick}>
  Save Changes
</Button>

<Button variant="secondary" size="sm">
  Cancel
</Button>

<Button variant="danger" loading>
  Deleting...
</Button>

Props:

  • variant?: 'primary' | 'secondary' | 'danger' | 'ghost'
  • size?: 'xs' | 'sm' | 'md' | 'lg'
  • loading?: boolean - Shows spinner
  • disabled?: boolean
  • icon?: ReactNode - Left icon
  • iconRight?: ReactNode - Right icon

LinkButton

Button styled as a link.

import { LinkButton } from '@cloudillo/react'

<LinkButton to="/settings">Go to Settings</LinkButton>

Dialog & Overlay Components

Dialog

Modal dialog with backdrop.

import { Dialog, useDialog } from '@cloudillo/react'

function MyComponent() {
  const dialog = useDialog()

  return (
    <>
      <Button onClick={dialog.open}>Open Dialog</Button>

      <Dialog {...dialog} title="Confirm Action">
        <p>Are you sure you want to proceed?</p>
        <HBox gap="sm">
          <Button variant="secondary" onClick={dialog.close}>Cancel</Button>
          <Button variant="primary" onClick={handleConfirm}>Confirm</Button>
        </HBox>
      </Dialog>
    </>
  )
}

useDialog returns:

interface UseDialogReturn {
  isOpen: boolean
  open: () => void
  close: () => void
  toggle: () => void
}

DialogContainer

Wrapper component for rendering dialogs at the root level.

import { DialogContainer } from '@cloudillo/react'

function App() {
  return (
    <>
      <MainContent />
      <DialogContainer />
    </>
  )
}

Low-level modal wrapper.

import { Modal } from '@cloudillo/react'

<Modal isOpen={isOpen} onClose={handleClose}>
  <div className="modal-content">
    Custom modal content
  </div>
</Modal>

BottomSheet

Mobile-friendly bottom sheet.

import { BottomSheet } from '@cloudillo/react'

<BottomSheet
  isOpen={isOpen}
  onClose={handleClose}
  snapPoints={['50%', '90%']}
>
  <div>Sheet content</div>
</BottomSheet>

Dropdown menu container.

import { Dropdown } from '@cloudillo/react'

<Dropdown
  trigger={<Button>Options</Button>}
  align="end"
>
  <MenuItem onClick={handleEdit}>Edit</MenuItem>
  <MenuItem onClick={handleDelete}>Delete</MenuItem>
</Dropdown>

Popper

Floating positioned element (tooltips, popovers).

import { Popper } from '@cloudillo/react'

<Popper
  trigger={<Button>Hover me</Button>}
  placement="top"
>
  <div>Tooltip content</div>
</Popper>

QRCodeDialog

Dialog showing a QR code.

import { QRCodeDialog } from '@cloudillo/react'

<QRCodeDialog
  isOpen={isOpen}
  onClose={handleClose}
  value="https://cloudillo.net/share/abc123"
  title="Share Link"
/>

Context menu and dropdown menu items.

import { Menu, MenuItem, MenuDivider, MenuHeader } from '@cloudillo/react'

<Menu>
  <MenuHeader>Actions</MenuHeader>
  <MenuItem icon={<EditIcon />} onClick={handleEdit}>
    Edit
  </MenuItem>
  <MenuItem icon={<CopyIcon />} onClick={handleCopy}>
    Copy
  </MenuItem>
  <MenuDivider />
  <MenuItem icon={<TrashIcon />} variant="danger" onClick={handleDelete}>
    Delete
  </MenuItem>
</Menu>

ActionSheet

Mobile action sheet (bottom menu).

import { ActionSheet, ActionSheetItem, ActionSheetDivider } from '@cloudillo/react'

<ActionSheet isOpen={isOpen} onClose={handleClose}>
  <ActionSheetItem onClick={handleShare}>Share</ActionSheetItem>
  <ActionSheetItem onClick={handleCopy}>Copy Link</ActionSheetItem>
  <ActionSheetDivider />
  <ActionSheetItem variant="danger" onClick={handleDelete}>
    Delete
  </ActionSheetItem>
</ActionSheet>

Feedback Components

Toast

Toast notification system.

import { useToast, ToastContainer } from '@cloudillo/react'

// In your app root
function App() {
  return (
    <>
      <ToastContainer />
      <MainContent />
    </>
  )
}

// In any component
function MyComponent() {
  const toast = useToast()

  const handleSave = async () => {
    try {
      await saveData()
      toast.success('Saved successfully!')
    } catch (err) {
      toast.error('Failed to save')
    }
  }

  return <Button onClick={handleSave}>Save</Button>
}

useToast methods:

  • toast.success(message, options?) - Success toast
  • toast.error(message, options?) - Error toast
  • toast.warning(message, options?) - Warning toast
  • toast.info(message, options?) - Info toast
  • toast.custom(content, options?) - Custom toast

Toast sub-components:

For building custom toast layouts:

import {
  Toast,
  ToastIcon,
  ToastContent,
  ToastTitle,
  ToastMessage,
  ToastActions,
  ToastClose,
  ToastProgress
} from '@cloudillo/react'

// Custom toast layout
<Toast variant="success">
  <ToastIcon />
  <ToastContent>
    <ToastTitle>Success!</ToastTitle>
    <ToastMessage>Your changes have been saved.</ToastMessage>
  </ToastContent>
  <ToastActions>
    <Button size="sm">Undo</Button>
  </ToastActions>
  <ToastClose />
  <ToastProgress />
</Toast>

Context: ToastContext and useToastContext are available for building custom toast providers.

LoadingSpinner

Spinning loader indicator.

import { LoadingSpinner } from '@cloudillo/react'

<LoadingSpinner size="md" />

{isLoading && <LoadingSpinner />}

Skeleton, SkeletonText, SkeletonCard, SkeletonList

Loading skeleton placeholders.

import { Skeleton, SkeletonText, SkeletonCard, SkeletonList } from '@cloudillo/react'

// Basic skeleton
<Skeleton width={200} height={20} />

// Text skeleton
<SkeletonText lines={3} />

// Card skeleton
<SkeletonCard />

// List skeleton
<SkeletonList count={5} />

Progress

Progress bar.

import { Progress } from '@cloudillo/react'

<Progress value={75} max={100} />

<Progress value={uploadProgress} showLabel />

EmptyState

Empty state placeholder.

import { EmptyState } from '@cloudillo/react'

<EmptyState
  icon={<FilesIcon />}
  title="No files yet"
  description="Upload your first file to get started"
  action={<Button>Upload File</Button>}
/>

Profile Components

Avatar, AvatarStatus, AvatarBadge, AvatarGroup

User avatar components.

import { Avatar, AvatarStatus, AvatarBadge, AvatarGroup } from '@cloudillo/react'

// Basic avatar
<Avatar src={profilePic} name="Alice" size="md" />

// With status indicator
<Avatar src={profilePic}>
  <AvatarStatus status="online" />
</Avatar>

// With badge
<Avatar src={profilePic}>
  <AvatarBadge>3</AvatarBadge>
</Avatar>

// Group of avatars
<AvatarGroup max={3}>
  <Avatar src={user1.pic} name={user1.name} />
  <Avatar src={user2.pic} name={user2.name} />
  <Avatar src={user3.pic} name={user3.name} />
  <Avatar src={user4.pic} name={user4.name} />
</AvatarGroup>

ProfilePicture, UnknownProfilePicture

Cloudillo profile picture components.

import { ProfilePicture, UnknownProfilePicture } from '@cloudillo/react'

<ProfilePicture idTag="alice.cloudillo.net" size="lg" />

<UnknownProfilePicture size="md" />

IdentityTag

Display identity tag with icon.

import { IdentityTag } from '@cloudillo/react'

<IdentityTag idTag="alice.cloudillo.net" />

ProfileCard

Profile card with picture, name, and actions.

import { ProfileCard } from '@cloudillo/react'

<ProfileCard
  profile={profile}
  onConnect={handleConnect}
  onFollow={handleFollow}
/>

ProfileAudienceCard

Profile card optimized for audience selection.

import { ProfileAudienceCard } from '@cloudillo/react'

<ProfileAudienceCard
  profile={profile}
  selected={isSelected}
  onSelect={handleSelect}
/>

EditProfileList

Editable list of profiles (for managing connections, etc.).

import { EditProfileList } from '@cloudillo/react'

<EditProfileList
  profiles={connections}
  onRemove={handleRemove}
  emptyMessage="No connections yet"
/>

Data Display Components

TreeView, TreeItem

Hierarchical tree view.

import { TreeView, TreeItem } from '@cloudillo/react'

<TreeView>
  <TreeItem label="Documents" icon={<FolderIcon />}>
    <TreeItem label="Report.pdf" icon={<FileIcon />} />
    <TreeItem label="Notes.txt" icon={<FileIcon />} />
  </TreeItem>
  <TreeItem label="Images" icon={<FolderIcon />}>
    <TreeItem label="Photo.jpg" icon={<ImageIcon />} />
  </TreeItem>
</TreeView>

Accordion, AccordionItem

Collapsible accordion sections.

import { Accordion, AccordionItem } from '@cloudillo/react'

<Accordion>
  <AccordionItem title="General Settings">
    <GeneralSettings />
  </AccordionItem>
  <AccordionItem title="Advanced Settings">
    <AdvancedSettings />
  </AccordionItem>
</Accordion>

PropertyPanel, PropertySection, PropertyField

Property inspector panel (like IDE properties).

import { PropertyPanel, PropertySection, PropertyField } from '@cloudillo/react'

<PropertyPanel>
  <PropertySection title="Appearance">
    <PropertyField label="Width">
      <NumberInput value={width} onChange={setWidth} />
    </PropertyField>
    <PropertyField label="Color">
      <ColorInput value={color} onChange={setColor} />
    </PropertyField>
  </PropertySection>
</PropertyPanel>

TimeFormat

Formatted time display (relative or absolute).

import { TimeFormat } from '@cloudillo/react'

<TimeFormat value={createdAt} />
// Renders: "2 hours ago" or "Jan 15, 2025"

Badge

Status badge.

import { Badge } from '@cloudillo/react'

<Badge variant="success">Active</Badge>
<Badge variant="warning">Pending</Badge>
<Badge variant="danger">Error</Badge>

Tag, TagList

Tags and tag lists.

import { Tag, TagList } from '@cloudillo/react'

<Tag>React</Tag>

<TagList
  tags={['react', 'typescript', 'cloudillo']}
  onRemove={handleRemoveTag}
/>

Filter & Toolbar Components

FilterBar

Filter bar with search and filters.

import {
  FilterBar,
  FilterBarSearch,
  FilterBarItem,
  FilterBarSection,
  FilterBarDivider
} from '@cloudillo/react'

<FilterBar>
  <FilterBarSearch
    value={search}
    onChange={setSearch}
    placeholder="Search files..."
  />
  <FilterBarDivider />
  <FilterBarSection>
    <FilterBarItem
      label="Type"
      value={typeFilter}
      onChange={setTypeFilter}
      options={typeOptions}
    />
    <FilterBarItem
      label="Date"
      value={dateFilter}
      onChange={setDateFilter}
      options={dateOptions}
    />
  </FilterBarSection>
</FilterBar>

Alternative export: FilterBarComponent is also exported for cases where you need to avoid naming conflicts.

Toolbar

Action toolbar.

import { Toolbar, ToolbarGroup, ToolbarDivider, ToolbarSpacer } from '@cloudillo/react'

<Toolbar>
  <ToolbarGroup>
    <Button icon={<BoldIcon />} />
    <Button icon={<ItalicIcon />} />
    <Button icon={<UnderlineIcon />} />
  </ToolbarGroup>
  <ToolbarDivider />
  <ToolbarGroup>
    <Button icon={<AlignLeftIcon />} />
    <Button icon={<AlignCenterIcon />} />
  </ToolbarGroup>
  <ToolbarSpacer />
  <Button>Save</Button>
</Toolbar>

Infinite Scroll Components

LoadMoreTrigger

Intersection observer trigger for infinite scroll.

import { LoadMoreTrigger } from '@cloudillo/react'
import { useInfiniteScroll } from '@cloudillo/react'

function FileList() {
  const { items, isLoading, sentinelRef } = useInfiniteScroll({
    fetchPage: async (cursor, limit) => {
      const result = await api.files.list({ cursor, limit })
      return {
        items: result.data,
        nextCursor: result.cursorPagination?.nextCursor ?? null,
        hasMore: result.cursorPagination?.hasMore ?? false
      }
    }
  })

  return (
    <div>
      {items.map(file => <FileCard key={file.fileId} file={file} />)}
      <LoadMoreTrigger ref={sentinelRef} isLoading={isLoading} />
    </div>
  )
}

Utility Components

FormattedText

Render formatted text with markdown support.

import { FormattedText } from '@cloudillo/react'

<FormattedText content="**Bold** and *italic* text" />

FontPicker

Font selection component. Works with the @cloudillo/fonts library for font metadata and pairing suggestions.

import { FontPicker } from '@cloudillo/react'
import { getSuggestedBodyFonts } from '@cloudillo/fonts'

<FontPicker
  value={fontFamily}
  onChange={setFontFamily}
/>

// Use with font pairings
const suggestedBodies = getSuggestedBodyFonts(headingFont)

See @cloudillo/fonts for available fonts and pairing APIs.


Utility Hooks

useMergedRefs

Combine multiple refs into one.

import { useMergedRefs } from '@cloudillo/react'

function MyComponent({ forwardedRef }) {
  const localRef = useRef()
  const mergedRef = useMergedRefs(localRef, forwardedRef)

  return <div ref={mergedRef} />
}

useBodyScrollLock

Lock body scroll (for modals).

import { useBodyScrollLock } from '@cloudillo/react'

function Modal({ isOpen }) {
  useBodyScrollLock(isOpen)

  return isOpen ? <div className="modal">...</div> : null
}

useEscapeKey

Handle Escape key press.

import { useEscapeKey } from '@cloudillo/react'

function Modal({ onClose }) {
  useEscapeKey(onClose)

  return <div className="modal">...</div>
}

useOutsideClick

Detect clicks outside an element.

import { useOutsideClick } from '@cloudillo/react'

function Dropdown({ onClose }) {
  const ref = useRef()
  useOutsideClick(ref, onClose)

  return <div ref={ref} className="dropdown">...</div>
}

useMediaQuery

Responsive media query hook.

import { useMediaQuery } from '@cloudillo/react'

function MyComponent() {
  const isMobile = useMediaQuery('(max-width: 768px)')

  return isMobile ? <MobileView /> : <DesktopView />
}

useIsMobile

Shorthand for mobile detection.

import { useIsMobile } from '@cloudillo/react'

function MyComponent() {
  const isMobile = useIsMobile()

  return isMobile ? <MobileNav /> : <DesktopNav />
}

usePrefersReducedMotion

Accessibility: detect reduced motion preference.

import { usePrefersReducedMotion } from '@cloudillo/react'

function AnimatedComponent() {
  const prefersReducedMotion = usePrefersReducedMotion()

  return (
    <div className={prefersReducedMotion ? 'no-animation' : 'animated'}>
      ...
    </div>
  )
}

See Also

@cloudillo/types

Overview

TypeScript type definitions for Cloudillo with runtime validation using @symbion/runtype.

Installation

pnpm add @cloudillo/types

Core Types

Profile

User or community profile information.

import type { Profile } from '@cloudillo/types'

const profile: Profile = {
  idTag: 'alice@example.com',
  name: 'Alice Johnson',
  profilePic: '/file/b1~abc123'
}

Fields:

  • idTag: string - Unique identity (DNS-based)
  • name?: string - Display name
  • profilePic?: string - Profile picture URL

Action

Represents a social action or activity.

import type { Action } from '@cloudillo/types'

const action: Action = {
  actionId: 'act_123',
  type: 'POST',
  issuerTag: 'alice@example.com',
  content: {
    text: 'Hello, world!',
    title: 'My First Post'
  },
  createdAt: 1735000000,
  status: 'A'
}

Fields:

  • actionId: string - Unique action identifier
  • type: ActionType - Type of action (POST, CMNT, REACT, etc.)
  • issuerTag: string - Who created the action
  • content?: unknown - Action-specific content
  • createdAt: number - Unix timestamp (seconds)
  • status?: ActionStatus - P/A/D/C/N
  • subType?: string - Action subtype/category
  • parentId?: string - Parent action (for threads)
  • rootId?: string - Root action (for deep threads)
  • audienceTag?: string - Target audience
  • subject?: string - Subject/target (e.g., who to follow)
  • attachments?: string[] - File IDs
  • expiresAt?: number - Expiration timestamp

ActionType

Literal union of all action types.

import type { ActionType } from '@cloudillo/types'

const type: ActionType =
  | 'CONN'   // Connection request
  | 'FLLW'   // Follow user
  | 'POST'   // Create post
  | 'REPOST' // Repost/share
  | 'REACT'  // Reaction
  | 'CMNT'   // Comment
  | 'SHRE'   // Share resource
  | 'MSG'    // Message
  | 'FSHR'   // File share
Additional Action Types

ACK (Acknowledgment) and RSTAT (Reaction Statistics) exist as action variants in the tagged union (tBaseAction) but are not part of the ActionType literal type. They are used internally for specific action handling.

ActionStatus

Action status enumeration.

import type { ActionStatus } from '@cloudillo/types'

const status: ActionStatus =
  | 'P' // Pending (draft/unpublished)
  | 'A' // Active (default - published/finalized)
  | 'D' // Deleted (soft delete)
  | 'C' // Created (pending approval, e.g., connection requests)
  | 'N' // New (notification awaiting acknowledgment)

ProfileStatus

Profile status codes.

import type { ProfileStatus } from '@cloudillo/types'

const status: ProfileStatus =
  | 'A' // Active
  | 'T' // Trusted
  | 'B' // Blocked
  | 'M' // Muted
  | 'S' // Suspended

ProfileConnectionStatus

Connection status between profiles.

import type { ProfileConnectionStatus } from '@cloudillo/types'

// true = connected
// 'R' = request pending
// undefined = not connected

CommunityRole

Community role hierarchy.

import type { CommunityRole } from '@cloudillo/types'
import { ROLE_LEVELS } from '@cloudillo/types'

type CommunityRole =
  | 'public'      // Level 0 - Anyone
  | 'follower'    // Level 1 - Following the community
  | 'supporter'   // Level 2 - Supporter/subscriber
  | 'contributor' // Level 3 - Can create content
  | 'moderator'   // Level 4 - Can moderate
  | 'leader'      // Level 5 - Full admin access

// Use ROLE_LEVELS for permission checks
console.log(ROLE_LEVELS)
// { public: 0, follower: 1, supporter: 2, contributor: 3, moderator: 4, leader: 5 }

// Check if user has sufficient role
function hasPermission(userRole: CommunityRole, required: CommunityRole): boolean {
  return ROLE_LEVELS[userRole] >= ROLE_LEVELS[required]
}

NewAction

Data for creating a new action.

import type { NewAction } from '@cloudillo/types'

const newAction: NewAction = {
  type: 'POST',
  content: {
    text: 'Hello!',
    title: 'Greeting'
  },
  attachments: ['file_123']
}

Fields:

  • type: string - Action type
  • subType?: string - Action subtype
  • parentId?: string - Parent action ID
  • rootId?: string - Root action ID
  • audienceTag?: string - Target audience
  • content?: unknown - Action content
  • attachments?: string[] - File IDs
  • subject?: string - Subject/target
  • expiresAt?: number - Expiration time

ActionView

Extended action with resolved references and statistics.

import type { ActionView } from '@cloudillo/types'

const actionView: ActionView = {
  actionId: 'act_123',
  type: 'POST',
  issuerTag: 'alice@example.com',
  issuer: {
    idTag: 'alice@example.com',
    name: 'Alice Johnson',
    profilePic: '/file/b1~abc'
  },
  content: { text: 'Hello!' },
  createdAt: 1735000000,
  stat: {
    reactions: 5,
    comments: 3,
    ownReaction: 'LOVE'
  }
}

Additional Fields:

  • issuer: Profile - Resolved issuer profile
  • audience?: Profile - Resolved audience profile
  • stat?: ActionStat - Statistics

FileView

File metadata with owner information.

import type { FileView } from '@cloudillo/types'

const file: FileView = {
  fileId: 'f1~abc123',
  status: 'M',
  contentType: 'image/png',
  fileName: 'photo.png',
  fileTp: 'BLOB',
  createdAt: new Date('2025-01-01'),
  tags: ['vacation', 'beach'],
  owner: {
    idTag: 'alice@example.com',
    name: 'Alice'
  }
}

Fields:

  • fileId: string - File identifier
  • status: 'P' | 'M' - Pending or Metadata-ready
  • contentType: string - MIME type
  • fileName: string - Original filename
  • fileTp?: string - File type (CRDT/RTDB/BLOB)
  • createdAt: string | Date - Creation time
  • tags?: string[] - Tags
  • owner?: Profile - Owner profile
  • preset?: string - Image preset configuration

Runtime Validation

All types include runtime validators using @symbion/runtype:

import { ProfileValidator, ActionValidator } from '@cloudillo/types'

// Validate data at runtime
const data = await api.profiles.getOwn()

if (ProfileValidator.validate(data)) {
  // TypeScript knows data is a valid Profile
  console.log(data.idTag)
} else {
  console.error('Invalid profile data')
}

Type Guards

Use type guards to check types at runtime:

import { isAction, isProfile } from '@cloudillo/types'

function processData(data: unknown) {
  if (isAction(data)) {
    console.log('Action:', data.type)
  } else if (isProfile(data)) {
    console.log('Profile:', data.idTag)
  }
}

Enum Constants

Action Types

import { ACTION_TYPES } from '@cloudillo/types'

console.log(ACTION_TYPES)
// ['CONN', 'FLLW', 'POST', 'REPOST', 'REACT', 'CMNT', 'SHRE', 'MSG', 'FSHR']

// Use in UI
{ACTION_TYPES.map(type => (
  <option key={type} value={type}>{type}</option>
))}

Action Statuses

import { ACTION_STATUSES } from '@cloudillo/types'

console.log(ACTION_STATUSES)
// ['P', 'A', 'D', 'C', 'N']

Action Type Variants

Typed action structures for type-safe action creation.

User Relationships

import type { ConnectAction, FollowAction } from '@cloudillo/types'

// Connection request
const connect: ConnectAction = {
  type: 'CONN',
  subject: 'bob.cloudillo.net',      // Who to connect with
  content: 'Would love to connect!'  // Optional message
}

// Follow relationship
const follow: FollowAction = {
  type: 'FLLW',
  subject: 'news.cloudillo.net'      // Who to follow
}

Content Actions

import type { PostAction, CommentAction, ReactAction } from '@cloudillo/types'

// Create a post
const post: PostAction = {
  type: 'POST',
  subType: 'IMG',                    // Optional: IMG, VID, etc.
  content: 'Check out this photo!',
  attachments: ['fileId123'],        // Optional attachments
  audience: 'friends.cloudillo.net'  // Optional target audience
}

// Add a comment
const comment: CommentAction = {
  type: 'CMNT',
  parentId: 'actionId123',           // Parent action to comment on
  content: 'Great post!',
  attachments: []                    // Optional
}

// Add a reaction
const reaction: ReactAction = {
  type: 'REACT',
  parentId: 'actionId123',           // Action to react to
  content: 'LOVE'                    // Reaction type
}

Content Spreading

import type { AckAction, RepostAction, ShareAction } from '@cloudillo/types'

// Acknowledge content (accept to feed)
const ack: AckAction = {
  type: 'ACK',
  parentId: 'actionId123'            // Action to acknowledge
}

// Repost content
const repost: RepostAction = {
  type: 'REPOST',
  parentId: 'actionId123',           // Original action
  content: 'Adding my thoughts...'   // Optional comment
}

// Share directly to someone
const share: ShareAction = {
  type: 'SHRE',
  subject: 'actionId123',            // What to share
  audience: 'bob.cloudillo.net',     // Who to share with
  content: 'You might like this!'    // Optional message
}

Messages

import type { MessageAction } from '@cloudillo/types'

// Direct message
const dm: MessageAction = {
  type: 'MSG',
  subType: 'TEXT',
  content: 'Hello!',
  audience: 'bob.cloudillo.net'      // Recipient
}

// Group message (reply to conversation)
const groupMsg: MessageAction = {
  type: 'MSG',
  subType: 'TEXT',
  content: 'Thanks everyone!',
  parentId: 'conversationId123'      // Conversation ID
}

File Sharing

import type { FileShareAction } from '@cloudillo/types'

// Share a file with read access
const fileShare: FileShareAction = {
  type: 'FSHR',
  subType: 'READ',                   // 'READ' or 'WRITE'
  subject: 'fileId123',              // File to share
  audience: 'bob.cloudillo.net',     // Who to share with
  content: {
    fileName: 'document.pdf',
    contentType: 'application/pdf',
    fileTp: 'BLOB'
  }
}

Reaction Statistics

import type { ReactionStatAction } from '@cloudillo/types'

// Aggregated reaction stats (usually system-generated)
const stats: ReactionStatAction = {
  type: 'RSTAT',
  parentId: 'actionId123',
  content: {
    comment: 5,                      // Number of comments
    reactions: [10, 3, 1]            // Reaction counts by type
  }
}

ProfileInfo

Embedded profile information in actions.

import type { ProfileInfo } from '@cloudillo/types'

const info: ProfileInfo = {
  idTag: 'alice.cloudillo.net',
  name: 'Alice',                     // Optional
  profilePic: 'picId123',            // Optional
  type: 'person'                     // 'person' or 'community'
}

See Also

@cloudillo/canvas-tools

Overview

React components and hooks for interactive object manipulation in SVG canvas applications. Provides transform gizmos, rotation handles, pivot controls, and gradient pickers for building drawing and design tools.

Installation

pnpm add @cloudillo/canvas-tools

Peer Dependencies:

  • react >= 18
  • react-svg-canvas

Components

TransformGizmo

Complete transform control for SVG objects with rotation, scaling, and positioning.

import { TransformGizmo } from '@cloudillo/canvas-tools'

function CanvasEditor() {
  const [bounds, setBounds] = useState({ x: 100, y: 100, width: 200, height: 150 })
  const [rotation, setRotation] = useState(0)

  return (
    <svg width={800} height={600}>
      <TransformGizmo
        bounds={bounds}
        rotation={rotation}
        onBoundsChange={setBounds}
        onRotationChange={setRotation}
        showRotationHandle
        showPivotHandle
      />
    </svg>
  )
}

Props (TransformGizmoProps):

  • bounds: Bounds - Object position and size { x, y, width, height }
  • rotation?: number - Rotation angle in degrees
  • pivot?: Point - Pivot point { x, y }
  • onBoundsChange?: (bounds: Bounds) => void - Bounds change callback
  • onRotationChange?: (angle: number) => void - Rotation change callback
  • onPivotChange?: (pivot: Point) => void - Pivot change callback
  • showRotationHandle?: boolean - Show rotation arc handle
  • showPivotHandle?: boolean - Show pivot point control

RotationHandle

Circular arc handle for rotating objects.

import { RotationHandle } from '@cloudillo/canvas-tools'

<RotationHandle
  center={{ x: 200, y: 200 }}
  radius={80}
  currentAngle={rotation}
  onRotate={(angle) => setRotation(angle)}
  snapAngles={[0, 45, 90, 135, 180, 225, 270, 315]}
/>

Props (RotationHandleProps):

  • center: Point - Center point of rotation
  • radius: number - Arc radius
  • currentAngle: number - Current rotation angle
  • onRotate: (angle: number) => void - Rotation callback
  • snapAngles?: number[] - Angles to snap to (default: 45-degree increments)
  • snapZoneRatio?: number - Snap zone size ratio

PivotHandle

Draggable handle for setting the pivot/rotation center point.

import { PivotHandle } from '@cloudillo/canvas-tools'

<PivotHandle
  position={{ x: 200, y: 200 }}
  bounds={{ x: 100, y: 100, width: 200, height: 200 }}
  onPivotChange={(point) => setPivot(point)}
  snapToCenter
  snapThreshold={10}
/>

Props (PivotHandleProps):

  • position: Point - Current pivot position
  • bounds: Bounds - Object bounds for snapping
  • onPivotChange: (point: Point) => void - Position change callback
  • snapToCenter?: boolean - Snap to center when close
  • snapThreshold?: number - Snap distance threshold

GradientPicker

Complete gradient editor with color stops, angle control, and presets.

import { GradientPicker } from '@cloudillo/canvas-tools'
import type { Gradient } from '@cloudillo/canvas-tools'

function GradientEditor() {
  const [gradient, setGradient] = useState<Gradient>({
    type: 'linear',
    angle: 90,
    stops: [
      { offset: 0, color: '#ff0000' },
      { offset: 1, color: '#0000ff' }
    ]
  })

  return (
    <GradientPicker
      value={gradient}
      onChange={setGradient}
      showPresets
      showAngleControl
    />
  )
}

Props (GradientPickerProps):

  • value: Gradient - Current gradient value
  • onChange: (gradient: Gradient) => void - Change callback
  • showPresets?: boolean - Show gradient preset grid
  • showAngleControl?: boolean - Show angle rotation control
  • showPositionControl?: boolean - Show radial gradient position control

GradientBar

Horizontal bar for editing gradient color stops.

import { GradientBar } from '@cloudillo/canvas-tools'

<GradientBar
  stops={gradient.stops}
  onStopsChange={(stops) => setGradient({ ...gradient, stops })}
  selectedStop={selectedIndex}
  onSelectStop={setSelectedIndex}
/>

GradientPresetGrid

Grid of predefined gradient presets.

import { GradientPresetGrid, GRADIENT_PRESETS } from '@cloudillo/canvas-tools'

<GradientPresetGrid
  presets={GRADIENT_PRESETS}
  onSelect={(preset) => setGradient(expandGradient(preset.gradient))}
  category="warm"
/>

AngleControl

Circular control for setting gradient angle.

import { AngleControl, DEFAULT_ANGLE_PRESETS } from '@cloudillo/canvas-tools'

<AngleControl
  value={angle}
  onChange={setAngle}
  presets={DEFAULT_ANGLE_PRESETS}
/>

PositionControl

XY position control for radial gradient centers.

import { PositionControl } from '@cloudillo/canvas-tools'

<PositionControl
  value={{ x: 0.5, y: 0.5 }}
  onChange={setPosition}
/>

Hooks

useTransformGizmo

Hook for managing transform gizmo state and interactions.

import { useTransformGizmo } from '@cloudillo/canvas-tools'

function CanvasObject({ object, onUpdate }) {
  const {
    state,
    handlers,
    isDragging,
    isRotating,
    isResizing
  } = useTransformGizmo({
    bounds: object.bounds,
    rotation: object.rotation,
    pivot: object.pivot,
    onBoundsChange: (bounds) => onUpdate({ ...object, bounds }),
    onRotationChange: (rotation) => onUpdate({ ...object, rotation }),
    onPivotChange: (pivot) => onUpdate({ ...object, pivot }),
    snapAngles: [0, 45, 90, 135, 180, 225, 270, 315],
    maintainAspectRatio: true
  })

  return (
    <g {...handlers}>
      {/* Object rendering */}
    </g>
  )
}

Options (TransformGizmoOptions):

  • bounds: Bounds - Initial bounds
  • rotation?: number - Initial rotation
  • pivot?: Point - Initial pivot
  • onBoundsChange?: (bounds: Bounds) => void
  • onRotationChange?: (angle: number) => void
  • onPivotChange?: (pivot: Point) => void
  • snapAngles?: number[] - Rotation snap angles
  • snapZoneRatio?: number - Snap sensitivity
  • maintainAspectRatio?: boolean - Lock aspect during resize
  • minWidth?: number - Minimum width constraint
  • minHeight?: number - Minimum height constraint

Returns (UseTransformGizmoReturn):

  • state: TransformGizmoState - Current transform state
  • handlers: TransformGizmoHandlers - Event handlers
  • isDragging: boolean - Move operation active
  • isRotating: boolean - Rotation operation active
  • isResizing: boolean - Resize operation active

Gradient Utilities

Creating Gradients

import {
  DEFAULT_LINEAR_GRADIENT,
  DEFAULT_RADIAL_GRADIENT,
  expandGradient,
  compactGradient
} from '@cloudillo/canvas-tools'

// Start with defaults
const linear = { ...DEFAULT_LINEAR_GRADIENT }
const radial = { ...DEFAULT_RADIAL_GRADIENT }

// Expand compact notation to full gradient
const full = expandGradient({ t: 'l', a: 90, s: [[0, '#f00'], [1, '#00f']] })

// Compact for storage
const compact = compactGradient(full)

Manipulating Stops

import {
  addStop,
  removeStop,
  updateStop,
  sortStops,
  reverseStops
} from '@cloudillo/canvas-tools'

// Add a stop at 50%
const newStops = addStop(gradient.stops, 0.5, '#00ff00')

// Remove stop at index 1
const filtered = removeStop(gradient.stops, 1)

// Update stop color
const updated = updateStop(gradient.stops, 1, { color: '#ff00ff' })

// Sort by offset
const sorted = sortStops(stops)

// Reverse direction
const reversed = reverseStops(stops)

Converting to CSS/SVG

import {
  gradientToCSS,
  createLinearGradientDef,
  createRadialGradientDef
} from '@cloudillo/canvas-tools'

// CSS background value
const css = gradientToCSS(gradient)
// "linear-gradient(90deg, #ff0000 0%, #0000ff 100%)"

// SVG gradient definition
const svgLinear = createLinearGradientDef('myGradient', gradient)
const svgRadial = createRadialGradientDef('myGradient', gradient)

Color Utilities

import {
  interpolateColor,
  getColorAtPosition
} from '@cloudillo/canvas-tools'

// Blend two colors
const mixed = interpolateColor('#ff0000', '#0000ff', 0.5)
// "#800080"

// Get color at position in gradient
const colorAt25 = getColorAtPosition(gradient.stops, 0.25)

Geometry Utilities

Coordinate Transformation

import {
  getCanvasCoordinates,
  getCanvasCoordinatesWithElement,
  getSvgElement
} from '@cloudillo/canvas-tools'

function handleClick(event: MouseEvent) {
  const svg = getSvgElement(event.target as Element)
  const point = getCanvasCoordinates(event, svg)
  console.log('Canvas position:', point.x, point.y)
}

Rotation Matrix

Pre-calculated trigonometry for performance-critical operations.

import {
  createRotationMatrix,
  rotatePointWithMatrix,
  unrotatePointWithMatrix,
  rotateDeltaWithMatrix,
  unrotateDeltaWithMatrix
} from '@cloudillo/canvas-tools'

// Create matrix once
const matrix = createRotationMatrix(45) // 45 degrees

// Rotate points efficiently
const rotated = rotatePointWithMatrix({ x: 100, y: 0 }, matrix, center)
const original = unrotatePointWithMatrix(rotated, matrix, center)

// Rotate deltas (movement vectors)
const rotatedDelta = rotateDeltaWithMatrix({ x: 10, y: 0 }, matrix)

View Coordinates

import {
  canvasToView,
  viewToCanvas,
  isPointInView,
  boundsIntersectsView
} from '@cloudillo/canvas-tools'

// Convert between canvas and view coordinates
const viewPoint = canvasToView(canvasPoint, viewTransform)
const canvasPoint = viewToCanvas(viewPoint, viewTransform)

// Visibility checks
const visible = isPointInView(point, viewport)
const intersects = boundsIntersectsView(objectBounds, viewport)

Resize Calculations

import {
  initResizeState,
  calculateResizeBounds,
  getAnchorForHandle,
  getRotatedAnchorPosition
} from '@cloudillo/canvas-tools'

// Initialize resize operation
const resizeState = initResizeState(
  bounds,
  rotation,
  'se', // corner handle
  startMousePosition
)

// Calculate new bounds during drag
const newBounds = calculateResizeBounds(
  resizeState,
  currentMousePosition,
  { maintainAspectRatio: true }
)

Types

Core Types

import type {
  Point,
  Bounds,
  ResizeHandle,
  RotationState,
  PivotState,
  RotatedObjectBounds,
  TransformedObject,
  RotationMatrix,
  ResizeState
} from '@cloudillo/canvas-tools'

type Point = { x: number; y: number }

type Bounds = {
  x: number
  y: number
  width: number
  height: number
}

type ResizeHandle = 'n' | 's' | 'e' | 'w' | 'ne' | 'nw' | 'se' | 'sw'

Gradient Types

import type {
  GradientType,
  GradientStop,
  Gradient,
  CompactGradient,
  GradientPreset,
  GradientPresetCategory
} from '@cloudillo/canvas-tools'

type GradientType = 'linear' | 'radial'

type GradientStop = {
  offset: number  // 0-1
  color: string   // hex color
}

type Gradient = {
  type: GradientType
  angle?: number        // linear: degrees
  cx?: number           // radial: center x (0-1)
  cy?: number           // radial: center y (0-1)
  stops: GradientStop[]
}

type CompactGradient = {
  t: 'l' | 'r'          // type
  a?: number            // angle
  cx?: number
  cy?: number
  s: [number, string][] // stops as tuples
}

Constants

Rotation Defaults

import {
  DEFAULT_SNAP_ANGLES,
  DEFAULT_SNAP_ZONE_RATIO,
  DEFAULT_PIVOT_SNAP_POINTS,
  DEFAULT_PIVOT_SNAP_THRESHOLD
} from '@cloudillo/canvas-tools'

// [0, 45, 90, 135, 180, 225, 270, 315]
console.log(DEFAULT_SNAP_ANGLES)

// 0.1 (10% of arc)
console.log(DEFAULT_SNAP_ZONE_RATIO)

Arc Sizing

import {
  ARC_RADIUS_MIN_VIEWPORT_RATIO,
  ARC_RADIUS_MAX_VIEWPORT_RATIO,
  DEFAULT_ARC_PADDING,
  calculateArcRadius
} from '@cloudillo/canvas-tools'

const radius = calculateArcRadius({
  objectBounds: bounds,
  viewportSize: { width: 800, height: 600 },
  padding: DEFAULT_ARC_PADDING
})

Angle Presets

import { DEFAULT_ANGLE_PRESETS } from '@cloudillo/canvas-tools'

// [0, 45, 90, 135, 180, 225, 270, 315]
console.log(DEFAULT_ANGLE_PRESETS)

Gradient Presets

import {
  GRADIENT_PRESETS,
  getPresetsByCategory,
  getPresetById,
  getCategories
} from '@cloudillo/canvas-tools'

// Get all categories
const categories = getCategories()
// ['warm', 'cool', 'vibrant', 'subtle', 'monochrome']

// Get presets in a category
const warmGradients = getPresetsByCategory('warm')

// Get specific preset
const sunset = getPresetById('sunset')

See Also

@cloudillo/fonts

Overview

The @cloudillo/fonts library provides a curated collection of Google Fonts metadata and pairing suggestions for Cloudillo applications. It enables font selection UIs, typography systems, and design tools.

Key Features:

  • Curated font metadata for 22 Google Fonts
  • Pre-defined font pairings (heading + body combinations)
  • Helper functions for filtering and lookup
  • Full TypeScript support

Installation

pnpm add @cloudillo/fonts

Font Metadata API

FONTS Constant

The FONTS array contains metadata for all available fonts.

import { FONTS } from '@cloudillo/fonts'

// List all fonts
FONTS.forEach(font => {
  console.log(font.displayName, font.category, font.roles)
})

Each font entry includes:

  • family - CSS font-family value (e.g., 'Roboto')
  • displayName - Human-readable name
  • category - 'sans-serif' | 'serif' | 'display' | 'monospace'
  • roles - Suitable uses: 'heading' | 'body' | 'display' | 'mono'
  • weights - Available font weights
  • hasItalic - Whether italic variants exist
  • license - 'OFL' or 'Apache-2.0'
  • directory - Font directory name

getFontByFamily

Look up a font by its family name.

import { getFontByFamily } from '@cloudillo/fonts'

const roboto = getFontByFamily('Roboto')
// { family: 'Roboto', category: 'sans-serif', roles: ['body', 'heading'], ... }

const unknown = getFontByFamily('Unknown Font')
// undefined

getFontsByCategory

Filter fonts by category.

import { getFontsByCategory } from '@cloudillo/fonts'

const serifFonts = getFontsByCategory('serif')
// Returns: Playfair Display, Merriweather, Lora, Crimson Pro, Source Serif 4, DM Serif Display

const displayFonts = getFontsByCategory('display')
// Returns: Oswald, Bebas Neue, Abril Fatface, Permanent Marker

Available categories:

  • sans-serif - Clean, modern fonts (Roboto, Open Sans, Montserrat, etc.)
  • serif - Traditional fonts with serifs (Playfair Display, Merriweather, etc.)
  • display - Decorative fonts for headlines (Oswald, Bebas Neue, etc.)
  • monospace - Fixed-width fonts (JetBrains Mono)

getFontsByRole

Filter fonts by intended use.

import { getFontsByRole } from '@cloudillo/fonts'

const headingFonts = getFontsByRole('heading')
// Fonts suitable for headings: Roboto, Montserrat, Poppins, Playfair Display, etc.

const bodyFonts = getFontsByRole('body')
// Fonts suitable for body text: Roboto, Open Sans, Lato, Inter, etc.

Available roles:

  • heading - Suitable for titles and headings
  • body - Suitable for body text
  • display - Decorative, for large display text
  • mono - Monospace, for code

Font Pairings API

The library includes curated heading + body font combinations that work well together.

FONT_PAIRINGS Constant

import { FONT_PAIRINGS } from '@cloudillo/fonts'

FONT_PAIRINGS.forEach(pairing => {
  console.log(`${pairing.name}: ${pairing.heading} + ${pairing.body}`)
})

Available pairings:

ID Name Heading Body Description
modern-professional Modern Professional Oswald Roboto Business presentations
elegant-editorial Elegant Editorial Playfair Display Source Sans 3 Articles and long-form
clean-modern Clean Modern Montserrat Open Sans Tech and startups
readable-classic Readable Classic Merriweather Lato Blogs and docs
contemporary-tech Contemporary Tech Poppins Inter Digital products
literary-warm Literary Warm Lora Nunito Sans Classic with friendly body
light-minimalist Light Minimalist Raleway Work Sans Minimal designs
academic-formal Academic Formal Crimson Pro DM Sans Scholarly content
bold-impact Bold Impact Bebas Neue Source Serif 4 Impactful headlines
geometric-harmony Geometric Harmony DM Serif Display DM Sans Cohesive DM family

getPairingById

Look up a specific pairing.

import { getPairingById } from '@cloudillo/fonts'

const pairing = getPairingById('modern-professional')
// { id: 'modern-professional', name: 'Modern Professional', heading: 'Oswald', body: 'Roboto', ... }

getPairingsForFont

Find pairings that use a specific font.

import { getPairingsForFont } from '@cloudillo/fonts'

const robotoPairings = getPairingsForFont('Roboto')
// Returns pairings where Roboto is used as heading or body

getSuggestedBodyFonts

Get body font suggestions for a heading font.

import { getSuggestedBodyFonts } from '@cloudillo/fonts'

const bodyOptions = getSuggestedBodyFonts('Oswald')
// ['Roboto']

const playfairBodies = getSuggestedBodyFonts('Playfair Display')
// ['Source Sans 3']

getSuggestedHeadingFonts

Get heading font suggestions for a body font.

import { getSuggestedHeadingFonts } from '@cloudillo/fonts'

const headingOptions = getSuggestedHeadingFonts('Inter')
// ['Poppins']

const latoHeadings = getSuggestedHeadingFonts('Lato')
// ['Merriweather']

TypeScript Types

import type {
  FontCategory,
  FontRole,
  FontWeight,
  FontMetadata,
  FontPairing
} from '@cloudillo/fonts'

FontCategory

type FontCategory = 'sans-serif' | 'serif' | 'display' | 'monospace'

FontRole

type FontRole = 'heading' | 'body' | 'display' | 'mono'

FontWeight

interface FontWeight {
  value: number    // CSS font-weight value (400, 700, etc.)
  label: string    // Display name ('Regular', 'Bold', etc.)
  italic?: boolean // Whether this is an italic variant
}

FontMetadata

interface FontMetadata {
  family: string           // CSS font-family value
  displayName: string      // Human-readable name
  category: FontCategory   // Font category
  roles: FontRole[]        // Suitable roles
  weights: FontWeight[]    // Available weights
  hasItalic: boolean       // Has italic variants
  license: 'OFL' | 'Apache-2.0'
  directory: string        // Font directory name
}

FontPairing

interface FontPairing {
  id: string          // Unique identifier
  name: string        // Human-readable name
  heading: string     // Heading font family
  body: string        // Body font family
  description: string // Pairing description
}

Integration with FontPicker

The @cloudillo/fonts library works with the FontPicker component from @cloudillo/react.

import { FontPicker } from '@cloudillo/react'
import { FONTS, FONT_PAIRINGS, getSuggestedBodyFonts } from '@cloudillo/fonts'

function TypographySettings() {
  const [headingFont, setHeadingFont] = useState('Montserrat')
  const [bodyFont, setBodyFont] = useState('Open Sans')

  // Get suggested body fonts when heading changes
  const suggestedBodies = getSuggestedBodyFonts(headingFont)

  return (
    <div>
      <FontPicker
        label="Heading Font"
        value={headingFont}
        onChange={setHeadingFont}
      />
      <FontPicker
        label="Body Font"
        value={bodyFont}
        onChange={setBodyFont}
      />
      {suggestedBodies.length > 0 && (
        <p>Suggested body fonts: {suggestedBodies.join(', ')}</p>
      )}
    </div>
  )
}

Available Fonts

Sans-Serif

Font Roles Weights
Roboto body, heading 400, 700
Open Sans body 400, 700
Montserrat heading, body 400, 700
Lato body 400, 700
Poppins heading, body 400, 700
Inter body 400, 700
Nunito Sans body 400, 700
Work Sans body 400, 700
Raleway heading 400, 700
DM Sans body 400, 700
Source Sans 3 body 400, 700

Serif

Font Roles Weights
Playfair Display heading, display 400, 700
Merriweather heading, body 400, 700
Lora heading, body 400, 700
Crimson Pro heading, body 400, 700
Source Serif 4 body 400, 700
DM Serif Display heading, display 400

Display

Font Roles Weights
Oswald heading, display 400, 700
Bebas Neue heading, display 400
Abril Fatface display 400
Permanent Marker display 400

Monospace

Font Roles Weights
JetBrains Mono mono 400, 700

See Also

Common patterns

Practical integration patterns for building Cloudillo applications.

Infinite Scroll with Actions

Load actions (posts, comments) with cursor-based pagination.

import { useApi, useInfiniteScroll, LoadMoreTrigger } from '@cloudillo/react'

function Feed() {
  const { api } = useApi()

  const { items, isLoading, sentinelRef } = useInfiniteScroll({
    fetchPage: async (cursor, limit) => {
      const result = await api.actions.listPaginated({
        type: 'POST',
        cursor,
        limit
      })
      return {
        items: result.data,
        nextCursor: result.cursorPagination?.nextCursor ?? null,
        hasMore: result.cursorPagination?.hasMore ?? false
      }
    },
    pageSize: 20
  })

  return (
    <div className="feed">
      {items.map(action => (
        <PostCard key={action.actionId} action={action} />
      ))}
      <LoadMoreTrigger ref={sentinelRef} isLoading={isLoading} />
    </div>
  )
}

Collaborative Document Editor

Set up a collaborative editing session with CRDT sync.

import { useCloudilloEditor } from '@cloudillo/react'
import { useEffect, useRef } from 'react'
import Quill from 'quill'
import { QuillBinding } from 'y-quill'

function CollaborativeEditor() {
  const { yDoc, provider, synced, ownerTag, fileId } = useCloudilloEditor('quillo')
  const editorRef = useRef<HTMLDivElement>(null)
  const quillRef = useRef<Quill | null>(null)

  useEffect(() => {
    if (!synced || !editorRef.current) return

    // Initialize Quill editor
    const quill = new Quill(editorRef.current, {
      theme: 'snow',
      modules: { toolbar: true }
    })
    quillRef.current = quill

    // Bind to Yjs document
    const yText = yDoc.getText('content')
    const binding = new QuillBinding(yText, quill, provider.awareness)

    return () => {
      binding.destroy()
      quill.disable()
    }
  }, [synced, yDoc, provider])

  if (!synced) {
    return <LoadingSpinner />
  }

  return (
    <div>
      <div className="editor-header">
        <span>Editing: {fileId}</span>
        <span>Owner: {ownerTag}</span>
      </div>
      <div ref={editorRef} />
    </div>
  )
}

Toast Notifications

Show feedback for user actions.

import { useToast, ToastContainer, Button } from '@cloudillo/react'

// In your app root
function App() {
  return (
    <>
      <ToastContainer position="bottom-right" />
      <MainContent />
    </>
  )
}

// In any component
function SaveButton({ data }) {
  const { api } = useApi()
  const toast = useToast()

  const handleSave = async () => {
    try {
      await api.files.update(data.fileId, data)
      toast.success('Changes saved successfully')
    } catch (err) {
      if (err.code === 'E-AUTH-UNAUTH') {
        toast.error('Session expired. Please log in again.')
      } else {
        toast.error('Failed to save changes')
      }
    }
  }

  return <Button onClick={handleSave}>Save</Button>
}

File Upload with Progress

Upload files with variant generation.

import { useApi } from '@cloudillo/react'
import { useState } from 'react'

function ImageUpload({ onUploaded }) {
  const { api } = useApi()
  const [uploading, setUploading] = useState(false)

  const handleFileSelect = async (e: React.ChangeEvent<HTMLInputElement>) => {
    const file = e.target.files?.[0]
    if (!file) return

    setUploading(true)
    try {
      // Upload with preset for automatic variant generation
      const result = await api.files.uploadBlob(
        'gallery',      // Preset: generates thumbnail + SD variants
        file.name,
        file,
        file.type
      )

      onUploaded({
        fileId: result.fileId,
        thumbnailId: result.variantId  // Thumbnail variant ID
      })
    } catch (err) {
      console.error('Upload failed:', err)
    } finally {
      setUploading(false)
    }
  }

  return (
    <div>
      <input
        type="file"
        accept="image/*"
        onChange={handleFileSelect}
        disabled={uploading}
      />
      {uploading && <LoadingSpinner />}
    </div>
  )
}

Real-Time Presence

Show who’s currently viewing/editing.

import { useCloudilloEditor, AvatarGroup, Avatar } from '@cloudillo/react'
import { useEffect, useState } from 'react'

function PresenceIndicator() {
  const { provider } = useCloudilloEditor('my-app')
  const [users, setUsers] = useState<Map<number, any>>(new Map())

  useEffect(() => {
    if (!provider) return

    const updateUsers = () => {
      setUsers(new Map(provider.awareness.getStates()))
    }

    provider.awareness.on('change', updateUsers)
    updateUsers()

    return () => {
      provider.awareness.off('change', updateUsers)
    }
  }, [provider])

  const otherUsers = Array.from(users.entries())
    .filter(([clientId]) => clientId !== provider?.awareness.clientID)
    .map(([, state]) => state.user)

  return (
    <AvatarGroup max={5}>
      {otherUsers.map((user, i) => (
        <Avatar key={i} name={user.name} src={user.profilePic} size="sm" />
      ))}
    </AvatarGroup>
  )
}

Role-Based UI

Show different UI based on user roles.

import { useAuth } from '@cloudillo/react'
import { ROLE_LEVELS, CommunityRole } from '@cloudillo/types'

function hasRole(userRoles: string[] | undefined, requiredRole: CommunityRole): boolean {
  if (!userRoles?.length) return false
  const userLevel = Math.max(...userRoles.map(r => ROLE_LEVELS[r as CommunityRole] ?? 0))
  return userLevel >= ROLE_LEVELS[requiredRole]
}

function CommunitySettings() {
  const [auth] = useAuth()

  const canModerate = hasRole(auth?.roles, 'moderator')
  const canAdmin = hasRole(auth?.roles, 'leader')

  return (
    <div>
      <h2>Community Settings</h2>

      {/* Everyone can see */}
      <GeneralSettings />

      {/* Moderators and above */}
      {canModerate && <ModerationPanel />}

      {/* Leaders only */}
      {canAdmin && <AdminPanel />}
    </div>
  )
}

Optimistic Updates

Update UI immediately while syncing in background.

import { useApi, useToast } from '@cloudillo/react'
import { useState } from 'react'

function LikeButton({ actionId, initialLiked, initialCount }) {
  const { api } = useApi()
  const toast = useToast()
  const [liked, setLiked] = useState(initialLiked)
  const [count, setCount] = useState(initialCount)

  const handleLike = async () => {
    // Optimistic update
    const wasLiked = liked
    const oldCount = count
    setLiked(!liked)
    setCount(liked ? count - 1 : count + 1)

    try {
      await api.actions.addReaction(actionId, { type: 'LOVE' })
    } catch (err) {
      // Rollback on error
      setLiked(wasLiked)
      setCount(oldCount)
      toast.error('Failed to update reaction')
    }
  }

  return (
    <Button variant={liked ? 'primary' : 'ghost'} onClick={handleLike}>
      {liked ? '❀️' : '🀍'} {count}
    </Button>
  )
}

Search with debouncing to reduce API calls.

import { useApi, Input } from '@cloudillo/react'
import { useState, useEffect, useRef } from 'react'

function SearchBox({ onResults }) {
  const { api } = useApi()
  const [query, setQuery] = useState('')
  const [loading, setLoading] = useState(false)
  const debounceRef = useRef<NodeJS.Timeout>()

  useEffect(() => {
    if (debounceRef.current) {
      clearTimeout(debounceRef.current)
    }

    if (!query.trim()) {
      onResults([])
      return
    }

    debounceRef.current = setTimeout(async () => {
      setLoading(true)
      try {
        const results = await api.profiles.list({ q: query, limit: 10 })
        onResults(results)
      } finally {
        setLoading(false)
      }
    }, 300)

    return () => {
      if (debounceRef.current) {
        clearTimeout(debounceRef.current)
      }
    }
  }, [query, api, onResults])

  return (
    <Input
      value={query}
      onChange={(e) => setQuery(e.target.value)}
      placeholder="Search profiles..."
      icon={loading ? <LoadingSpinner size="sm" /> : <SearchIcon />}
    />
  )
}

Connection Request Flow

Handle bi-directional connection requests.

import { useApi, useToast, Button, ProfileCard } from '@cloudillo/react'

function ConnectionButton({ profile }) {
  const { api } = useApi()
  const toast = useToast()

  const handleConnect = async () => {
    try {
      await api.actions.create({
        type: 'CONN',
        subject: profile.idTag,
        content: 'Would love to connect!'
      })
      toast.success('Connection request sent')
    } catch (err) {
      toast.error('Failed to send request')
    }
  }

  const handleAccept = async (actionId: string) => {
    try {
      await api.actions.accept(actionId)
      toast.success('Connection accepted')
    } catch (err) {
      toast.error('Failed to accept connection')
    }
  }

  // Render based on connection state
  if (profile.connected === true) {
    return <Badge variant="success">Connected</Badge>
  }

  if (profile.connected === 'R') {
    return <Badge variant="info">Request Pending</Badge>
  }

  return <Button onClick={handleConnect}>Connect</Button>
}

Error Boundary Pattern

Handle errors gracefully in components.

import { Component, ReactNode } from 'react'
import { EmptyState, Button } from '@cloudillo/react'

interface Props {
  children: ReactNode
  fallback?: ReactNode
}

interface State {
  hasError: boolean
  error?: Error
}

class ErrorBoundary extends Component<Props, State> {
  state: State = { hasError: false }

  static getDerivedStateFromError(error: Error): State {
    return { hasError: true, error }
  }

  handleRetry = () => {
    this.setState({ hasError: false, error: undefined })
  }

  render() {
    if (this.state.hasError) {
      return this.props.fallback || (
        <EmptyState
          icon={<AlertIcon />}
          title="Something went wrong"
          description={this.state.error?.message}
          action={<Button onClick={this.handleRetry}>Try Again</Button>}
        />
      )
    }

    return this.props.children
  }
}

// Usage
function App() {
  return (
    <ErrorBoundary>
      <MyComponent />
    </ErrorBoundary>
  )
}

See Also

REST API

REST API Reference

Cloudillo provides a comprehensive REST API for building applications. All endpoints return JSON and use standard HTTP methods.

Base URL

https://your-cloudillo-server.com

For local development:

http://localhost:3000

Authentication

Most endpoints require authentication via JWT tokens in the Authorization header:

Authorization: Bearer eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9...

See Authentication for details on obtaining and managing tokens.

Response Format

All successful responses follow this format:

{
  "data": <payload>,
  "time": "2025-01-01T12:00:00Z",
  "reqId": "req_abc123"
}

For list endpoints, cursor-based pagination is recommended:

{
  "data": [...],
  "cursorPagination": {
    "nextCursor": "eyJzIjoiY3JlYXRlZCIsInYiOjE3MzUwMDAwMDAsImlkIjoiYTF-YWJjMTIzIn0",
    "hasMore": true
  },
  "time": "2025-01-01T12:00:00Z"
}

Legacy offset-based pagination (deprecated):

{
  "data": [...],
  "pagination": {
    "total": 150,
    "offset": 0,
    "limit": 20
  },
  "time": "2025-01-01T12:00:00Z"
}

Error Format

Errors return this structure:

{
  "error": {
    "code": "E-AUTH-UNAUTH",
    "message": "Unauthorized access",
    "details": {...}
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

See Error Handling for all error codes.

Common Query Parameters

Many list endpoints support these parameters:

  • limit - Maximum number of results (default: 20)
  • cursor - Opaque cursor for pagination (from previous response)
  • sort - Sort field (e.g., created, modified, name)
  • sortDir - Sort direction (asc or desc)

Endpoint Categories

Authentication

User authentication and token management.

  • POST /api/auth/login - Login and get token
  • POST /api/auth/logout - Logout
  • POST /api/auth/password - Change password
  • GET /api/auth/login-token - Refresh token
  • GET /api/auth/access-token - Get scoped token
  • GET /api/auth/proxy-token - Get federation token
  • GET /api/me - Get tenant profile (public)
  • GET /.well-known/cloudillo/id-tag - Resolve identity

Profiles

User and community profiles.

  • POST /api/profiles/register - Register new user
  • POST /api/profiles/verify - Verify identity availability
  • GET /api/me - Get own profile
  • PATCH /api/me - Update own profile
  • PUT /api/me/image - Upload profile image
  • PUT /api/me/cover - Upload cover image
  • GET /api/profiles - List profiles
  • GET /api/profiles/:idTag - Get specific profile
  • PATCH /api/profiles/:idTag - Update relationship
  • PATCH /api/admin/profiles/:idTag - Admin update profile

Actions

Social features: posts, comments, reactions, connections.

  • GET /api/actions - List actions
  • POST /api/actions - Create action
  • GET /api/actions/:actionId - Get action
  • DELETE /api/actions/:actionId - Delete action
  • POST /api/actions/:actionId/accept - Accept action
  • POST /api/actions/:actionId/reject - Reject action
  • POST /api/actions/:actionId/stat - Update statistics
  • POST /api/actions/:actionId/reaction - Add reaction
  • POST /api/inbox - Federation inbox (async)
  • POST /api/inbox/sync - Federation inbox (sync)

Files

File upload, download, and management.

  • GET /api/files - List files
  • POST /api/files - Create file metadata (CRDT/RTDB)
  • POST /api/files/{preset}/{file_name} - Upload file (BLOB)
  • GET /api/files/:fileId - Download file
  • GET /api/files/:fileId/descriptor - Get file info
  • PATCH /api/files/:fileId - Update file
  • DELETE /api/files/:fileId - Delete file
  • PUT /api/files/:fileId/tag/:tag - Add tag
  • DELETE /api/files/:fileId/tag/:tag - Remove tag
  • GET /api/files/variant/:variantId - Get variant

Settings

User preferences and configuration.

  • GET /api/settings - List all settings
  • GET /api/settings/:name - Get setting
  • PUT /api/settings/:name - Update setting

References

Bookmarks and shortcuts.

  • GET /api/refs - List references
  • POST /api/refs - Create reference
  • GET /api/refs/:refId - Get reference
  • DELETE /api/refs/:refId - Delete reference

Tags

File and content tagging.

  • GET /api/tags - List tags
  • PUT /api/files/:fileId/tag/:tag - Add tag
  • DELETE /api/files/:fileId/tag/:tag - Remove tag

Collections

Favorites, bookmarks, and pins.

  • GET /api/collections/:collType - List collection items
  • POST /api/collections/:collType/:itemId - Add to collection
  • DELETE /api/collections/:collType/:itemId - Remove from collection

Trash

Trash management.

  • GET /api/files?parentId=__trash__ - List trashed files
  • POST /api/files/:fileId/restore - Restore from trash
  • DELETE /api/files/:fileId?permanent=true - Permanently delete
  • DELETE /api/trash - Empty trash

Communities

Community creation and management.

  • PUT /api/profiles/:idTag - Create community
  • POST /api/profiles/verify - Verify availability

Admin

System administration (requires admin role).

  • GET /api/admin/tenants - List tenants
  • POST /api/admin/tenants/:idTag/password-reset - Send password reset
  • POST /api/admin/email/test - Test SMTP
  • PATCH /api/admin/profiles/:idTag - Admin profile update

IDP Management

Identity provider administration.

  • GET /api/idp/identities - List managed identities
  • POST /api/idp/identities - Create identity
  • GET /api/idp/identities/:idTag - Get identity
  • PATCH /api/idp/identities/:idTag - Update identity
  • PUT /api/idp/identities/:idTag/address - Update identity address (DNS)
  • DELETE /api/idp/identities/:idTag - Delete identity
  • GET /api/idp/api-keys - List API keys
  • POST /api/idp/api-keys - Create API key
  • DELETE /api/idp/api-keys/:keyId - Delete API key

Rate Limiting

API requests are rate-limited per tenant:

  • Default: 1000 requests per minute
  • Authenticated: 5000 requests per minute
  • Admin: Unlimited

Rate limit headers:

X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4950
X-RateLimit-Reset: 1735000000

CORS

CORS is enabled for all origins in development mode. In production, configure allowed origins in the server settings.

Timestamps

All response timestamps are in ISO 8601 format:

{
  "createdAt": "2025-01-01T12:00:00Z"
}

Query parameter timestamps accept both ISO 8601 strings and Unix seconds:

GET /api/actions?createdAfter=2025-01-01T00:00:00Z
GET /api/actions?createdAfter=1735689600

Content Types

Request Content-Type

Most endpoints accept:

Content-Type: application/json

File uploads use:

Content-Type: multipart/form-data

Response Content-Type

All responses return:

Content-Type: application/json; charset=utf-8

Except file downloads which return the appropriate MIME type.

HTTP Methods

  • GET - Retrieve resources
  • POST - Create resources
  • PATCH - Partially update resources
  • PUT - Replace resources
  • DELETE - Delete resources

Idempotency

PUT, PATCH, and DELETE operations are idempotent. POST operations are not idempotent unless you provide an idempotencyKey:

{
  "idempotencyKey": "unique-key-123",
  "type": "POST",
  "content": {...}
}

Pagination

List endpoints use cursor-based pagination for stable results:

GET /api/actions?limit=20

Response includes pagination info:

{
  "data": [...],
  "cursorPagination": {
    "nextCursor": "eyJzIjoiY3JlYXRlZCIsInYiOjE3MzUwMDAwMDAsImlkIjoiYTF-YWJjMTIzIn0",
    "hasMore": true
  },
  "time": "2025-01-01T12:00:00Z"
}

To fetch the next page, use the cursor:

GET /api/actions?limit=20&cursor=eyJzIjoiY3JlYXRlZCIsInYiOjE3MzUwMDAwMDAsImlkIjoiYTF-YWJjMTIzIn0

Filtering

Many endpoints support filtering via query parameters. Each endpoint documents its available filters.

GET /api/actions?type=POST&status=A&createdAfter=2025-01-01T00:00:00Z

Batch Operations

Some endpoints support batch operations:

POST /actions/batch
Content-Type: application/json

{
  "operations": [
    { "method": "POST", "data": {...} },
    { "method": "PATCH", "id": "act_123", "data": {...} }
  ]
}

WebSocket Endpoints

For real-time features, use WebSocket connections:

  • WSS /ws/crdt - Collaborative documents
  • WSS /ws/rtdb/:fileId - Real-time database
  • WSS /ws/bus - Message bus

See WebSocket API for details.

Quick Start

Using @cloudillo/core

import * as cloudillo from '@cloudillo/core'

await cloudillo.init('my-app')
const api = cloudillo.createApiClient()

// Make requests
const profile = await api.profiles.getOwn()
const posts = await api.actions.list({ type: 'POST' })

Using fetch directly

const response = await fetch('/api/me', {
  headers: {
    'Authorization': `Bearer ${token}`,
    'Content-Type': 'application/json'
  }
})

const data = await response.json()
console.log(data.data) // Profile object
console.log(data.time) // Timestamp
console.log(data.reqId) // Request ID

Cursor Pagination Details

Cursors are opaque base64-encoded strings containing:

  • Sort field (s)
  • Sort value (v)
  • Last item ID (id)
{
  "s": "created",
  "v": 1735000000,
  "id": "a1~abc123"
}

Benefits:

  • Stable: Results don’t shift when new items are added
  • Efficient: No offset scanning in database
  • Reliable: Works with large datasets

Use the SDK for easier pagination:

let cursor = undefined
while (true) {
  const result = await api.actions.list({ type: 'POST', limit: 20, cursor })
  // Process result.data
  if (!result.cursorPagination?.hasMore) break
  cursor = result.cursorPagination.nextCursor
}

Next Steps

Explore specific endpoint categories:

Subsections of REST API

Authentication API

Overview

User authentication and token management endpoints. For user registration, see Profiles API.

Endpoints

Login

POST /api/auth/login

Authenticate with email/password and receive an access token.

Authentication: Not required

Request:

{
  "idTag": "alice@example.com",
  "password": "secure-password"
}

Response:

{
  "data": {
    "tnId": 12345,
    "idTag": "alice@example.com",
    "name": "Alice Johnson",
    "token": "eyJhbGc...",
    "roles": ["user"]
  },
  "time": "2025-01-01T12:00:00Z"
}

Logout

POST /api/auth/logout

Invalidate the current session.

Authentication: Required

Change Password

POST /api/auth/password

Change the authenticated user’s password.

Authentication: Required

Request:

{
  "currentPassword": "current-password",
  "newPassword": "new-secure-password"
}

Response:

{
  "data": {
    "success": true
  },
  "time": "2025-01-01T12:00:00Z"
}

Set Password

POST /api/auth/set-password

Set a new password using a reset token. Used during password recovery flow.

Authentication: Not required

Request:

{
  "token": "reset-token-from-email",
  "password": "new-secure-password"
}

Response:

{
  "data": {
    "success": true
  },
  "time": "2025-01-01T12:00:00Z"
}

Forgot Password

POST /api/auth/forgot-password

Request a password reset email.

Authentication: Not required

Request:

{
  "idTag": "alice@example.com"
}

Response:

{
  "data": {
    "sent": true
  },
  "time": "2025-01-01T12:00:00Z"
}

Refresh Login Token

GET /api/auth/login-token

Refresh the authentication token before it expires.

Authentication: Not required (uses existing valid token)

Response:

{
  "data": {
    "token": "eyJhbGc...",
    "expiresAt": 1735086400
  },
  "time": "2025-01-01T12:00:00Z"
}

Get Access Token

GET /api/auth/access-token

Exchange credentials or tokens for a scoped access token. Supports multiple authentication methods.

Authentication: Not required

Query Parameters:

  • token - Existing token to exchange
  • refId - Reference ID for share links
  • apiKey - API key for programmatic access
  • scope - Requested scope (optional)
  • refresh - Set to true to refresh an existing token

Response:

{
  "data": {
    "token": "eyJhbGc...",
    "expiresAt": 1735086400,
    "scope": "read:files"
  },
  "time": "2025-01-01T12:00:00Z"
}

Get Proxy Token

GET /api/auth/proxy-token

Get a proxy token for accessing remote resources via federation.

Authentication: Required

Query Parameters:

  • target - Target identity for federation

Response:

{
  "data": {
    "token": "eyJhbGc...",
    "expiresAt": 1735555555
  },
  "time": "2025-01-01T12:00:00Z"
}

WebAuthn (Passkey) Authentication

WebAuthn enables passwordless authentication using passkeys (biometrics, security keys, etc.).

List Passkey Registrations

GET /api/auth/wa/reg

List all registered passkeys for the authenticated user.

Authentication: Required

Response:

{
  "data": [
    {
      "keyId": "abc123",
      "name": "MacBook Touch ID",
      "createdAt": "2025-01-01T12:00:00Z",
      "lastUsedAt": "2025-01-15T09:30:00Z"
    }
  ],
  "time": "2025-01-01T12:00:00Z"
}

Get Registration Challenge

GET /api/auth/wa/reg/challenge

Get a challenge for registering a new passkey.

Authentication: Required

Response:

{
  "data": {
    "challenge": "base64-encoded-challenge",
    "rpId": "example.com",
    "rpName": "Cloudillo",
    "userId": "base64-user-id",
    "userName": "alice@example.com"
  },
  "time": "2025-01-01T12:00:00Z"
}

Register Passkey

POST /api/auth/wa/reg

Complete passkey registration with the WebAuthn response.

Authentication: Required

Request:

{
  "name": "MacBook Touch ID",
  "credential": {
    "id": "credential-id",
    "rawId": "base64-raw-id",
    "response": {
      "clientDataJSON": "base64-client-data",
      "attestationObject": "base64-attestation"
    },
    "type": "public-key"
  }
}

Response:

{
  "data": {
    "keyId": "abc123",
    "name": "MacBook Touch ID",
    "createdAt": "2025-01-01T12:00:00Z"
  },
  "time": "2025-01-01T12:00:00Z"
}

Delete Passkey

DELETE /api/auth/wa/reg/{key_id}

Remove a registered passkey.

Authentication: Required

Path Parameters:

  • key_id - The passkey identifier to delete

Response:

{
  "data": "ok",
  "time": "2025-01-01T12:00:00Z"
}

Get Login Challenge

GET /api/auth/wa/login/challenge

Get a challenge for passkey authentication.

Authentication: Not required

Query Parameters:

  • idTag - User identity (optional, for usernameless flow)

Response:

{
  "data": {
    "challenge": "base64-encoded-challenge",
    "rpId": "example.com",
    "allowCredentials": [
      {
        "type": "public-key",
        "id": "credential-id"
      }
    ]
  },
  "time": "2025-01-01T12:00:00Z"
}

Login with Passkey

POST /api/auth/wa/login

Authenticate using a passkey.

Authentication: Not required

Request:

{
  "credential": {
    "id": "credential-id",
    "rawId": "base64-raw-id",
    "response": {
      "clientDataJSON": "base64-client-data",
      "authenticatorData": "base64-auth-data",
      "signature": "base64-signature"
    },
    "type": "public-key"
  }
}

Response:

{
  "data": {
    "tnId": 12345,
    "idTag": "alice@example.com",
    "name": "Alice Johnson",
    "token": "eyJhbGc...",
    "roles": ["user"]
  },
  "time": "2025-01-01T12:00:00Z"
}

API Key Management

API keys enable programmatic access without interactive login.

List API Keys

GET /api/auth/api-keys

List all API keys for the authenticated user.

Authentication: Required

Response:

{
  "data": [
    {
      "keyId": "key_abc123",
      "name": "CI/CD Pipeline",
      "scope": "read:files,write:files",
      "createdAt": "2025-01-01T12:00:00Z",
      "lastUsedAt": "2025-01-15T09:30:00Z",
      "expiresAt": "2026-01-01T12:00:00Z"
    }
  ],
  "time": "2025-01-01T12:00:00Z"
}

Create API Key

POST /api/auth/api-keys

Create a new API key.

Authentication: Required

Request:

{
  "name": "CI/CD Pipeline",
  "scope": "read:files,write:files",
  "expiresAt": "2026-01-01T12:00:00Z"
}

Response:

{
  "data": {
    "keyId": "key_abc123",
    "name": "CI/CD Pipeline",
    "key": "ck_live_abc123xyz...",
    "scope": "read:files,write:files",
    "createdAt": "2025-01-01T12:00:00Z",
    "expiresAt": "2026-01-01T12:00:00Z"
  },
  "time": "2025-01-01T12:00:00Z"
}
Warning

The key field is only returned once at creation. Store it securely.

Get API Key

GET /api/auth/api-keys/{key_id}

Get details of a specific API key.

Authentication: Required

Path Parameters:

  • key_id - The API key identifier

Response:

{
  "data": {
    "keyId": "key_abc123",
    "name": "CI/CD Pipeline",
    "scope": "read:files,write:files",
    "createdAt": "2025-01-01T12:00:00Z",
    "lastUsedAt": "2025-01-15T09:30:00Z",
    "expiresAt": "2026-01-01T12:00:00Z"
  },
  "time": "2025-01-01T12:00:00Z"
}

Update API Key

PATCH /api/auth/api-keys/{key_id}

Update an API key’s metadata.

Authentication: Required

Path Parameters:

  • key_id - The API key identifier

Request:

{
  "name": "Production Pipeline",
  "scope": "read:files"
}

Response:

{
  "data": {
    "keyId": "key_abc123",
    "name": "Production Pipeline",
    "scope": "read:files",
    "createdAt": "2025-01-01T12:00:00Z",
    "expiresAt": "2026-01-01T12:00:00Z"
  },
  "time": "2025-01-01T12:00:00Z"
}

Delete API Key

DELETE /api/auth/api-keys/{key_id}

Revoke and delete an API key.

Authentication: Required

Path Parameters:

  • key_id - The API key identifier

Response:

{
  "data": "ok",
  "time": "2025-01-01T12:00:00Z"
}

Public Endpoints

Get Tenant Profile (Public)

GET /api/me
GET /api/me/full

Get the tenant (server) profile with public keys. This is a public endpoint that returns the server’s identity information.

Note: Both paths return the same data; /full is an alias for compatibility.

Authentication: Not required

Response:

{
  "data": {
    "idTag": "server@example.com",
    "name": "Example Server",
    "publicKey": "-----BEGIN PUBLIC KEY-----...",
    "serverInfo": {
      "version": "1.0.0",
      "features": ["federation", "crdt", "rtdb"]
    }
  },
  "time": "2025-01-01T12:00:00Z"
}

Resolve Identity Tag

GET /.well-known/cloudillo/id-tag

Resolve a domain-based identity to a Cloudillo server. This is part of the DNS-based identity system.

Authentication: Not required

Query Parameters:

  • idTag - The identity to resolve (e.g., alice@example.com)

Response:

{
  "data": {
    "idTag": "alice@example.com",
    "serverUrl": "https://cloudillo.example.com",
    "publicKey": "-----BEGIN PUBLIC KEY-----..."
  },
  "time": "2025-01-01T12:00:00Z"
}

Get VAPID Public Key

GET /api/auth/vapid

Get the VAPID public key for push notification subscriptions.

Authentication: Required

Response:

{
  "data": {
    "publicKey": "BNxwfD..."
  },
  "time": "2025-01-01T12:00:00Z"
}

See Also

Profiles API

Overview

Manage user and community profiles, including registration and community creation.

Registration

Verify Profile

POST /api/profiles/verify

Verify identity availability before registration or community creation. This endpoint checks if an identity tag is available and can be used.

Authentication: Not required

Request:

{
  "idTag": "alice@example.com",
  "type": "idp"
}
Field Type Required Description
idTag string Yes The identity tag to verify
type string Yes Identity type: idp (hosted) or domain (self-hosted)
appDomain string No Application domain (for domain type)
token string No Verification token (for unauthenticated requests)

Response:

{
  "data": {
    "available": true,
    "idTag": "alice@example.com"
  },
  "time": "2025-01-01T12:00:00Z"
}

Example:

const result = await api.profile.verify({
  idTag: 'alice@example.com',
  type: 'idp'
})
if (result.data.available) {
  // Proceed with registration
}

Register User

POST /api/profiles/register

Register a new user account. This initiates the registration process and sends a verification email.

Authentication: Not required

Request:

{
  "type": "idp",
  "idTag": "alice@example.com",
  "email": "alice@gmail.com",
  "token": "verification-token",
  "lang": "en"
}
Field Type Required Description
type string Yes Identity type: idp or domain
idTag string Yes Desired identity tag
email string Yes Email address for verification
token string Yes Registration token
appDomain string No Application domain (for domain type)
lang string No Preferred language code

Response:

{
  "data": {
    "tnId": 12345,
    "idTag": "alice@example.com",
    "name": "Alice",
    "token": "eyJhbGc..."
  },
  "time": "2025-01-01T12:00:00Z"
}

Example:

const result = await api.profile.register({
  type: 'idp',
  idTag: 'alice@example.com',
  email: 'alice@gmail.com',
  token: 'registration-token'
})
// User is now registered and logged in
console.log('Registered:', result.data.idTag)

Create Community

PUT /api/profiles/{id_tag}

Create a new community profile. The authenticated user becomes the community owner.

Authentication: Required

Path Parameters:

  • id_tag - The identity tag for the new community

Request:

{
  "type": "idp",
  "name": "Developer Community",
  "profilePic": "b1~abc123",
  "appDomain": "devs.example.com"
}
Field Type Required Description
type string Yes Identity type: idp or domain
name string No Community display name
profilePic string No Profile picture file ID
appDomain string No Application domain (for domain type)

Response:

{
  "data": {
    "idTag": "devs@example.com",
    "name": "Developer Community",
    "type": "community",
    "createdAt": "2025-01-01T12:00:00Z"
  },
  "time": "2025-01-01T12:00:00Z"
}

Example:

// First verify the community name is available
const check = await api.profile.verify({
  idTag: 'devs@example.com',
  type: 'idp'
})

if (check.data.available) {
  const response = await fetch('/api/profiles/devs@example.com', {
    method: 'PUT',
    headers: {
      'Authorization': `Bearer ${token}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      type: 'idp',
      name: 'Developer Community'
    })
  })
  const community = await response.json()
  console.log('Created community:', community.data.idTag)
}
Community Ownership

The authenticated user who creates the community becomes the owner. Only the owner can manage community settings and membership.

Profile Management

Get Own Profile

GET /api/me

Get the authenticated user’s profile (requires authentication). For the public server identity endpoint, see GET /api/me in the Authentication API.

Authentication: Required

Example:

const api = cloudillo.createApiClient()
const profile = await api.profiles.getOwn()

Response:

{
  "data": {
    "tnId": 12345,
    "idTag": "alice@example.com",
    "name": "Alice Johnson",
    "profilePic": "/file/b1~abc123",
    "x": {
      "bio": "Software developer",
      "location": "San Francisco"
    }
  }
}

Update Own Profile

PATCH /api/me

Update the authenticated user’s profile.

Authentication: Required

Request:

await api.profiles.updateOwn({
  name: 'Alice Smith',
  x: {
    bio: 'Senior developer',
    website: 'https://alice.example.com'
  }
})

Upload Profile Image

PUT /api/me/image

Upload or update your profile picture.

Authentication: Required

Content-Type: Image type (e.g., image/jpeg, image/png)

Request Body: Raw image binary data

Example:

const imageFile = document.querySelector('input[type="file"]').files[0]

const response = await fetch('/api/me/image', {
  method: 'PUT',
  headers: {
    'Authorization': `Bearer ${token}`,
    'Content-Type': imageFile.type
  },
  body: imageFile
})

const result = await response.json()
console.log('Profile image updated:', result.data.profilePic)

Response:

{
  "data": {
    "profilePic": "/api/file/b1~abc123",
    "fileId": "b1~abc123"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Upload Cover Image

PUT /api/me/cover

Upload or update your cover photo.

Authentication: Required

Content-Type: Image type (e.g., image/jpeg, image/png)

Request Body: Raw image binary data

Example:

const coverFile = document.querySelector('input[type="file"]').files[0]

const response = await fetch('/api/me/cover', {
  method: 'PUT',
  headers: {
    'Authorization': `Bearer ${token}`,
    'Content-Type': coverFile.type
  },
  body: coverFile
})

const result = await response.json()
console.log('Cover image updated:', result.data.coverPic)

Response:

{
  "data": {
    "coverPic": "/api/file/b1~xyz789",
    "fileId": "b1~xyz789"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Get Profile

GET /api/profiles/:idTag

Get a specific user or community profile.

Example:

const profile = await api.profiles.get('bob@example.com')

List Profiles

GET /api/profiles

List all accessible profiles.

Authentication: Required

Query Parameters:

  • type - Filter by type (person, community)
  • status - Filter by status (comma-separated)
  • connected - Filter by connection status (disconnected, pending, connected)
  • following - Filter by follow status (true/false)
  • q - Search query for name
  • idTag - Filter by specific identity tag

Example:

// List all communities
const communities = await api.profiles.list({
  type: 'community'
})

// List connected profiles
const friends = await api.profiles.list({
  connected: 'connected'
})

// Search for profiles
const results = await api.profiles.list({
  q: 'alice'
})

Update Relationship

PATCH /api/profiles/:idTag

Update your relationship with another profile (follow, block, etc.).

Authentication: Required

Request:

{
  "relationship": "follow"
}

Response:

{
  "data": {
    "idTag": "bob@example.com",
    "relationship": "follow",
    "updatedAt": 1735000000
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Admin: Update Profile

PATCH /api/admin/profiles/:idTag

Admin endpoint to update any user’s profile.

Authentication: Required (admin role)

Request:

{
  "name": "Updated Name",
  "status": "active",
  "roles": ["user", "moderator"]
}

Response:

{
  "data": {
    "tnId": 12345,
    "idTag": "bob@example.com",
    "name": "Updated Name",
    "status": "active",
    "roles": ["user", "moderator"]
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

See Also

Actions API

Overview

Actions represent social interactions and activities in Cloudillo. This includes posts, comments, reactions, connections, and more.

The Actions API allows you to:

  • Create posts, comments, and reactions
  • Manage user connections and follows
  • Share files and resources
  • Send messages
  • Track action statistics

Action Types

Content Actions

Type Description Audience Examples
POST Create a post or content Followers, Public, Custom Blog posts, status updates
CMNT Comment on an action Parent action audience Thread replies
REPOST Share existing content Followers Retweets
SHRE Share resource/link Followers, Custom Link sharing

User Actions

Type Description Audience Examples
FLLW Follow a user/community Target user Subscribe to updates
CONN Connection request Target user Friend requests

Communication Actions

Type Description Audience Examples
MSG Private message Specific user Direct messages
FSHR File sharing Specific user(s) Share documents

Metadata Actions

Type Description Audience Examples
REACT React to content None (broadcast) Likes, loves
ACK Acknowledgment Parent action issuer Read receipts

Endpoints

List Actions

GET /api/actions

Query all actions with optional filters. Uses cursor-based pagination for stable results.

Authentication: Optional (visibility-based access control)

Query Parameters:

Filtering:

  • type - Filter by action type(s), comma-separated (e.g., POST,CMNT)
  • status - Filter by status(es), comma-separated (P=Pending, A=Active, D=Deleted, C=Created, N=New)
  • issuer - Filter by issuer identity (e.g., alice@example.com)
  • audience - Filter by audience identity
  • parentId - Filter by parent action (for comments/reactions)
  • rootId - Filter by root/thread ID (for nested comments)
  • subject - Filter by subject identity (for CONN, FLLW actions)
  • involved - Filter by actions involving a specific identity (issuer, audience, or subject)
  • actionId - Filter by specific action ID
  • tag - Filter by content tag

Time-based:

  • createdAfter - ISO 8601 timestamp or Unix seconds

Pagination:

  • limit - Max results (default: 20)
  • cursor - Opaque cursor for next page (from previous response)

Sorting:

  • sort - Sort field: created (default)
  • sortDir - Sort direction: asc or desc (default: desc)

Examples:

Get recent posts:

const api = cloudillo.createApiClient()

const posts = await api.actions.list({
  type: 'POST',
  status: 'A',
  limit: 20,
  sort: 'created',
  sortDir: 'desc'
})

Get comments on a specific post:

const comments = await api.actions.list({
  type: 'CMNT',
  parentId: 'act_post123',
  sort: 'created',
  sortDir: 'asc'
})

Get all actions involving a specific user:

const userActivity = await api.actions.list({
  involved: 'alice@example.com',
  limit: 50
})

Get posts from a time range:

const lastWeek = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString()

const recentPosts = await api.actions.list({
  type: 'POST',
  status: 'A',
  createdAfter: lastWeek,
  limit: 100
})

Get pending connection requests:

const connectionRequests = await api.actions.list({
  type: 'CONN',
  status: 'P',
  subject: cloudillo.idTag // Requests to me
})

Get thread with all nested comments:

const thread = await api.actions.list({
  rootId: 'act_original_post',
  type: 'CMNT',
  sort: 'created',
  sortDir: 'asc',
  limit: 200
})

Cursor-based pagination:

// First page
const page1 = await api.actions.list({ type: 'POST', limit: 20 })

// Next page using cursor
if (page1.cursorPagination?.hasMore) {
  const page2 = await api.actions.list({
    type: 'POST',
    limit: 20,
    cursor: page1.cursorPagination.nextCursor
  })
}

Response:

{
  "data": [
    {
      "actionId": "act_abc123",
      "type": "POST",
      "issuer": {
        "idTag": "alice@example.com",
        "name": "Alice Johnson",
        "profilePic": "/file/b1~abc"
      },
      "content": {
        "text": "Hello, Cloudillo!",
        "title": "My First Post"
      },
      "createdAt": "2025-01-01T12:00:00Z",
      "visibility": "P",
      "stat": {
        "reactions": 5,
        "comments": 3,
        "ownReaction": "LOVE"
      }
    }
  ],
  "cursorPagination": {
    "nextCursor": "eyJzIjoiY3JlYXRlZCIsInYiOjE3MzUwMDAwMDAsImlkIjoiYWN0X2FiYzEyMyJ9",
    "hasMore": true
  },
  "time": "2025-01-01T12:00:00Z"
}

Create Action

POST /api/actions

Create a new action (post, comment, reaction, etc.).

Authentication: Required

Request Body:

interface NewAction {
  type: string // Action type (POST, CMNT, etc.)
  subType?: string // Optional subtype/category
  parentId?: string // For comments, reactions
  rootId?: string // For deep threads
  audienceTag?: string // Target audience
  content?: unknown // Action-specific content
  attachments?: string[] // File IDs
  subject?: string // Target (e.g., who to follow)
  expiresAt?: number // Expiration timestamp
}

Examples:

Create a Post:

const post = await api.actions.create({
  type: 'POST',
  content: {
    text: 'Hello, Cloudillo!',
    title: 'My First Post'
  },
  attachments: ['file_123', 'file_456']
})

Create a Comment:

const comment = await api.actions.create({
  type: 'CMNT',
  parentId: 'act_post123',
  content: {
    text: 'Great post!'
  }
})

Create a Reaction:

const reaction = await api.actions.create({
  type: 'REACT',
  subType: 'LOVE',
  parentId: 'act_post123'
})

Follow a User:

const follow = await api.actions.create({
  type: 'FLLW',
  subject: 'bob@example.com'
})

Connect with a User:

const connection = await api.actions.create({
  type: 'CONN',
  subject: 'bob@example.com',
  content: {
    message: 'Would like to connect!'
  }
})

Share a File:

const share = await api.actions.create({
  type: 'FSHR',
  subject: 'bob@example.com',
  attachments: ['file_789'],
  content: {
    permission: 'READ', // or 'WRITE'
    message: 'Check out this document'
  }
})

Response:

{
  "data": {
    "actionId": "act_newpost123",
    "type": "POST",
    "issuerTag": "alice@example.com",
    "content": {
      "text": "Hello, Cloudillo!",
      "title": "My First Post"
    },
    "createdAt": 1735000000,
    "status": "A"
  }
}

Get Action

GET /api/actions/:actionId

Retrieve a specific action by ID.

Example:

const action = await api.actions.get('act_123')

Response:

{
  "data": {
    "actionId": "act_123",
    "type": "POST",
    "issuerTag": "alice@example.com",
    "issuer": {
      "idTag": "alice@example.com",
      "name": "Alice Johnson",
      "profilePic": "/file/b1~abc"
    },
    "content": {
      "text": "Hello!",
      "title": "Greeting"
    },
    "createdAt": 1735000000,
    "stat": {
      "reactions": 10,
      "comments": 5,
      "ownReaction": null
    }
  }
}
Actions are Immutable

Actions are signed JWTs and cannot be modified after creation. There is no PATCH endpoint for actions. To “edit” content, delete the original and create a new action.

Delete Action

DELETE /api/actions/:actionId

Delete an action. Only the issuer can delete their actions.

Authentication: Required (must be issuer)

Example:

await api.actions.delete('act_123')

Response:

{
  "data": "ok"
}

Accept Action

POST /api/actions/:actionId/accept

Accept a pending action (e.g., connection request, follow request).

Authentication: Required (must be the subject/target)

DSL Hooks: When an action is accepted, the server triggers the on_accept hook defined in the action type’s DSL configuration. This can execute custom logic such as:

  • Creating reciprocal connections (CONN actions)
  • Adding the user to groups (INVT actions)
  • Granting permissions (FSHR actions)

Example:

// Accept a connection request
await api.actions.accept('act_connreq123')

// Accept a follow request (for private accounts)
await api.actions.accept('act_follow456')

Response:

{
  "data": {
    "actionId": "act_connreq123",
    "status": "A"
  }
}

Reject Action

POST /api/actions/:actionId/reject

Reject a pending action.

Authentication: Required (must be the subject/target)

DSL Hooks: When an action is rejected, the server triggers the on_reject hook defined in the action type’s DSL configuration. This can execute cleanup logic such as:

  • Removing pending permissions
  • Notifying the issuer of the rejection
  • Cleaning up temporary resources

Example:

await api.actions.reject('act_connreq123')

Response:

{
  "data": {
    "actionId": "act_connreq123",
    "status": "D"
  }
}

Update Statistics

POST /api/actions/:actionId/stat

Update action statistics (typically called by the system).

Authentication: Required (admin)

Request Body:

{
  reactions?: number
  comments?: number
  views?: number
}

Add Reaction

POST /api/actions/:actionId/reaction

Add a reaction to an action. Creates a REACT action.

Authentication: Required

Request Body:

{
  type: string // Reaction type (e.g., 'LOVE')
}

Example:

await api.actions.addReaction('act_post123', {
  type: 'LOVE'
})

Response:

{
  "data": {
    "actionId": "act_reaction789",
    "type": "REACT",
    "subType": "LOVE",
    "parentId": "act_post123",
    "issuerTag": "alice@example.com"
  }
}

Federation Inbox

POST /api/inbox

Receive federated actions from other Cloudillo instances. This is the primary endpoint for cross-instance action delivery. Processing is asynchronous.

Authentication: Not required (actions are verified via signatures)

Request Body: Action token (JWT)

Example:

// This is typically called by other Cloudillo servers
const actionToken = 'eyJhbGc...' // Signed action token

const response = await fetch('/api/inbox', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/jwt'
  },
  body: actionToken
})

Response:

{
  "data": {
    "actionId": "act_federated123",
    "status": "received",
    "verified": true
  },
  "time": "2025-01-01T12:00:00Z"
}

Federation Inbox (Synchronous)

POST /api/inbox/sync

Receive federated actions with synchronous processing. Unlike the standard inbox, this endpoint processes the action immediately and returns the result.

Authentication: Not required (actions are verified via signatures)

Request Body: Action token (JWT)

Response:

{
  "data": {
    "actionId": "act_federated123",
    "status": "processed",
    "verified": true
  },
  "time": "2025-01-01T12:00:00Z"
}
Info

Both inbox endpoints verify the action signature against the issuer’s public key before accepting it.

Action Status Flow

Actions have a lifecycle represented by status:

C (Created) β†’ A (Active/Accepted)
            β†˜ D (Deleted)

P (Pending) β†’ A (Active)
            β†˜ D (Deleted)
  • P (Pending): Draft or unpublished content
  • A (Active): Published, visible, and finalized
  • D (Deleted): Soft-deleted or rejected
  • C (Created): Awaiting acceptance (e.g., connection requests)
  • N (New): Notification awaiting acknowledgment

Status transitions:

  • Only Created or Pending actions can be accepted/rejected
  • Only Pending actions can be updated before publishing
  • Any action can be deleted by issuer (changes status to Deleted)
  • Accepting changes status to Active
  • Rejecting changes status to Deleted

Content Schemas

Different action types have different content structures:

POST Content

{
  title?: string
  text?: string
  summary?: string
  category?: string
  tags?: string[]
}

CMNT Content

{
  text: string
}

MSG Content

{
  text: string
  subject?: string
}

FSHR Content

{
  permission: 'READ' | 'WRITE'
  message?: string
  expiresAt?: number
}

CONN Content

{
  message?: string
}

Action Statistics

Actions can have aggregated statistics:

interface ActionStat {
  reactions?: number // Total reactions
  comments?: number // Total comments
  commentsRead?: number // Comments user has read
  ownReaction?: string // User's own reaction type
  views?: number // View count
  shares?: number // Share count
}

Accessing statistics:

const action = await api.actions.get('act_123')

console.log('Reactions:', action.stat?.reactions)
console.log('Comments:', action.stat?.comments)
console.log('My reaction:', action.stat?.ownReaction)

Threading

Actions support threading via parentId and rootId:

POST (root action)
└── CMNT (comment)
    └── CMNT (reply to comment)
        └── CMNT (nested reply)

Creating a thread:

// Original post
const post = await api.actions.create({
  type: 'POST',
  content: { text: 'Main post' }
})

// First level comment
const comment1 = await api.actions.create({
  type: 'CMNT',
  parentId: post.actionId,
  rootId: post.actionId,
  content: { text: 'A comment' }
})

// Reply to comment
const reply = await api.actions.create({
  type: 'CMNT',
  parentId: comment1.actionId,
  rootId: post.actionId, // Still points to original post
  content: { text: 'A reply' }
})

Fetching a thread:

// Get all comments in a thread
const thread = await api.actions.list({
  rootId: 'act_post123',
  type: 'CMNT',
  sort: 'created',
  sortDir: 'asc'
})

Federation

Actions are the core of Cloudillo’s federation model. Each action is cryptographically signed and can be verified across instances.

Action Token Structure:

Header: { alg: "ES384", typ: "JWT" }
Payload: {
  actionId: "act_123",
  type: "POST",
  issuerTag: "alice@example.com",
  content: {...},
  createdAt: 1735000000,
  iat: 1735000000,
  exp: 1735086400
}
Signature: <ES384 signature>

This enables:

  • Trust-free verification
  • Cross-instance action delivery
  • Tamper-proof audit trails

Best Practices

1. Use Cursor-Based Pagination

// βœ… Use cursor pagination for stable results
async function fetchAllPosts() {
  const allPosts = []
  let cursor = undefined

  while (true) {
    const result = await api.actions.list({
      type: 'POST',
      limit: 50,
      cursor
    })

    allPosts.push(...result.data)

    if (!result.cursorPagination?.hasMore) break
    cursor = result.cursorPagination.nextCursor
  }

  return allPosts
}

2. Optimistic UI Updates

// Update UI immediately, rollback on error
const optimisticAction = {
  actionId: 'temp_' + Date.now(),
  type: 'POST',
  content: { text: 'New post' },
  issuer: { idTag: cloudillo.idTag },
  createdAt: new Date().toISOString()
}

setPosts([optimisticAction, ...posts])

try {
  const created = await api.actions.create(optimisticAction)
  setPosts(posts => posts.map(p =>
    p.actionId === optimisticAction.actionId ? created : p
  ))
} catch (error) {
  setPosts(posts => posts.filter(p =>
    p.actionId !== optimisticAction.actionId
  ))
}

3. Handle Visibility

Actions have visibility levels that control who can see them:

Code Level Description
P Public Anyone can view
V Verified Authenticated users only
F Follower User’s followers only
C Connected Mutual connections only
null Direct Owner + explicit audience
// Create a public post
const publicPost = await api.actions.create({
  type: 'POST',
  visibility: 'P',
  content: { text: 'Hello everyone!' }
})

// Create a followers-only post
const followersPost = await api.actions.create({
  type: 'POST',
  visibility: 'F',
  content: { text: 'Just for my followers' }
})

See Also

Files API

Overview

The Files API handles file upload, download, and management in Cloudillo. It supports four file types: BLOB (binary files), CRDT (collaborative documents), RTDB (real-time databases), and FLDR (folders).

File Types

Type Description Use Cases
BLOB Binary files (images, PDFs, etc.) Photos, documents, attachments
CRDT Collaborative documents Rich text, spreadsheets, diagrams
RTDB Real-time databases Structured data, forms, todos
FLDR Folders Organize files hierarchically

File Status

Code Status Description
A Active File is available
P Pending File is being processed
D Deleted File is in trash

Image Variants

For BLOB images, Cloudillo automatically generates 5 variants:

Variant Code Max Dimension Quality Format
Thumbnail tn 150px Medium JPEG/WebP
Icon ic 64px Medium JPEG/WebP
SD sd 640px Medium JPEG/WebP
HD hd 1920px High JPEG/WebP
Original orig Original Original Original

Automatic format selection:

  • Modern browsers: WebP or AVIF
  • Fallback: JPEG or PNG

Endpoints

List Files

GET /api/files

List all files accessible to the user. Uses cursor-based pagination.

Authentication: Optional (visibility-based access control)

Query Parameters:

Filtering:

  • fileTp - Filter by file type (BLOB, CRDT, RTDB, FLDR)
  • contentType - Filter by MIME type
  • tags - Filter by tags (comma-separated)
  • parentId - Filter by parent folder (for hierarchical listing)
  • status - Filter by status (A, P, D)

Pagination:

  • limit - Max results (default: 20)
  • cursor - Opaque cursor for next page

Sorting:

  • sort - Sort field: created, modified, name, recent
  • sortDir - Sort direction: asc or desc

Example:

const api = cloudillo.createApiClient()

// List all images
const images = await api.files.list({
  fileTp: 'BLOB',
  contentType: 'image/*',
  limit: 20
})

// List files in a folder
const folderContents = await api.files.list({
  parentId: 'f1~folder123',
  sort: 'name',
  sortDir: 'asc'
})

// List tagged files
const projectFiles = await api.files.list({
  tags: 'project-alpha,important'
})

// Cursor-based pagination
if (images.cursorPagination?.hasMore) {
  const page2 = await api.files.list({
    fileTp: 'BLOB',
    cursor: images.cursorPagination.nextCursor
  })
}

Response:

{
  "data": [
    {
      "fileId": "b1~abc123",
      "parentId": null,
      "status": "A",
      "contentType": "image/png",
      "fileName": "photo.png",
      "fileTp": "BLOB",
      "createdAt": "2025-01-01T12:00:00Z",
      "tags": ["vacation", "beach"],
      "visibility": "P",
      "owner": {
        "idTag": "alice@example.com",
        "name": "Alice"
      },
      "userData": {
        "pinned": false,
        "starred": true,
        "accessedAt": "2025-01-15T09:30:00Z"
      }
    }
  ],
  "cursorPagination": {
    "nextCursor": "eyJzIjoiY3JlYXRlZCIsInYiOjE3MzUwMDAwMDAsImlkIjoiYjF-YWJjMTIzIn0",
    "hasMore": true
  },
  "time": "2025-01-01T12:00:00Z"
}

Create File Metadata (CRDT/RTDB)

POST /api/files

Create file metadata for CRDT or RTDB files. For BLOB files, use the one-step upload endpoint instead (see “Upload File (BLOB)” below).

Authentication: Required

Request Body:

{
  fileTp: string // CRDT or RTDB
  fileName?: string // Optional filename
  tags?: string // Comma-separated tags
}

Example:

// Create CRDT document
const file = await api.files.create({
  fileTp: 'CRDT',
  fileName: 'team-doc.crdt',
  tags: 'collaborative,document'
})

console.log('File ID:', file.fileId) // e.g., "f1~abc123"

// Now connect via WebSocket to edit the document
import * as Y from 'yjs'
const yDoc = new Y.Doc()
const { provider } = await cloudillo.openYDoc(yDoc, file.fileId)

Response:

{
  "data": {
    "fileId": "f1~abc123",
    "status": "P",
    "fileTp": "CRDT",
    "fileName": "team-doc.crdt",
    "createdAt": 1735000000
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Note: For BLOB files (images, PDFs, etc.), use POST /api/files/{preset}/{file_name} instead, which creates metadata and uploads the file in a single step.

Upload File (BLOB)

POST /api/files/{preset}/{file_name}

Upload binary file data directly. This creates the file metadata and uploads the binary in a single operation.

Authentication: Required

Path Parameters:

  • preset (string, required) - Image processing preset (e.g., default, profile-picture, cover-photo)
  • file_name (string, required) - Original filename with extension

Query Parameters:

  • created_at (number, optional) - Unix timestamp (seconds) for when the file was created
  • tags (string, optional) - Comma-separated tags (e.g., vacation,beach,2025)

Content-Type: Binary content type (e.g., image/png, image/jpeg, application/pdf)

Request Body: Raw binary file data

Example:

const api = cloudillo.createApiClient()

// Upload image file
const imageFile = document.querySelector('input[type="file"]').files[0]
const blob = await imageFile.arrayBuffer()

const response = await fetch('/api/files/default/vacation-photo.jpg?tags=vacation,beach', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${token}`,
    'Content-Type': 'image/jpeg'
  },
  body: blob
})

const result = await response.json()
console.log('File ID:', result.data.fileId) // e.g., "b1~abc123"

Complete upload with File object:

async function uploadFile(file: File, preset = 'default', tags?: string) {
  const queryParams = new URLSearchParams()
  if (tags) queryParams.set('tags', tags)

  const url = `/api/files/${preset}/${encodeURIComponent(file.name)}${
    queryParams.toString() ? '?' + queryParams.toString() : ''
  }`

  const response = await fetch(url, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${token}`,
      'Content-Type': file.type
    },
    body: file
  })

  const result = await response.json()
  return result.data.fileId
}

// Usage
const fileInput = document.querySelector('input[type="file"]')
fileInput.addEventListener('change', async (e) => {
  const file = e.target.files[0]
  const fileId = await uploadFile(file, 'default', 'vacation,beach')
  console.log('Uploaded:', fileId)

  // Display uploaded image
  const img = document.createElement('img')
  img.src = `/api/files/${fileId}?variant=sd`
  document.body.appendChild(img)
})

Response:

{
  "data": {
    "fileId": "b1~abc123",
    "status": "M",
    "fileTp": "BLOB",
    "contentType": "image/jpeg",
    "fileName": "vacation-photo.jpg",
    "createdAt": 1735000000,
    "tags": ["vacation", "beach"]
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Download File

GET /api/files/:fileId

Download a file. Returns binary data with appropriate Content-Type.

Query Parameters:

  • variant - Image variant (tn, ic, sd, hd, orig)

Example:

// Direct URL usage
<img src="/api/files/b1~abc123" />

// Get specific variant
<img src="/api/files/b1~abc123?variant=sd" />

// Fetch with API
const response = await fetch(`/api/files/${fileId}`)
const blob = await response.blob()
const url = URL.createObjectURL(blob)

Responsive images:

<picture>
  <source srcset="/api/files/b1~abc123?variant=hd" media="(min-width: 1200px)">
  <source srcset="/api/files/b1~abc123?variant=sd" media="(min-width: 600px)">
  <img src="/api/files/b1~abc123?variant=tn" alt="Photo">
</picture>

Get File Descriptor

GET /api/files/:fileId/descriptor

Get file metadata and available variants.

Example:

const descriptor = await api.files.getDescriptor('b1~abc123')

console.log('File type:', descriptor.contentType)
console.log('Size:', descriptor.size)
console.log('Variants:', descriptor.variants)

Response:

{
  "data": {
    "fileId": "b1~abc123",
    "contentType": "image/png",
    "fileName": "photo.png",
    "size": 1048576,
    "createdAt": "2025-01-01T12:00:00Z",
    "variants": [
      {
        "id": "tn",
        "width": 150,
        "height": 100,
        "size": 5120,
        "format": "webp"
      },
      {
        "id": "sd",
        "width": 640,
        "height": 426,
        "size": 51200,
        "format": "webp"
      },
      {
        "id": "hd",
        "width": 1920,
        "height": 1280,
        "size": 204800,
        "format": "webp"
      },
      {
        "id": "orig",
        "width": 3840,
        "height": 2560,
        "size": 1048576,
        "format": "png"
      }
    ]
  }
}

Update File Metadata

PATCH /api/files/:fileId

Update file metadata (tags, filename, etc.).

Authentication: Required (must be owner)

Request Body:

{
  fileName?: string
  tags?: string // Comma-separated
}

Example:

await api.files.update('b1~abc123', {
  fileName: 'renamed-photo.png',
  tags: 'vacation,beach,2025'
})

Delete File

DELETE /api/files/{file_id}

Move a file to trash (soft delete).

Authentication: Required (must have write access)

Example:

await api.files.delete('b1~abc123')

Response:

{
  "data": "ok",
  "time": "2025-01-01T12:00:00Z"
}

Restore File

POST /api/files/{file_id}/restore

Restore a file from trash.

Authentication: Required (must have write access)

Example:

await fetch('/api/files/b1~abc123/restore', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${token}` }
})

Response:

{
  "data": "ok",
  "time": "2025-01-01T12:00:00Z"
}

Empty Trash

DELETE /api/trash

Permanently delete all files in trash.

Authentication: Required

Example:

await fetch('/api/trash', {
  method: 'DELETE',
  headers: { 'Authorization': `Bearer ${token}` }
})

Response:

{
  "data": {
    "deleted": 15
  },
  "time": "2025-01-01T12:00:00Z"
}

Update User File Data

PATCH /api/files/{file_id}/user

Update user-specific file metadata (pinned, starred). This updates only the authenticated user’s relationship with the file, not the file itself. Users can pin/star any file they have read access to.

Authentication: Required

Request:

{
  "pinned": true,
  "starred": false
}

Response:

{
  "data": {
    "pinned": true,
    "starred": false,
    "accessedAt": "2025-01-01T12:00:00Z"
  },
  "time": "2025-01-01T12:00:00Z"
}

Example:

// Star a file
await fetch('/api/files/b1~abc123/user', {
  method: 'PATCH',
  headers: {
    'Authorization': `Bearer ${token}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({ starred: true })
})

List Tags

GET /api/tags

List all tags used in files owned by the authenticated user.

Authentication: Required

Response:

{
  "data": [
    {
      "tag": "vacation",
      "count": 15
    },
    {
      "tag": "project-alpha",
      "count": 8
    },
    {
      "tag": "important",
      "count": 3
    }
  ]
}

Example:

const response = await fetch('/api/tags', {
  headers: {
    'Authorization': `Bearer ${token}`
  }
})
const { data: tags } = await response.json()

// Display tags with counts
tags.forEach(({ tag, count }) => {
  console.log(`${tag}: ${count} files`)
})

Add Tag

PUT /api/files/:fileId/tag/:tag

Add a tag to a file.

Authentication: Required (must be owner)

Example:

await fetch(`/api/files/b1~abc123/tag/important`, {
  method: 'PUT',
  headers: {
    'Authorization': `Bearer ${token}`
  }
})

Remove Tag

DELETE /api/files/:fileId/tag/:tag

Remove a tag from a file.

Authentication: Required (must be owner)

Example:

await fetch(`/api/files/b1~abc123/tag/important`, {
  method: 'DELETE',
  headers: {
    'Authorization': `Bearer ${token}`
  }
})

Get File Variant

GET /api/files/variant/:variantId

Get a specific image variant directly.

Example:

// Variant IDs are in the format: {fileId}~{variant}
<img src="/api/files/variant/b1~abc123~sd" />

File Identifiers

Cloudillo uses content-addressable identifiers:

Format: {prefix}{version}~{hash}

Prefixes:

  • b - BLOB files
  • f - CRDT files
  • r - RTDB files

Examples:

  • b1~abc123def - BLOB file
  • f1~xyz789ghi - CRDT file
  • r1~mno456pqr - RTDB file

Variants:

  • b1~abc123def~sd - SD variant of BLOB
  • b1~abc123def~tn - Thumbnail variant

CRDT Files

CRDT files store collaborative documents using Yjs.

Creating a CRDT file:

// 1. Create file metadata
const file = await api.files.create({
  fileTp: 'CRDT',
  tags: 'document,collaborative'
})

// 2. Open for editing
import * as Y from 'yjs'

const yDoc = new Y.Doc()
const { provider } = await cloudillo.openYDoc(yDoc, file.fileId)

// 3. Use shared types
const yText = yDoc.getText('content')
yText.insert(0, 'Hello, CRDT!')

See CRDT documentation for details.

RTDB Files

RTDB files store structured real-time databases.

Creating an RTDB file:

// 1. Create file metadata
const file = await api.files.create({
  fileTp: 'RTDB',
  tags: 'database,todos'
})

// 2. Connect to database
import { RtdbClient } from '@cloudillo/rtdb'

const rtdb = new RtdbClient({
  fileId: file.fileId,
  token: cloudillo.accessToken,
  url: 'wss://server.com/ws/rtdb'
})

// 3. Use collections
const todos = rtdb.collection('todos')
await todos.create({ title: 'Learn Cloudillo', done: false })

See RTDB documentation for details.

Image Presets

Configure automatic image processing with presets:

const file = await api.files.create({
  fileTp: 'BLOB',
  contentType: 'image/jpeg',
  preset: 'profile-picture' // Custom preset
})

Default presets:

  • default - Standard 5-variant generation
  • profile-picture - Square crop, 400x400 max
  • cover-photo - 16:9 crop, 1920x1080 max
  • thumbnail-only - Only generate thumbnails

Tagging

Tags help organize and filter files.

Best practices:

  • Use lowercase tags
  • Use hyphens for multi-word tags (e.g., project-alpha)
  • Limit to 3-5 tags per file
  • Use namespaced tags for projects (e.g., proj:alpha, proj:beta)

Tag filtering:

// Files with ANY of these tags
const files = await api.files.list({
  tags: 'vacation,travel'
})

// Files with ALL of these tags (use multiple requests)
const vacationFiles = await api.files.list({ tags: 'vacation' })
const summerFiles = vacationFiles.data.filter(f =>
  f.tags?.includes('summer')
)

Permissions

File access is controlled by:

  1. Ownership - Owner has full access
  2. FSHR actions - Files shared via FSHR actions grant temporary access
  3. Public files - Files attached to public actions are publicly accessible
  4. Audience - Files attached to actions inherit action audience permissions

Sharing a file:

// Share file with read access
await api.actions.create({
  type: 'FSHR',
  subject: 'bob@example.com',
  attachments: ['b1~abc123'],
  content: {
    permission: 'READ', // or 'WRITE'
    message: 'Check out this photo!'
  }
})

Storage Considerations

File size limits:

  • Free tier: 100 MB per file
  • Pro tier: 1 GB per file
  • Enterprise: Configurable

Total storage:

  • Free tier: 10 GB
  • Pro tier: 100 GB
  • Enterprise: Unlimited

Variant generation:

  • Only for image files (JPEG, PNG, WebP, AVIF, GIF)
  • Automatic async processing
  • Lanczos3 filtering for high quality
  • Progressive JPEG for faster loading

Best Practices

1. Always Create Metadata First

// βœ… Correct order
const metadata = await api.files.create({ fileTp: 'BLOB', contentType: 'image/png' })
await uploadBinary(metadata.fileId, imageBlob)

// ❌ Wrong - upload will fail without metadata
await uploadBinary('unknown-id', imageBlob)

2. Use Appropriate Variants

// βœ… Use thumbnails in lists
<img src={`/api/files/${fileId}?variant=tn`} />

// βœ… Use HD for detail views
<img src={`/api/files/${fileId}?variant=hd`} />

// ❌ Don't use original for thumbnails (wastes bandwidth)
<img src={`/api/files/${fileId}`} width="100" />

3. Handle Upload Errors

async function uploadWithRetry(file: File, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      const result = await api.files.uploadBlob('default', file.name, file, file.type)
      return result.fileId
    } catch (error) {
      if (i === maxRetries - 1) throw error
      await new Promise(r => setTimeout(r, 1000 * (i + 1)))
    }
  }
}

4. Progressive Enhancement

// Show preview immediately, upload in background
function previewAndUpload(file: File) {
  // Show local preview
  const reader = new FileReader()
  reader.onload = (e) => {
    setPreview(e.target.result)
  }
  reader.readAsDataURL(file)

  // Upload in background
  uploadImage(file).then(fileId => {
    setUploadedId(fileId)
  })
}

5. Clean Up Unused Files

// Delete old temporary files
const oldFiles = await api.files.list({
  tags: 'temp',
  createdBefore: Date.now() / 1000 - 86400 // 24 hours ago
})

for (const file of oldFiles.data) {
  await api.files.delete(file.fileId)
}

See Also

Settings API

Settings API

User preferences and configuration key-value store.

Settings Scopes

Settings operate at different scopes with cascading resolution:

Scope Description Permission
system Server-wide defaults (read-only) Requires restart to change
global Shared across all tenants (tn_id=0) Admin only
tenant User-specific settings Owner

Permission Levels

Level Access
system Non-changeable (compile-time)
admin Requires SADM role
user Any authenticated user

Resolution Order

When retrieving a setting value, the system resolves in this order:

  1. Tenant-specific value (if exists)
  2. Global value (if exists)
  3. System default

Endpoints

List Settings

GET /api/settings

Get all settings for the authenticated user.

Authentication: Required

Response:

{
  "data": [
    ["theme", "dark"],
    ["language", "en"],
    ["notifications", "true"]
  ]
}

Get Setting

GET /api/settings/:name

Get a specific setting value.

Example:

const theme = await api.settings.name('theme').get()

Update Setting

PUT /api/settings/:name

Set or update a setting value.

Request:

await api.settings.name('theme').put('dark')

Usage

// Get all settings
const settings = await api.settings.get()

// Get specific setting
const theme = await api.settings.name('theme').get()

// Update setting
await api.settings.name('theme').put('light')
await api.settings.name('fontSize').put(16)
await api.settings.name('notifications').put(true)

Common Settings

  • theme - UI theme preference
  • language - User language
  • fontSize - Text size
  • notifications - Enable/disable notifications
  • darkMode - Dark mode preference

References API

Overview

References (refs) are shareable tokens for various workflows including file sharing, email verification, password reset, and invitations. They support configurable expiration, usage limits, and access levels.

Endpoints

List References

GET /api/refs

List references for the current tenant.

Authentication: Required

Query Parameters:

Parameter Type Required Description
type string No Filter by type (e.g., share.file, email-verify)
filter string No Status filter: active, used, expired, all (default: active)
resourceId string No Filter by resource ID (e.g., file ID)

Response:

{
  "data": [
    {
      "refId": "abc123def456",
      "type": "share.file",
      "description": "Project document",
      "createdAt": 1735000000,
      "expiresAt": 1735604800,
      "count": null,
      "resourceId": "f1~xyz789",
      "accessLevel": "read"
    }
  ],
  "pagination": {
    "offset": 0,
    "limit": 10,
    "total": 1
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Example:

// List all active file shares
const shares = await api.refs.list({
  type: 'share.file',
  filter: 'active'
})

// List shares for a specific file
const fileShares = await api.refs.list({
  resourceId: 'f1~xyz789'
})

Create Reference

POST /api/refs

Create a new reference token.

Authentication: Required

Request:

{
  "type": "share.file",
  "description": "Project document",
  "expiresAt": 1735604800,
  "count": null,
  "resourceId": "f1~xyz789",
  "accessLevel": "read"
}
Field Type Required Description
type string Yes Reference type
description string No Human-readable description
expiresAt number No Expiration timestamp (Unix seconds)
count number/null No Usage count: omit for 1, null for unlimited, or number
resourceId string No Resource ID (required for share.file)
accessLevel string No Access level: read or write (default: read)

Response:

{
  "data": {
    "refId": "abc123def456",
    "type": "share.file",
    "description": "Project document",
    "createdAt": 1735000000,
    "expiresAt": 1735604800,
    "count": null,
    "resourceId": "f1~xyz789",
    "accessLevel": "read"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Example:

// Create a read-only share link with unlimited uses
const shareLink = await api.refs.create({
  type: 'share.file',
  description: 'Team document',
  resourceId: 'f1~xyz789',
  accessLevel: 'read',
  count: null  // Unlimited uses
})

// Create a write-access share link (single use)
const editLink = await api.refs.create({
  type: 'share.file',
  resourceId: 'f1~xyz789',
  accessLevel: 'write'
  // count omitted = single use
})

Get Reference

GET /api/refs/{refId}

Get details of a specific reference. Returns full details if authenticated, minimal details (only refId and type) if not.

Authentication: Optional

Path Parameters:

Parameter Type Description
refId string Reference ID

Response (authenticated):

{
  "data": {
    "refId": "abc123def456",
    "type": "share.file",
    "description": "Project document",
    "createdAt": 1735000000,
    "expiresAt": 1735604800,
    "count": 5,
    "resourceId": "f1~xyz789",
    "accessLevel": "read"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Response (unauthenticated):

{
  "data": {
    "refId": "abc123def456",
    "type": "share.file"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Delete Reference

DELETE /api/refs/{refId}

Delete/revoke a reference. The reference will immediately become invalid.

Authentication: Required

Path Parameters:

Parameter Type Description
refId string Reference ID to delete

Response:

{
  "data": null,
  "time": 1735000000,
  "reqId": "req_abc123"
}

Guest Access with References

References can be used to grant guest access to resources without requiring authentication.

Exchange Reference for Access Token

GET /api/auth/access-token

Exchange a reference for a scoped access token. This enables unauthenticated users to access shared resources.

Authentication: None

Query Parameters:

Parameter Type Required Description
ref string Yes Reference ID

Response:

{
  "data": {
    "token": "eyJhbGc...",
    "expiresAt": 1735086400,
    "accessLevel": "read",
    "resourceId": "f1~xyz789"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Example:

// Guest accessing a shared file
const { token, accessLevel } = await api.auth.getAccessToken({
  ref: 'abc123def456'
})

// Use token for subsequent API calls
const file = await api.files.get('f1~xyz789', {
  headers: { 'Authorization': `Bearer ${token}` }
})

Reference Types

Type Description Requires resourceId
share.file File sharing link Yes
email-verify Email verification No
password-reset Password reset link No
invite User invitation No
welcome Welcome/onboarding link No

Usage Count Behavior

The count field controls how many times a reference can be used:

Value Behavior
Omitted Single use (default = 1)
null Unlimited uses
Number That many uses remaining

Each use decrements the count. When count reaches 0, the reference becomes invalid.

Access Levels

For share.file references, the accessLevel controls permissions:

Level Description
read View-only access (default)
write Edit access to the resource

Complete File Sharing Example

// 1. Create a shareable link
const share = await api.refs.create({
  type: 'share.file',
  description: 'Quarterly report',
  resourceId: 'f1~xyz789',
  accessLevel: 'read',
  count: null,  // Unlimited
  expiresAt: Math.floor(Date.now() / 1000) + 7 * 24 * 60 * 60  // 1 week
})

// 2. Generate share URL
const shareUrl = `https://example.cloudillo.net/share/${share.data.refId}`

// 3. Guest accesses the share (on their side)
const { token } = await fetch('/api/auth/access-token?ref=' + refId)
  .then(r => r.json())
  .then(r => r.data)

// 4. Guest uses token to access file
const file = await fetch('/api/files/f1~xyz789', {
  headers: { 'Authorization': `Bearer ${token}` }
})

See Also

Collections API

Overview

The Collections API manages user-specific item collections like favorites, recent items, bookmarks, and pins.

Collection Types

Type Description
FAVR Favorites - user’s favorite items
RCNT Recent - recently accessed items
BKMK Bookmarks - saved references
PIND Pinned - pinned items for quick access

Endpoints

List Collection Items

GET /api/collections/{coll_type}

List all items in a collection.

Authentication: Required

Path Parameters:

  • coll_type - Collection type: FAVR, RCNT, BKMK, or PIND

Query Parameters:

  • limit - Maximum items to return (default: 20)
  • cursor - Opaque cursor for pagination

Response:

{
  "data": [
    {
      "itemId": "f1~abc123",
      "addedAt": "2025-01-15T10:30:00Z"
    }
  ],
  "cursorPagination": {
    "nextCursor": "eyJzIjoiYWRkZWRfYXQiLCJ2IjoxNzA1MzE1ODAwfQ",
    "hasMore": false
  },
  "time": "2025-01-15T10:30:00Z"
}

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/collections/FAVR?limit=20"

Add Item to Collection

POST /api/collections/{coll_type}/{item_id}

Add an item to a collection.

Authentication: Required

Path Parameters:

  • coll_type - Collection type: FAVR, RCNT, BKMK, or PIND
  • item_id - ID of the item to add (file ID or action ID)

Response:

{
  "data": {
    "itemId": "f1~abc123",
    "addedAt": "2025-01-15T10:30:00Z"
  },
  "time": "2025-01-15T10:30:00Z"
}

Example:

curl -X POST -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/collections/FAVR/f1~abc123"

Remove Item from Collection

DELETE /api/collections/{coll_type}/{item_id}

Remove an item from a collection.

Authentication: Required

Path Parameters:

  • coll_type - Collection type: FAVR, RCNT, BKMK, or PIND
  • item_id - ID of the item to remove

Response:

{
  "data": "ok",
  "time": "2025-01-15T10:30:00Z"
}

Example:

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/collections/FAVR/f1~abc123"

Client SDK Usage

import { createApiClient } from '@cloudillo/core'

const api = createApiClient({ idTag: 'alice.cloudillo.net', authToken: token })

// List favorites
const favorites = await api.collections.list('FAVR', { limit: 20 })

// Add to bookmarks
await api.collections.add('BKMK', 'file_abc123')

// Remove from pins
await api.collections.remove('PIND', 'file_abc123')

// Shorthand for files
await api.files.favorite('file_abc123')
await api.files.unfavorite('file_abc123')

Use Cases

Favorites

Store user’s favorite files for quick access.

// Add file to favorites
await api.files.favorite(fileId)

// List favorite files
const favs = await api.files.listFavorites({ limit: 50 })

Recent Items

Track recently accessed files (managed automatically by the system).

// List recent files
const recent = await api.files.listRecent({ limit: 20 })

Bookmarks

Save references for later.

// Add bookmark
await api.collections.add('BKMK', actionId)

// List bookmarks
const bookmarks = await api.collections.list('BKMK')

Pinned Items

Pin important items for quick access.

// Pin a file
await api.files.setPinned(fileId, true)

// List pinned items
const pinned = await api.collections.list('PIND')

See Also

Identity Provider API

Overview

The Identity Provider (IDP) API enables identity management for Cloudillo’s DNS-based identity system. It allows identity providers to register, manage, and activate identities within their domain.

IDP Availability

IDP functionality must be enabled for the tenant. When disabled, all IDP endpoints return 404 Not Found.

Public Endpoints

These endpoints are available without authentication.

Get IDP Info

GET /api/idp/info

Get public information about this Identity Provider. Used by registration UIs to help users choose a provider.

Authentication: None

Response:

{
  "data": {
    "domain": "cloudillo.net",
    "name": "Cloudillo",
    "info": "Free identity hosting for the Cloudillo network",
    "url": "https://cloudillo.net/identity"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}
Field Type Description
domain string Provider domain (e.g., cloudillo.net)
name string Display name of the provider
info string Short description (pricing, terms)
url string Optional URL for more information

Check Availability

GET /api/idp/check-availability

Check if an identity tag is available for registration.

Authentication: Required

Query Parameters:

Parameter Type Required Description
idTag string Yes Identity to check (e.g., alice.cloudillo.net)

Response:

{
  "data": {
    "available": true,
    "idTag": "alice.cloudillo.net"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Example:

const result = await api.idp.checkAvailability({
  idTag: 'alice.cloudillo.net'
})
if (result.data.available) {
  // Identity is available for registration
}
Domain Restriction

The identity domain must match the authenticated registrar’s domain. You cannot check availability for identities in other domains.

Activate Identity

POST /api/idp/activate

Activate an identity using a reference token. This is used when the identity owner activates their identity after the registrar has created it.

Authentication: None (uses reference token)

Request:

{
  "refId": "ref_abc123def456"
}
Field Type Required Description
refId string Yes The activation reference ID

Response:

{
  "data": {
    "idTag": "alice.cloudillo.net",
    "status": "active",
    "address": "cloudillo.example.com"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Identity Management

List Identities

GET /api/idp/identities

List identities managed by the authenticated user.

Authentication: Required

Query Parameters:

Parameter Type Required Description
email string No Filter by email (partial match)
registrarIdTag string No Filter by registrar
ownerIdTag string No Filter by owner
status string No Filter by status: pending, active, suspended
limit number No Maximum results to return
offset number No Pagination offset

Response:

{
  "data": [
    {
      "idTag": "alice.cloudillo.net",
      "email": "alice@example.com",
      "registrarIdTag": "admin.cloudillo.net",
      "ownerIdTag": null,
      "address": "cloudillo.example.com",
      "addressUpdatedAt": 1735000000,
      "status": "active",
      "createdAt": 1734900000,
      "updatedAt": 1735000000,
      "expiresAt": 1766436000
    }
  ],
  "time": 1735000000,
  "reqId": "req_abc123"
}

Example:

const identities = await api.idp.identities.list({
  status: 'active',
  limit: 20
})

Create Identity

POST /api/idp/identities

Create a new identity. The authenticated user becomes the registrar.

Authentication: Required

Request:

{
  "idTag": "alice.cloudillo.net",
  "email": "alice@example.com",
  "ownerIdTag": null,
  "address": "cloudillo.example.com"
}
Field Type Required Description
idTag string Yes Identity tag (must be in registrar’s domain)
email string No Email address (required if no ownerIdTag)
ownerIdTag string No Owner identity for community-owned identities
address string No Initial server address

Response:

{
  "data": {
    "idTag": "alice.cloudillo.net",
    "email": "alice@example.com",
    "registrarIdTag": "admin.cloudillo.net",
    "status": "pending",
    "createdAt": 1735000000,
    "updatedAt": 1735000000,
    "expiresAt": 1766436000,
    "apiKey": "clid_abc123..."
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}
API Key

The apiKey field is only returned during creation. Store it securely - it cannot be retrieved later.

Example:

const identity = await api.idp.identities.create({
  idTag: 'alice.cloudillo.net',
  email: 'alice@example.com'
})
console.log('Created identity:', identity.data.idTag)
console.log('API key:', identity.data.apiKey) // Store this!

Get Identity

GET /api/idp/identities/{id}

Get details of a specific identity.

Authentication: Required (must be owner or registrar)

Path Parameters:

Parameter Type Description
id string Identity tag (e.g., alice.cloudillo.net)

Response:

{
  "data": {
    "idTag": "alice.cloudillo.net",
    "email": "alice@example.com",
    "registrarIdTag": "admin.cloudillo.net",
    "ownerIdTag": null,
    "address": "cloudillo.example.com",
    "addressUpdatedAt": 1735000000,
    "status": "active",
    "createdAt": 1734900000,
    "updatedAt": 1735000000,
    "expiresAt": 1766436000
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Update Identity Address

PUT /api/idp/identities/{id}/address

Update the server address for an identity.

Authentication: Required (must be owner or registrar while pending)

Path Parameters:

Parameter Type Description
id string Identity tag

Request:

{
  "address": "new-server.example.com",
  "autoAddress": false
}
Field Type Required Description
address string No New server address
autoAddress boolean No If true and no address provided, use requester’s IP

Response:

{
  "data": {
    "idTag": "alice.cloudillo.net",
    "address": "new-server.example.com",
    "addressUpdatedAt": 1735000000,
    "status": "active"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Delete Identity

DELETE /api/idp/identities/{id}

Delete an identity. This action cannot be undone.

Authentication: Required (must be owner or registrar while pending)

Path Parameters:

Parameter Type Description
id string Identity tag to delete

Response:

{
  "data": {
    "deleted": true,
    "idTag": "alice.cloudillo.net"
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}
Permanent Deletion

Deleting an identity is permanent. The identity tag may be reused after deletion.

API Key Management

API keys provide programmatic access to identity operations.

Create API Key

POST /api/idp/api-keys

Create a new API key for the authenticated identity.

Authentication: Required

Request:

{
  "name": "Server automation",
  "expiresAt": 1766436000
}
Field Type Required Description
name string No Human-readable name
expiresAt number No Expiration timestamp (Unix seconds)

Response:

{
  "data": {
    "apiKey": {
      "id": 123,
      "idTag": "alice.cloudillo.net",
      "keyPrefix": "clid_abc",
      "name": "Server automation",
      "createdAt": 1735000000,
      "lastUsedAt": null,
      "expiresAt": 1766436000
    },
    "plaintextKey": "clid_abc123def456ghi789..."
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}
Store Key Securely

The plaintextKey is only shown once during creation. Store it securely - it cannot be retrieved later.

List API Keys

GET /api/idp/api-keys

List API keys for the authenticated identity.

Authentication: Required

Query Parameters:

Parameter Type Required Description
limit number No Maximum results
offset number No Pagination offset

Response:

{
  "data": [
    {
      "id": 123,
      "idTag": "alice.cloudillo.net",
      "keyPrefix": "clid_abc",
      "name": "Server automation",
      "createdAt": 1735000000,
      "lastUsedAt": 1735050000,
      "expiresAt": 1766436000
    }
  ],
  "time": 1735000000,
  "reqId": "req_abc123"
}

Get API Key

GET /api/idp/api-keys/{id}

Get details of a specific API key.

Authentication: Required

Path Parameters:

Parameter Type Description
id number API key ID

Response:

{
  "data": {
    "id": 123,
    "idTag": "alice.cloudillo.net",
    "keyPrefix": "clid_abc",
    "name": "Server automation",
    "createdAt": 1735000000,
    "lastUsedAt": 1735050000,
    "expiresAt": 1766436000
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Delete API Key

DELETE /api/idp/api-keys/{id}

Delete an API key. The key will immediately become invalid.

Authentication: Required

Path Parameters:

Parameter Type Description
id number API key ID to delete

Response:

{
  "data": {
    "deleted": true,
    "id": 123
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Authorization Model

IDP operations use a two-tier authorization model:

Role Access Duration
Owner Full control Permanent
Registrar Create, view, update, delete Only while status is pending

After an identity is activated (status changes from pending to active), the registrar loses control. Only the owner retains access.

Identity Status

Status Description
pending Created but not yet activated by owner
active Activated and operational
suspended Temporarily disabled by administrator

See Also

Tags API

The Tags API provides tag management and listing functionality for organizing files and content.

Endpoints

List Tags

GET /api/tags

List all tags used by the current user.

Query Parameters:

  • prefix (optional) - Filter tags by prefix
  • withCounts (optional) - Include usage counts
  • limit (optional) - Maximum tags to return

Response:

{
  "data": [
    {
      "tag": "project-alpha",
      "count": 15
    },
    {
      "tag": "important",
      "count": 8
    }
  ],
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/tags?prefix=proj-&withCounts=true"

File Tag Endpoints

Tags are primarily managed through the Files API.

Add Tag to File

PUT /api/files/{fileId}/tag/{tag}

Add a tag to a file.

Path Parameters:

  • fileId - File ID
  • tag - Tag name (URL-encoded)

Response:

{
  "data": {
    "tags": ["project-alpha", "important", "new-tag"]
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -X PUT -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/files/file_abc123/tag/important"

Remove Tag from File

DELETE /api/files/{fileId}/tag/{tag}

Remove a tag from a file.

Path Parameters:

  • fileId - File ID
  • tag - Tag name (URL-encoded)

Response:

{
  "data": {
    "tags": ["project-alpha", "important"]
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/files/file_abc123/tag/old-tag"

Client SDK Usage

import { createApiClient } from '@cloudillo/core'

const api = createApiClient({ idTag: 'alice.cloudillo.net', authToken: token })

// List all tags
const tags = await api.tags.list()

// List tags with prefix
const projectTags = await api.tags.list({ prefix: 'proj-' })

// Add tag to file
await api.files.addTag(fileId, 'important')

// Remove tag from file
await api.files.removeTag(fileId, 'old-tag')

Tag Naming Conventions

  • Tags are case-sensitive
  • Use lowercase with hyphens for consistency: project-alpha, q1-2025
  • Avoid special characters except hyphens and underscores
  • Keep tags concise but descriptive

Filtering Files by Tag

// List files with a specific tag
const files = await api.files.list({ tag: 'important' })

// Multiple tags (all must match)
const files = await api.files.list({ tags: ['project-alpha', 'active'] })

See Also

Push Notifications API

Overview

The Push Notifications API enables Web Push notifications for Cloudillo. It uses the VAPID (Voluntary Application Server Identification) protocol to securely deliver notifications to users’ browsers when they’re offline.

Push notifications are sent when actions are received for the user while they are not connected via WebSocket.

Endpoints

Get VAPID Public Key

GET /api/notification/vapid-public-key

Get the VAPID public key for subscribing to push notifications. The key is automatically generated on first request if it doesn’t exist.

Alternative path: GET /api/auth/vapid (alias)

Authentication: Required

Response:

{
  "vapidPublicKey": "BM5..."
}
Field Type Description
vapidPublicKey string Base64-encoded VAPID public key

Example:

const response = await api.notifications.getVapidPublicKey()
const vapidPublicKey = response.vapidPublicKey

// Use with browser Push API
const subscription = await registration.pushManager.subscribe({
  userVisibleOnly: true,
  applicationServerKey: vapidPublicKey
})

Register Subscription

POST /api/notification/subscription

Register a push notification subscription. The subscription is stored and used to send notifications when the user is offline.

Authentication: Required

Request:

{
  "subscription": {
    "endpoint": "https://fcm.googleapis.com/fcm/send/...",
    "expirationTime": null,
    "keys": {
      "p256dh": "BKgS...",
      "auth": "Qs..."
    }
  }
}
Field Type Required Description
subscription.endpoint string Yes Push service endpoint URL
subscription.expirationTime number No Expiration timestamp (Unix milliseconds)
subscription.keys.p256dh string Yes P-256 public key (base64url)
subscription.keys.auth string Yes Auth secret (base64url)

Response:

{
  "id": 12345
}
Field Type Description
id number Subscription ID for later deletion

Example:

// After getting VAPID public key and subscribing via Push API
const browserSubscription = await registration.pushManager.subscribe({
  userVisibleOnly: true,
  applicationServerKey: vapidPublicKey
})

// Send subscription to server
const result = await api.notifications.subscribe({
  subscription: browserSubscription.toJSON()
})

// Store subscription ID for later unsubscription
localStorage.setItem('pushSubscriptionId', result.id)

Unregister Subscription

DELETE /api/notification/subscription/{id}

Remove a push notification subscription. The subscription will no longer receive notifications.

Authentication: Required

Path Parameters:

Parameter Type Description
id number Subscription ID to delete

Response: 204 No Content

Example:

const subscriptionId = localStorage.getItem('pushSubscriptionId')
if (subscriptionId) {
  await api.notifications.unsubscribe(subscriptionId)
  localStorage.removeItem('pushSubscriptionId')
}

Complete Integration Example

// 1. Check if push is supported
if (!('PushManager' in window)) {
  console.log('Push notifications not supported')
  return
}

// 2. Get service worker registration
const registration = await navigator.serviceWorker.ready

// 3. Get VAPID public key from server
const { vapidPublicKey } = await api.notifications.getVapidPublicKey()

// 4. Subscribe via browser Push API
const subscription = await registration.pushManager.subscribe({
  userVisibleOnly: true,
  applicationServerKey: urlBase64ToUint8Array(vapidPublicKey)
})

// 5. Send subscription to Cloudillo server
const result = await api.notifications.subscribe({
  subscription: subscription.toJSON()
})

console.log('Push subscription registered with ID:', result.id)

// Helper function to convert base64 key
function urlBase64ToUint8Array(base64String: string): Uint8Array {
  const padding = '='.repeat((4 - base64String.length % 4) % 4)
  const base64 = (base64String + padding)
    .replace(/-/g, '+')
    .replace(/_/g, '/')
  const rawData = atob(base64)
  return Uint8Array.from([...rawData].map(char => char.charCodeAt(0)))
}

Notification Settings

Users can control which notification types they receive through the Settings API:

Setting Type Description
notify.push.message boolean Receive notifications for new messages
notify.push.mention boolean Receive notifications when mentioned
notify.push.reaction boolean Receive notifications for reactions
notify.push.connection boolean Receive notifications for connection requests
notify.push.follow boolean Receive notifications for new followers

Example:

// Disable reaction notifications
await api.settings.set('notify.push.reaction', false)

// Get current notification settings
const settings = await api.settings.list()
const pushSettings = Object.entries(settings)
  .filter(([key]) => key.startsWith('notify.push.'))

Web Push Standards

The implementation follows these RFCs:

RFC Title
RFC 8292 VAPID for Web Push
RFC 8188 Encrypted Content-Encoding for HTTP
RFC 8291 Message Encryption for Web Push

See Also

Trash API

The Trash API manages soft-deleted files, allowing users to restore or permanently delete items.

Concepts

When a file is deleted via DELETE /api/files/{fileId}, it is moved to the trash rather than being permanently deleted. This allows users to:

  • Recover accidentally deleted files
  • Review deleted items before permanent removal
  • Empty the trash to free up storage

Files remain in trash until explicitly restored or permanently deleted.

Endpoints

List Trashed Files

GET /api/files?parentId=__trash__

List all files currently in the trash.

Query Parameters:

  • parentId=__trash__ - Required to list trash
  • limit (optional) - Maximum files to return

Response:

{
  "data": [
    {
      "fileId": "file_abc123",
      "fileName": "document.pdf",
      "contentType": "application/pdf",
      "deletedAt": "2025-01-15T10:30:00Z",
      "originalParentId": "folder_xyz"
    }
  ],
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/files?parentId=__trash__&limit=50"

Restore File from Trash

POST /api/files/{fileId}/restore

Restore a file from the trash.

Path Parameters:

  • fileId - File ID to restore

Request Body:

{
  "parentId": "folder_xyz"
}

If parentId is omitted or null, the file is restored to the root folder.

Response:

{
  "data": {
    "fileId": "file_abc123",
    "parentId": "folder_xyz",
    "restoredAt": "2025-01-15T11:00:00Z"
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"parentId": "folder_xyz"}' \
  "https://cl-o.alice.cloudillo.net/api/files/file_abc123/restore"

Permanently Delete File

DELETE /api/files/{fileId}?permanent=true

Permanently delete a file from trash. The file must already be in trash.

Path Parameters:

  • fileId - File ID to permanently delete

Query Parameters:

  • permanent=true - Required for permanent deletion

Response:

{
  "data": {
    "fileId": "file_abc123",
    "permanent": true
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/files/file_abc123?permanent=true"

Empty Trash

DELETE /api/trash

Permanently delete all files in the trash.

Response:

{
  "data": {
    "deletedCount": 15
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://cl-o.alice.cloudillo.net/api/trash"

Client SDK Usage

import { createApiClient } from '@cloudillo/core'

const api = createApiClient({ idTag: 'alice.cloudillo.net', authToken: token })

// Move file to trash (soft delete)
await api.files.delete(fileId)

// List trashed files
const trashed = await api.trash.list()

// Restore a file
await api.files.restore(fileId, 'folder_xyz')

// Restore to root
await api.files.restore(fileId)

// Permanently delete a single file
await api.files.permanentDelete(fileId)

// Empty entire trash
const result = await api.trash.empty()
console.log(`Deleted ${result.deletedCount} files`)

File Lifecycle

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”     DELETE      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚    Active    β”‚ ───────────────>β”‚    Trash     β”‚
β”‚    File      β”‚                 β”‚    (soft)    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚
                                        β”‚ restore
                                        β”‚
                β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                β”‚
                v
        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
        β”‚    Active    β”‚
        β”‚    File      β”‚
        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜


β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  DELETE?permanent  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚    Trash     β”‚ ──────────────────>β”‚   Deleted    β”‚
β”‚    (soft)    β”‚                    β”‚  (permanent) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Automatic Trash Cleanup

Info

Automatic trash cleanup (retention period) is planned for future releases. Currently, files remain in trash indefinitely until manually restored or deleted.

See Also

Communities API

The Communities API enables creation and management of community profiles.

Concepts

Communities are special profile types that can have multiple members with different roles. They provide:

  • Shared content feeds
  • Member management with role hierarchy
  • Community-specific settings

Community Roles

Roles form a hierarchy from least to most privileged:

Role Level Description
public 0 Anyone (non-member)
follower 1 Following the community
supporter 2 Supporter/subscriber
contributor 3 Can create content
moderator 4 Can moderate content
leader 5 Full administrative access

Endpoints

Create Community

PUT /api/profiles/{idTag}

Create a new community profile.

Path Parameters:

  • idTag - Identity tag for the new community (e.g., mygroup.cloudillo.net)

Request Body:

{
  "type": "community",
  "name": "My Community",
  "bio": "A community for enthusiasts",
  "ownerIdTag": "alice.cloudillo.net"
}

Response:

{
  "data": {
    "idTag": "mygroup.cloudillo.net",
    "name": "My Community",
    "type": "community",
    "bio": "A community for enthusiasts",
    "createdAt": "2025-01-15T10:30:00Z"
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -X PUT -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"type":"community","name":"My Community","ownerIdTag":"alice.cloudillo.net"}' \
  "https://cl-o.alice.cloudillo.net/api/profiles/mygroup.cloudillo.net"

Verify Community Availability

POST /api/profiles/verify

Check if a community identity is available.

Request Body:

{
  "type": "community",
  "idTag": "mygroup.cloudillo.net"
}

Response:

{
  "data": {
    "available": true,
    "errors": [],
    "serverAddresses": ["192.168.1.100"]
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

If not available:

{
  "data": {
    "available": false,
    "errors": ["E-PROFILE-EXISTS"],
    "serverAddresses": []
  }
}

Client SDK Usage

import { createApiClient } from '@cloudillo/core'

const api = createApiClient({ idTag: 'alice.cloudillo.net', authToken: token })

// Verify community name is available
const verification = await api.profile.verify({
  type: 'community',
  idTag: 'mygroup.cloudillo.net'
})

if (verification.available) {
  // Create the community
  const community = await api.communities.create('mygroup.cloudillo.net', {
    type: 'community',
    name: 'My Community',
    ownerIdTag: 'alice.cloudillo.net'
  })
}

Member Management

Member roles are managed through the Profiles API.

Update Member Role

// As community leader/moderator
await api.profiles.adminUpdate('member.cloudillo.net', {
  roles: ['contributor']
})

List Community Members

const members = await api.profiles.list({
  community: 'mygroup.cloudillo.net',
  role: 'contributor'
})

Role-Based Permissions

Use the ROLE_LEVELS constant from @cloudillo/types for permission checks:

import { ROLE_LEVELS, CommunityRole } from '@cloudillo/types'

function hasPermission(userRole: CommunityRole, requiredRole: CommunityRole): boolean {
  return ROLE_LEVELS[userRole] >= ROLE_LEVELS[requiredRole]
}

// Check if user can moderate
if (hasPermission(userRole, 'moderator')) {
  // Allow moderation actions
}

See Also

Admin API

Overview

The Admin API provides administrative operations for system administrators.

Warning

Admin endpoints require elevated privileges. These operations are restricted to users with the SADM (system admin) role.

Endpoints

List Tenants

GET /api/admin/tenants

List all tenants (identities) managed by this server.

Authentication: Required (admin role)

Query Parameters:

  • limit - Maximum results (default: 20)
  • offset - Skip N results for pagination

Response:

{
  "data": [
    {
      "tnId": 12345,
      "idTag": "alice.cloudillo.net",
      "name": "Alice",
      "type": "person",
      "profilePic": "b1~abc123",
      "createdAt": "2025-01-01T00:00:00Z"
    },
    {
      "tnId": 12346,
      "idTag": "bob.cloudillo.net",
      "name": "Bob",
      "type": "person",
      "createdAt": "2025-01-02T00:00:00Z"
    }
  ],
  "time": "2025-01-15T10:30:00Z"
}

Example:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  "https://cl-o.admin.cloudillo.net/api/admin/tenants?limit=50"

Send Password Reset

POST /api/admin/tenants/{id_tag}/password-reset

Send a password reset email to a tenant.

Authentication: Required (admin role)

Path Parameters:

  • id_tag - Identity tag of the tenant

Response:

{
  "data": {
    "sent": true
  },
  "time": "2025-01-15T10:30:00Z"
}

Example:

curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  "https://cl-o.admin.cloudillo.net/api/admin/tenants/alice.cloudillo.net/password-reset"

Send Test Email

POST /api/admin/email/test

Send a test email to verify SMTP configuration.

Authentication: Required (admin role)

Request Body:

{
  "to": "admin@example.com"
}

Response:

{
  "data": {
    "sent": true
  },
  "time": "2025-01-15T10:30:00Z"
}

Example:

curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"to":"admin@example.com"}' \
  "https://cl-o.admin.cloudillo.net/api/admin/email/test"

Update Profile (Admin)

PATCH /api/admin/profiles/{id_tag}

Admin update of a profile (roles, status, ban metadata).

Authentication: Required (admin role)

Path Parameters:

  • id_tag - Identity tag of the profile to update

Request Body:

{
  "name": "Updated Name",
  "status": "Suspended",
  "roles": ["user", "moderator"],
  "banExpiresAt": "2025-02-01T00:00:00Z",
  "banReason": "Terms of service violation"
}

Profile Status Values:

Status Description
Active Normal active account
Trusted Verified/trusted account
Blocked Blocked from interactions
Muted Content hidden from feeds
Suspended Account suspended (temporary)
Banned Account banned (permanent)

Response:

{
  "data": {
    "idTag": "user.cloudillo.net",
    "name": "Updated Name",
    "status": "Suspended",
    "roles": ["user", "moderator"]
  },
  "time": "2025-01-15T10:30:00Z"
}

Client SDK Usage

import { createApiClient } from '@cloudillo/core'

const api = createApiClient({ idTag: 'admin.cloudillo.net', authToken: adminToken })

// List all tenants
const tenants = await api.admin.listTenants({ limit: 100 })

// Search tenants
const results = await api.admin.listTenants({ q: 'alice' })

// Send password reset
await api.admin.sendPasswordReset('alice.cloudillo.net')

// Send test email
await api.admin.sendTestEmail('admin@example.com')

// Suspend a user
await api.profiles.adminUpdate('baduser.cloudillo.net', {
  status: 'S',
  ban_reason: 'Spam'
})

Security Considerations

  • Admin endpoints require admin role authentication
  • All admin actions are logged for audit purposes
  • Password reset emails are rate-limited
  • Suspension requires a reason for accountability

See Also

IDP Management API

The IDP (Identity Provider) Management API enables identity provider administrators to manage identities and API keys for their hosted identities.

Info

This API is for identity provider administrators who host identities for other users (e.g., cloudillo.net service). For end-user identity operations, see IDP API.

Endpoints

List Managed Identities

GET /api/idp/identities

List all identities managed by this identity provider.

Query Parameters:

  • q (optional) - Search query
  • status (optional) - Filter by status
  • cursor (optional) - Pagination cursor
  • limit (optional) - Maximum results

Response:

{
  "data": [
    {
      "idTag": "alice.cloudillo.net",
      "email": "alice@example.com",
      "status": "active",
      "createdAt": "2025-01-01T00:00:00Z",
      "ownerIdTag": null,
      "dyndns": true
    },
    {
      "idTag": "bob.cloudillo.net",
      "email": "bob@example.com",
      "status": "active",
      "createdAt": "2025-01-02T00:00:00Z",
      "ownerIdTag": "alice.cloudillo.net",
      "dyndns": false
    }
  ],
  "time": 1705315800,
  "req_id": "req_xyz"
}

Example:

curl -H "Authorization: Bearer $IDP_TOKEN" \
  "https://cl-o.cloudillo.net/api/idp/identities?limit=50"

Create Identity

POST /api/idp/identities

Create a new identity under this provider.

Request Body:

{
  "idTag": "newuser.cloudillo.net",
  "email": "newuser@example.com",
  "ownerIdTag": "alice.cloudillo.net",
  "createApiKey": true,
  "apiKeyName": "Initial Key"
}

Response:

{
  "data": {
    "idTag": "newuser.cloudillo.net",
    "email": "newuser@example.com",
    "status": "pending",
    "createdAt": "2025-01-15T10:30:00Z",
    "apiKey": "clak_abc123xyz..."
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}
Warning

The apiKey is only returned once when createApiKey: true. Store it securely.

Example:

curl -X POST -H "Authorization: Bearer $IDP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"idTag":"newuser.cloudillo.net","email":"newuser@example.com","createApiKey":true}' \
  "https://cl-o.cloudillo.net/api/idp/identities"

Get Identity Details

GET /api/idp/identities/{idTag}

Get details for a specific managed identity.

Path Parameters:

  • idTag - Identity tag (URL-encoded)

Response:

{
  "data": {
    "idTag": "alice.cloudillo.net",
    "email": "alice@example.com",
    "status": "active",
    "createdAt": "2025-01-01T00:00:00Z",
    "lastLoginAt": "2025-01-15T10:30:00Z",
    "ownerIdTag": null,
    "dyndns": true
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Update Identity

PATCH /api/idp/identities/{idTag}

Update identity settings.

Path Parameters:

  • idTag - Identity tag (URL-encoded)

Request Body:

{
  "dyndns": true
}

Response:

{
  "data": {
    "idTag": "alice.cloudillo.net",
    "dyndns": true
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Update Identity Address

PUT /api/idp/identities/{idTag}/address

Update the DNS address mapping for a managed identity. This is used for dynamic DNS updates when self-hosting.

Path Parameters:

  • idTag - Identity tag (URL-encoded)

Request Body:

{
  "address": "192.168.1.100"
}

Response:

{
  "data": {
    "idTag": "alice.cloudillo.net",
    "address": "192.168.1.100"
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Delete Identity

DELETE /api/idp/identities/{idTag}

Delete a managed identity.

Path Parameters:

  • idTag - Identity tag (URL-encoded)

Response:

{
  "data": null,
  "time": 1705315800,
  "req_id": "req_xyz"
}

List API Keys for Identity

GET /api/idp/api-keys?idTag={idTag}

List API keys for a specific managed identity.

Query Parameters:

  • idTag - Identity tag to list keys for

Response:

{
  "data": [
    {
      "keyId": 1,
      "name": "Production Key",
      "createdAt": "2025-01-01T00:00:00Z",
      "lastUsedAt": "2025-01-15T10:30:00Z"
    }
  ],
  "time": 1705315800,
  "req_id": "req_xyz"
}

Create API Key for Identity

POST /api/idp/api-keys

Create a new API key for a managed identity.

Request Body:

{
  "idTag": "alice.cloudillo.net",
  "name": "New API Key"
}

Response:

{
  "data": {
    "keyId": 2,
    "name": "New API Key",
    "apiKey": "clak_newkey123...",
    "createdAt": "2025-01-15T10:30:00Z"
  },
  "time": 1705315800,
  "req_id": "req_xyz"
}

Revoke API Key

DELETE /api/idp/api-keys/{keyId}?idTag={idTag}

Revoke an API key for a managed identity.

Path Parameters:

  • keyId - Key ID to revoke

Query Parameters:

  • idTag - Identity tag the key belongs to

Response:

{
  "data": null,
  "time": 1705315800,
  "req_id": "req_xyz"
}

Client SDK Usage

import { createApiClient } from '@cloudillo/core'

const api = createApiClient({ idTag: 'cloudillo.net', authToken: idpToken })

// List managed identities
const identities = await api.idpManagement.listIdentities({ limit: 100 })

// Create new identity with API key
const newIdentity = await api.idpManagement.createIdentity({
  idTag: 'newuser.cloudillo.net',
  email: 'newuser@example.com',
  createApiKey: true,
  apiKeyName: 'Initial Key'
})
console.log('API Key:', newIdentity.apiKey) // Store securely!

// Get identity details
const identity = await api.idpManagement.getIdentity('alice.cloudillo.net')

// Update identity settings
await api.idpManagement.updateIdentity('alice.cloudillo.net', { dyndns: true })

// Delete identity
await api.idpManagement.deleteIdentity('olduser.cloudillo.net')

// Manage API keys
const keys = await api.idpManagement.listApiKeys('alice.cloudillo.net')
const newKey = await api.idpManagement.createApiKey({
  idTag: 'alice.cloudillo.net',
  name: 'Backup Key'
})
await api.idpManagement.deleteApiKey(keyId, 'alice.cloudillo.net')

Use Cases

Provisioning New Users

async function provisionUser(email: string, subdomain: string) {
  const idTag = `${subdomain}.cloudillo.net`

  // Create identity with initial API key
  const result = await api.idpManagement.createIdentity({
    idTag,
    email,
    createApiKey: true,
    apiKeyName: 'Setup Key'
  })

  // Send setup instructions with API key
  await sendSetupEmail(email, {
    idTag,
    apiKey: result.apiKey
  })

  return result
}

Automating Identity Management

// Deactivate inactive users
const identities = await api.idpManagement.listIdentities()

for (const identity of identities.data) {
  const daysSinceLogin = daysSince(identity.lastLoginAt)
  if (daysSinceLogin > 365) {
    await api.idpManagement.deleteIdentity(identity.idTag)
  }
}

See Also

WebSocket API

Overview

Cloudillo provides three WebSocket endpoints for real-time features: Bus (pub/sub messaging), RTDB (real-time database), and CRDT (collaborative documents).

Endpoint Purpose
/ws/bus Event bus for notifications and batch updates
/ws/rtdb/{file_id} Real-time database sync
/ws/crdt/{doc_id} CRDT collaboration (Yjs protocol)

Message Bus (/ws/bus)

The message bus provides pub/sub messaging, presence tracking, and notifications.

Connection

import * as cloudillo from '@cloudillo/core'

await cloudillo.init('my-app')

const bus = cloudillo.openMessageBus({
  channels: ['notifications', 'presence']
})

Protocol

Client β†’ Server:

// Subscribe to channel
{
  type: 'subscribe',
  channel: 'notifications'
}

// Unsubscribe
{
  type: 'unsubscribe',
  channel: 'notifications'
}

// Publish message
{
  type: 'publish',
  channel: 'notifications',
  data: {
    event: 'new-post',
    actionId: 'act_123'
  }
}

// Set presence
{
  type: 'presence',
  status: 'online',
  data: {
    currentPage: '/posts'
  }
}

Server β†’ Client:

// Message received
{
  type: 'message',
  channel: 'notifications',
  data: {
    event: 'new-post',
    actionId: 'a1~abc123'
  },
  from: 'alice@example.com',
  timestamp: '2025-01-01T12:00:00Z'
}

// Presence update
{
  type: 'presence',
  userId: 'bob@example.com',
  status: 'online',
  data: {...}
}

// Subscription confirmed
{
  type: 'subscribed',
  channel: 'notifications'
}

Usage Example

const bus = cloudillo.openMessageBus()

// Subscribe to notifications
bus.subscribe('notifications', (message) => {
  console.log('Notification:', message)
  showToast(message.data)
})

// Publish typing indicator
bus.publish('typing', {
  conversationId: 'conv_123',
  typing: true
})

// Set presence
bus.setPresence('online', {
  currentPage: window.location.pathname
})

// Listen for presence changes
bus.on('presence', (update) => {
  console.log(`${update.userId} is ${update.status}`)
})

Real-Time Database (/ws/rtdb/:fileId)

Real-time synchronization of structured data. See RTDB for full documentation.

Connection

import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

const rtdb = new RtdbClient({
  dbId: 'my-database-id',
  auth: {
    getToken: () => bus.accessToken
  },
  serverUrl: getRtdbUrl(bus.idTag!, 'my-database-id', bus.accessToken!)
})

await rtdb.connect()

Protocol

Client β†’ Server:

// Query documents
{
  type: 'query',
  id: 1,
  path: 'todos',
  filter: { equals: { field: 'completed', value: false } },
  sort: { field: 'createdAt', direction: 'desc' },
  limit: 20
}

// Subscribe to path
{
  type: 'subscribe',
  id: 2,
  path: 'todos',
  filter: { equals: { field: 'completed', value: false } }
}

// Get single document
{
  type: 'get',
  id: 3,
  path: 'todos/todo_123'
}

// Transaction (wraps create/update/delete)
{
  type: 'transaction',
  id: 4,
  operations: [
    { type: 'create', path: 'todos', data: { title: 'Learn Cloudillo', completed: false } },
    { type: 'update', path: 'todos/todo_123', data: { completed: true } },
    { type: 'delete', path: 'todos/todo_456' }
  ]
}

// Lock / unlock document
{ type: 'lock', id: 5, path: 'todos/todo_123', mode: 'hard' }
{ type: 'unlock', id: 6, path: 'todos/todo_123' }

// Ping
{ type: 'ping', id: 7 }

Server β†’ Client:

// Query result
{
  type: 'queryResult',
  id: 1,
  data: [
    { _id: 'todo_123', title: 'Learn Cloudillo', completed: false },
    { _id: 'todo_456', title: 'Build app', completed: false }
  ],
  total: 2
}

// Get result
{
  type: 'getResult',
  id: 3,
  data: { _id: 'todo_123', title: 'Learn Cloudillo', completed: false }
}

// Subscribe result (initial data)
{
  type: 'subscribeResult',
  id: 2,
  subscriptionId: 'sub_abc123',
  data: [
    { _id: 'todo_123', title: 'Learn Cloudillo', completed: false }
  ]
}

// Change event (real-time update)
{
  type: 'change',
  subscriptionId: 'sub_abc123',
  event: {
    action: 'create',  // create | update | delete | lock | unlock | ready
    path: 'todos',
    data: { _id: 'todo_789', title: 'New todo', completed: false }
  }
}

// Transaction result
{
  type: 'transactionResult',
  id: 4,
  results: [
    { id: 'todo_new_001' },
    { id: 'todos/todo_123' },
    { id: 'todos/todo_456' }
  ]
}

// Lock result
{ type: 'lockResult', id: 5, locked: true }

// Error
{
  type: 'error',
  id: 4,
  code: 'permission_denied',
  message: 'Insufficient permissions'
}

// Pong
{ type: 'pong', id: 7 }

Collaborative Documents (/ws/crdt/:docId)

CRDT synchronization using Yjs protocol. See CRDT for full documentation.

Connection

import * as Y from 'yjs'
import * as cloudillo from '@cloudillo/core'

const yDoc = new Y.Doc()
const { provider } = await cloudillo.openYDoc(yDoc, 'my-document')

Protocol

The CRDT endpoint uses the y-websocket protocol:

Binary messages:

  • Sync Step 1: Client sends document state
  • Sync Step 2: Server responds with missing updates
  • Updates: Incremental document changes
  • Awareness: Cursor positions, selections, user info

Awareness format:

{
  user: {
    name: 'Alice Johnson',
    idTag: 'alice@example.com',
    color: '#ff6b6b'
  },
  cursor: {
    line: 10,
    column: 5
  },
  selection: {
    start: { line: 10, column: 5 },
    end: { line: 10, column: 10 }
  }
}

Usage Example

const yDoc = new Y.Doc()
const { provider } = await cloudillo.openYDoc(yDoc, 'doc_123')

// Use shared types
const yText = yDoc.getText('content')
yText.insert(0, 'Hello, collaborative world!')

// Listen for remote changes
yText.observe((event) => {
  console.log('Text changed by:', event.transaction.origin)
})

// Awareness (see other users)
provider.awareness.on('change', () => {
  const states = provider.awareness.getStates()
  states.forEach((state, clientId) => {
    console.log(`User ${state.user.name} at cursor ${state.cursor}`)
  })
})

// Set own awareness
provider.awareness.setLocalState({
  user: {
    name: cloudillo.name,
    idTag: cloudillo.idTag,
    color: '#' + Math.random().toString(16).slice(2, 8)
  },
  cursor: { line: 5, column: 10 }
})

Authentication

All WebSocket connections require authentication via query parameter:

wss://server.com/ws/bus?token=eyJhbGc...
wss://server.com/ws/rtdb/file_123?token=eyJhbGc...
wss://server.com/ws/crdt/doc_123?token=eyJhbGc...

The client libraries handle this automatically.

Reconnection

All WebSocket connections implement automatic reconnection with exponential backoff:

  • Initial retry: 1 second
  • Max retry delay: 30 seconds
  • Exponential factor: 1.5

Handling reconnection:

provider.on('status', ({ status }) => {
  if (status === 'connected') {
    console.log('Connected to server')
  } else if (status === 'disconnected') {
    console.log('Disconnected, will retry...')
  }
})

Best Practices

1. Clean Up Connections

// React example
useEffect(() => {
  const bus = cloudillo.openMessageBus()

  bus.subscribe('notifications', handleNotification)

  return () => {
    bus.close() // Clean up on unmount
  }
}, [])

2. Handle Connection State

const [connected, setConnected] = useState(false)

provider.on('status', ({ status }) => {
  setConnected(status === 'connected')
})

// Show offline indicator
{!connected && <div className="offline-banner">Reconnecting...</div>}

3. Batch Updates

// ❌ Don't send updates individually
for (let i = 0; i < 100; i++) {
  yText.insert(i, 'x')
}

// βœ… Batch in a transaction
yDoc.transact(() => {
  for (let i = 0; i < 100; i++) {
    yText.insert(i, 'x')
  }
})

See Also

  • RTDB - Real-time database documentation
  • CRDT - Collaborative editing documentation
  • Authentication - WebSocket authentication

RTDB (Real-Time Database)

The Cloudillo RTDB provides a Firebase-like real-time database with TypeScript support.

RTDB vs CRDT

RTDB is best for structured data with queries (todos, settings, lists). For collaborative document editing where multiple users edit simultaneously, see CRDT. Compare both in Data Storage & Access.

Installation

pnpm add @cloudillo/rtdb

Quick Start

import { getAppBus } from '@cloudillo/core'
import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

// Initialize Cloudillo
const bus = getAppBus()
await bus.init('my-app')

// Create RTDB client
const rtdb = new RtdbClient({
  dbId: 'my-database-id',
  auth: {
    getToken: () => bus.accessToken
  },
  serverUrl: getRtdbUrl(bus.idTag!, 'my-database-id', bus.accessToken!)
})

// Connect to the database
await rtdb.connect()

// Get collection reference
const todos = rtdb.collection('todos')

// Create document
const batch = rtdb.batch()
batch.create(todos, {
  title: 'Learn Cloudillo RTDB',
  completed: false,
  createdAt: Date.now()
})
await batch.commit()

// Subscribe to changes
todos.onSnapshot((snapshot) => {
  console.log('Todos:', snapshot.docs.map(doc => doc.data()))
})

RtdbClient Constructor

interface RtdbClientOptions {
  dbId: string                    // Database/file ID
  auth: {
    getToken: () => string | undefined | Promise<string | undefined>
  }
  serverUrl: string               // WebSocket URL
  options?: {
    enableCache?: boolean         // Default: false
    reconnect?: boolean           // Default: true
    reconnectDelay?: number       // Default: 1000ms
    maxReconnectDelay?: number    // Default: 30000ms
    debug?: boolean               // Default: false
  }
}

Example with all options:

import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

const rtdb = new RtdbClient({
  dbId: 'my-database-id',
  auth: {
    // Token provider function - called when connection needs auth
    getToken: async () => {
      // Can be sync or async
      return bus.accessToken
    }
  },
  serverUrl: getRtdbUrl(bus.idTag!, 'my-database-id', bus.accessToken!),
  options: {
    enableCache: true,        // Enable local caching
    reconnect: true,          // Auto-reconnect on disconnect
    reconnectDelay: 1000,     // Initial reconnect delay
    maxReconnectDelay: 30000, // Max reconnect delay (exponential backoff)
    debug: false              // Enable debug logging
  }
})

Connection Management

// Connect to database
await rtdb.connect()

// Disconnect
await rtdb.disconnect()

// Check connection status
if (rtdb.isConnected()) {
  console.log('Connected')
}

// Diagnostics
console.log('Pending requests:', rtdb.getPendingRequests())
console.log('Active subscriptions:', rtdb.getActiveSubscriptions())

Core Concepts

Collections

Collections are groups of documents, similar to tables in SQL.

const users = rtdb.collection('users')
const posts = rtdb.collection('posts')
const comments = rtdb.collection('comments')

// Typed collections
interface Todo {
  title: string
  completed: boolean
  createdAt: number
}
const todos = rtdb.collection<Todo>('todos')

Documents

Documents are individual records accessed by path.

// Reference a document by path
const userDoc = rtdb.ref('users/alice')

// Get document data
const snapshot = await userDoc.get()
if (snapshot.exists) {
  console.log(snapshot.data())
}

CRUD Operations

Create

Use batch operations to create documents:

const todos = rtdb.collection('todos')
const batch = rtdb.batch()

// Create with auto-generated ID
batch.create(todos, {
  title: 'New task',
  completed: false
})

// Create with ref for tracking
batch.create(todos, {
  title: 'Another task',
  completed: false
}, { ref: 'task-ref' })

// Commit returns results with IDs
const results = await batch.commit()
console.log('Created IDs:', results.map(r => r.id))

Read

// Get single document
const doc = rtdb.ref('todos/task_123')
const snapshot = await doc.get()
if (snapshot.exists) {
  console.log(snapshot.data())
}

// Query collection
const todos = rtdb.collection('todos')
const results = await todos.get()
results.docs.forEach(doc => {
  console.log(doc.id, doc.data())
})

Update

const batch = rtdb.batch()
const todoRef = rtdb.ref('todos/task_123')

// Partial update
batch.update(todoRef, {
  completed: true
})

await batch.commit()

Delete

const batch = rtdb.batch()
const todoRef = rtdb.ref('todos/task_123')

batch.delete(todoRef)

await batch.commit()

Queries

Query Options

interface QueryOptions {
  filter?: {
    equals?: Record<string, any>
    notEquals?: Record<string, any>
    greaterThan?: Record<string, any>
    lessThan?: Record<string, any>
    greaterThanOrEqual?: Record<string, any>
    lessThanOrEqual?: Record<string, any>
    in?: Record<string, any[]>
    notIn?: Record<string, any[]>
    arrayContains?: Record<string, any>
    arrayContainsAny?: Record<string, any[]>
    arrayContainsAll?: Record<string, any[]>
  }
  sort?: Array<{ field: string; ascending: boolean }>
  limit?: number
  offset?: number
}

You can also build queries using the chainable .where() builder:

type WhereFilterOp =
  | '=='  | '!='
  | '<'   | '>'
  | '<='  | '>='
  | 'in'  | 'not-in'
  | 'array-contains'
  | 'array-contains-any'
  | 'array-contains-all'

collection.where(field: string, op: WhereFilterOp, value: any)

Filtering

const todos = rtdb.collection('todos')

// Filter with options object
const incomplete = await todos.query({
  filter: {
    equals: { completed: false }
  }
})

// Filter with chainable .where() builder
const incomplete2 = await todos
  .where('completed', '==', false)
  .get()

Filter Operators

// Equality
const active = await todos.where('status', '==', 'active').get()
const notDone = await todos.where('status', '!=', 'done').get()

// Comparison
const highPriority = await todos.where('priority', '>', 3).get()
const recent = await todos.where('createdAt', '>=', lastWeek).get()

// Set membership
const selected = await todos
  .where('status', 'in', ['active', 'pending'])
  .get()
const excluded = await todos
  .where('status', 'not-in', ['archived', 'deleted'])
  .get()

// Array filters
const tagged = await todos
  .where('tags', 'array-contains', 'urgent')
  .get()
const anyTag = await todos
  .where('tags', 'array-contains-any', ['urgent', 'important'])
  .get()
const allTags = await todos
  .where('tags', 'array-contains-all', ['frontend', 'bug'])
  .get()

// Chain multiple filters
const filtered = await todos
  .where('completed', '==', false)
  .where('priority', '>', 2)
  .get()

Sorting

// Sort ascending
const sorted = await todos.query({
  sort: [{ field: 'createdAt', ascending: true }]
})

// Sort descending
const newest = await todos.query({
  sort: [{ field: 'createdAt', ascending: false }]
})

// Multiple sorts
const prioritized = await todos.query({
  sort: [
    { field: 'priority', ascending: false },
    { field: 'createdAt', ascending: true }
  ]
})

Pagination

// Limit results
const first10 = await todos.query({
  limit: 10
})

// Pagination with offset
const page2 = await todos.query({
  limit: 20,
  offset: 20
})

Combined Queries

const results = await todos.query({
  filter: {
    equals: { completed: false }
  },
  sort: [{ field: 'createdAt', ascending: false }],
  limit: 10
})

Real-Time Subscriptions

Collection Subscriptions

const todos = rtdb.collection('todos')

// Subscribe to all documents
const unsubscribe = todos.onSnapshot((snapshot) => {
  console.log('Total todos:', snapshot.size)

  snapshot.docs.forEach(doc => {
    console.log(doc.id, doc.data())
  })

  // Track changes
  const changes = snapshot.docChanges()
  changes.forEach(change => {
    switch (change.type) {
      case 'added':
        console.log('New:', change.doc.id)
        break
      case 'modified':
        console.log('Updated:', change.doc.id)
        break
      case 'removed':
        console.log('Deleted:', change.doc.id)
        break
    }
  })
})

// Unsubscribe later
unsubscribe()

Document Subscriptions

const todoRef = rtdb.ref('todos/task_123')

const unsubscribe = todoRef.onSnapshot((snapshot) => {
  if (snapshot.exists) {
    console.log('Todo updated:', snapshot.data())
  } else {
    console.log('Todo deleted')
  }
})

Filtered Subscriptions

// Using options object
const unsubscribe = todos.subscribe({
  filter: {
    equals: { completed: false }
  }
}, (snapshot) => {
  console.log('Incomplete todos:', snapshot.size)
})

// Using .where() builder
const unsubscribe2 = todos
  .where('completed', '==', false)
  .onSnapshot((snapshot) => {
    console.log('Incomplete todos:', snapshot.size)
  })

Document Locking

Lock documents for exclusive or advisory editing access.

Lock Modes

  • soft β€” Advisory lock. Other clients can still write, but are notified that the document is locked.
  • hard β€” Enforced lock. The server rejects writes from other clients while the lock is held.

Locking and Unlocking

const docRef = rtdb.ref('todos/task_123')

// Acquire a soft (advisory) lock
const result = await docRef.lock('soft')
// result: { locked: true }

// Acquire a hard (exclusive) lock
const result2 = await docRef.lock('hard')
// result: { locked: true }

// If already locked by another client
const result3 = await docRef.lock('hard')
// result: { locked: false, holder: 'bob@example.com', mode: 'hard' }

// Release the lock
await docRef.unlock()

Lock Result

interface LockResult {
  locked: boolean         // Whether the lock was acquired
  holder?: string         // Identity of current lock holder (if denied)
  mode?: 'soft' | 'hard' // Lock mode of existing lock (if denied)
}

Lock Events

Listen for lock changes on a document using the onLock callback in snapshot options:

const docRef = rtdb.ref('todos/task_123')

const unsubscribe = docRef.onSnapshot({
  onLock: (event) => {
    if (event.action === 'lock') {
      console.log(`Locked by ${event.holder} (${event.mode})`)
    } else if (event.action === 'unlock') {
      console.log('Document unlocked')
    }
  }
}, (snapshot) => {
  console.log('Data:', snapshot.data())
})

Example: Exclusive Editing

async function startEditing(docRef: DocumentRef) {
  const result = await docRef.lock('hard')

  if (!result.locked) {
    alert(`Document is locked by ${result.holder}`)
    return false
  }

  // Edit the document...
  return true
}

async function stopEditing(docRef: DocumentRef) {
  await docRef.unlock()
}
Info

Locks have a TTL (time-to-live) and expire automatically if the client disconnects or fails to renew them. This prevents permanently locked documents from abandoned sessions.

Aggregate Queries

Perform server-side aggregations on collections.

Aggregate API

interface AggregateOptions {
  groupBy?: string             // Field to group results by
  ops: AggregateOp[]           // Aggregation operations
}

type AggregateOp = 'sum' | 'avg' | 'min' | 'max'

interface AggregateGroupEntry {
  group: any                   // Value of the groupBy field
  count: number                // Number of documents in the group
  [key: string]: any           // Aggregate results (e.g., sum_hours, avg_hours)
}

interface AggregateSnapshot {
  groups: AggregateGroupEntry[]
}

Basic Aggregation

const todos = rtdb.collection('todos')

// Aggregate with filters
const result = await todos
  .where('completed', '==', false)
  .aggregate({
    groupBy: 'status',
    ops: ['sum', 'avg']
  })
  .get()

result.groups.forEach(group => {
  console.log(`${group.group}: ${group.count} items`)
})

Real-Time Aggregate Subscriptions

Aggregate queries support real-time updates via onSnapshot:

const unsubscribe = todos
  .where('completed', '==', false)
  .aggregate({
    groupBy: 'status',
    ops: ['sum']
  })
  .onSnapshot((snapshot: AggregateSnapshot) => {
    snapshot.groups.forEach(group => {
      console.log(`${group.group}: ${group.count} items`)
    })
  })

Example: Task Dashboard

const tasks = rtdb.collection('tasks')

// Group tasks by status with count and total estimated hours
const unsubscribe = tasks
  .aggregate({
    groupBy: 'status',
    ops: ['sum', 'avg']
  })
  .onSnapshot((snapshot) => {
    snapshot.groups.forEach(({ group, count, sum_hours, avg_hours }) => {
      console.log(`${group}: ${count} tasks, ${sum_hours}h total, ${avg_hours}h avg`)
    })
    // Output:
    // todo: 12 tasks, 36h total, 3h avg
    // in_progress: 5 tasks, 20h total, 4h avg
    // done: 28 tasks, 84h total, 3h avg
  })

Batch Operations

Perform multiple operations atomically:

const todos = rtdb.collection('todos')
const batch = rtdb.batch()

// Create multiple
batch.create(todos, { title: 'Task 1', completed: false })
batch.create(todos, { title: 'Task 2', completed: false })

// Update existing
batch.update(rtdb.ref('todos/task_123'), { completed: true })

// Delete
batch.delete(rtdb.ref('todos/task_456'))

// Commit all operations atomically
const results = await batch.commit()
console.log('Batch results:', results)

BatchResult:

interface BatchResult {
  ref?: string   // Reference ID if provided
  id?: string    // Generated document ID
}

Indexes

Create indexes for efficient queries:

// Create index on a field
await rtdb.createIndex('todos', 'createdAt')
await rtdb.createIndex('todos', 'completed')

TypeScript Support

Full type safety with generics:

interface Todo {
  title: string
  completed: boolean
  createdAt: number
  tags?: string[]
}

const todos = rtdb.collection<Todo>('todos')

// TypeScript knows the shape
const snapshot = await todos.get()
snapshot.docs.forEach(doc => {
  const data = doc.data()
  console.log(data.title)     // string
  console.log(data.completed) // boolean
  // @ts-error: Property 'invalid' does not exist
  // console.log(data.invalid)
})

React Integration

import { useEffect, useState } from 'react'
import { useAuth } from '@cloudillo/react'
import { RtdbClient } from '@cloudillo/rtdb'
import { getRtdbUrl } from '@cloudillo/core'

interface Todo {
  title: string
  completed: boolean
}

function TodoList({ dbId }: { dbId: string }) {
  const [auth] = useAuth()
  const [todos, setTodos] = useState<Todo[]>([])
  const [rtdb, setRtdb] = useState<RtdbClient | null>(null)

  // Initialize RTDB client
  useEffect(() => {
    if (!auth?.token || !auth?.idTag) return

    const client = new RtdbClient({
      dbId,
      auth: { getToken: () => auth.token },
      serverUrl: getRtdbUrl(auth.idTag, dbId, auth.token!)
    })

    setRtdb(client)

    return () => {
      client.disconnect()
    }
  }, [auth?.token, auth?.idTag, dbId])

  // Subscribe to todos
  useEffect(() => {
    if (!rtdb) return

    const todos = rtdb.collection<Todo>('todos')
    const unsubscribe = todos.onSnapshot((snapshot) => {
      setTodos(snapshot.docs.map(doc => ({
        id: doc.id,
        ...doc.data()
      })))
    })

    return () => unsubscribe()
  }, [rtdb])

  return (
    <ul>
      {todos.map(todo => (
        <li key={todo.id}>{todo.title}</li>
      ))}
    </ul>
  )
}

Error Handling

The RTDB client provides typed errors:

import {
  RtdbError,
  ConnectionError,
  AuthError,
  PermissionError,
  NotFoundError,
  ValidationError,
  TimeoutError
} from '@cloudillo/rtdb'

try {
  await rtdb.connect()
} catch (error) {
  if (error instanceof ConnectionError) {
    console.log('Connection failed:', error.message)
  } else if (error instanceof AuthError) {
    console.log('Authentication failed:', error.message)
  } else if (error instanceof PermissionError) {
    console.log('Permission denied:', error.message)
  } else if (error instanceof NotFoundError) {
    console.log('Not found:', error.message)
  } else if (error instanceof TimeoutError) {
    console.log('Request timed out:', error.message)
  }
}

Best Practices

1. Use Connection Management

// Initialize once, reuse
const rtdb = new RtdbClient({ ... })
await rtdb.connect()

// Use throughout your app
const todos = rtdb.collection('todos')

2. Clean Up Subscriptions

// Always unsubscribe to prevent memory leaks
useEffect(() => {
  const unsubscribe = todos.onSnapshot(callback)
  return () => unsubscribe()
}, [])

3. Use Batch for Multiple Operations

// Good: Atomic batch operation
const batch = rtdb.batch()
batch.create(todos, { title: 'Task 1' })
batch.create(todos, { title: 'Task 2' })
await batch.commit()

// Avoid: Multiple separate requests
// await create({ title: 'Task 1' })
// await create({ title: 'Task 2' })

4. Use Typed Collections

// Define your types
interface Todo {
  title: string
  completed: boolean
}

// Get type safety
const todos = rtdb.collection<Todo>('todos')

5. Handle Connection State

// Check connection before operations
if (!rtdb.isConnected()) {
  await rtdb.connect()
}

// Handle reconnection in UI
const [connected, setConnected] = useState(false)

See Also

CRDT (Collaborative Editing)

Cloudillo uses Yjs for real-time collaborative editing with CRDT (Conflict-Free Replicated Data Types).

CRDT vs RTDB

CRDT is best for collaborative editing where multiple users edit simultaneously. For structured data with queries (todos, settings, lists), see RTDB. Compare all storage types in Data Storage & Access.

Installation

pnpm add yjs y-websocket

Quick Start

import * as cloudillo from '@cloudillo/core'
import * as Y from 'yjs'

// Initialize
await cloudillo.init('my-app')

// Create Yjs document
const yDoc = new Y.Doc()

// Open collaborative document
const { provider } = await cloudillo.openYDoc(yDoc, 'my-document-id')

// Use shared text
const yText = yDoc.getText('content')
yText.insert(0, 'Hello, collaborative world!')

// Listen for changes
yText.observe(() => {
  console.log('Text updated:', yText.toString())
})

Shared Types

Yjs provides several shared data types:

YText - Shared Text

Best for plain text or rich text content.

const yText = yDoc.getText('content')

// Insert text
yText.insert(0, 'Hello ')
yText.insert(6, 'world!')

// Delete text
yText.delete(0, 5) // Delete 5 characters from position 0

// Format text (for rich text)
yText.format(0, 5, { bold: true })

// Get text content
console.log(yText.toString()) // "world!"

// Observe changes
yText.observe((event) => {
  event.changes.delta.forEach(change => {
    if (change.insert) {
      console.log('Inserted:', change.insert)
    }
    if (change.delete) {
      console.log('Deleted', change.delete, 'characters')
    }
  })
})

YMap - Shared Object

Best for key-value data like form fields or settings.

const yMap = yDoc.getMap('settings')

// Set values
yMap.set('theme', 'dark')
yMap.set('fontSize', 14)
yMap.set('notifications', true)

// Get values
console.log(yMap.get('theme')) // "dark"

// Delete keys
yMap.delete('fontSize')

// Check existence
console.log(yMap.has('theme')) // true

// Iterate
yMap.forEach((value, key) => {
  console.log(`${key}: ${value}`)
})

// Observe changes
yMap.observe((event) => {
  event.changes.keys.forEach((change, key) => {
    if (change.action === 'add') {
      console.log(`Added ${key}:`, yMap.get(key))
    } else if (change.action === 'update') {
      console.log(`Updated ${key}:`, yMap.get(key))
    } else if (change.action === 'delete') {
      console.log(`Deleted ${key}`)
    }
  })
})

YArray - Shared Array

Best for lists like todos, comments, or items.

const yArray = yDoc.getArray('todos')

// Push items
yArray.push([
  { title: 'Task 1', done: false },
  { title: 'Task 2', done: false }
])

// Insert at position
yArray.insert(0, [{ title: 'Urgent task', done: false }])

// Delete
yArray.delete(0, 1) // Delete 1 item at position 0

// Get items
console.log(yArray.get(0)) // First item
console.log(yArray.toArray()) // All items as array

// Iterate
yArray.forEach((item, index) => {
  console.log(index, item)
})

// Observe changes
yArray.observe((event) => {
  console.log('Array changed:', event.changes)
})

YXmlFragment - Shared XML/HTML

Best for rich text editors with complex formatting.

const yXml = yDoc.getXmlFragment('document')

// Create elements
const paragraph = new Y.XmlElement('p')
paragraph.setAttribute('class', 'text')
paragraph.insert(0, [new Y.XmlText('Hello world')])

yXml.insert(0, [paragraph])

Awareness

Awareness tracks user presence, cursors, and selections in real-time.

const { provider } = await cloudillo.openYDoc(yDoc, 'doc_123')

// Set local awareness state
provider.awareness.setLocalState({
  user: {
    name: cloudillo.name,
    idTag: cloudillo.idTag,
    color: '#ff6b6b'
  },
  cursor: {
    line: 10,
    column: 5
  },
  selection: {
    from: 100,
    to: 150
  }
})

// Listen for awareness changes
provider.awareness.on('change', () => {
  const states = provider.awareness.getStates()

  states.forEach((state, clientId) => {
    if (state.user) {
      console.log(`User ${state.user.name} at cursor ${state.cursor}`)
    }
  })
})

// Get specific client state
const clientId = provider.awareness.clientID
const state = provider.awareness.getStates().get(clientId)

Editor Bindings

Yjs provides bindings for popular editors:

Quill (Rich Text)

pnpm add quill y-quill
import Quill from 'quill'
import { QuillBinding } from 'y-quill'

// Create Yjs document
const yDoc = new Y.Doc()
const { provider } = await cloudillo.openYDoc(yDoc, 'doc_123')

const yText = yDoc.getText('content')

// Create Quill editor
const editor = new Quill('#editor', {
  theme: 'snow'
})

// Bind Yjs to Quill
const binding = new QuillBinding(yText, editor, provider.awareness)

// Quill now syncs with Yjs automatically

CodeMirror (Code Editor)

pnpm add codemirror y-codemirror
import { EditorView, basicSetup } from 'codemirror'
import { yCollab } from 'y-codemirror.next'

const yText = yDoc.getText('content')

const editor = new EditorView({
  extensions: [
    basicSetup,
    yCollab(yText, provider.awareness)
  ],
  parent: document.querySelector('#editor')
})

Monaco (VS Code Editor)

pnpm add monaco-editor y-monaco
import * as monaco from 'monaco-editor'
import { MonacoBinding } from 'y-monaco'

const yText = yDoc.getText('content')

const editor = monaco.editor.create(document.getElementById('editor'), {
  value: '',
  language: 'javascript'
})

const binding = new MonacoBinding(
  yText,
  editor.getModel(),
  new Set([editor]),
  provider.awareness
)

ProseMirror (Rich Text)

pnpm add prosemirror-view prosemirror-state y-prosemirror
import { EditorState } from 'prosemirror-state'
import { EditorView } from 'prosemirror-view'
import { ySyncPlugin, yCursorPlugin, yUndoPlugin } from 'y-prosemirror'

const yXml = yDoc.getXmlFragment('prosemirror')

const state = EditorState.create({
  schema,
  plugins: [
    ySyncPlugin(yXml),
    yCursorPlugin(provider.awareness),
    yUndoPlugin()
  ]
})

const view = new EditorView(document.querySelector('#editor'), {
  state
})

Offline Support

Yjs documents work offline and sync when reconnected.

import { IndexeddbPersistence } from 'y-indexeddb'

const yDoc = new Y.Doc()

// Persist to IndexedDB
const indexeddbProvider = new IndexeddbPersistence('my-doc-id', yDoc)

indexeddbProvider.on('synced', () => {
  console.log('Loaded from IndexedDB')
})

// Also connect to server
const { provider } = await cloudillo.openYDoc(yDoc, 'my-doc-id')

// Now works offline with local persistence
// Syncs to server when connection available

Transactions

Group multiple changes into a single transaction:

yDoc.transact(() => {
  const yText = yDoc.getText('content')
  yText.insert(0, 'Hello ')
  yText.insert(6, 'world!')
  yText.format(0, 11, { bold: true })
})

// All changes sync as one update
// Only one observer event fired

Undo/Redo

import { UndoManager } from 'yjs'

const yText = yDoc.getText('content')
const undoManager = new UndoManager(yText)

// Make changes
yText.insert(0, 'Hello')

// Undo
undoManager.undo()

// Redo
undoManager.redo()

// Track who made changes
undoManager.on('stack-item-added', (event) => {
  console.log('Change by:', event.origin)
})

Document Lifecycle

// Create document
const yDoc = new Y.Doc()

// Open collaborative connection
const { provider } = await cloudillo.openYDoc(yDoc, 'doc_123')

// Use document...

// Close connection
provider.destroy()

// Destroy document
yDoc.destroy()

Best Practices

1. Use Subdocs for Large Documents

const yDoc = new Y.Doc()
const yMap = yDoc.getMap('pages')

// Create subdocument for each page
const page1 = new Y.Doc()
yMap.set('page1', page1)

const page1Text = page1.getText('content')
page1Text.insert(0, 'Page 1 content')

2. Batch Operations in Transactions

// βœ… Single transaction
yDoc.transact(() => {
  for (let i = 0; i < 100; i++) {
    yText.insert(i, 'x')
  }
})

// ❌ Many transactions
for (let i = 0; i < 100; i++) {
  yText.insert(i, 'x') // Sends 100 updates!
}

3. Clean Up Observers

// Add observer
const observer = (event) => {
  console.log('Changed:', event)
}
yText.observe(observer)

// Remove observer when done
yText.unobserve(observer)

4. Handle Connection State

provider.on('status', ({ status }) => {
  if (status === 'connected') {
    setConnectionStatus('online')
  } else {
    setConnectionStatus('offline')
  }
})

React Example

Complete collaborative editor in React:

import { useEffect, useState } from 'react'
import { useApi } from '@cloudillo/react'
import * as Y from 'yjs'
import * as cloudillo from '@cloudillo/core'

function CollaborativeEditor({ docId }) {
  const [yDoc, setYDoc] = useState(null)
  const [provider, setProvider] = useState(null)
  const [connected, setConnected] = useState(false)

  useEffect(() => {
    const doc = new Y.Doc()
    setYDoc(doc)

    cloudillo.openYDoc(doc, docId).then(({ provider: p }) => {
      setProvider(p)

      p.on('status', ({ status }) => {
        setConnected(status === 'connected')
      })
    })

    return () => {
      provider?.destroy()
      doc.destroy()
    }
  }, [docId])

  if (!yDoc) return <div>Loading...</div>

  return (
    <div>
      <div className="status">
        {connected ? '🟒 Connected' : 'πŸ”΄ Disconnected'}
      </div>
      <Editor yDoc={yDoc} provider={provider} />
    </div>
  )
}

See Also

Error Handling

Cloudillo uses structured error codes and standardized error responses for consistent error handling across the platform.

Error Response Format

All API errors return this structure:

{
  "error": {
    "code": "E-AUTH-UNAUTH",
    "message": "Unauthorized access",
    "details": {
      "reason": "Token expired",
      "expiredAt": 1735000000
    }
  },
  "time": 1735000000,
  "reqId": "req_abc123"
}

Fields:

  • error.code - Structured error code (see below)
  • error.message - Human-readable error message
  • error.details - Optional additional context
  • time - Unix timestamp (seconds)
  • reqId - Request ID for tracing

Error Code Format

Error codes follow the pattern: E-MODULE-ERRTYPE

  • E- - Prefix (all error codes start with this)
  • MODULE - Module identifier (AUTH, CORE, SYS, etc.)
  • ERRTYPE - Error type (UNAUTH, NOTFOUND, etc.)

Error Codes

Authentication Errors (AUTH)

Code HTTP Status Description
E-AUTH-UNAUTH 401 Unauthorized - Invalid or missing token
E-AUTH-FORBID 403 Forbidden - Insufficient permissions
E-AUTH-EXPIRED 401 Token expired
E-AUTH-INVALID 400 Invalid credentials
E-AUTH-MISMATCH 400 Password mismatch
E-AUTH-EXISTS 409 User already exists

Core Errors (CORE)

Code HTTP Status Description
E-CORE-NOTFOUND 404 Resource not found
E-CORE-CONFLICT 409 Resource conflict (duplicate)
E-CORE-INVALID 400 Invalid request data
E-CORE-BADREQ 400 Malformed request
E-CORE-LIMIT 429 Rate limit exceeded

System Errors (SYS)

Code HTTP Status Description
E-SYS-UNAVAIL 503 Service unavailable
E-SYS-INTERNAL 500 Internal server error
E-SYS-TIMEOUT 504 Request timeout
E-SYS-STORAGE 507 Storage full

File Errors (FILE)

Code HTTP Status Description
E-FILE-NOTFOUND 404 File not found
E-FILE-TOOLARGE 413 File too large
E-FILE-BADTYPE 415 Unsupported file type
E-FILE-CORRUPT 422 Corrupted file

Action Errors (ACTION)

Code HTTP Status Description
E-ACTION-NOTFOUND 404 Action not found
E-ACTION-INVALID 400 Invalid action data
E-ACTION-DENIED 403 Action not allowed
E-ACTION-EXPIRED 410 Action expired

Federation Errors (FED)

Code HTTP Status Description
E-FED-SIGFAIL 400 Action signature verification failed
E-FED-KEYNOTFOUND 400 Issuer public key not found
E-FED-EXPIRED 400 Action token expired
E-FED-NOTRUST 403 No trust relationship with issuer
E-POW-REQUIRED 428 Proof-of-work required (CONN actions)

Validation Errors (VAL)

Code HTTP Status Description
E-VAL-INVALID 400 Validation error
E-AUTH-NOPERM 403 Permission denied

Network Errors (NET)

Code HTTP Status Description
E-NET-TIMEOUT 408 Network timeout

Database Errors (CORE)

Code HTTP Status Description
E-CORE-DBERR 500 Database error
E-CORE-PARSE 500 Parse error

Identity Provider Errors (IDP)

Code HTTP Status Description
E-IDP-NOTFOUND 404 Identity not found
E-IDP-EXISTS 409 Identity already exists
E-IDP-INVALID 400 Invalid identity format

RTDB/CRDT Errors

Code HTTP Status Description
E-RTDB-NOTFOUND 404 RTDB document not found
E-CRDT-NOTFOUND 404 CRDT document not found
E-CRDT-CONFLICT 409 CRDT merge conflict

Handling Errors with FetchError

The @cloudillo/core library provides FetchError for consistent error handling:

import { FetchError } from '@cloudillo/core'

try {
  const api = cloudillo.createApiClient()
  const profile = await api.profiles.getOwn()
} catch (error) {
  if (error instanceof FetchError) {
    console.error('Error code:', error.code)
    console.error('Message:', error.message)
    console.error('HTTP status:', error.status)
    console.error('Details:', error.details)

    // Handle specific errors
    switch (error.code) {
      case 'E-AUTH-UNAUTH':
        // Redirect to login
        window.location.href = '/login'
        break

      case 'E-AUTH-FORBID':
        // Show permission error
        alert('You do not have permission to access this resource')
        break

      case 'E-CORE-NOTFOUND':
        // Show not found message
        console.log('Resource not found')
        break

      case 'E-CORE-LIMIT':
        // Rate limited
        console.log('Too many requests, please slow down')
        break

      default:
        // Generic error
        console.error('An error occurred:', error.message)
    }
  } else {
    // Non-API error (network, parsing, etc.)
    console.error('Unexpected error:', error)
  }
}

Error Handling Patterns

Pattern 1: Global Error Handler

function createApiWithErrorHandling() {
  const api = cloudillo.createApiClient()

  // Wrap API methods with error handling
  return new Proxy(api, {
    get(target, prop) {
      const original = target[prop]

      if (typeof original === 'function') {
        return async (...args) => {
          try {
            return await original.apply(target, args)
          } catch (error) {
            handleApiError(error)
            throw error
          }
        }
      }

      return original
    }
  })
}

function handleApiError(error) {
  if (error instanceof FetchError) {
    switch (error.code) {
      case 'E-AUTH-UNAUTH':
      case 'E-AUTH-EXPIRED':
        // Redirect to login
        window.location.href = '/login'
        break

      case 'E-CORE-LIMIT':
        // Show rate limit toast
        showToast('Too many requests. Please slow down.')
        break

      default:
        // Log to error tracking service
        logErrorToSentry(error)
    }
  }
}

Pattern 2: React Error Boundary

import { Component } from 'react'
import { FetchError } from '@cloudillo/core'

class ApiErrorBoundary extends Component {
  state = { error: null }

  static getDerivedStateFromError(error) {
    return { error }
  }

  componentDidCatch(error, errorInfo) {
    if (error instanceof FetchError) {
      console.error('API Error:', error.code, error.message)

      // Handle specific errors
      if (error.code === 'E-AUTH-UNAUTH') {
        window.location.href = '/login'
      }
    }
  }

  render() {
    if (this.state.error) {
      return (
        <div className="error">
          <h1>Something went wrong</h1>
          <p>{this.state.error.message}</p>
          <button onClick={() => this.setState({ error: null })}>
            Try again
          </button>
        </div>
      )
    }

    return this.props.children
  }
}

Pattern 3: Retry Logic

async function fetchWithRetry(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn()
    } catch (error) {
      if (error instanceof FetchError) {
        // Retry on transient errors
        if (error.code === 'E-SYS-UNAVAIL' ||
            error.code === 'E-SYS-TIMEOUT') {
          if (i < maxRetries - 1) {
            // Exponential backoff
            await new Promise(r => setTimeout(r, 1000 * Math.pow(2, i)))
            continue
          }
        }

        // Don't retry on auth or client errors
        if (error.code.startsWith('E-AUTH-') ||
            error.code.startsWith('E-CORE-')) {
          throw error
        }
      }

      throw error
    }
  }
}

// Usage
const data = await fetchWithRetry(() => api.profiles.getOwn())

Pattern 4: User-Friendly Messages

function getUserFriendlyMessage(error: FetchError): string {
  const messages: Record<string, string> = {
    'E-AUTH-UNAUTH': 'Please log in to continue',
    'E-AUTH-FORBID': 'You don\'t have permission to do that',
    'E-AUTH-EXPIRED': 'Your session has expired. Please log in again',
    'E-CORE-NOTFOUND': 'The item you\'re looking for doesn\'t exist',
    'E-CORE-LIMIT': 'You\'re making too many requests. Please slow down',
    'E-FILE-TOOLARGE': 'This file is too large. Maximum size is 100MB',
    'E-FILE-BADTYPE': 'This file type is not supported',
    'E-SYS-UNAVAIL': 'The service is temporarily unavailable. Please try again later',
  }

  return messages[error.code] || 'An unexpected error occurred. Please try again'
}

// Usage
try {
  await api.files.uploadBlob('default', file.name, file)
} catch (error) {
  if (error instanceof FetchError) {
    alert(getUserFriendlyMessage(error))
  }
}

Pattern 5: Typed Error Handling

type ApiError = {
  code: string
  message: string
  status: number
}

function isAuthError(error: ApiError): boolean {
  return error.code.startsWith('E-AUTH-')
}

function isClientError(error: ApiError): boolean {
  return error.status >= 400 && error.status < 500
}

function isServerError(error: ApiError): boolean {
  return error.status >= 500
}

// Usage
try {
  const data = await api.actions.create(newAction)
} catch (error) {
  if (error instanceof FetchError) {
    if (isAuthError(error)) {
      handleAuthError(error)
    } else if (isServerError(error)) {
      showRetryPrompt()
    }
  }
}

Validation Errors

Client-side validation can prevent many errors:

import type { NewAction } from '@cloudillo/types'

function validateAction(action: NewAction): string[] {
  const errors: string[] = []

  if (!action.type) {
    errors.push('Action type is required')
  }

  if (action.type === 'POST' && !action.content) {
    errors.push('Post content is required')
  }

  if (action.attachments && action.attachments.length > 10) {
    errors.push('Maximum 10 attachments allowed')
  }

  return errors
}

// Usage
const newAction = {
  type: 'POST',
  content: { text: 'Hello!' }
}

const validationErrors = validateAction(newAction)
if (validationErrors.length > 0) {
  alert('Validation errors:\n' + validationErrors.join('\n'))
  return
}

// Proceed with API call
await api.actions.create(newAction)

Logging and Monitoring

function logApiError(error: FetchError, context?: any) {
  const logData = {
    code: error.code,
    message: error.message,
    status: error.status,
    url: error.response?.url,
    context,
    timestamp: new Date().toISOString(),
    userAgent: navigator.userAgent,
    user: cloudillo.idTag
  }

  // Send to logging service
  fetch('/api/logs/error', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(logData)
  })

  // Also log to console in development
  if (process.env.NODE_ENV === 'development') {
    console.error('API Error:', logData)
  }
}

Best Practices

1. Always Handle Errors

// βœ… Good - handle errors
try {
  await api.actions.create(newAction)
} catch (error) {
  handleError(error)
}

// ❌ Bad - unhandled errors crash the app
await api.actions.create(newAction)

2. Provide User Feedback

// βœ… Good - user knows what happened
try {
  await api.files.uploadBlob('default', file.name, file)
  showToast('File uploaded successfully!')
} catch (error) {
  showToast('Upload failed. Please try again.')
}

// ❌ Bad - silent failure
try {
  await api.files.uploadBlob('default', file.name, file)
} catch (error) {
  console.error(error)
}

3. Differentiate Error Types

// βœ… Good - handle different errors appropriately
if (error.code === 'E-AUTH-UNAUTH') {
  redirectToLogin()
} else if (error.code === 'E-CORE-NOTFOUND') {
  show404Page()
} else {
  showGenericError()
}

// ❌ Bad - same handling for all errors
alert('Error!')

4. Log Errors for Debugging

// βœ… Good - logs help debug issues
catch (error) {
  console.error('Failed to create post:', {
    error,
    action: newAction,
    user: cloudillo.idTag
  })
  showError('Failed to create post')
}

// ❌ Bad - no debugging info
catch (error) {
  showError('Failed')
}

See Also

Microfrontend Integration

Cloudillo uses a microfrontend architecture where applications run as sandboxed iframes and communicate with the shell via postMessage.

Overview

Benefits:

  • Isolation - Apps are sandboxed for security
  • Technology agnostic - Use any framework (React, Vue, vanilla JS)
  • Independent deployment - Update apps without redeploying the shell
  • Resource sharing - Share authentication and state via the shell

Communication Protocol

Shell β†’ App (Init Message)

When an app loads, the shell sends an init message:

{
  cloudillo: true,
  type: 'init',
  idTag: 'alice@example.com',
  tnId: 12345,
  roles: ['user', 'admin'],
  theme: 'glass',
  darkMode: false,
  token: 'eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9...'
}

Fields:

  • cloudillo: true - Identifies Cloudillo messages
  • type: 'init' - Message type
  • idTag - User’s identity
  • tnId - Tenant ID
  • roles - User roles
  • theme - UI theme variant
  • darkMode - Dark mode preference
  • token - Access token for API calls

App β†’ Shell (Init Request)

Apps request initialization by sending:

{
  cloudillo: true,
  type: 'initReq'
}

Bidirectional Communication

Apps and shell can send requests and replies:

// Shell β†’ App (request)
{
  cloudillo: true,
  type: 'request',
  id: 123,
  method: 'getData',
  params: { key: 'value' }
}

// App β†’ Shell (reply)
{
  cloudillo: true,
  type: 'reply',
  id: 123,
  data: { result: 'success' }
}

Basic App Setup

Step 1: Create index.html

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>My Cloudillo App</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="styles.css">
</head>
<body>
  <div id="app">Loading...</div>
  <script type="module" src="./app.js"></script>
</body>
</html>

Step 2: Initialize in app.js

import * as cloudillo from '@cloudillo/core'

async function main() {
  // Request init from shell
  const token = await cloudillo.init('my-app')

  // Now you have access to:
  console.log('User:', cloudillo.idTag)
  console.log('Tenant:', cloudillo.tnId)
  console.log('Roles:', cloudillo.roles)
  console.log('Token:', token)

  // Create API client
  const api = cloudillo.createApiClient()

  // Your app logic here...
  initializeApp(api)
}

main().catch(console.error)

Step 3: Build and Deploy

# Build your app (example with Rollup)
rollup -c

# Deploy to shell's apps directory
cp -r dist /path/to/cloudillo/shell/public/apps/my-app

Complete Example

Here’s a complete microfrontend app:

import * as cloudillo from '@cloudillo/core'
import * as Y from 'yjs'

// Initialize
async function init() {
  const token = await cloudillo.init('quillo')

  // Check dark mode
  if (cloudillo.darkMode) {
    document.body.classList.add('dark-theme')
  }

  // Create API client
  const api = cloudillo.createApiClient()

  // Get document ID from URL
  const params = new URLSearchParams(window.location.search)
  const docId = params.get('docId') || 'default-doc'

  // Open collaborative document
  const yDoc = new Y.Doc()
  const { provider } = await cloudillo.openYDoc(yDoc, docId)

  // Initialize editor
  initEditor(yDoc, provider)

  // Handle save
  document.getElementById('save-btn')?.addEventListener('click', async () => {
    await saveDocument(api, docId, yDoc)
  })
}

function initEditor(yDoc, provider) {
  const yText = yDoc.getText('content')

  // Simple textarea editor
  const textarea = document.getElementById('editor')

  // Load content
  textarea.value = yText.toString()

  // Listen for remote changes
  yText.observe(() => {
    if (document.activeElement !== textarea) {
      textarea.value = yText.toString()
    }
  })

  // Send local changes
  textarea.addEventListener('input', (e) => {
    yDoc.transact(() => {
      yText.delete(0, yText.length)
      yText.insert(0, e.target.value)
    })
  })
}

async function saveDocument(api, docId, yDoc) {
  const content = yDoc.getText('content').toString()

  await api.actions.create({
    type: 'POST',
    content: {
      text: content,
      docId: docId
    }
  })

  alert('Saved!')
}

// Start app
init().catch(console.error)

React Microfrontend

For React apps, use CloudilloProvider:

import React from 'react'
import ReactDOM from 'react-dom/client'
import { CloudilloProvider, useAuth, useApi } from '@cloudillo/react'

function App() {
  return (
    <CloudilloProvider appName="my-react-app">
      <AppContent />
    </CloudilloProvider>
  )
}

function AppContent() {
  const auth = useAuth()
  const api = useApi()

  if (!auth.idTag) {
    return <div>Loading...</div>
  }

  return (
    <div>
      <h1>Welcome, {auth.name}!</h1>
      <YourComponents />
    </div>
  )
}

const root = ReactDOM.createRoot(document.getElementById('root'))
root.render(<App />)

Loading Apps in the Shell

Use MicrofrontendContainer to load apps:

import { MicrofrontendContainer } from '@cloudillo/react'

function Shell() {
  const [currentApp, setCurrentApp] = useState('quillo')
  const [docId, setDocId] = useState('doc_123')

  return (
    <div className="shell">
      <nav>
        <button onClick={() => setCurrentApp('quillo')}>Quillo</button>
        <button onClick={() => setCurrentApp('prello')}>Prello</button>
        <button onClick={() => setCurrentApp('todollo')}>Todollo</button>
      </nav>

      <main>
        <MicrofrontendContainer
          appName={currentApp}
          src={`/apps/${currentApp}/index.html`}
          docId={docId}
        />
      </main>
    </div>
  )
}

URL Parameters

Pass data to apps via URL parameters:

// Shell passes docId
<MicrofrontendContainer
  src={`/apps/quillo/index.html?docId=${docId}`}
/>

// App reads docId
const params = new URLSearchParams(window.location.search)
const docId = params.get('docId')

Styling

Apps should be responsive and adapt to the shell’s theme:

/* Base styles */
body {
  margin: 0;
  padding: 16px;
  font-family: system-ui, sans-serif;
  background: var(--bg-color, #fff);
  color: var(--text-color, #000);
}

/* Dark mode */
body.dark-theme {
  --bg-color: #1a1a1a;
  --text-color: #fff;
}

/* Responsive */
@media (max-width: 768px) {
  body {
    padding: 8px;
  }
}

Security Considerations

1. Sandbox Attributes

The shell loads apps with these sandbox attributes:

<iframe
  sandbox="allow-scripts allow-same-origin allow-forms"
  src="/apps/my-app/index.html"
></iframe>

This prevents:

  • Navigation of the top frame
  • Popup windows
  • Download without user gesture

2. Content Security Policy

Apps should set a strict CSP:

<meta http-equiv="Content-Security-Policy"
      content="default-src 'self'; connect-src 'self' wss:">

3. Token Handling

Never store tokens in localStorage (vulnerable to XSS):

// βœ… Good - token in memory only
const token = await cloudillo.init('my-app')

// ❌ Bad - vulnerable to XSS
localStorage.setItem('token', token)

4. Message Validation

Validate all postMessage events:

window.addEventListener('message', (event) => {
  // Verify origin
  if (event.origin !== window.location.origin) {
    return
  }

  // Verify message structure
  if (!event.data?.cloudillo) {
    return
  }

  // Handle message
  handleCloudilloMessage(event.data)
})

Debugging

Console Logging

// Enable debug logging
cloudillo.debug = true

// All postMessage events will be logged
await cloudillo.init('my-app')

DevTools

Use browser DevTools to inspect iframe:

  1. Open DevTools
  2. Select iframe context in console dropdown
  3. Inspect app state

Error Handling

try {
  const token = await cloudillo.init('my-app')
} catch (error) {
  console.error('Init failed:', error)

  // Check if running in shell
  if (window.parent === window) {
    console.error('App must run inside Cloudillo shell')
  }
}

Example Apps

Cloudillo includes several example apps:

Quillo - Rich Text Editor

  • Location: /apps/quillo
  • Tech: Quill + Yjs
  • Features: Collaborative rich text editing

Prello - Presentations

  • Location: /apps/prello
  • Tech: Custom drag-drop + Yjs
  • Features: Slides, drawings, animations

Sheello - Spreadsheet

  • Location: /apps/sheello
  • Tech: Fortune Sheet + Yjs
  • Features: Excel-like spreadsheets

Formillo - Forms

  • Location: /apps/formillo
  • Tech: React + RTDB
  • Features: Form builder and responses

Todollo - Tasks

  • Location: /apps/todollo
  • Tech: React + RTDB
  • Features: Task management

All use the same patterns described in this guide.

Best Practices

1. Request Init Immediately

// βœ… Initialize on load
async function main() {
  const token = await cloudillo.init('my-app')
  // Continue...
}
main()

// ❌ Delay init
setTimeout(() => {
  cloudillo.init('my-app')
}, 1000)

2. Handle Theme Changes

// Listen for theme changes from shell
window.addEventListener('message', (event) => {
  if (event.data?.cloudillo && event.data.type === 'themeChange') {
    document.body.classList.toggle('dark-theme', event.data.darkMode)
  }
})

3. Clean Up on Unload

window.addEventListener('beforeunload', () => {
  // Close WebSocket connections
  provider?.destroy()

  // Cancel pending requests
  abortController.abort()
})

4. Progressive Enhancement

// Show loading state while initializing
document.getElementById('app').innerHTML = 'Loading...'

await cloudillo.init('my-app')

// Show app content
document.getElementById('app').innerHTML = '<div>App ready!</div>'

See Also

Testing

Comprehensive guide to testing applications built with the Cloudillo API.

Testing Strategy

Testing Pyramid

        /\
       /E2E\       Few, slow, expensive
      /------\
     /  API  \     More, medium speed
    /----------\
   / Unit Tests \  Many, fast, cheap
  /--------------\

Unit Tests: Test individual functions and components in isolation API/Integration Tests: Test API interactions and service integration E2E Tests: Test complete user workflows

Unit Testing

Testing API Client Functions

import { describe, test, expect, vi } from 'vitest'
import { ActionService } from './action-service'

describe('ActionService', () => {
  test('formats action query parameters correctly', () => {
    const service = new ActionService(mockClient)

    const query = service.buildQuery({
      type: 'POST',
      status: 'A',
      _limit: 20,
      _offset: 0
    })

    expect(query.toString()).toBe('type=POST&status=A&_limit=20&_offset=0')
  })

  test('converts timestamps to Unix seconds', () => {
    const service = new ActionService(mockClient)

    const action = service.prepareAction({
      type: 'POST',
      content: { text: 'Hello' },
      published: new Date('2025-01-01T00:00:00Z')
    })

    expect(action.published).toBe(1735689600)
  })

  test('validates action type', () => {
    const service = new ActionService(mockClient)

    expect(() => {
      service.validateAction({ type: 'INVALID' })
    }).toThrow('Invalid action type')
  })
})

Testing State Management

import { describe, test, expect } from 'vitest'
import { TokenManager } from './token-manager'

describe('TokenManager', () => {
  test('detects expired tokens', () => {
    const manager = new TokenManager()

    // Token expired 1 hour ago
    const expiredToken = createToken({ exp: Math.floor(Date.now() / 1000) - 3600 })

    expect(manager.isExpired(expiredToken)).toBe(true)
  })

  test('schedules refresh before expiry', () => {
    const manager = new TokenManager()

    // Token expires in 20 minutes
    const token = createToken({ exp: Math.floor(Date.now() / 1000) + 1200 })

    const refreshTime = manager.getRefreshTime(token)
    const now = Date.now()

    // Should refresh in ~10 minutes (10 min before expiry)
    expect(refreshTime).toBeGreaterThan(now)
    expect(refreshTime).toBeLessThan(now + 11 * 60 * 1000)
  })

  test('caches tokens by scope', () => {
    const manager = new TokenManager()

    manager.setToken('read', 'token1')
    manager.setToken('write', 'token2')

    expect(manager.getToken('read')).toBe('token1')
    expect(manager.getToken('write')).toBe('token2')
  })
})

Testing Utility Functions

import { describe, test, expect } from 'vitest'
import { validateIdTag, parseActionId, formatTimestamp } from './utils'

describe('Utility Functions', () => {
  test('validates idTag format', () => {
    expect(validateIdTag('alice@example.com')).toBe(true)
    expect(validateIdTag('invalid')).toBe(false)
    expect(validateIdTag('@example.com')).toBe(false)
  })

  test('parses action ID components', () => {
    const parsed = parseActionId('act_abc123')

    expect(parsed.prefix).toBe('act')
    expect(parsed.id).toBe('abc123')
  })

  test('formats Unix timestamp to readable date', () => {
    const timestamp = 1735689600  // 2025-01-01 00:00:00 UTC
    const formatted = formatTimestamp(timestamp)

    expect(formatted).toBe('2025-01-01 00:00:00')
  })
})

Integration Testing

Mocking Fetch Requests

import { describe, test, expect, beforeEach, afterEach, vi } from 'vitest'
import { CloudilloClient } from './client'

describe('CloudilloClient Integration', () => {
  let client: CloudilloClient

  beforeEach(() => {
    global.fetch = vi.fn()
    client = new CloudilloClient('http://localhost:3000', tokenManager)
  })

  afterEach(() => {
    vi.restoreAllMocks()
  })

  test('creates action via API', async () => {
    const mockResponse = {
      data: {
        actionId: 'act_123',
        type: 'POST',
        content: { text: 'Hello' }
      },
      time: 1735689600,
      reqId: 'req_abc'
    }

    ;(global.fetch as any).mockResolvedValueOnce({
      ok: true,
      json: async () => mockResponse
    })

    const result = await client.actions.create({
      type: 'POST',
      content: { text: 'Hello' },
      audience: ['public'],
      published: 1735689600
    })

    expect(fetch).toHaveBeenCalledWith(
      'http://localhost:3000/api/action',
      expect.objectContaining({
        method: 'POST',
        headers: expect.objectContaining({
          'Content-Type': 'application/json',
          'Authorization': expect.stringContaining('Bearer ')
        })
      })
    )

    expect(result.data.actionId).toBe('act_123')
  })

  test('handles API errors', async () => {
    ;(global.fetch as any).mockResolvedValueOnce({
      ok: false,
      status: 401,
      json: async () => ({
        error: {
          code: 'E-AUTH-UNAUTH',
          message: 'Unauthorized access'
        }
      })
    })

    await expect(
      client.actions.list()
    ).rejects.toThrow('Unauthorized access')
  })

  test('retries on 5xx errors', async () => {
    ;(global.fetch as any)
      .mockResolvedValueOnce({
        ok: false,
        status: 503,
        json: async () => ({ error: { message: 'Service unavailable' } })
      })
      .mockResolvedValueOnce({
        ok: true,
        json: async () => ({ data: [] })
      })

    const result = await client.actions.list()

    expect(fetch).toHaveBeenCalledTimes(2)
    expect(result.data).toEqual([])
  })
})

Testing with MSW (Mock Service Worker)

import { rest } from 'msw'
import { setupServer } from 'msw/node'
import { describe, test, expect, beforeAll, afterAll, afterEach } from 'vitest'

const server = setupServer(
  // Mock login endpoint
  rest.post('http://localhost:3000/api/auth/login', (req, res, ctx) => {
    const { email, password } = req.body as any

    if (email === 'test@example.com' && password === 'password') {
      return res(
        ctx.json({
          data: {
            token: 'mock-jwt-token',
            tenant: { idTag: 'test@example.com' }
          }
        })
      )
    }

    return res(
      ctx.status(401),
      ctx.json({
        error: {
          code: 'E-AUTH-BADCRED',
          message: 'Invalid credentials'
        }
      })
    )
  }),

  // Mock actions list endpoint
  rest.get('http://localhost:3000/api/action', (req, res, ctx) => {
    const type = req.url.searchParams.get('type')
    const limit = parseInt(req.url.searchParams.get('_limit') || '50')

    return res(
      ctx.json({
        data: [
          {
            actionId: 'act_1',
            type: type || 'POST',
            content: { text: 'Test post' },
            createdAt: 1735689600
          }
        ],
        pagination: {
          total: 1,
          offset: 0,
          limit,
          hasMore: false
        }
      })
    )
  })
)

beforeAll(() => server.listen())
afterEach(() => server.resetHandlers())
afterAll(() => server.close())

describe('API Integration with MSW', () => {
  test('login flow', async () => {
    const result = await fetch('http://localhost:3000/api/auth/login', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        email: 'test@example.com',
        password: 'password'
      })
    })

    const data = await result.json()
    expect(data.data.token).toBe('mock-jwt-token')
  })

  test('fetch filtered actions', async () => {
    const result = await fetch('http://localhost:3000/api/action?type=POST&_limit=20')
    const data = await result.json()

    expect(data.data).toHaveLength(1)
    expect(data.data[0].type).toBe('POST')
    expect(data.pagination.limit).toBe(20)
  })
})

E2E Testing

Testing with Playwright

import { test, expect } from '@playwright/test'

test.describe('Cloudillo Social Features', () => {
  test.beforeEach(async ({ page }) => {
    // Login before each test
    await page.goto('http://localhost:3000/login')
    await page.fill('[name="email"]', 'test@example.com')
    await page.fill('[name="password"]', 'password')
    await page.click('button[type="submit"]')
    await page.waitForURL('http://localhost:3000/feed')
  })

  test('create a post', async ({ page }) => {
    // Navigate to create post
    await page.click('[data-testid="create-post-button"]')

    // Fill in post content
    await page.fill('[data-testid="post-textarea"]', 'This is a test post')

    // Submit post
    await page.click('[data-testid="submit-post-button"]')

    // Wait for post to appear
    await page.waitForSelector('[data-testid="post-item"]')

    // Verify post content
    const postContent = await page.textContent('[data-testid="post-content"]')
    expect(postContent).toContain('This is a test post')
  })

  test('comment on a post', async ({ page }) => {
    // Find first post
    const firstPost = page.locator('[data-testid="post-item"]').first()

    // Click comment button
    await firstPost.locator('[data-testid="comment-button"]').click()

    // Fill comment
    await page.fill('[data-testid="comment-textarea"]', 'Great post!')

    // Submit comment
    await page.click('[data-testid="submit-comment-button"]')

    // Verify comment appears
    await page.waitForSelector('[data-testid="comment-item"]')
    const commentText = await page.textContent('[data-testid="comment-content"]')
    expect(commentText).toContain('Great post!')
  })

  test('upload file', async ({ page }) => {
    await page.goto('http://localhost:3000/files')

    // Upload file
    const fileInput = page.locator('input[type="file"]')
    await fileInput.setInputFiles('./test-fixtures/image.jpg')

    // Wait for upload to complete
    await page.waitForSelector('[data-testid="upload-success"]')

    // Verify file appears in list
    const fileName = await page.textContent('[data-testid="file-name"]')
    expect(fileName).toContain('image.jpg')
  })

  test('real-time updates via WebSocket', async ({ page, context }) => {
    // Open two pages
    const page2 = await context.newPage()
    await page2.goto('http://localhost:3000/feed')

    // Create post on first page
    await page.click('[data-testid="create-post-button"]')
    await page.fill('[data-testid="post-textarea"]', 'Real-time test')
    await page.click('[data-testid="submit-post-button"]')

    // Verify post appears on second page (via WebSocket)
    await page2.waitForSelector(':has-text("Real-time test")', { timeout: 5000 })
    const postOnPage2 = await page2.textContent(':has-text("Real-time test")')
    expect(postOnPage2).toContain('Real-time test')
  })
})

Testing API Directly in E2E

import { test, expect } from '@playwright/test'

test.describe('API E2E Tests', () => {
  let token: string

  test.beforeAll(async ({ request }) => {
    // Login to get token
    const response = await request.post('http://localhost:3000/api/auth/login', {
      data: {
        email: 'test@example.com',
        password: 'password'
      }
    })

    const data = await response.json()
    token = data.data.token
  })

  test('complete post lifecycle', async ({ request }) => {
    // Create post
    const createResponse = await request.post('http://localhost:3000/api/action', {
      headers: {
        'Authorization': `Bearer ${token}`,
        'Content-Type': 'application/json'
      },
      data: {
        type: 'POST',
        content: { text: 'E2E test post' },
        audience: ['public'],
        published: Math.floor(Date.now() / 1000)
      }
    })

    expect(createResponse.ok()).toBeTruthy()
    const createData = await createResponse.json()
    const actionId = createData.data.actionId

    // Fetch post
    const getResponse = await request.get(
      `http://localhost:3000/api/action/${actionId}`,
      {
        headers: { 'Authorization': `Bearer ${token}` }
      }
    )

    const getData = await getResponse.json()
    expect(getData.data.content.text).toBe('E2E test post')

    // Delete post
    const deleteResponse = await request.delete(
      `http://localhost:3000/api/action/${actionId}`,
      {
        headers: { 'Authorization': `Bearer ${token}` }
      }
    )

    expect(deleteResponse.ok()).toBeTruthy()

    // Verify deleted
    const verifyResponse = await request.get(
      `http://localhost:3000/api/action/${actionId}`,
      {
        headers: { 'Authorization': `Bearer ${token}` }
      }
    )

    expect(verifyResponse.status()).toBe(404)
  })

  test('pagination works correctly', async ({ request }) => {
    // Fetch first page
    const page1 = await request.get(
      'http://localhost:3000/api/action?_limit=10&_offset=0',
      {
        headers: { 'Authorization': `Bearer ${token}` }
      }
    )

    const page1Data = await page1.json()
    expect(page1Data.data).toHaveLength(10)
    expect(page1Data.pagination.offset).toBe(0)

    // Fetch second page
    const page2 = await request.get(
      'http://localhost:3000/api/action?_limit=10&_offset=10',
      {
        headers: { 'Authorization': `Bearer ${token}` }
      }
    )

    const page2Data = await page2.json()
    expect(page2Data.pagination.offset).toBe(10)

    // Verify different results
    const page1Ids = page1Data.data.map((a: any) => a.actionId)
    const page2Ids = page2Data.data.map((a: any) => a.actionId)
    expect(page1Ids).not.toEqual(page2Ids)
  })
})

Component Testing

Testing React Components with API

import { render, screen, waitFor } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { vi } from 'vitest'
import { ActionFeed } from './ActionFeed'

describe('ActionFeed Component', () => {
  test('renders loading state', () => {
    render(<ActionFeed />)
    expect(screen.getByText('Loading...')).toBeInTheDocument()
  })

  test('renders actions after loading', async () => {
    const mockActions = [
      {
        actionId: 'act_1',
        type: 'POST',
        content: { text: 'Test post 1' },
        createdAt: 1735689600
      }
    ]

    global.fetch = vi.fn().mockResolvedValue({
      ok: true,
      json: async () => ({ data: mockActions })
    })

    render(<ActionFeed />)

    await waitFor(() => {
      expect(screen.getByText('Test post 1')).toBeInTheDocument()
    })
  })

  test('loads more on scroll', async () => {
    const page1 = [
      { actionId: 'act_1', type: 'POST', content: { text: 'Post 1' } }
    ]
    const page2 = [
      { actionId: 'act_2', type: 'POST', content: { text: 'Post 2' } }
    ]

    global.fetch = vi.fn()
      .mockResolvedValueOnce({
        ok: true,
        json: async () => ({
          data: page1,
          pagination: { hasMore: true }
        })
      })
      .mockResolvedValueOnce({
        ok: true,
        json: async () => ({
          data: page2,
          pagination: { hasMore: false }
        })
      })

    const { container } = render(<ActionFeed />)

    // Wait for first page
    await waitFor(() => {
      expect(screen.getByText('Post 1')).toBeInTheDocument()
    })

    // Scroll to bottom
    const scrollElement = container.querySelector('[data-testid="feed-container"]')
    scrollElement?.scrollTo(0, scrollElement.scrollHeight)

    // Wait for second page
    await waitFor(() => {
      expect(screen.getByText('Post 2')).toBeInTheDocument()
    })
  })

  test('handles create post', async () => {
    global.fetch = vi.fn()
      .mockResolvedValueOnce({
        ok: true,
        json: async () => ({ data: [] })  // Initial load
      })
      .mockResolvedValueOnce({
        ok: true,
        json: async () => ({
          data: {
            actionId: 'act_new',
            type: 'POST',
            content: { text: 'New post' }
          }
        })
      })

    render(<ActionFeed />)

    // Fill in post
    const textarea = screen.getByPlaceholderText('What\'s on your mind?')
    await userEvent.type(textarea, 'New post')

    // Submit
    const submitButton = screen.getByRole('button', { name: 'Post' })
    await userEvent.click(submitButton)

    // Verify API call
    await waitFor(() => {
      expect(fetch).toHaveBeenCalledWith(
        expect.stringContaining('/api/action'),
        expect.objectContaining({
          method: 'POST',
          body: expect.stringContaining('New post')
        })
      )
    })
  })
})

Load Testing

Testing with k6

import http from 'k6/http'
import { check, sleep } from 'k6'

export const options = {
  stages: [
    { duration: '1m', target: 50 },   // Ramp up to 50 users
    { duration: '3m', target: 50 },   // Stay at 50 users
    { duration: '1m', target: 100 },  // Ramp up to 100 users
    { duration: '3m', target: 100 },  // Stay at 100 users
    { duration: '1m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95% of requests should be below 500ms
    http_req_failed: ['rate<0.01'],    // Error rate should be below 1%
  },
}

const BASE_URL = 'http://localhost:3000'
let token = ''

export function setup() {
  // Login to get token
  const loginRes = http.post(`${BASE_URL}/api/auth/login`, JSON.stringify({
    email: 'test@example.com',
    password: 'password'
  }), {
    headers: { 'Content-Type': 'application/json' }
  })

  const loginData = JSON.parse(loginRes.body)
  return { token: loginData.data.token }
}

export default function (data) {
  const headers = {
    'Authorization': `Bearer ${data.token}`,
    'Content-Type': 'application/json'
  }

  // Test 1: Fetch actions
  const listRes = http.get(`${BASE_URL}/api/action?_limit=20`, { headers })
  check(listRes, {
    'status is 200': (r) => r.status === 200,
    'has data array': (r) => JSON.parse(r.body).data !== undefined,
  })

  sleep(1)

  // Test 2: Create action
  const createRes = http.post(
    `${BASE_URL}/api/action`,
    JSON.stringify({
      type: 'POST',
      content: { text: 'Load test post' },
      audience: ['public'],
      published: Math.floor(Date.now() / 1000)
    }),
    { headers }
  )

  check(createRes, {
    'create status is 200': (r) => r.status === 200,
    'has actionId': (r) => JSON.parse(r.body).data.actionId !== undefined,
  })

  const actionId = JSON.parse(createRes.body).data.actionId

  sleep(1)

  // Test 3: Get specific action
  const getRes = http.get(`${BASE_URL}/api/action/${actionId}`, { headers })
  check(getRes, {
    'get status is 200': (r) => r.status === 200,
  })

  sleep(1)

  // Test 4: Delete action
  const delRes = http.del(`${BASE_URL}/api/action/${actionId}`, null, { headers })
  check(delRes, {
    'delete status is 200': (r) => r.status === 200,
  })

  sleep(1)
}

Test Data Management

Fixtures and Factories

// test/fixtures/actions.ts
export const actionFixtures = {
  post: {
    actionId: 'act_post_1',
    type: 'POST',
    content: { text: 'Test post content' },
    audience: ['public'],
    issuerTag: 'alice@example.com',
    createdAt: 1735689600,
    published: 1735689600,
    status: 'A'
  },
  comment: {
    actionId: 'act_comment_1',
    type: 'COMMENT',
    content: { text: 'Test comment' },
    parentId: 'act_post_1',
    issuerTag: 'bob@example.com',
    createdAt: 1735689700,
    status: 'A'
  }
}

// test/factories/action-factory.ts
export class ActionFactory {
  static create(overrides: Partial<Action> = {}): Action {
    return {
      actionId: `act_${Math.random().toString(36).substr(2, 9)}`,
      type: 'POST',
      content: { text: 'Default post content' },
      audience: ['public'],
      issuerTag: 'test@example.com',
      createdAt: Math.floor(Date.now() / 1000),
      published: Math.floor(Date.now() / 1000),
      status: 'A',
      ...overrides
    }
  }

  static createMany(count: number, overrides: Partial<Action> = {}): Action[] {
    return Array.from({ length: count }, () => this.create(overrides))
  }

  static createPost(text: string): Action {
    return this.create({
      type: 'POST',
      content: { text }
    })
  }

  static createComment(parentId: string, text: string): Action {
    return this.create({
      type: 'COMMENT',
      parentId,
      content: { text }
    })
  }
}

// Usage in tests
const post = ActionFactory.createPost('Hello world')
const comment = ActionFactory.createComment(post.actionId, 'Nice post!')
const manyPosts = ActionFactory.createMany(10)

Database Seeding for Tests

// test/seed.ts
export async function seedTestData(token: string) {
  const baseUrl = 'http://localhost:3000'

  // Create test users
  const users = [
    { email: 'alice@example.com', name: 'Alice' },
    { email: 'bob@example.com', name: 'Bob' }
  ]

  // Create test posts
  const posts = []
  for (let i = 0; i < 10; i++) {
    const response = await fetch(`${baseUrl}/api/action`, {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${token}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({
        type: 'POST',
        content: { text: `Test post ${i}` },
        audience: ['public'],
        published: Math.floor(Date.now() / 1000)
      })
    })
    const { data } = await response.json()
    posts.push(data)
  }

  return { users, posts }
}

// Use in tests
beforeAll(async () => {
  const { posts } = await seedTestData(testToken)
  testData = { posts }
})

Continuous Integration

GitHub Actions Workflow

# .github/workflows/test.yml
name: Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Run unit tests
        run: npm run test:unit

      - name: Upload coverage
        uses: codecov/codecov-action@v3

  integration-tests:
    runs-on: ubuntu-latest
    services:
      cloudillo:
        image: cloudillo/server:latest
        ports:
          - 3000:3000

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3

      - name: Install dependencies
        run: npm ci

      - name: Wait for server
        run: npx wait-on http://localhost:3000/api/auth/login

      - name: Run integration tests
        run: npm run test:integration

  e2e-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright
        run: npx playwright install --with-deps

      - name: Run E2E tests
        run: npm run test:e2e

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: playwright-results
          path: test-results/

Best Practices

Test Organization

tests/
β”œβ”€β”€ unit/
β”‚   β”œβ”€β”€ services/
β”‚   β”‚   β”œβ”€β”€ action-service.test.ts
β”‚   β”‚   β”œβ”€β”€ profile-service.test.ts
β”‚   β”‚   └── file-service.test.ts
β”‚   └── utils/
β”‚       β”œβ”€β”€ validation.test.ts
β”‚       └── formatting.test.ts
β”œβ”€β”€ integration/
β”‚   β”œβ”€β”€ auth-flow.test.ts
β”‚   β”œβ”€β”€ action-crud.test.ts
β”‚   └── file-upload.test.ts
β”œβ”€β”€ e2e/
β”‚   β”œβ”€β”€ social-features.spec.ts
β”‚   β”œβ”€β”€ file-management.spec.ts
β”‚   └── real-time.spec.ts
β”œβ”€β”€ fixtures/
β”‚   β”œβ”€β”€ actions.ts
β”‚   β”œβ”€β”€ profiles.ts
β”‚   └── files.ts
β”œβ”€β”€ factories/
β”‚   └── action-factory.ts
└── helpers/
    β”œβ”€β”€ setup.ts
    └── teardown.ts

Writing Maintainable Tests

// βœ… Good - descriptive test names
test('should return 401 when token is expired', async () => { ... })

// ❌ Bad - vague test name
test('auth test', async () => { ... })

// βœ… Good - test one thing
test('validates email format', () => {
  expect(validateEmail('test@example.com')).toBe(true)
})

test('rejects invalid email format', () => {
  expect(validateEmail('invalid')).toBe(false)
})

// ❌ Bad - tests multiple things
test('email validation', () => {
  expect(validateEmail('test@example.com')).toBe(true)
  expect(validateEmail('invalid')).toBe(false)
  expect(validateEmail('')).toBe(false)
})

// βœ… Good - clear arrange-act-assert
test('creates action successfully', async () => {
  // Arrange
  const action = ActionFactory.createPost('Hello')

  // Act
  const result = await api.actions.create(action)

  // Assert
  expect(result.data.actionId).toBeDefined()
  expect(result.data.type).toBe('POST')
})

Summary

Key testing strategies for Cloudillo API:

  1. Unit Tests: Test business logic and utilities in isolation
  2. Integration Tests: Test API interactions with mocked services
  3. E2E Tests: Test complete user workflows
  4. Component Tests: Test UI components with API interactions
  5. Load Tests: Test performance under load
  6. CI/CD: Automate testing in CI pipeline

Aim for 80%+ code coverage while focusing on critical paths and edge cases.

CRDT Design Guide

A comprehensive guide to designing collaborative data structures using Yjs and CRDTs for real-time applications.

Overview

Building collaborative applications requires careful consideration of how data structures behave when multiple users edit simultaneously. This guide covers the design patterns, best practices, and common pitfalls when working with Conflict-free Replicated Data Types (CRDTs) in Cloudillo applications.

Early Stage Documentation

The patterns in this guide are based on internal Cloudillo applications (Calcillo, Ideallo, Prezillo, Quillo) which have shown promising results in our testing. However, Cloudillo has not yet achieved wide adoption, so these recommendations should be considered with appropriate caution. We’re sharing our experience to help the community, but real-world usage at scale may reveal patterns that need adjustment.

Who This Guide Is For

This guide is for developers who:

  • Are building collaborative features in Cloudillo applications
  • Need to understand how to structure data for real-time synchronization
  • Want to learn from real-world patterns used in Cloudillo apps (Calcillo, Ideallo, Prezillo, Quillo)

Prerequisites

Before diving in, you should be familiar with:

  • Basic JavaScript/TypeScript
  • The CRDT API for Cloudillo integration
  • General concepts of collaborative editing

Topics

Fundamentals

Core Yjs concepts including shared types, document structure, and how CRDTs work internally.

Design Patterns

Proven patterns for structuring collaborative data: ID-based storage, nested maps, content separation, and more.

Application Types

Specific guidance for different document types: text editors, spreadsheets, canvas/whiteboard apps, and presentations.

Collaboration Features

Implementing multi-user features: transactions, undo/redo, presence awareness, and conflict resolution.

Pitfalls

Common mistakes that cause data corruption, sync issues, or unexpected behaviorβ€”and how to avoid them.

Quick Start

If you’re new to CRDT design, start with:

  1. Shared Types - Understand Y.Map, Y.Array, Y.Text
  2. ID-Based Storage - The most important pattern for collaborative apps
  3. Transactions - Properly batching changes

See Also

Subsections of CRDT Design Guide

Fundamentals

Core Yjs concepts you need to understand before designing collaborative data structures.

Overview

Yjs provides a set of shared data types that automatically synchronize across clients and resolve conflicts. Understanding these primitives is essential for building robust collaborative applications.

Topics

Key Concepts

Shared Types Are Not Regular Objects

Yjs shared types look similar to JavaScript objects and arrays, but they behave differently:

// Regular JavaScript - changes are local only
const obj = { name: 'Alice' }
obj.name = 'Bob'  // Only visible locally

// Yjs shared type - changes synchronize
const yMap = yDoc.getMap('user')
yMap.set('name', 'Alice')
yMap.set('name', 'Bob')  // Syncs to all connected clients

Documents Contain All Shared Data

A Y.Doc is the root container for all collaborative data. Different parts of your application access different top-level keys:

const yDoc = new Y.Doc()

// Each getMap/getArray/getText creates a named root-level shared type
const cells = yDoc.getMap('cells')      // Spreadsheet data
const order = yDoc.getArray('rowOrder') // Row ordering
const meta = yDoc.getMap('metadata')    // Document metadata

Changes Must Go Through Yjs APIs

Yjs only tracks changes made through its APIs. Direct mutation of extracted values does not synchronize:

// WRONG - changes not tracked
const obj = yMap.get('config')
obj.theme = 'dark'  // This change is lost!

// CORRECT - use Yjs API
yMap.set('config', { ...yMap.get('config'), theme: 'dark' })

Subsections of Fundamentals

Shared Types

Understanding Yjs shared types: Y.Map, Y.Array, Y.Text, and Y.XmlFragment.

Overview

Yjs provides four primary shared types. Each has specific characteristics for different use cases.

Y.Map

A key-value store similar to JavaScript’s Map. Supports nested shared types for hierarchical structures.

Key behavior: Setting a plain object snapshots itβ€”later mutations don’t sync. For granular updates, nest Y.Map instances.

Best For:

  • Keyed data where order doesn’t matter
  • Configuration objects
  • Entity storage (keyed by ID)
  • Hierarchical structures (nested maps)

Y.Array

An ordered list similar to JavaScript’s Array. Handles concurrent insertions gracefully with position-aware merging.

Key behavior: Reordering items (delete + insert) creates copies, not moves. For reorderable collections, store only IDs in the array.

Best For:

  • Ordered lists where sequence matters
  • ID arrays for ordering (with content in a separate Y.Map)
  • Text editor paragraphs
  • Timeline events
Avoid Complex Objects in Arrays

Don’t store complex objects in Y.Array if you need to reorder them. Store IDs in the array and content in a Y.Map instead. See ID-Based Storage.

Y.Text

A shared string optimized for collaborative text editing. Supports character-level insertions, deletions, and formatting attributes.

Key behavior: Concurrent edits merge at character positions. Formatting uses Quill-style delta operations.

// Rich text formatting
yText.insert(0, 'Bold text', { bold: true })
yText.format(0, 4, { italic: true })  // Apply to range

Editor Bindings: Integrates with Quill, ProseMirror, Monaco, CodeMirror, and TipTap through official bindings.

Best For:

  • Rich text documents
  • Code editors
  • Chat messages
  • Any content needing character-level merging

Y.XmlFragment

An XML-like structure for representing DOM or document trees. Used primarily with ProseMirror for complex rich text with nested elements.

Best For:

  • ProseMirror integration
  • DOM-like document structures
  • Complex rich text with nested elements

Type Selection Guide

Data Type Use Case Conflict Resolution
Y.Map Key-value data, entities Last-writer-wins per key
Y.Array Ordered lists, sequences Position-aware insertion
Y.Text Text content, rich text Character-level merging
Y.XmlFragment DOM structures Element-level operations

See Also

Document Structure

Organizing your Y.Doc for maintainable and efficient collaborative applications.

Overview

A Y.Doc is the root container for all collaborative data. Structure affects performance, maintainability, and how concurrent edits merge.

Structure by Purpose

Organize data into distinct categories:

Document
β”œβ”€β”€ Content Data       (cells, objects, text)
β”œβ”€β”€ Ordering Data      (arrays of IDs)
β”œβ”€β”€ Metadata           (title, createdAt, version)
└── Configuration      (user preferences, settings)

Example: Spreadsheet

Document
β”œβ”€β”€ cells: Map<rowId, Map<colId, CellData>>
β”œβ”€β”€ rowOrder: Array<rowId>
β”œβ”€β”€ colOrder: Array<colId>
β”œβ”€β”€ rowProps: Map<rowId, {height}>
β”œβ”€β”€ colProps: Map<colId, {width}>
β”œβ”€β”€ namedRanges: Map<name, Range>
└── styles: Map<styleId, Style>

Example: Canvas App

Document
β”œβ”€β”€ objects: Map<objectId, DrawingObject>
β”œβ”€β”€ layers: Map<layerId, LayerData>
β”œβ”€β”€ layerOrder: Map<layerId, Array<objectId>>
β”œβ”€β”€ paths: Map<pathId, PathData>
β”œβ”€β”€ textBoxes: Map<textId, Y.Text>
└── metadata: Map<key, value>

Example: Presentation

Document
β”œβ”€β”€ slides: Map<slideId, SlideData>
β”œβ”€β”€ slideOrder: Array<slideId>
β”œβ”€β”€ containers: Map<containerId, Container>
β”œβ”€β”€ masters: Map<masterId, MasterTemplate>
β”œβ”€β”€ styles: Map<styleId, Style>
└── notes: Map<slideId, string>

Design Guidelines

Separate Content from Order

  • Store content in Y.Map by ID
  • Store ordering in Y.Array (IDs only)
  • Never put complex objects in reorderable arrays

Flatten When Possible

  • Deep nesting has overhead
  • Prefer 2-3 levels max
  • Use flat maps with composite keys when appropriate

Use TypeScript Interfaces

  • Define types for your document structure
  • Create helper functions for typed access

Initialize Required Structure

  • Access top-level types on document creation
  • Set defaults in a transaction

Performance Notes

  • Many top-level types is fineβ€”enables granular sync
  • Deeply nested types (4+ levels) add overhead
  • Large arrays (10,000+ items): consider pagination or chunking

See Also

Internals

How Yjs works under the hoodβ€”understanding the mechanics behind CRDT synchronization.

The CRDT Model

Simplified Overview

This page provides a practical understanding of Yjs internals. For definitive details, consult the Yjs documentation.

Yjs is primarily operation-based: it stores and transmits operations (inserts, deletes). However, it also supports state encoding for snapshots and initial sync. This combination provides:

  • Efficient sync - Only missing operations are transmitted based on vector clock comparison
  • Full state snapshots - New clients can receive complete state without operation replay
  • Compact updates - Ongoing changes are small binary operation deltas

Items and the Item List

Internally, all data is a linked list of “items”:

[item1] <-> [item2] <-> [item3] <-> [item4]

Each item contains:

  • ID - Unique (clientId, clock) pair
  • Content - The actual data
  • Origin - Item this was inserted after
  • Right Origin - Item this was inserted before

Client IDs and Clocks

Every client has a unique ID and logical clock. Item IDs are (clientId, clock) pairs, ensuring globally unique identifiers without coordination.

Vector Clocks

State vectors track what each client has seen:

{
  clientA: 15,  // Has seen A's operations up to clock 15
  clientB: 8,   // Has seen B's operations up to clock 8
}

When syncing, only missing operations are sent based on vector clock comparison.

Conflict Resolution

Y.Map: Last-writer-wins by logical timestamp. Higher clock wins.

Y.Array: Concurrent insertions at same position are both preserved. Order determined by client ID.

Y.Text: Character-level merging. Concurrent insertions both appear; order by position and client ID.

The Update Format

Changes are encoded as compact binary:

yDoc.on('update', (update: Uint8Array) => {
  // Send over network or store for persistence
})

Y.applyUpdate(yDoc, update)  // Apply received update
Y.mergeUpdates([u1, u2, u3]) // Compact multiple updates

Garbage Collection

Deleted items become tombstones (needed for concurrent operation resolution). Tombstones are eventually garbage collected when all clients have moved past them.

GC Implications

Heavy editing accumulates tombstones until GC. Very long-lived, heavily-edited documents may grow larger than expected.

Subdocuments

For large documents, split into subdocuments for lazy loading:

const mainDoc = new Y.Doc()
const subDoc = new Y.Doc({ guid: 'chapter-1' })
mainDoc.getMap('subdocs').set('chapter1', subDoc)

Performance Characteristics

The following are approximate complexities for typical use cases. Actual performance varies by implementation details, document structure, and operation history:

Operation Approximate Complexity
Map get/set O(1) average
Array push O(1)
Array insert at index O(n)
Text insert O(log n) typical*
Sync (diff) O(changes)

* Text insertion complexity depends on the document’s internal structure and edit history.

Space overhead: For typical documents, expect 2-10x the raw data size due to CRDT metadata and tombstones. Very long-lived, heavily-edited documents may accumulate more overhead. Documents with minimal edits will be closer to the lower bound.

See Also

Design Patterns

Proven patterns for structuring collaborative data that scales and handles concurrent edits gracefully.

Overview

These patterns emerge from real-world collaborative applications. They solve common problems like maintaining order while allowing concurrent edits, organizing complex hierarchies, and keeping data structures efficient.

Topics

Pattern Selection Guide

Your Need Recommended Pattern
Items that can be reordered ID-Based Storage + Ordering Arrays
Many small entities vs complex objects JSON vs Y.Map Tradeoffs
Hierarchical data (trees) ID-Based Storage with parent references
Large documents with many object types Separate Content Maps
Theming or default values Style Inheritance
Simple key-value storage Direct Y.Map (no special pattern needed)

The Most Important Rule

Never store complex objects in arrays that will be reordered.

This single rule prevents the most common and destructive CRDT bugs. When you need to reorder items:

  1. Store the actual content in a Y.Map keyed by ID
  2. Store only the IDs in a Y.Array for ordering
  3. Reorder by moving IDs in the array, not the content

See ID-Based Storage for complete examples.

Subsections of Design Patterns

ID-Based Storage

The most important pattern for collaborative applications: storing content by ID and referencing by ID.

Overview

ID-based storage separates what content exists from where it appears. Store content in a map keyed by unique IDs, and reference those IDs from elsewhere.

This prevents data loss during concurrent reordering and enables reliable cross-references.

The Problem

Storing content directly in a reorderable array:

// WRONG - content in array
const slides = yDoc.getArray('slides')
slides.push([{ id: 'k8d2fn3m', title: 'Intro', content: [...] }])

When two users reorder the same slide concurrently, delete + insert creates copiesβ€”resulting in duplicated or lost content.

The Solution

Separate content storage from ordering:

const slideContent = yDoc.getMap('slides')
const slideOrder = yDoc.getArray('slideOrder')

// Store content by ID
slideContent.set('k8d2fn3m', { title: 'Intro', content: [...] })

// Store only ID in ordering array
slideOrder.push(['k8d2fn3m'])

Now reordering only moves IDsβ€”lightweight operations that merge cleanly.

Core Operations

All operations should be wrapped in yDoc.transact() for atomicity:

Operation Steps
Create content.set(id, data) + order.push([id])
Reorder Delete ID from old index, insert at new index
Delete order.delete(index, 1) + content.delete(id)
Read order.toArray().map(id => content.get(id))

Always delete from both order and content to avoid orphaned data.

Benefits

  • Safe Reordering: Moving items only moves IDs, which merge cleanly
  • Stable References: Other parts can reference by ID without breaking
  • Efficient Updates: Content changes don’t affect order, and vice versa
  • Easy Deletion: References become stale IDs that can be filtered
  • Undo Granularity: Content and order changes are separate undo steps

Hierarchical Data (Trees)

ID-based storage extends naturally to trees. Store all nodes flat with parent references:

nodes/
β”œβ”€β”€ k8d2fn3m β†’ { name: 'Documents', parentId: null }
β”œβ”€β”€ m4x9pt2q β†’ { name: 'Work', parentId: 'k8d2fn3m' }
β”œβ”€β”€ j7n3ks8w β†’ { name: 'Report.md', parentId: 'm4x9pt2q' }
└── p2r6vm4c β†’ { name: 'Notes.md', parentId: 'm4x9pt2q' }

Why flat beats nested maps for trees:

  • Moving nodes is a single parentId update, not delete + insert
  • Cross-references (shortcuts, symlinks) work naturally
  • Depth changes don’t require restructuring

For sibling ordering, add a childOrder array per parent or a separate ordering map.

Common Mistakes

Mistake Problem Solution
Using array indices as references Indices shift when items are inserted/deleted Use stable IDs for references
Deleting from order only Orphaned content accumulates in the map Delete from both order and content
Not using transactions Sync may occur between operations Wrap related changes in yDoc.transact()

See Also

JSON vs Y.Map Tradeoffs

Choosing between plain JSON objects and nested Y.Maps is a critical design decision.

The Core Tradeoff

Aspect JSON Objects Nested Y.Maps
Update granularity Whole object replaced Per-field
Concurrent field edits Last-write-wins (data loss!) Merge correctly
Memory overhead Lower Higher (CRDT metadata)
Code complexity Simpler More complex

The Concurrent Edit Problem

This is the critical consideration. With JSON objects, concurrent field edits cause data loss.

// Two users editing same entity
entities.set('player', { name: 'Alice', score: 100 })

// User A updates name:
entities.set('player', { ...entities.get('player'), name: 'Bob' })

// User B updates score (concurrently):
entities.set('player', { ...entities.get('player'), score: 200 })

// Result: ONE UPDATE IS LOST!
// Either { name: 'Bob', score: 100 } or { name: 'Alice', score: 200 }

With Y.Map, both changes merge correctly β†’ { name: 'Bob', score: 200 }

Decision Framework

Use JSON when:

  • Thousands of small entities (memory matters)
  • Entities rarely updated after creation
  • Users won’t edit same entity simultaneously
  • Values are primitives

Use Y.Map when:

  • Concurrent editing of same entity is likely
  • Entities have many fields
  • Frequent partial updates
  • Field-level merging matters

Hybrid: Separate by Volatility

Different parts of entities may have different update patterns:

// Cell values: frequent, small, different users β†’ different cells
const values = yDoc.getMap('values')     // JSON primitives
values.set('A1', 42)

// Styles: rare updates, can replace whole
const styles = yDoc.getMap('styles')     // JSON objects
styles.set('A1', { bold: true, color: '#000' })

// Canvas objects: concurrent x/y/size edits on same object
const objects = yDoc.getMap('objects')   // Nested Y.Maps

Memory Impact

Structure ~Overhead per item
JSON in Y.Map ~50 bytes
Nested Y.Map (5 fields) ~300 bytes

For 10,000 items: ~500KB (JSON) vs ~3MB (Y.Map)

Guidelines Summary

Scenario Recommendation
Large collection of small values JSON
Complex objects, unlikely concurrent edits JSON
Complex objects, likely concurrent edits Y.Map required
Mixed update patterns Separate maps by volatility

See Also

Separate Content Maps

Splitting data by type into separate Y.Maps for better organization and performance.

The Pattern

Instead of one nested structure, use separate maps by type:

// Instead of one big data.objects with mixed types...
const shapes = yDoc.getMap('shapes')
const images = yDoc.getMap('images')
const textBoxes = yDoc.getMap('textBoxes')
const paths = yDoc.getMap('paths')

Benefits

  • Targeted observers: Subscribe only to relevant types (shapes.observe() won’t fire for image changes)
  • Faster lookups: images.toJSON() is faster than filtering a mixed collection by type
  • Type safety: Each map can have its own TypeScript interface (Y.Map<ShapeData>, Y.Map<ImageData>)

Shared Ordering

Objects from different maps can share ordering via layer arrays:

const layerOrder = yDoc.getMap('layerOrder')  // Map<layerId, Array<objectId>>

// Object IDs in layers, regardless of type
layerOrder.get('default').push([shapeId])
layerOrder.get('default').push([imageId])

Getting All Objects

When you need all objects regardless of type, spread the values from each map or check each map when looking up by ID. With consistent ID formats (e.g., type-prefixed IDs like shape-xxx, image-xxx), lookups can go directly to the right map.

When to Separate

Separate when:

  • Different types have different properties
  • You query by type frequently
  • Different UI areas handle different types

Keep together when:

  • Objects are always processed together
  • You have very few objects total

See Also

Ordering with Arrays

Using Y.Array to maintain element order in collaborative applications.

Overview

Y.Array provides ordered sequences that handle concurrent insertions gracefully. Combined with ID-based storage, Y.Array becomes the standard way to represent ordered collections.

The Pattern

Store only IDs in Y.Array; store content in Y.Map. See ID-Based Storage for the complete pattern and core operations.

Concurrent Insert Behavior

When two users insert at the same position simultaneously:

Initial: [A, B, C]

User 1 inserts X after A: [A, X, B, C]
User 2 inserts Y after A: [A, Y, B, C]

Merged: [A, X, Y, B, C] or [A, Y, X, B, C]

The order of X and Y is determined by client IDs (arbitrary but consistent). Both items appearβ€”neither is lost.

Grouped Ordering

For items grouped into categories, use a Map of Arrays:

items/           β†’ Map<itemId, ItemData>
groupOrders/     β†’ Map<groupId, Y.Array<itemId>>
  β”œβ”€β”€ 'todo'     β†’ ['k8d2fn3m', 'j7n3ks8w']
  β”œβ”€β”€ 'doing'    β†’ ['m4x9pt2q']
  └── 'done'     β†’ ['p2r6vm4c', 'q5t8wn2x']

Moving an item between groups: delete ID from source array, insert into target array, update item’s groupId fieldβ€”all in one transaction.

Performance Tips

Large arrays (10,000+ items):

  • Cache order.toArray() instead of calling repeatedly
  • Use UI pagination/virtualization

Batch operations:

// WRONG: multiple syncs
for (const id of idsToRemove) {
  order.delete(order.toArray().indexOf(id), 1)
}

// CORRECT: single transaction, delete from end first
yDoc.transact(() => {
  const indices = idsToRemove
    .map(id => order.toArray().indexOf(id))
    .filter(i => i !== -1)
    .sort((a, b) => b - a)  // Descending
  for (const i of indices) order.delete(i, 1)
})

See Also

Style Inheritance

Implementing prototype chains and style inheritance for themes, defaults, and templates.

Application Pattern

Style inheritance is an application-level pattern built on top of CRDTs, not a CRDT feature itself. It leverages CRDT properties (automatic sync, conflict resolution) while keeping the inheritance logic in application code.

The Pattern

Instead of storing all properties on each element:

  1. Store a reference to a parent style/template
  2. Store only the overridden properties
const styles = yDoc.getMap('styles')
styles.set('heading', {
  fontSize: 24,
  fontWeight: 'bold',
  color: '#333333'
})

const elements = yDoc.getMap('elements')
elements.set('title', {
  text: 'Welcome',
  styleId: 'heading',
  overrides: { color: '#0066cc' }  // Override just this
})

Resolving Styles

function resolveStyle(element: ElementData): ResolvedStyle {
  const baseStyle = styles.get(element.styleId) || {}
  return { ...baseStyle, ...element.overrides }
}

// { fontSize: 24, fontWeight: 'bold', color: '#0066cc' }

Multi-Level Inheritance

Chain multiple levels for complex theming:

theme/colors     β†’ { primary: '#0066cc', text: '#333' }
      ↓
templates/card   β†’ { padding: 16, titleColor: 'theme.colors.primary' }
      ↓
elements/card-1  β†’ { templateId: 'card', overrides: { padding: 24 } }

Resolution cascades: element overrides β†’ template defaults β†’ theme values.

Named Style Classes

Like CSS classes, elements can reference multiple styles by name. Store class definitions in a map (styleClasses), and let elements specify an array of class names. Resolution merges classes in order, then applies overrides.

Cascading Updates

When a base style changes, all dependent elements automatically get new values:

theme.observeDeep(() => {
  renderAll()  // Elements resolve to new values
})

See Also

Application Types

Specific design guidance for different types of collaborative documents.

Overview

Different application types have different data structure needs. A text editor has very different requirements from a spreadsheet or drawing canvas. This section provides tailored guidance for each category.

Application Categories

  • Text Editors - Rich text with Y.Text and editor bindings
  • Spreadsheets - 2D grids with cells, rows, and columns (Calcillo patterns)
  • Canvas Apps - Drawing and whiteboard applications (Ideallo patterns)
  • Presentations - Slide-based documents with containers and views (Prezillo patterns)

Choosing the Right Approach

Application Type Primary Data Structure Key Pattern
Text editor Y.Text Editor binding (Quill, ProseMirror)
Spreadsheet Y.Map of cells ID-based cells, ordered rows/columns
Canvas/Whiteboard Y.Map of objects Separate content maps by type
Presentations Y.Map of containers Style inheritance, templates

Common Themes

Despite their differences, all collaborative applications share these needs:

  1. Stable references - Use IDs, not indices, to reference other elements
  2. Separate ordering from content - Store order in arrays, content in maps
  3. Batch changes - Use transactions to group related modifications
  4. Local state separation - Keep UI state (selection, scroll) out of the CRDT

Each application type page shows how to apply these principles to that specific domain.

Subsections of Application Types

Text Editors

Designing collaborative rich text editors with Y.Text.

Document Structure

Document
β”œβ”€β”€ content: Y.Text             (main document)
β”œβ”€β”€ metadata: Map<key, value>   (title, created, etc.)
└── comments: Map<commentId, Comment>

Use Y.Text for the main content. Editor bindings (y-quill, y-prosemirror, y-tiptap) handle synchronization automatically.

Comments with Relative Positions

Comments reference text ranges that must survive edits. Use Y.RelativePosition:

interface Comment {
  id: string
  text: string
  startPosition: Y.RelativePosition  // Survives text changes
  endPosition: Y.RelativePosition
}

Convert between absolute and relative positions when creating/reading comments. Relative positions stay valid even when text is inserted or deleted around them.

Common Mistakes

Storing HTML as plain text:

// WRONG - loses semantic structure
yText.insert(0, '<p><strong>Hello</strong></p>')

// CORRECT - use formatting attributes
yText.insert(0, 'Hello', { bold: true })

Absolute positions for persistent references:

// WRONG: position 42 becomes invalid when text changes
const savedPosition = 42

// CORRECT: relative position adapts to edits
const relPos = Y.createRelativePositionFromTypeIndex(yText, 42)

See Also

Spreadsheets

Designing collaborative spreadsheets with 2D grids, cell addressing, and formula support.

Document Structure

There are two common approaches for cell storage, each with tradeoffs:

Approach 1: Nested Maps

Document
β”œβ”€β”€ cells: Map<rowId, Map<colId, CellData>>
β”œβ”€β”€ rowOrder: Array<rowId>
β”œβ”€β”€ colOrder: Array<colId>
...

Pros: Efficient row operations (iterate all cells in a row), natural grouping. Cons: More complex access pattern, nested map setup for each row.

Approach 2: Flat Map with Composite Keys

Document
β”œβ”€β”€ cells: Map<"rowId:colId", CellData>
β”œβ”€β”€ rowOrder: Array<rowId>
β”œβ”€β”€ colOrder: Array<colId>
...

Pros: Simpler access (cells.get(`${rowId}:${colId}`)), flat structure. Cons: Row iteration requires filtering all keys, slightly more parsing overhead.

Common Structure (Both Approaches)

Document
β”œβ”€β”€ cells: (see above)
β”œβ”€β”€ rowOrder: Array<rowId>
β”œβ”€β”€ colOrder: Array<colId>
β”œβ”€β”€ rowProps: Map<rowId, {height}>
β”œβ”€β”€ colProps: Map<colId, {width}>
β”œβ”€β”€ namedRanges: Map<name, Range>
└── styles: Map<styleId, Style>

Choose based on your access patterns: nested if you frequently operate on entire rows, flat if you primarily access individual cells.

Cell Addressing

Cells are addressed via row/column IDs, not traditional A1 notation. Convert for display:

// Internal: ID-based
const cellRef = { rowId: 'r8k2mf9n', colId: 'c3nd8k2m' }

// Display: convert to A1 when showing to user
function idsToA1(rowId: string, colId: string): string {
  const colIndex = colOrder.toArray().indexOf(colId)
  const rowIndex = rowOrder.toArray().indexOf(rowId)
  return indexToLetter(colIndex) + (rowIndex + 1)  // "B3"
}

Formula References

Use IDs, not indices. When rows/columns are inserted or deleted, A1-style references break. Store references as cell IDs:

interface ParsedFormula {
  expression: string
  references: Array<{
    type: 'cell' | 'range'
    rowId: string
    colId: string
    endRowId?: string  // For ranges
    endColId?: string
  }>
}

// Example: =SUM(B1:B10) stored as
{
  expression: 'SUM(range1)',
  references: [{
    type: 'range',
    rowId: 'r8k2mf9n', colId: 'c3nd8k2m',
    endRowId: 'r9m3kf2n', endColId: 'c3nd8k2m'
  }]
}

Dependency Tracking

Track which cells depend on which for efficient recalculation:

const dependencies = new Map<string, Set<string>>()

// On cell change, only recalculate affected cells
function onCellChange(cellKey: string) {
  const dependents = dependencies.get(cellKey)
  if (dependents) {
    for (const dep of dependents) {
      recalculateCell(dep)
    }
  }
}
Circular References

Implement cycle detection when evaluating formulas. Circular references should show an error rather than causing infinite loops.

Collaborative Features

Selection awareness: Share current cell selection via awareness (not CRDT):

awareness.setLocalStateField('selection', {
  user: { name: 'Alice', color: '#f783ac' },
  cell: { rowId: 'r8k2mf9n', colId: 'c3nd8k2m' }
})

Virtual rendering: For large sheets, only render visible cells based on viewport.

Common Mistakes

Using indices in formulas:

// WRONG: indices shift
formula: '=A1 * B2'

// CORRECT: use stable IDs
formula: { refs: ['r3k9mf8n:c8m2pt3q', 'r3k9mf8n:c2n7ks4w'], expr: '*' }

Recalculating everything:

// WRONG: O(n) on every change
cells.observe(() => recalculateAllCells())

// CORRECT: only affected cells
cells.observe(event => {
  event.changes.keys.forEach((_, key) => {
    recalculateDependents(key)
  })
})

See Also

Canvas Apps

Designing collaborative drawing and whiteboard applications.

Document Structure

Document
β”œβ”€β”€ objects: Map<objectId, DrawingObject>
β”œβ”€β”€ layers: Map<layerId, LayerData>
β”œβ”€β”€ layerOrder: Map<layerId, Array<objectId>>
β”œβ”€β”€ paths: Map<pathId, PathData>        (large path data)
β”œβ”€β”€ textContent: Map<textId, Y.Text>    (rich text)
└── metadata: Map<key, value>

Store large data (path points, rich text) separately from object metadataβ€”reference by ID.

Object Model

interface BaseObject {
  id: string
  type: 'rect' | 'ellipse' | 'path' | 'text' | 'image' | 'group'
  layerId: string
  x: number; y: number
  width: number; height: number
  rotation: number
  opacity: number
  visible: boolean
  locked: boolean
}

// Large data stored separately
interface PathObject extends BaseObject {
  type: 'path'
  pathDataId: string  // Reference to paths map
}

interface TextObject extends BaseObject {
  type: 'text'
  textContentId: string  // Reference to Y.Text
}

Z-Order (Layering)

Each layer has an ordered array of object IDs:

function bringToFront(objectId: string) {
  const obj = objects.get(objectId)
  const order = layerOrder.get(obj.layerId) as Y.Array<string>
  const index = order.toArray().indexOf(objectId)

  if (index !== -1 && index < order.length - 1) {
    yDoc.transact(() => {
      order.delete(index, 1)
      order.push([objectId])
    })
  }
}

Freehand Drawing

For performance, collect points locally, then commit on stroke end:

let currentPoints: Point[] = []

function onPointerMove(x: number, y: number) {
  currentPoints.push({ x, y })
  renderLocalPreview()  // Don't sync yet
}

function onPointerUp() {
  const simplified = simplifyPath(currentPoints)  // Reduce points
  const pathId = nanoid(8)
  const objectId = nanoid(8)

  yDoc.transact(() => {
    paths.set(pathId, { points: simplified })
    objects.set(objectId, {
      type: 'path',
      pathDataId: pathId,
      // ... other properties
    })
    layerOrder.get('default').push([objectId])
  })

  currentPoints = []
}

Collaborative Cursors

Throttle cursor updates to avoid flooding:

const updateCursor = throttle((x: number, y: number) => {
  awareness.setLocalStateField('cursor', {
    x, y,
    user: { name: 'Alice', color: '#f783ac' }
  })
}, 50)  // Max 20 updates/second

Common Mistakes

Storing render state in CRDT:

// WRONG: UI state in CRDT
objects.set(id, { ...obj, isSelected: true, isDragging: true })

// CORRECT: keep render state local
const localState = new Map<string, { isSelected: boolean }>()

Large path data in object:

// WRONG: thousands of points in object
objects.set(id, { ...obj, points: hugePointArray })

// CORRECT: store separately
paths.set(pathId, { points: hugePointArray })
objects.set(id, { ...obj, pathDataId: pathId })

See Also

Presentations

Designing collaborative presentation software with slides, templates, and views.

Document Structure

Document
β”œβ”€β”€ slides: Map<slideId, SlideData>
β”œβ”€β”€ slideOrder: Array<slideId>
β”œβ”€β”€ containers: Map<containerId, Container>
β”œβ”€β”€ masters: Map<masterId, MasterTemplate>
β”œβ”€β”€ styles: Map<styleId, Style>
└── notes: Map<slideId, string | Y.Text>

Slides contain references to containers (text boxes, shapes, images). Masters define reusable layouts and styles.

Slide/Container Relationship

interface Slide {
  id: string
  masterId: string
  containerIds: string[]
  background?: Background
}

interface Container {
  id: string
  slideId: string
  type: 'text' | 'image' | 'shape'
  x: number; y: number
  width: number; height: number
  contentId?: string  // For text: Y.Text ID; for image: blob ID
  styleId?: string
  zIndex: number
}

When duplicating a slide, duplicate its containers too:

function duplicateSlide(slideId: string): string {
  const slide = slides.get(slideId)
  const newId = nanoid(8)
  const newContainerIds: string[] = []

  yDoc.transact(() => {
    for (const cid of slide.containerIds) {
      const container = containers.get(cid)
      const newCid = nanoid(8)
      containers.set(newCid, { ...container, id: newCid, slideId: newId })
      newContainerIds.push(newCid)
    }

    slides.set(newId, {
      ...slide, id: newId,
      containerIds: newContainerIds
    })

    const index = slideOrder.toArray().indexOf(slideId)
    slideOrder.insert(index + 1, [newId])
  })

  return newId
}

Style Inheritance

Resolve styles by cascading: master β†’ slide β†’ container:

function resolveStyle(container: Container): ResolvedStyle {
  const slide = slides.get(container.slideId)
  const master = masters.get(slide.masterId)

  let style = {}

  // 1. Master base style
  if (master?.styles?.[container.styleId]) {
    style = { ...master.styles[container.styleId] }
  }

  // 2. Slide overrides
  if (slide?.styleOverrides?.[container.styleId]) {
    style = { ...style, ...slide.styleOverrides[container.styleId] }
  }

  // 3. Container overrides
  if (container.styleOverrides) {
    style = { ...style, ...container.styleOverrides }
  }

  return style
}

Presentation Mode

Presentation state is local (not in CRDT), but can be shared via awareness:

// Broadcast current slide for "follow presenter" mode
awareness.setLocalStateField('presenting', {
  slideIndex: currentIndex,
  user: getCurrentUser()
})

// Follow mode: sync to presenter's slide
awareness.on('change', () => {
  if (followingClientId) {
    const state = awareness.getStates().get(followingClientId)
    if (state?.presenting) {
      goToSlide(state.presenting.slideIndex)
    }
  }
})

Common Mistakes

Slide content directly in slideOrder:

// WRONG: content in array
slideOrder.push([{ title: 'Intro', containers: [...] }])

// CORRECT: content separate
slides.set(id, { title: 'Intro', containerIds: [...] })
slideOrder.push([id])

Not cleaning up containers on slide delete:

function deleteSlide(slideId: string) {
  const slide = slides.get(slideId)
  yDoc.transact(() => {
    // Delete containers first
    for (const cid of slide.containerIds) {
      containers.delete(cid)
    }
    // Then slide
    const index = slideOrder.toArray().indexOf(slideId)
    if (index !== -1) slideOrder.delete(index, 1)
    slides.delete(slideId)
  })
}

See Also

Collaboration Features

Implementing multi-user collaboration features: transactions, undo/redo, presence, and conflict handling.

Overview

Beyond basic data synchronization, collaborative applications need features that make the multi-user experience smooth and intuitive. This section covers the key collaboration patterns.

Topics

Why These Features Matter

Transactions

Without transactions, each change triggers a separate sync and observer event. This causes:

  • Performance issues from excessive network traffic
  • Visual flickering as partial changes render
  • Undo capturing individual operations instead of logical units

Undo/Redo

Users expect undo to reverse their own changes, not their collaborators’. Proper undo implementation requires:

  • Per-user tracking with trackedOrigins
  • Integration with transactions for atomic operations
  • Cursor restoration for text editing

Awareness

Seeing where other users are working prevents conflicts and enables coordination:

  • Cursor positions and selections
  • User names and colors
  • Typing indicators and active states

Conflict Resolution

Understanding how CRDTs resolve conflicts helps you design structures that merge intuitively:

  • Last-writer-wins for simple values
  • Character-level merging for text
  • Concurrent array insertions

Subsections of Collaboration Features

Transactions

Batching changes with yDoc.transact() for atomic operations and better performance.

Why Transactions

Transactions ensure that:

  • All changes sync together (not partially)
  • Observers fire once (not for each operation)
  • Undo captures the entire transaction
// Without transaction: 2 syncs, 2 observer events
items.set('k8d2fn3m', { name: 'Task 1' })
order.push(['k8d2fn3m'])

// With transaction: 1 sync, 1 observer event
yDoc.transact(() => {
  items.set('k8d2fn3m', { name: 'Task 1' })
  order.push(['k8d2fn3m'])
})

Atomic Sync

Without transactions, remote clients might see partial state:

// WRONG: order might sync before content
order.push(['k8d2fn3m'])
items.set('k8d2fn3m', data)

// CORRECT: atomic
yDoc.transact(() => {
  items.set('k8d2fn3m', data)
  order.push(['k8d2fn3m'])
})

Transaction Origins

The second argument identifies the change source:

yDoc.transact(() => {
  content.set('title', 'New Title')
}, 'user-action')

With UndoManager

Filter which changes to track:

const undoManager = new UndoManager([content], {
  trackedOrigins: new Set(['user-action'])
})

// Tracked for undo
yDoc.transact(() => content.set('title', 'New'), 'user-action')

// NOT tracked (remote sync, system updates)
yDoc.transact(() => metadata.set('modified', Date.now()), 'system')

In Observers

items.observe((event, transaction) => {
  if (transaction.origin === 'import') return  // Skip re-render
  renderItems()
})

Nested Transactions

Nested transactions merge into the outermost:

function addItem(data) {
  yDoc.transact(() => {
    items.set(data.id, data)
    order.push([data.id])
  })
}

function addMultiple(dataArray) {
  yDoc.transact(() => {
    for (const data of dataArray) {
      addItem(data)  // Inner transact merges into outer
    }
  })
  // Single sync, single observer event
}

Anti-Patterns

Async inside transactions:

// WRONG: async breaks transaction
yDoc.transact(async () => {
  items.set('a', { ... })
  await saveToServer()  // Transaction already ended!
  items.set('b', { ... })  // NOT in same transaction
})

// CORRECT
yDoc.transact(() => {
  items.set('a', { ... })
  items.set('b', { ... })
})
await saveToServer()

Over-large transactions:

// For huge imports, chunk to avoid blocking UI
async function importLarge(data: any[]) {
  const CHUNK = 1000
  for (let i = 0; i < data.length; i += CHUNK) {
    yDoc.transact(() => {
      data.slice(i, i + CHUNK).forEach(item => items.set(item.id, item))
    }, 'import')
    await new Promise(r => setTimeout(r, 0))  // Yield to UI
  }
}

See Also

  • Undo/Redo - How transactions integrate with UndoManager
  • API Usage - Transaction-related mistakes

Undo/Redo

Implementing per-user undo stacks with Yjs UndoManager.

Overview

In collaborative apps, users expect undo to reverse their own changes, not collaborators’. Yjs UndoManager provides this through “tracked origins.”

Basic Setup

import { UndoManager } from 'yjs'

const undoManager = new UndoManager([content])

undoManager.undo()
undoManager.redo()
undoManager.undoStack.length > 0  // canUndo
undoManager.redoStack.length > 0  // canRedo

Tracked Origins

Without tracked origins, undo captures remote changes too:

// WRONG: captures everything
const undoManager = new UndoManager([content])

// CORRECT: only capture local changes
const undoManager = new UndoManager([content], {
  trackedOrigins: new Set(['user-action'])
})

// Mark local changes
yDoc.transact(() => {
  content.set('title', 'New Title')
}, 'user-action')  // Tracked

// Remote changes have no origin β€” not tracked

With Editor Bindings

const binding = new QuillBinding(yText, quill, awareness)
const undoManager = new UndoManager(yText, {
  trackedOrigins: new Set([binding])
})

Scoping to Shared Types

// Only track changes to cells, rowOrder, colOrder
const undoManager = new UndoManager([cells, rowOrder, colOrder], {
  trackedOrigins: new Set(['user-action'])
})

// Changes to other types (metadata) not tracked

Capturing Metadata

Restore cursor position after undo:

const undoManager = new UndoManager([content], {
  trackedOrigins: new Set(['user-action']),
  captureTransaction: (transaction) => ({
    cursorPosition: getCursorPosition()
  })
})

undoManager.on('stack-item-popped', (event) => {
  if (event.stackItem.meta.cursorPosition) {
    setCursorPosition(event.stackItem.meta.cursorPosition)
  }
})

Transaction Grouping

Each transaction becomes one undo item:

yDoc.transact(() => {
  items.set('a', data1)
  items.set('b', data2)
  items.set('c', data3)
}, 'user-action')

undoManager.undo()  // Reverts all three at once

For typing, use captureTimeout to group rapid changes:

const undoManager = new UndoManager([content], {
  trackedOrigins: new Set([binding]),
  captureTimeout: 500  // Group changes within 500ms
})

Clear History

undoManager.clear()  // Clear undo/redo stacks
undoManager.stopCapturing()  // End current group, start new one

Stack Events

undoManager.on('stack-item-added', (event) => {
  updateUndoButtons()
})

undoManager.on('stack-item-popped', (event) => {
  // Restore metadata
  updateUndoButtons()
})

Common Mistakes

Forgetting trackedOrigins:

// WRONG
const undoManager = new UndoManager([content])

// CORRECT
const undoManager = new UndoManager([content], {
  trackedOrigins: new Set(['user-action'])
})

Inconsistent origins:

// WRONG: one tracked, one not
content.set('title', 'New')  // Not tracked
yDoc.transact(() => metadata.set('time', now()), 'user-action')  // Tracked

// CORRECT: consistent
yDoc.transact(() => {
  content.set('title', 'New')
  metadata.set('time', now())
}, 'user-action')

See Also

Awareness

Implementing presence, cursors, and real-time user status with the Yjs awareness protocol.

Overview

Awareness provides ephemeral state synchronizationβ€”data that syncs in real-time but doesn’t persist: cursor positions, user presence, selections, typing indicators.

Basic Setup

const awareness = provider.awareness

awareness.setLocalStateField('user', {
  name: 'Alice',
  color: '#f783ac'
})

Setting Local State

// Set single field (preserves others)
awareness.setLocalStateField('cursor', { x: 100, y: 200 })

// Replace entire state
awareness.setLocalState({
  user: { name: 'Alice', color: '#f783ac' },
  cursor: { x: 100, y: 200 }
})

// Clear state (on disconnect)
awareness.setLocalState(null)

Observing Other Users

awareness.on('change', ({ added, updated, removed }) => {
  console.log('Joined:', added)
  console.log('Updated:', updated)
  console.log('Left:', removed)
  renderPresence()
})

function renderPresence() {
  awareness.getStates().forEach((state, clientId) => {
    if (clientId === yDoc.clientID) return  // Skip self
    renderCursor(state.cursor, state.user, clientId)
  })
}

Cursor Synchronization

// Throttle updates to avoid flooding
const updateCursor = throttle((x: number, y: number) => {
  awareness.setLocalStateField('cursor', { x, y })
}, 50)

function onMouseMove(e: MouseEvent) {
  updateCursor(e.clientX, e.clientY)
}

function onMouseLeave() {
  awareness.setLocalStateField('cursor', null)
}

Text Editor Cursors

Use relative positions so cursors survive text edits:

function updateTextCursor(selection) {
  awareness.setLocalStateField('cursor', {
    anchor: Y.createRelativePositionFromTypeIndex(yText, selection.anchor),
    head: Y.createRelativePositionFromTypeIndex(yText, selection.head),
    user: awareness.getLocalState()?.user
  })
}

Activity Status

let idleTimeout: NodeJS.Timeout

function setActive() {
  awareness.setLocalStateField('status', 'active')
  clearTimeout(idleTimeout)
  idleTimeout = setTimeout(() => {
    awareness.setLocalStateField('status', 'idle')
  }, 60000)
}

document.addEventListener('mousemove', setActive)
document.addEventListener('visibilitychange', () => {
  awareness.setLocalStateField('status', document.hidden ? 'away' : 'active')
})

Typing Indicator

let typingTimeout: NodeJS.Timeout

function onTextInput() {
  awareness.setLocalStateField('activity', { type: 'typing' })
  clearTimeout(typingTimeout)
  typingTimeout = setTimeout(() => {
    awareness.setLocalStateField('activity', null)
  }, 2000)
}

Cleanup

window.addEventListener('beforeunload', () => {
  awareness.setLocalState(null)
})

Performance Tips

  • Throttle updates: Max 10-20 cursor updates/second
  • Minimal state: Don’t put large objects in awareness
  • Conditional updates: Only send when position changes significantly:
if (Math.abs(e.clientX - lastX) > 5 || Math.abs(e.clientY - lastY) > 5) {
  awareness.setLocalStateField('cursor', { x: e.clientX, y: e.clientY })
}

Common Mistakes

Forgetting to filter self:

// WRONG: renders own cursor
awareness.getStates().forEach(state => renderCursor(state))

// CORRECT
awareness.getStates().forEach((state, clientId) => {
  if (clientId !== yDoc.clientID) renderCursor(state)
})

Not cleaning up removed users:

awareness.on('change', ({ removed }) => {
  for (const clientId of removed) {
    removeCursor(clientId)
  }
})

See Also

Conflict Resolution

Understanding how Yjs CRDTs automatically merge concurrent changes.

Core Principle

CRDTs guarantee eventual consistency: all clients receiving the same operations converge to identical state, regardless of operation order.

Y.Map: Last-Writer-Wins

When multiple clients set the same key, the highest logical timestamp wins. When timestamps are equal, the client ID breaks ties (arbitrary but consistent ordering):

Alice: map.set('color', 'blue')   // timestamp: 100
Bob:   map.set('color', 'green')  // timestamp: 101

Result: 'green' (Bob's timestamp higher)

// When timestamps match:
Alice: map.set('color', 'blue')   // timestamp: 100, clientId: 'abc'
Bob:   map.set('color', 'green')  // timestamp: 100, clientId: 'xyz'

Result: determined by client ID comparison (consistent across all peers)

Design tip: To preserve both values, use unique keys:

map.set(`comment-${odieId}`, 'X')  // Both preserved
map.set(`comment-${bobId}`, 'Y')

Y.Array: Position-Aware Merge

Concurrent insertions at the same position are both preserved:

Initial: [A, B, C]
Alice inserts X after A
Bob inserts Y after A

Result: [A, X, Y, B, C] or [A, Y, X, B, C]

Order of X/Y is deterministic (by client ID) but arbitrary. Design UIs that tolerate this.

Y.Text: Character-Level Merge

Initial: "Hello World"
Alice: "Hello Beautiful World"
Bob: "Hello Amazing World"

Result: "Hello Beautiful Amazing World" (or reversed)

Different formatting attributes merge (bold + italic). Same attribute uses LWW (both set color β†’ one wins).

Designing for Good Merges

Use ID-based references - Indices shift; IDs are stable.

Separate order from content - Content edits and reordering merge independently.

Avoid computed data - Don’t store totals; compute when needed.

Use granular keys - user.name, user.email instead of one user object.

Accept non-determinism - Concurrent inserts at same position have arbitrary order.

When Automatic Merge Isn’t Enough

For critical data, detect conflicts and show resolution UI:

yMap.observe((event, transaction) => {
  if (!transaction.local && hasConflict(event)) {
    showConflictDialog(localValue, remoteValue)
  }
})

Example: Counters and votes. Rather than storing a single number (where concurrent increments get lost to LWW), track each user’s contribution separately:

// WRONG: concurrent increments overwrite each other
counter.set('votes', counter.get('votes') + 1)

// CORRECT: track per-user contributions, sum at read time
const userVotes = yDoc.getMap('votes')
userVotes.set(userId, (userVotes.get(userId) || 0) + 1)

function getTotalVotes(): number {
  let total = 0
  userVotes.forEach(count => total += count)
  return total
}

This pattern preserves all concurrent operations by giving each user their own key.

See Also

Pitfalls

Common mistakes that cause data corruption, sync issues, or unexpected behaviorβ€”and how to avoid them.

Overview

CRDT-based applications can fail in subtle ways that are hard to debug. Many issues only appear when multiple users edit simultaneously, making them difficult to reproduce in development. This section documents the most common pitfalls organized by what you’re doing when you encounter them.

Topics

The Cost of CRDT Bugs

CRDT bugs are particularly problematic because:

  1. Data corruption spreads - Once corrupted data syncs to other clients, it’s everywhere
  2. Hard to reproduce - Issues may only occur with specific timing of concurrent edits
  3. Silent failures - Changes may be lost without any error messages
  4. Difficult to fix - Corrupted documents may need manual repair or data loss

Quick Reference

What You’re Doing Common Issues
Designing schema Complex objects in arrays, index references, binary data in CRDT
Calling Yjs methods Mutating after set, wrong getMap/new confusion, missing transactions
Handling events Not checking local flag, flooding awareness, assuming sync order

Quick Checklist

Before deploying a collaborative feature, verify:

  • Complex content in Y.Map with IDs, order in Y.Array
  • All references use IDs, not indices
  • Never mutating objects after set()
  • Using yDoc.getMap() not new Y.Map()
  • Changes batched in transactions
  • Undo uses trackedOrigins to isolate users
  • Observers check transaction.local when needed
  • Awareness updates are debounced
  • Large binary data uses blob storage, not CRDT

Most Common Issues

Symptom Likely Cause See
Data disappears after reorder Objects in array Data Modeling
Changes don’t sync Mutating after insertion API Usage
Undo affects other users Missing trackedOrigins API Usage
Slow sync, high memory Large data in CRDT Data Modeling
Duplicate events Not checking local flag Events & Sync

Subsections of Pitfalls

Data Modeling

Schema decisions that cause data corruption or sync issues.

Complex Objects in Reorderable Arrays

// WRONG
const slides = yDoc.getArray('slides')
slides.push([{ id: 'k8d2fn3m', title: 'Intro', content: [...] }])

Y.Array delete + insert creates copies, not moves. Concurrent reordering causes duplicates or data loss.

Fix: Store content in Y.Map, only IDs in Y.Array. See ID-Based Storage.

Index-Based References

// WRONG
cells.set('A1', { formula: '=B1 * C1' })  // References shift when columns inserted

Array indices change on insert/delete. References break silently.

Fix: Use stable IDs:

cells.set('r8k2mf9n:c3nd8k2m', { formula: { refs: ['r8k2mf9n:c9m2pt3q', 'r8k2mf9n:c2n7ks4w'], expr: '*' } })

Large Binary Data in CRDT

// WRONG
images.set('hero', { data: base64EncodedImage })  // 500KB+ in CRDT

CRDT metadata adds 2-10x overhead. History accumulates. Sync becomes slow.

Fix: Use blob storage, store only reference:

images.set('hero', { blobId: 'b8k2mf9n', width: 1920, height: 1080 })

Checklist

  • Complex content in Y.Map with IDs, order in Y.Array
  • All references use stable IDs, not indices
  • Binary data external, only references in CRDT

See Also

API Usage

Mistakes when calling Yjs methods.

Mutating Objects After Insertion

// WRONG
const config = { theme: 'light' }
yMap.set('config', config)
config.theme = 'dark'  // NOT synced!

Yjs snapshots on set(). Later mutations don’t propagate.

Fix: Create new objects, or use nested Y.Map for granular updates:

yMap.set('config', { ...yMap.get('config'), theme: 'dark' })

Moving Shared Types Between Parents

// WRONG
const items = folder1.get('items')  // Y.Array
folder2.set('items', items)  // Error!

Shared types can only have one parent.

Fix: Copy data to new shared type, delete old.

Creating New Instances Instead of Accessing

// WRONG
function addItem(item) {
  const items = new Y.Map()  // Disconnected!
  items.set(item.id, item)
}

new Y.Map() is unattached. Use yDoc.getMap().

Not Using Transactions

// WRONG: 2N syncs, 2N observer events
for (const item of items) {
  content.set(item.id, item)
  order.push([item.id])
}

Fix: Wrap in transact():

yDoc.transact(() => {
  for (const item of items) {
    content.set(item.id, item)
    order.push([item.id])
  }
})

Forgetting trackedOrigins

// WRONG: captures remote changes too
const undoManager = new UndoManager([content])

Without trackedOrigins, undo captures collaborators’ changes.

Fix:

const undoManager = new UndoManager([content], {
  trackedOrigins: new Set(['user-action'])
})
yDoc.transact(() => { ... }, 'user-action')

Checklist

  • Never mutate after set()
  • Don’t move shared types between parents
  • Use yDoc.getMap(), not new Y.Map()
  • Wrap related changes in transact()
  • Configure UndoManager with trackedOrigins

See Also

Events & Sync

Mistakes when handling events and the distributed nature of CRDTs.

Ignoring Local vs Remote

// WRONG: runs twice (local + sync confirmation)
content.observe(event => {
  saveToLocalStorage(content.toJSON())
  sendAnalytics('changed')
})

Observers fire for both local and remote changes.

Fix: Check transaction.local:

content.observe((event, transaction) => {
  updateUI()  // Always
  if (transaction.local) {
    saveToLocalStorage(content.toJSON())  // Only local
  }
})

Flooding Awareness Updates

// WRONG: 60+ messages/second
document.addEventListener('mousemove', e => {
  awareness.setLocalStateField('cursor', { x: e.clientX, y: e.clientY })
})

Floods network, overwhelms clients.

Fix: Throttle to 10-20 updates/second:

const updateCursor = throttle((x, y) => {
  awareness.setLocalStateField('cursor', { x, y })
}, 50)

Expecting Sync Order

// WRONG: assumes remote sees operations in order
yDoc.transact(() => items.set('config', { init: false }))
yDoc.transact(() => items.set('config', { init: true }))

CRDT sync order isn’t guaranteed across clients.

Fix: Design for eventual consistency. Use timestamps if order matters.

Checklist

  • Observers check transaction.local for side effects
  • Awareness updates throttled (10-20/sec max)
  • No assumptions about sync order

See Also