RTDB Overview

Cloudillo’s RTDB (Real-Time Database) provides Firebase-like functionality for structured JSON data with queries, subscriptions, and real-time synchronization. It integrates seamlessly with Cloudillo’s federated architecture while maintaining privacy and user control.

CRDT Collaborative Editing (Separate System)

Cloudillo also provides a separate CRDT API for collaborative editing:

Technology: Yrs - Rust implementation of Yjs CRDT

Features:

• Conflict-free replicated data types (CRDTs)
• Rich data structures (Text, Map, Array, XML)
• Automatic conflict resolution
• Time-travel and versioning
• Awareness (presence, cursors)
• Yjs ecosystem compatibility

Use Cases:

• Collaborative text editors (Google Docs-like)
• Shared whiteboards
• Real-time collaborative forms
• Collaborative spreadsheets
• Multiplayer game state

Learn more: CRDT Collaborative Editing

Comparison: RTDB vs CRDT

Note: These are separate, complementary systems. Use RTDB for structured data with queries, and CRDT for collaborative editing scenarios.

Core Concept: Database-as-File

Both systems use the same foundational concept: databases/documents are special files in the Cloudillo file system.

How It Works

1. File Metadata (MetaAdapter) stores:

   • Database ID, name, owner
   • Creation timestamp, last accessed
   • Permission rules
   • Configuration (max size, retention policy)

2. Database Content (RtdbAdapter or CrdtAdapter) stores:

   • Actual data (documents, CRDT state)
   • Indexes (for query performance)
   • Snapshots (for fast loading)

3. File ID serves as database identifier:

   /ws/rtdb/:fileId  // WebSocket connection endpoint

Benefits

• Natural Integration: Databases managed like files
• Permission Reuse: File permissions apply to databases
• Federation Ready: Databases can be shared across instances
• Content Addressing: Database snapshots are tamper-proof
• Discoverable: Find databases through file APIs

Example

// Create database file
const response = await fetch('/api/db', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${token}` },
  body: JSON.stringify({
    name: 'My Tasks',
    type: 'redb',  // or 'yrs' for CRDT
    permissions: {
      public_read: false,
      readers: ['bob.example.com'],
      writers: ['bob.example.com']
    }
  })
});

const { fileId } = await response.json();
// fileId: "f1~abc123..."

// Connect to database via WebSocket
const ws = new WebSocket(`wss://cl-o.alice.example.com/ws/rtdb/${fileId}`);

Architecture Overview

Components

┌─────────────────────────────────────────────────────┐
│ Client Application                                  │
│  - JavaScript/TypeScript                            │
│  - React hooks / Vue composables                    │
│  - WebSocket connection                             │
└─────────────────────────────────────────────────────┘
                          ↓ WebSocket
┌─────────────────────────────────────────────────────┐
│ Cloudillo Server                                    │
│  ┌─────────────────────────────────────────────┐    │
│  │ WebSocket Handler                           │    │
│  │  - Authentication                           │    │
│  │  - Message routing                          │    │
│  │  - Subscription management                  │    │
│  └─────────────────────────────────────────────┘    │
│                        ↓                            │
│  ┌─────────────────────────────────────────────┐    │
│  │ Database Manager                            │    │
│  │  - Instance lifecycle (load/evict)          │    │
│  │  - Snapshot management                      │    │
│  │  - Memory limits                            │    │
│  └─────────────────────────────────────────────┘    │
│         ↓                           ↓               │
│  ┌─────────────┐           ┌──────────────┐         │
│  │ RtdbAdapter │           │ CrdtAdapter  │         │
│  │  (redb)     │           │  (Yrs)       │         │
│  └─────────────┘           └──────────────┘         │
│         ↓                           ↓               │
│  ┌──────────────────────────────────────────────┐   │
│  │ Storage Layer                                │   │
│  │  - MetaAdapter (metadata)                    │   │
│  │  - BlobAdapter (snapshots, data)             │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

Permission Model

Connection-Time Access Check

Permissions are checked once at WebSocket connection time using file_access::check_file_access_with_scope(). This function evaluates multiple access sources:

1. Scoped tokens: Share links with restricted access
2. Ownership: File owner has full access
3. Tenant roles: Role-based access within the tenant
4. FSHR action tokens: Federation-based file sharing permissions

The result determines whether the connection operates in read_only or read_write mode. Clients can also request a specific access level via the ?access=read or ?access=write query parameter.

Future: Fine-Grained Permissions

Planned for future releases:

• Per-collection permissions: Different access per table
• Per-document permissions: Filter queries by ownership
• Runtime rules: JavaScript-like expressions evaluated at runtime
• Attribute-based: Permissions based on user attributes

WebSocket Protocol

Both systems (RTDB and CRDT) use WebSocket for real-time communication, though with different protocols:

Connection

const ws = new WebSocket(
  `wss://cl-o.example.com/ws/rtdb/${fileId}`,
  {
    headers: {
      'Authorization': `Bearer ${accessToken}`
    }
  }
);

Message Format

JSON messages with type field:

// Client → Server
{
  "type": "query",      // or "subscribe", "create", "update", "delete"
  "id": 123,            // Request ID for correlation
  // ... type-specific fields
}

// Server → Client
{
  "type": "queryResult", // or "change", "error"
  "id": 123,             // Matches request ID
  // ... response data
}

Storage Strategy

RTDB data is stored directly in per-tenant redb database files. The RtdbAdapter handles persistence through ACID transactions – each write operation is committed atomically. Snapshots use zstd compression, and inactive databases are evicted via LRU.

For details on the storage layout, see RTDB with redb.

Federation Support

Databases can be shared across Cloudillo instances through the file sharing mechanism (FSHR action tokens). Access from remote users is granted via the same check_file_access_with_scope() system used for local access control.

Security Considerations

• WebSocket connections require valid access tokens, validated on establishment
• Permissions checked on connection; every read/write operation validated
• Optional schema validation, size limits, and rate limiting per user
• TLS/WSS for all connections; content-addressed snapshots prevent tampering

Choosing Between RTDB and CRDT

Use RTDB (redb) for structured data with schemas, complex queries (filters, sorts, aggregates), computed values, document locking, and atomic transactions.

Use CRDT (Yrs) for concurrent multi-user editing, conflict-free merging, rich text editing, offline-first design, and Yjs ecosystem compatibility.

Both can be used together – for example, Yrs for collaborative document editing and redb for structured metadata.

API Overview

Database Management

POST /api/db                  # Create database
GET /api/db                   # List databases
GET /api/db/:fileId          # Get metadata
PATCH /api/db/:fileId        # Update metadata
DELETE /api/db/:fileId       # Delete database

WebSocket Connection

GET /ws/rtdb/:fileId
Upgrade: websocket
Authorization: Bearer <token>

Export/Import

GET /api/db/:fileId/export?format=json
POST /api/db/:fileId/import

Next Steps

• RTDB with redb - Query-based database with WebSocket protocol
• CRDT Collaborative Editing - Yrs-based conflict-free editing
• System Architecture - Overall architecture context
• File Storage - Database-as-file implementation