RTDB Overview

Cloudillo’s RTDB (Real-Time Database) provides Firebase-like functionality for structured JSON data with queries, subscriptions, and real-time synchronization. It integrates seamlessly with Cloudillo’s federated architecture while maintaining privacy and user control.

Overview

The RTDB system provides:

  • Real-time synchronization: Changes propagate to all connected clients instantly
  • Offline support: Works offline, syncs when connection returns
  • Collaborative editing: Multiple users can edit the same data concurrently
  • Query capabilities: Filter, sort, and paginate data
  • WebSocket-based: Efficient, bidirectional communication
  • Privacy-focused: Data stored on user’s chosen node

Real-Time Database (RTDB)

The RTDB system provides Firebase-like functionality for structured data:

Technology: redb - Lightweight embedded database (171 KiB package size)

Features:

  • JSON document storage
  • Query filters (equals, greater than, less than)
  • Sorting and pagination
  • Computed values (increment, aggregate, functions)
  • Atomic transactions with temporary references
  • Real-time subscriptions via WebSocket

Use Cases:

  • User profiles and settings
  • Task lists and project management
  • E-commerce catalogs
  • Analytics and reporting
  • Structured forms and surveys

Learn more: RTDB with redb

CRDT Collaborative Editing (Separate System)

Cloudillo also provides a separate CRDT API for collaborative editing:

Technology: Yrs - Rust implementation of Yjs CRDT

Features:

  • Conflict-free replicated data types (CRDTs)
  • Rich data structures (Text, Map, Array, XML)
  • Automatic conflict resolution
  • Time-travel and versioning
  • Awareness (presence, cursors)
  • Yjs ecosystem compatibility

Use Cases:

  • Collaborative text editors (Google Docs-like)
  • Shared whiteboards
  • Real-time collaborative forms
  • Collaborative spreadsheets
  • Multiplayer game state

Learn more: CRDT Collaborative Editing

Comparison: RTDB vs CRDT

Feature RTDB (redb) CRDT (Yrs)
Purpose Structured data storage Concurrent editing
Queries Rich (filter, sort, paginate) Limited (document-based)
Conflict Resolution Last-write-wins Automatic merge (CRDT)
Best For Traditional database needs Collaborative editing
API Style Firebase-like Yjs-compatible

Note: These are separate, complementary systems. Use RTDB for structured data with queries, and CRDT for collaborative editing scenarios.

Core Concept: Database-as-File

Both systems use the same foundational concept: databases/documents are special files in the Cloudillo file system.

How It Works

  1. File Metadata (MetaAdapter) stores:

    • Database ID, name, owner
    • Creation timestamp, last accessed
    • Permission rules
    • Configuration (max size, retention policy)

  2. Database Content (RtdbAdapter or CrdtAdapter) stores:

    • Actual data (documents, CRDT state)
    • Indexes (for query performance)
    • Snapshots (for fast loading)

  3. File ID serves as database identifier:

    /ws/db/:fileId  // WebSocket connection endpoint

Benefits

✅ Natural Integration: Databases managed like files ✅ Permission Reuse: File permissions apply to databases ✅ Federation Ready: Databases can be shared across instances ✅ Content Addressing: Database snapshots are tamper-proof ✅ Discoverable: Find databases through file APIs

Example

// Create database file
const response = await fetch('/api/db', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${token}` },
  body: JSON.stringify({
    name: 'My Tasks',
    type: 'redb',  // or 'yrs' for CRDT
    permissions: {
      public_read: false,
      readers: ['bob.example.com'],
      writers: ['bob.example.com']
    }
  })
});

const { fileId } = await response.json();
// fileId: "f1~abc123..."

// Connect to database via WebSocket
const ws = new WebSocket(`wss://cl-o.alice.example.com/ws/db/${fileId}`);

Architecture Overview

Components

┌─────────────────────────────────────────────────────┐
│ Client Application                                  │
│  - JavaScript/TypeScript                            │
│  - React hooks / Vue composables                    │
│  - WebSocket connection                             │
└─────────────────────────────────────────────────────┘
                          ↓ WebSocket
┌─────────────────────────────────────────────────────┐
│ Cloudillo Server                                    │
│  ┌─────────────────────────────────────────────┐    │
│  │ WebSocket Handler                           │    │
│  │  - Authentication                           │    │
│  │  - Message routing                          │    │
│  │  - Subscription management                  │    │
│  └─────────────────────────────────────────────┘    │
│                        ↓                            │
│  ┌─────────────────────────────────────────────┐    │
│  │ Database Manager                            │    │
│  │  - Instance lifecycle (load/evict)          │    │
│  │  - Snapshot management                      │    │
│  │  - Memory limits                            │    │
│  └─────────────────────────────────────────────┘    │
│         ↓                           ↓               │
│  ┌─────────────┐           ┌──────────────┐         │
│  │ RtdbAdapter │           │ CrdtAdapter  │         │
│  │  (redb)     │           │  (Yrs)       │         │
│  └─────────────┘           └──────────────┘         │
│         ↓                           ↓               │
│  ┌──────────────────────────────────────────────┐   │
│  │ Storage Layer                                │   │
│  │  - MetaAdapter (metadata)                    │   │
│  │  - BlobAdapter (snapshots, data)             │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

Request Flow

Query-Based (redb):

Client → WebSocket message (query/subscribe)
  ↓
WebSocket Handler → Authenticate
  ↓
Database Manager → Get or load instance
  ↓
RtdbAdapter → Execute query
  ↓
Return results + subscribe to changes
  ↓
Client receives data + real-time updates

CRDT-Based (Yrs):

Client → WebSocket connection
  ↓
WebSocket Handler → Authenticate
  ↓
Database Manager → Get or load instance
  ↓
Yrs Sync Protocol → Exchange state vectors
  ↓
Bidirectional updates → Merge with CRDT algorithm
  ↓
Both clients stay in sync

Permission Model

File-Level Permissions

Inherited from file metadata:

pub struct DatabasePermissions {
    pub owner: String,              // Identity tag
    pub readers: Vec<String>,       // Can query data
    pub writers: Vec<String>,       // Can modify data
    pub public_read: bool,
    pub public_write: bool,
}

Runtime Permission Checking

Permissions checked on every operation:

fn check_permission(auth: &Auth, db_meta: &DatabaseMetadata, action: Action) -> bool {
    // Owner can do anything
    if db_meta.owner == auth.id_tag {
        return true;
    }

    match action {
        Action::Read => {
            db_meta.permissions.public_read ||
            db_meta.permissions.readers.contains(&auth.id_tag)
        }
        Action::Write => {
            db_meta.permissions.public_write ||
            db_meta.permissions.writers.contains(&auth.id_tag)
        }
    }
}

Future: Fine-Grained Permissions

Planned for future releases:

  • Per-collection permissions: Different access per table
  • Per-document permissions: Filter queries by ownership
  • Runtime rules: JavaScript-like expressions evaluated at runtime
  • Attribute-based: Permissions based on user attributes

WebSocket Protocol

Both systems (RTDB and CRDT) use WebSocket for real-time communication, though with different protocols:

Connection

const ws = new WebSocket(
  `wss://cl-o.example.com/ws/db/${fileId}`,
  {
    headers: {
      'Authorization': `Bearer ${accessToken}`
    }
  }
);

Message Format

JSON messages with type field:

// Client → Server
{
  "type": "query",      // or "subscribe", "create", "update", "delete"
  "id": 123,            // Request ID for correlation
  // ... type-specific fields
}

// Server → Client
{
  "type": "queryResult", // or "change", "error"
  "id": 123,             // Matches request ID
  // ... response data
}

Lifecycle

Client connects
  ↓
Server authenticates
  ↓
Server loads database instance
  ↓
Client sends queries/subscriptions
  ↓
Server sends results + change notifications
  ↓
Client disconnects
  ↓
Server cleans up subscriptions

Performance Characteristics

Query-Based RTDB (redb)

  • Package size: ~171 KiB
  • Database file size: ~50 KiB minimum
  • Query speed: ~10,000 queries/second (in-memory)
  • Write speed: ~1,000 writes/second
  • Connection capacity: ~1,000 concurrent per database
  • Memory usage: ~10 MB per active database instance

CRDT-Based (Yrs)

  • Sync speed: ~50 ms for typical documents
  • Conflict resolution: Automatic, deterministic
  • Memory usage: ~5-20 MB per active document
  • Update latency: <10 ms for local network
  • Scalability: Tested with 100+ concurrent editors

Optimization Strategies

  1. Snapshots: Periodic full-state saves reduce sync time
  2. Compression: zstd compression for storage
  3. Eviction: LRU eviction for inactive databases
  4. Indexing: Secondary indexes for fast queries
  5. Batching: Batch updates for efficiency

Storage Strategy

Snapshots

Periodic full-state backups:

pub struct SnapshotStrategy {
    update_threshold: u32,      // Snapshot every N updates
    time_threshold: Duration,   // Snapshot every T duration
    compression: u8,            // zstd compression level (0-9)
    retention: u32,             // Keep N historical snapshots
}

Delta Updates

Incremental changes stored separately:

{file_id}~snapshot~v001      // Full state
{file_id}~updates~v001       // Changes since snapshot
{file_id}~metadata~v001      // Database config

File Variants

Reuse file variant system:

  • snapshot: Full database state (primary)
  • delta: Incremental updates
  • compressed: Compressed snapshot
  • exported: Human-readable export (JSON)

Federation Support

Databases can be shared across Cloudillo instances:

Read-Only Federation

pub struct FederatedDatabase {
    origin_instance: String,    // Original instance
    local_replica: bool,        // Keep local copy
    sync_mode: SyncMode,
}

pub enum SyncMode {
    ReadOnly,          // Subscribe to updates only
    ReadWrite,         // Full bidirectional sync
    Periodic(Duration), // Sync every N seconds
}

Sync Protocol

Using existing action/inbox mechanism:

pub struct DatabaseSyncAction {
    db_file_id: String,
    updates: Vec<u8>,       // Serialized updates
    state_vector: Vec<u8>,  // CRDT state or query timestamp
}

// POST /api/inbox with action type "db.sync"

Security Considerations

Authentication

  • WebSocket connections require valid access token
  • Token validated on connection establishment
  • Token can expire during connection (disconnected)

Authorization

  • Permissions checked on connection
  • Every read/write operation validated
  • Subscriptions filtered by permissions

Data Validation

  • Schema validation (optional)
  • Size limits per database
  • Rate limiting per user
  • Malicious update detection

Encryption

  • TLS/WSS for all connections
  • Optional client-side encryption (future)
  • Content-addressed snapshots prevent tampering

Choosing Between RTDB and CRDT

Use RTDB (redb) When:

✅ You need structured data with schemas ✅ Complex queries are important (filters, joins) ✅ Computed values and aggregations are needed ✅ Traditional database patterns fit your use case ✅ Atomic transactions are required ✅ You want minimal package size

Use CRDT (Yrs) When:

✅ Multiple users edit the same data concurrently ✅ Conflict-free merging is critical ✅ Rich text editing is needed ✅ Offline-first design is important ✅ You want Yjs ecosystem compatibility ✅ Time-travel/versioning is valuable

Can You Use Both?

Yes! Many applications benefit from both:

  • Yrs for collaborative document editing
  • redb for user profiles, settings, and structured data

Example: A collaborative task management app might use:

  • Yrs for the task description (rich text, concurrent editing)
  • redb for task metadata (assignee, due date, status)

API Overview

Database Management

POST /api/db                  # Create database
GET /api/db                   # List databases
GET /api/db/:fileId          # Get metadata
PATCH /api/db/:fileId        # Update metadata
DELETE /api/db/:fileId       # Delete database

WebSocket Connection

GET /ws/db/:fileId
Upgrade: websocket
Authorization: Bearer <token>

Export/Import

GET /api/db/:fileId/export?format=json
POST /api/db/:fileId/import

Next Steps