RTDB Overview

Cloudillo’s RTDB (Real-Time Database) provides Firebase-like functionality for structured JSON data with queries, subscriptions, and real-time synchronization. It integrates seamlessly with Cloudillo’s federated architecture while maintaining privacy and user control.

Overview

The RTDB system provides:

• Real-time synchronization: Changes propagate to all connected clients instantly
• Offline support: Works offline, syncs when connection returns
• Collaborative editing: Multiple users can edit the same data concurrently
• Query capabilities: Filter, sort, and paginate data
• WebSocket-based: Efficient, bidirectional communication
• Privacy-focused: Data stored on user’s chosen node

Real-Time Database (RTDB)

The RTDB system provides Firebase-like functionality for structured data:

Technology: redb - Lightweight embedded database (171 KiB package size)

Features:

• JSON document storage
• Query filters (equals, greater than, less than, in, not-in, array-contains, array-contains-any, array-contains-all)
• Sorting and pagination
• Computed values (increment, decrement, multiply, concat, min, max, aggregate, functions)
• Atomic transactions with temporary references
• Real-time subscriptions via WebSocket
• Document locking (soft/advisory and hard/enforced)
• Aggregate queries with groupBy (sum, avg, min, max)

Use Cases:

• User profiles and settings
• Task lists and project management
• E-commerce catalogs
• Analytics and reporting
• Structured forms and surveys

Learn more: RTDB with redb

CRDT Collaborative Editing (Separate System)

Cloudillo also provides a separate CRDT API for collaborative editing:

Technology: Yrs - Rust implementation of Yjs CRDT

Features:

• Conflict-free replicated data types (CRDTs)
• Rich data structures (Text, Map, Array, XML)
• Automatic conflict resolution
• Time-travel and versioning
• Awareness (presence, cursors)
• Yjs ecosystem compatibility

Use Cases:

• Collaborative text editors (Google Docs-like)
• Shared whiteboards
• Real-time collaborative forms
• Collaborative spreadsheets
• Multiplayer game state

Learn more: CRDT Collaborative Editing

Comparison: RTDB vs CRDT

Note: These are separate, complementary systems. Use RTDB for structured data with queries, and CRDT for collaborative editing scenarios.

Core Concept: Database-as-File

Both systems use the same foundational concept: databases/documents are special files in the Cloudillo file system.

How It Works

1. File Metadata (MetaAdapter) stores:

   • Database ID, name, owner
   • Creation timestamp, last accessed
   • Permission rules
   • Configuration (max size, retention policy)

2. Database Content (RtdbAdapter or CrdtAdapter) stores:

   • Actual data (documents, CRDT state)
   • Indexes (for query performance)
   • Snapshots (for fast loading)

3. File ID serves as database identifier:

   /ws/rtdb/:fileId  // WebSocket connection endpoint

Benefits

✅ Natural Integration: Databases managed like files ✅ Permission Reuse: File permissions apply to databases ✅ Federation Ready: Databases can be shared across instances ✅ Content Addressing: Database snapshots are tamper-proof ✅ Discoverable: Find databases through file APIs

Example

// Create database file
const response = await fetch('/api/db', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${token}` },
  body: JSON.stringify({
    name: 'My Tasks',
    type: 'redb',  // or 'yrs' for CRDT
    permissions: {
      public_read: false,
      readers: ['bob.example.com'],
      writers: ['bob.example.com']
    }
  })
});

const { fileId } = await response.json();
// fileId: "f1~abc123..."

// Connect to database via WebSocket
const ws = new WebSocket(`wss://cl-o.alice.example.com/ws/rtdb/${fileId}`);

Architecture Overview

Components

┌─────────────────────────────────────────────────────┐
│ Client Application                                  │
│  - JavaScript/TypeScript                            │
│  - React hooks / Vue composables                    │
│  - WebSocket connection                             │
└─────────────────────────────────────────────────────┘
                          ↓ WebSocket
┌─────────────────────────────────────────────────────┐
│ Cloudillo Server                                    │
│  ┌─────────────────────────────────────────────┐    │
│  │ WebSocket Handler                           │    │
│  │  - Authentication                           │    │
│  │  - Message routing                          │    │
│  │  - Subscription management                  │    │
│  └─────────────────────────────────────────────┘    │
│                        ↓                            │
│  ┌─────────────────────────────────────────────┐    │
│  │ Database Manager                            │    │
│  │  - Instance lifecycle (load/evict)          │    │
│  │  - Snapshot management                      │    │
│  │  - Memory limits                            │    │
│  └─────────────────────────────────────────────┘    │
│         ↓                           ↓               │
│  ┌─────────────┐           ┌──────────────┐         │
│  │ RtdbAdapter │           │ CrdtAdapter  │         │
│  │  (redb)     │           │  (Yrs)       │         │
│  └─────────────┘           └──────────────┘         │
│         ↓                           ↓               │
│  ┌──────────────────────────────────────────────┐   │
│  │ Storage Layer                                │   │
│  │  - MetaAdapter (metadata)                    │   │
│  │  - BlobAdapter (snapshots, data)             │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

Request Flow

Query-Based (redb):

Client → WebSocket message (query/subscribe)
  ↓
WebSocket Handler → Authenticate
  ↓
Database Manager → Get or load instance
  ↓
RtdbAdapter → Execute query
  ↓
Return results + subscribe to changes
  ↓
Client receives data + real-time updates

CRDT-Based (Yrs):

Client → WebSocket connection
  ↓
WebSocket Handler → Authenticate
  ↓
Database Manager → Get or load instance
  ↓
Yrs Sync Protocol → Exchange state vectors
  ↓
Bidirectional updates → Merge with CRDT algorithm
  ↓
Both clients stay in sync

Permission Model

Connection-Time Access Check

Permissions are checked once at WebSocket connection time using file_access::check_file_access_with_scope(). This function evaluates multiple access sources:

1. Scoped tokens: Share links with restricted access
2. Ownership: File owner has full access
3. Tenant roles: Role-based access within the tenant
4. FSHR action tokens: Federation-based file sharing permissions

The result determines whether the connection operates in read_only or read_write mode. Clients can also request a specific access level via the ?access=read or ?access=write query parameter.

Future: Fine-Grained Permissions

Planned for future releases:

• Per-collection permissions: Different access per table
• Per-document permissions: Filter queries by ownership
• Runtime rules: JavaScript-like expressions evaluated at runtime
• Attribute-based: Permissions based on user attributes

WebSocket Protocol

Both systems (RTDB and CRDT) use WebSocket for real-time communication, though with different protocols:

Connection

const ws = new WebSocket(
  `wss://cl-o.example.com/ws/rtdb/${fileId}`,
  {
    headers: {
      'Authorization': `Bearer ${accessToken}`
    }
  }
);

Message Format

JSON messages with type field:

// Client → Server
{
  "type": "query",      // or "subscribe", "create", "update", "delete"
  "id": 123,            // Request ID for correlation
  // ... type-specific fields
}

// Server → Client
{
  "type": "queryResult", // or "change", "error"
  "id": 123,             // Matches request ID
  // ... response data
}

Lifecycle

Client connects
  ↓
Server authenticates
  ↓
Server loads database instance
  ↓
Client sends queries/subscriptions
  ↓
Server sends results + change notifications
  ↓
Client disconnects
  ↓
Server cleans up subscriptions

Performance Characteristics

Query-Based RTDB (redb)

• Package size: ~171 KiB
• Database file size: ~50 KiB minimum
• Query speed: ~10,000 queries/second (in-memory)
• Write speed: ~1,000 writes/second
• Connection capacity: ~1,000 concurrent per database
• Memory usage: ~10 MB per active database instance

CRDT-Based (Yrs)

• Sync speed: ~50 ms for typical documents
• Conflict resolution: Automatic, deterministic
• Memory usage: ~5-20 MB per active document
• Update latency: <10 ms for local network
• Scalability: Tested with 100+ concurrent editors

Optimization Strategies

1. Snapshots: Periodic full-state saves reduce sync time
2. Compression: zstd compression for storage
3. Eviction: LRU eviction for inactive databases
4. Indexing: Secondary indexes for fast queries
5. Batching: Batch updates for efficiency

Storage Strategy

RTDB data is stored directly in per-tenant redb database files. The RtdbAdapter handles persistence through ACID transactions — each write operation is committed atomically.

For details on the storage layout, see RTDB with redb.

Federation Support

Databases can be shared across Cloudillo instances through the file sharing mechanism (FSHR action tokens). Access from remote users is granted via the same check_file_access_with_scope() system used for local access control.

Security Considerations

Authentication

• WebSocket connections require valid access token
• Token validated on connection establishment
• Token can expire during connection (disconnected)

Authorization

• Permissions checked on connection
• Every read/write operation validated
• Subscriptions filtered by permissions

Data Validation

• Schema validation (optional)
• Size limits per database
• Rate limiting per user
• Malicious update detection

Encryption

• TLS/WSS for all connections
• Optional client-side encryption (future)
• Content-addressed snapshots prevent tampering

Choosing Between RTDB and CRDT

Use RTDB (redb) When:

✅ You need structured data with schemas ✅ Complex queries are important (filters, sorts, aggregates) ✅ Computed values and aggregations are needed ✅ Document locking for exclusive editing is required ✅ Traditional database patterns fit your use case ✅ Atomic transactions are required ✅ You want minimal package size

Use CRDT (Yrs) When:

✅ Multiple users edit the same data concurrently ✅ Conflict-free merging is critical ✅ Rich text editing is needed ✅ Offline-first design is important ✅ You want Yjs ecosystem compatibility ✅ Time-travel/versioning is valuable

Can You Use Both?

Yes! Many applications benefit from both:

• Yrs for collaborative document editing
• redb for user profiles, settings, and structured data

Example: A collaborative task management app might use:

• Yrs for the task description (rich text, concurrent editing)
• redb for task metadata (assignee, due date, status)

API Overview

Database Management

POST /api/db                  # Create database
GET /api/db                   # List databases
GET /api/db/:fileId          # Get metadata
PATCH /api/db/:fileId        # Update metadata
DELETE /api/db/:fileId       # Delete database

WebSocket Connection

GET /ws/rtdb/:fileId
Upgrade: websocket
Authorization: Bearer <token>

Export/Import

GET /api/db/:fileId/export?format=json
POST /api/db/:fileId/import

Next Steps

• RTDB with redb - Query-based database with WebSocket protocol
• CRDT Collaborative Editing - Yrs-based conflict-free editing
• System Architecture - Overall architecture context
• File Storage - Database-as-file implementation