Operations Guide

Monitoring, best practices, and troubleshooting for federation.

Monitoring & Observability

Federation activity is observable through structured tracing logs emitted by the action delivery and verification paths. Key events to watch for:

  • Delivery: each delivery task logs → DELIVER when attempted and ← DELIVERED on success, with the action ID and target instance. Failures log a warning with the error before the scheduler reschedules a retry.
  • Verification: inbound tokens log → VERIFY (issuer and key ID) and ← VERIFIED on success. Signature failures and key-fetch failures are logged as warnings, and repeated failures are throttled by the key failure cache.
  • Auto-approval: trusted-connection auto-approvals log AUTO-APPROVE with the action ID and issuer.
  • Rejections: actions blocked by permission or role checks are logged as warnings explaining why (e.g. insufficient role, audience mismatch).

Best Practices

For Instance Operators

  • ✅ Enable HTTPS: Always use TLS for federation
  • ✅ Monitor logs: Watch for spam or abuse
  • ✅ Set rate limits: Protect against DoS
  • ✅ Backup regularly: Federation doesn’t replace backups
  • ✅ Update promptly: Security patches are critical

For Developers

  • ✅ Verify signatures: Never trust unverified content
  • ✅ Check relationships: Enforce connection requirements
  • ✅ Handle failures: Network is unreliable
  • ✅ Cache wisely: Balance freshness vs performance
  • ✅ Test federation: Use multiple instances for testing

Troubleshooting

Actions Not Federating

Symptoms: Actions created locally but not received on remote instances

Checks:

  1. DNS resolution of target instance
  2. TLS certificate validity
  3. Firewall rules (port 443)
  4. Federation logs for errors

Common causes:

  • Target instance unreachable
  • Certificate expired
  • Rate limited

Signature Verification Failures

Symptoms: Inbound actions rejected with signature errors

Checks:

  1. Issuer’s public key is fetchable (GET /api/me)
  2. Key expiration
  3. Algorithm matches (ES384)

Common causes:

  • Key rotated without propagation
  • Clock skew between instances
  • Corrupted token in transit

File Sync Failures

Symptoms: Attachments not displaying, download errors

Checks:

  1. Content hash computation
  2. Blob adapter permissions
  3. Sufficient storage space

Common causes:

  • Hash mismatch (corrupted file)
  • Disk full
  • Permission denied

Profile Sync Issues

Symptoms: Stale profile data, missing avatars

Checks:

  1. Cache TTL (24h default)
  2. Remote instance reachable
  3. Profile endpoint returning valid data

Common causes:

  • Cache not invalidating
  • Remote instance down
  • Profile changed but not refreshed

See Also