Operations Guide

Monitoring, best practices, and troubleshooting for federation.

Monitoring & Observability

Metrics

FederationMetrics Structure:

  • outbound_actions_sent (AtomicU64) - Successfully sent federation actions
  • outbound_actions_failed (AtomicU64) - Failed outbound deliveries
  • inbound_actions_received (AtomicU64) - Received federated actions
  • inbound_actions_rejected (AtomicU64) - Rejected actions (spam, invalid, blocked)
  • profiles_synced (AtomicU64) - Remote profiles cached/updated
  • files_synced (AtomicU64) - Attachment files downloaded
  • active_federation_connections (AtomicUsize) - Open federation connections

Logging

Sent Action Log:

  • instance: Target instance domain
  • action_type: Type of action (POST, FLLW, etc.)
  • action_id: Unique action identifier

Rejected Action Log:

  • instance: Source instance domain
  • action_type: Type of action
  • reason: Why rejected (spam, invalid sig, blocked, etc.)

Best Practices

For Instance Operators

✅ Enable HTTPS: Always use TLS for federation ✅ Monitor logs: Watch for spam or abuse ✅ Set rate limits: Protect against DoS ✅ Backup regularly: Federation doesn’t replace backups ✅ Update promptly: Security patches are critical

For Developers

✅ Verify signatures: Never trust unverified content ✅ Check relationships: Enforce connection requirements ✅ Handle failures: Network is unreliable ✅ Cache wisely: Balance freshness vs performance ✅ Test federation: Use multiple instances for testing

Troubleshooting

Actions Not Federating

Symptoms: Actions created locally but not received on remote instances

Checks:

  1. DNS resolution of target instance
  2. TLS certificate validity
  3. Firewall rules (port 443)
  4. Federation logs for errors

Common causes:

  • Target instance unreachable
  • Certificate expired
  • Rate limited

Signature Verification Failures

Symptoms: Inbound actions rejected with signature errors

Checks:

  1. Issuer’s public key is fetchable (GET /api/me)
  2. Key expiration
  3. Algorithm matches (ES384)

Common causes:

  • Key rotated without propagation
  • Clock skew between instances
  • Corrupted token in transit

File Sync Failures

Symptoms: Attachments not displaying, download errors

Checks:

  1. Content hash computation
  2. Blob adapter permissions
  3. Sufficient storage space

Common causes:

  • Hash mismatch (corrupted file)
  • Disk full
  • Permission denied

Profile Sync Issues

Symptoms: Stale profile data, missing avatars

Checks:

  1. Cache TTL (24h default)
  2. Remote instance reachable
  3. Profile endpoint returning valid data

Common causes:

  • Cache not invalidating
  • Remote instance down
  • Profile changed but not refreshed

See Also