Backups & restore — what's on disk, what to restore, when

Backups & restore

Keystone takes a Postgres dump every night via the keystone-backup.service systemd unit. Dumps live under /var/backups/keystone/ and are organised into three retention tiers:

Tier	Retention	When written
Daily	Last 14 days	Every night at 02:00 local time
Weekly	Last 4 weeks	The first daily run of each ISO week
Monthly	Last 6 months	The first daily run on the 1st of each month

The /admin/backups page surfaces every dump on disk with its tier, size, creation timestamp, and per-dump Download and Restore actions.

Taking an ad-hoc snapshot

Before a risky change (a major migration, a config rewrite, a dependency upgrade), click Take a snapshot now. This invokes the same pg_dump the timer uses, writes a fresh dump to the daily tier, and records the action in the audit log.

The button calls a sudoers-wrapped script: /usr/local/bin/keystone-backup-snapshot. The bash installer drops the wrapper plus a one-line sudoers grant so the keystone PHP-FPM user can invoke it passwordlessly. If the wrapper isn’t installed (re-run keystone-install), the page tells you.

Downloading for off-host storage

Click Download on any dump. The file streams as application/octet-stream with the original filename. Useful for one-shot off-host safekeeping (e.g. an ops engineer wants to take a dump home before a live migration).

This isn’t how disaster recovery should work in production — for that, configure off-site sync via KEYSTONE_BACKUP_REMOTE in /etc/default/keystone-backup. Format:

# rsync over SSH
KEYSTONE_BACKUP_REMOTE=user@host:/path

# rclone — needs the rclone binary + a configured remote
KEYSTONE_BACKUP_REMOTE=s3://bucket/path

# disabled (default)
KEYSTONE_BACKUP_REMOTE=

When set, every successful backup is rsync’d / rcloned to the remote target on the same systemd run.

Restoring from a dump

Restore replaces the entire database with the chosen dump. All in-flight changes since the dump’s timestamp are lost. The operation typically takes 1–10 minutes; longer for very large trusts.

1. Click Restore next to the dump.
2. The modal asks you to type the dump’s filename verbatim. The button stays disabled until you do — defends against muscle-memory clicks.
3. Click Restore now. The privileged wrapper at /usr/local/bin/keystone-restore-runner does the work:
   • drops the keystone database
   • recreates it
   • pg_restore --clean --if-exists from the chosen dump
   • restarts PHP-FPM so caches don’t outlive the swap
4. Active sessions are invalidated by the PHP-FPM restart — you’ll see a fresh login screen on your next request.

CLI escape hatch

For incident response or scripted recovery:

# Take a snapshot now
sudo /usr/local/bin/keystone-backup-snapshot

# Download a dump (just SCP it from /var/backups/keystone)
scp keystone@host:/var/backups/keystone/<filename> ./

# Restore — the same wrapper the UI shells out to
sudo /usr/local/bin/keystone-restore-runner <filename>

Filename validation in the wrapper enforces the same anti-traversal rules as the controller (no /, no ..).

Demo mode

Snapshot and restore are both blocked in demo mode (the demo.readonly middleware wraps both endpoints). Operators on a demo install never accidentally drop the demo dataset.

What about the application data, not just the DB?

Attachments, branding files and the like are stored on the local filesystem under /var/lib/keystone/storage. The systemd backup unit only dumps Postgres; rsync the storage directory separately if your trust requires complete-system recovery.

Related

• /admin/audit — every snapshot and restore is logged with event=backup.*
• Update channel — same wrapper pattern as restore