Backup and restore

What this page covers

The backup scope is one folder root, one environment file, and the TLS certificate folder. This page lists what to back up, how to do it, how to restore, and the ten things to verify afterwards before a restore counts. Package deployments handle the whole flow with custodexa.sh backup and custodexa.sh restore; the manual procedure uses pg_dump, pg_restore, and tar, which come with the database and the operating system.

What you need

Three persistent locations, all bind mounts:

Host location	Content
`${DATA_PATH}/postgres`	All business data, audit records, and encrypted credentials and key wrappings
`${DATA_PATH}/recordings`	Session recording files
`${DATA_PATH}/audit`	Audit downgrade files and sealing period logs

Plus .env and the TLS certificate folder tls/. .env carries the key configuration; tls/ holds the private key of the external certificate and, in self-signed mode, the private key of the local certificate authority. When TLS_NGINX_TEMPLATE is set, back up the custom proxy template it names as well. Keep backups encrypted, and away from the master key material. Directory permissions on DATA_PATH should be 0750 or stricter.

The export directory and the offsite staging area fall outside the backup scope: an evidence bundle has a retention period of its own and is cleared when it ends, and the staging area holds nothing worth keeping once the upload completes.

How to set it up

Single-file backup (package deployments)

Choose Backup in the custodexa.sh menu, or run:

sudo /opt/custodexa/custodexa.sh backup

The script pauses backend, guacd, and frontend, writes one backup file that can be copied to another host, backups/custodexa-backup-<version>-<time>.tar, with a .sha256 checksum file next to it, and starts the services again. The file holds the database, the audit files, .env, tls/, and the custom proxy template that TLS_NGINX_TEMPLATE names; status shows the latest backup.

The recordings go in when chosen at the prompt or with --with-recordings.
The backup file can be encrypted with a passphrase of 12 to 256 characters, entered at the prompt or read from --passphrase-file <file>. The encrypted file ends in .tar.enc and opens with OpenSSL 1.1.1 or later and tar; a restore needs the same passphrase.
Deployments with an external database are backed up the same way. The release carries PostgreSQL 16, 17, and 18 clients and uses the one matching the database server’s major version. The database CA file goes into the backup and a client private key does not, so keep that key with the deployment’s other secrets.

An upgrade also writes a backup in the same format before switching versions; see Upgrade and rollback. Export artifacts persist at ${DATA_PATH}/exports, outside the default backup scope; download items needed beyond their retention period.

Manual procedure

The manual procedure below assumes the built-in database and local postgres directory. Use it for deployments not installed from the package, or when you need a backup flow of your own.

0. Read the variables first

Every procedure starts the same way. Read .env literally rather than sourcing it:

ENV_FILE="${ENV_FILE:-./.env}"
env_get() { sed -n "s/^[[:space:]]*$1=//p" "$ENV_FILE" | tail -n 1; }
DATA_PATH="${DATA_PATH:-$(env_get DATA_PATH)}"
DB_USER="${DB_USER:-$(env_get DB_USER)}"
DB_NAME="${DB_NAME:-$(env_get DB_NAME)}"
printf 'DATA_PATH=%s\nDB_USER=%s\nDB_NAME=%s\n' "$DATA_PATH" "$DB_USER" "$DB_NAME"

Refer to them afterwards as "${VAR:?}", so a missing value fails on the spot.

1. Cold backup

The backup before an upgrade has to be a cold one. The order is stop the application, back up the database, back up the files, back up the configuration, then start again:

docker compose stop backend guacd frontend
Export the database with pg_dump -Fc
Pack the recordings and audit directories with tar -czf
Copy .env and pack the tls/ folder
docker compose start backend guacd frontend
Confirm both artifacts are readable with pg_restore --list and tar -tzf

2. Hot backup

A routine backup can run hot: skip the stop and start steps, back up the database first, then copy the recording directory. The direction of the inconsistency is deliberate; better that the recording directory be newer than the database.

How to restore

Restoring a single-file backup with `restore`

Package deployments restore a single-file backup whose data belongs to 1.16.0 or later with restore, on the same host or on a host not yet installed:

sudo /opt/custodexa/custodexa.sh restore <backup file>

An installed host: before replacing anything, the script takes a safety backup of the current data and keeps the replaced folders under *.before-restore-<time> names. The deployment then runs the backup’s version.
A host not yet installed: place a release package of the backup’s version or later with get-custodexa.sh, then run restore with the script it placed. It installs the backup’s version and restores into it, asking for this host’s data path, TLS name and address, and public URL.
The master key: a restore finishes once the master key the backend unsealed with matches the one recorded in the backup. With KEK_PROVIDER=env this happens at startup. With ui or kms the command ends after the import; unseal on the unseal page, then run restore --resume to finish.
Stopping partway: a restore that fails or is interrupted stops where it is. restore --resume carries on, restore --revert goes back with the safety backup on an installed host, and restore --abandon returns a new host to not installed.
External database: the same command, on the same host or another one. The restore uses the database server and database named in the backup, empties that database and imports the backup in one transaction, so a failed import leaves the database as it was. When the backup’s connection used a client certificate, give it with --db-client-cert and --db-client-key.

Manual restore

Backups that are not a single backup file, and backup files whose data belongs to a version before 1.16.0, are restored by hand in this order. It differs from the backup in one place: .env is restored first, and the variables are read afterwards:

docker compose down
Restore .env and the tls/ folder
Read the variables from the restored .env
Unpack the recording and audit archives (the target postgres directory has to be empty)
Start the database service alone and wait until it accepts connections
Restore the database with pg_restore --clean --if-exists
On Linux, set the recordings directory ownership to 1000:0 and mode to 2770, then run docker compose up -d to start everything

Ten checks after a restore

Every service is up.
The backend health check passes.
The startup log holds no fatal error.
The frontend returns 200.
The login path works.
The four fingerprints on the key inventory match the ones from the backup.
Encrypted fields decrypt (open an asset’s editor and confirm the credential still works).
A recording picked at random plays back.
The audit chain verifies.
On deployments with offsite storage, reconcile the ledger against the bucket.

Suggested settings for the object storage bucket

The offsite bucket is managed by the deployer. Suggested:

S3 or a compatible service: turn on versioning; turn on object lock at bucket creation when you want to prevent deletion and modification (it cannot be added afterwards); align the expiry days of the lifecycle rule with the recording retention policy, and add an expiry rule for noncurrent versions; narrow the access permissions to writing under the given prefix, reading objects and their metadata, and reading the bucket configuration.
GCS: use the bucket retention policy as the baseline, and grant permissions at the level of object creation and object viewing.

What auditors can see

The chain verification result after a restore, issued per interval, shows that the restored audit data is still self-consistent.
Comparing key fingerprints is direct evidence that the cryptographic material was not replaced.
The offsite ledger records a hash and custody chain events per item; for the reconciliation, see Offsite evidence storage.