Upgrade and rollback

What this page covers

Package deployments upgrade with custodexa.sh upgrade. The script previews the downtime, takes a cold backup on its own before switching versions, and checks services and the evidence chain afterwards. To return to the version before an upgrade, use custodexa.sh rollback. See Backup and restore for backup steps.

What you need

• Records of .env, the data directory, and four key fingerprints.
• A maintenance window for existing sessions, the audit queue, and database migrations.
• Room in the Docker data folder for the new images and room in the backup folder for the pre-upgrade backup; the script checks free space before it starts.

How to set it up

Before upgrading

The preview shows the open connections and the expected downtime, so announce the window and let sessions end. Before stopping services, the script waits for the audit queue to drain; if it does not drain, the run stops where it is and the service keeps running. Run the upgrade inside tmux or screen on the host so that a dropped SSH connection does not interrupt it. Record the version, image IDs, key fingerprints, and table counts. Use an absolute DATA_PATH.

To stop the services ahead of time, custodexa.sh stop asks for confirmation and waits until pending audit records are written; custodexa.sh start starts the whole deployment and waits for the backend to become ready.

New export artifacts persist at ${DATA_PATH}/exports, outside the default backup scope. Before manually upgrading an older version, move any needed artifacts out of the container or download them.

Package deployment

Use the installed custodexa.sh to query for updates, choose a target in the menu, or run:

sudo /opt/custodexa/custodexa.sh upgrade             # query only
sudo /opt/custodexa/custodexa.sh upgrade <version>   # target version
sudo /opt/custodexa/custodexa.sh status

A downloaded package path is also accepted. A query without a target does not accept --images-from. With an installed 1.14.0 or later script, the menu offers an image source; a command may use --images-from auto|source, defaulting to auto. Auto checks this host, an offline bundle, GHCR, Docker Hub, then a local build. Source verifies the package source before building backend and frontend; upstream images and build dependencies still need to be obtained, so it is not fully offline. Downloads, checks and builds show progress before they run and a result after they finish.

For the first upgrade from 1.13.x to 1.14.0, the installed old script starts the upgrade. It does not recognize --images-from and its menu has no source choice. Omit the flag; the 1.14.0 target uses Auto. Later upgrades can select source building after 1.14.0 is installed. Check the backend startup log for the database migration result.

The backup before an upgrade

Before switching versions, the script stops backend, guacd, and frontend and writes a single-file cold backup that holds the recordings and is not encrypted, in the same format as custodexa.sh backup. Deployments with an external database are backed up by the script as well: the release carries PostgreSQL 16, 17, and 18 clients, uses the one matching the database server’s major version, and checks the database before any service stops.

You can use your own snapshot instead, such as a virtual machine or storage snapshot. On a terminal the script first asks whether it should take the backup; if you choose your own snapshot, start it after the services stop, cover the data folder, .env, and tls/ (and, with an external database, a database backup started after the stop), then enter the snapshot name, its start time, and where its restore procedure is written. Without a terminal, first stop the services with custodexa.sh stop, start the snapshot after the stop, then give --backup-ref, --backup-time, and --backup-restore together. When the script cannot obtain the database client or reach the external database, the preview says why and this step is completed with your own snapshot.

Old images

After a successful upgrade, the script keeps the backend and frontend images of the new version and the previous one and removes older ones, so the images rollback needs are always there. Upstream images, images used by a container, and images that also carry a tag of another repository stay. The final screen gives the number removed and the space freed; an image that cannot be removed is listed with the docker rmi command to run by hand, and the upgrade still counts as succeeded.

To keep every image, add --keep-images to upgrade. The installed script that starts the upgrade reads the option, so that script has to be 1.17.6 or later. To restore a backup of a version whose images were removed, plan for registry access or for that version’s offline image bundle.

Manual migration of an older git clone deployment

Running scripts/custodexa/custodexa.sh inside an old git clone, or pointing CUSTODEXA_HOME at it, explains that this is an older source deployment, leaves deployment files and services unchanged, and refuses the operation. Check backups and deployment settings first. Follow the manual migration section of the Upgrade SOP, or install the package in a separate clean directory and restore by hand using Backup and Restore. Do not run install against the existing data directory.

Verify and roll back

Check health, key fingerprints, the checkpoint chain, playback of an older recording, and auditing of a new connection before users connect again.

To return to the version before the upgrade, run:

sudo /opt/custodexa/custodexa.sh rollback

When the new version has not changed the database, rollback points the deployment back to the previous version and keeps the data, .env, and certificates. When it has, rollback does not switch versions and the screen prints the restore command for the backup that upgrade took. If a rollback is interrupted, rollback --resume finishes it and rollback --revert returns to the upgraded version. rollback goes back one version, to 1.16.0 or later; to go from 1.16.0 back to 1.15.x, restore the pre-upgrade backup by hand as described in Backup and Restore. Database migrations are not reversed.

When an upgrade fails after its backup, upgrade runs again once a restore has finished, and status shows that the restore dealt with the failure.

What auditors can see

Checkpoint verification and old and new sessions let reviewers check the evidence chain. The recording is the source of truth for text sessions, and input without echo leaves no command-text record. The query console’s structured statement record is the source of truth for that query.