Backups and restore

Cluster ERP has one backup implementation: the runner python -m cluster_erp.server.backup_tenants. Cron, the installer, the operator console, and tenant offboarding all call it. This page explains what it dumps, how a dump becomes a verified backup, how old dumps are pruned, and how a restore works. Backup locations of real nodes are deployment data and are omitted under the security disclosure policy; the page names the setting backup_dir instead.

In short¶

• The runner dumps the default database, every live tenant database from the server of its region, and the control-plane database (src/cluster_erp/server/backup_tenants.py:387-405).
• Before each dump it checks that the dump role can read past the row guard; otherwise it stops with a named error (src/cluster_erp/server/backup_tenants.py:105-144).
• A dump counts only after pg_restore --list reads it back; a dump that does not read back is renamed with the suffix .corrupt (src/cluster_erp/server/backup_tenants.py:147-173, src/cluster_erp/server/backup_tenants.py:287-304).
• With backup_encryption_key set, the dump stream is encrypted on its way to disk as .dump.enc (src/cluster_erp/server/backup_tenants.py:259-278).
• A restore goes only into a database without tables (src/cluster_erp/server/backup_tenants.py:477-532).

How it works¶

Commands¶

The exit code is 1 when any database failed, so cron reports the failure (src/cluster_erp/server/backup_tenants.py:570-573).

The backup and restore path¶

The diagram shows the path of one database.

flowchart TD
  start["backup_all"] --> check["assert_dump_role_can_read_everything"]
  check --> dump["pg_dump -Fc"]
  dump --> enc{"backup_encryption_key set"}
  enc -- "yes" --> sealed["encrypt_stream to .dump.enc"]
  enc -- "no" --> plain["write .dump"]
  sealed --> verify["verify_dump with pg_restore --list"]
  plain --> verify
  verify -- "fails" --> corrupt["rename to .corrupt, report"]
  verify -- "passes" --> prune["_prune by count and age"]
  rstart["restore_one"] --> rcheck["role check, create database"]
  rcheck --> empty{"database has tables"}
  empty -- "yes" --> refuse["refuse"]
  empty -- "no" --> restore["pg_restore --no-owner"]

In prose:

1. backup_all walks the targets in order and counts failures; one failing database does not stop the others (src/cluster_erp/server/backup_tenants.py:408-439).
2. _dump_one checks the dump role, then runs pg_dump -Fc against the region's server (src/cluster_erp/server/backup_tenants.py:241-258).
3. With an encryption key the output of pg_dump is encrypted chunk by chunk into a .dump.enc file, and no plain copy is written (src/cluster_erp/server/backup_tenants.py:259-278). Without a key the dump is written as .dump (src/cluster_erp/server/backup_tenants.py:279-286).
4. verify_dump lists the dump with pg_restore --list. An encrypted dump is decrypted straight into pg_restore, and every chunk is authenticated (src/cluster_erp/server/backup_tenants.py:147-173, src/cluster_erp/server/backup_tenants.py:187-238).
5. A dump that fails verification is renamed with .corrupt, so it never takes a retained slot (src/cluster_erp/server/backup_tenants.py:287-304).
6. After a good dump, _prune deletes older dumps only when they are beyond the newest backup_keep files and, when backup_keep_days is above zero, older than that many days (src/cluster_erp/server/backup_tenants.py:369-384).
7. restore_one checks the role, creates the database if it is missing, and refuses when the database already has tables. Then it runs pg_restore --no-owner, decrypting an encrypted dump straight into it (src/cluster_erp/server/backup_tenants.py:477-532).

Dump files are named <db>__<UTC stamp>.dump or .dump.enc under a folder per database inside backup_dir (src/cluster_erp/server/backup_tenants.py:242-243, src/cluster_erp/server/backup_tenants.py:264, src/cluster_erp/server/backup_tenants.py:280).

Callers¶

Optional archives¶

The cold export reports a table that is missing in a database by name, continues with the others, and exits with code 1 at the end (scripts/backup/pg_cold_storage_export.sh:72-79, scripts/backup/pg_cold_storage_export.sh:112-116). Its cron line is commented out by default (deploy/cluster-erp.cron:13-19).

Why it is built this way¶

• One implementation, because an earlier single pg_dump of one database looked like a backup and held none of the customer data (scripts/backup/pg_full_backup.sh:19-23).
• Verification separates "a dump exists" from "a backup exists" (src/cluster_erp/server/backup_tenants.py:147-160).
• Both retention guards, because a count alone loses history when the runner fires often, and an age alone empties a database's folder when the runner stops (deploy/env.example:117-120).
• Restore never overwrites, "so a typo cannot destroy a live customer" (src/cluster_erp/server/backup_tenants.py:21-24).

Rules and boundaries¶

• A restore of an encrypted dump needs the same backup_encryption_key (src/cluster_erp/server/backup_tenants.py:195-198).
• Database names must match the identifier pattern DB_NAME_RE before any dump or restore (src/cluster_erp/server/backup_tenants.py:415-418, src/cluster_erp/server/backup_tenants.py:479-481).
• The tests tests/test_backup_retention.py and tests/test_backup_pipe.py cover pruning and the encrypted pipe.

Interaction with other parts¶

• The erasure ledger lives under backup_dir and is read at every start-up. An erasure that the ledger records but the restored database does not show as completed is carried out again, and the log shows RESTORE DETECTED (src/cluster_erp/core/erasure_ledger.py:1-16, src/cluster_erp/server/app.py:255-300). See privacy and data-subject requests.
• The residency check reports a remote or relative backup_dir, wal_archive_dir, or cold_storage_dir (src/cluster_erp/server/residency_check.py:105-122, src/cluster_erp/server/residency_check.py:213-233). See data residency check.

Limits and open points¶

• The runner backs up databases only; attachment files under storage_path are not part of a dump. Their backup: Not determined from code. backup_tenants.py and scripts/backup/ were searched.
• Closed tenants are no longer targets, because the target query selects tenants whose deleted_at is empty (src/cluster_erp/server/backup_tenants.py:395-401, src/cluster_erp/core/tenancy.py:550-575).

Terms introduced¶

Related pages¶

• Operations
• PostgreSQL roles and tuning
• Scheduled jobs and retention
• Operator console

Verified against commit 92f8051 on 2026-11-13.