Ransomware-Proof Backups: Append-Only Storage, Object Lock and the Restore Test Nobody Runs

Ransomware crews learned years ago that encrypting servers is not enough: if the victim restores from backup, nobody pays. So the first thing a serious attack does after getting administrator access is find the backups and destroy them. The variant we described in ransomware that targets Proxmox directly removes the protection flag from backups and deletes them before it encrypts a single disk.

The lesson is uncomfortable. A backup that the compromised machine can delete is not a backup, it is a second copy that disappears with the first. And a backup nobody has restored is not a backup either: it is a hope.

This post shows how to build the other kind, with commands we ran in a lab: an append-only backup server that production can write to but never delete from, an off-site copy under Object Lock that nobody can destroy before its retention ends, and a monthly restore test that proves the data comes back and alerts you when it does not.

Why Backups Fail Against Ransomware

When backups are lost in a ransomware attack, it is almost always for one of three reasons:

• The same credentials. The backup job on the server holds a key that can delete what it wrote. The attacker who owns the server owns that key.
• The same administrators. The backup server joined to the same directory, managed from the same jump host, with the same admin accounts. Compromise the domain and the backup console comes with it.
• Untested restores. The backup ran green every night, and on the day it mattered the dump was empty, the encryption password was on the encrypted server, or the restore took four days.

The fix for the first two is one rule: production can add backups, never remove them. The fix for the third is a test that runs whether or not anyone remembers.

The Rule: Production Can Add, Never Remove

The classic 3-2-1 rule (three copies, on two kinds of media, one off-site) still holds, but on its own it says nothing about who can delete the copies. Add two conditions:

• At least one copy is immutable: nothing that production can reach is able to delete or overwrite it before a retention period ends.
• Restores are tested, with the result recorded and alerted on, not assumed.

Below are two ways to get the immutable copy, both with open-source tools. Use the first on your own infrastructure, the second for the off-site copy, or both.

Option 1: An Append-Only Backup Server

restic is an open-source backup tool, and its companion rest-server can run in append-only mode: clients can create snapshots and read them, but every request to delete or overwrite a file is refused. Pruning old snapshots happens only on the server, by someone logged in there.

On the backup server, one repository per host, with TLS:

rest-server --path /srv/restic --listen :8000 --append-only --private-repos \
  --tls --tls-cert /etc/rest-server/cert.pem --tls-key /etc/rest-server/key.pem
# one user per host; --private-repos limits each user to its own repository
htpasswd -B -c /srv/restic/.htpasswd db01

On each production host, the backup job only needs the repository URL and its password:

export RESTIC_REPOSITORY=rest:https://db01:USER_PASSWORD@backup.internal:8000/db01/
export RESTIC_PASSWORD_FILE=/root/.restic-repo-password
pg_dump -Fc -f /var/backups/postgres/app.dump app
restic backup --host db01 /var/backups/postgres /etc

We tested what an attacker with a host's backup credentials can do. Backing up works. Trying to remove snapshots fails, and the snapshots stay:

$ restic forget --host web01 --keep-last 1 --prune
remove 1 snapshots:
...
Remove(<snapshot/55f0cb7a99>) failed: unexpected HTTP response (403): 403 Forbidden
unable to remove snapshot/55f0cb7a99... from the repository

Retention runs on the backup server itself, against the local path, with a time-based policy:

restic -r /srv/restic/db01 forget --host db01 --keep-within 30d --prune

The time-based policy is not a detail. restic's documentation warns that an attacker who can add snapshots can add empty ones with timestamps slightly newer than the real ones, so that a count-based policy such as "keep the last three" deletes every legitimate snapshot for you. --keep-within 30d keeps everything from the 30 days before the newest snapshot, so the real ones survive. Run it while the last good backup is less than 30 days old, look at the snapshot list before you prune, and keep the alert on backup freshness below.

Two more points. The repository password protects the data's confidentiality; keep a copy of it somewhere that is neither the production host nor the backup server, because losing it means losing every backup. And the backup server should not share an admin domain, SSH keys or a management network path with production.

Option 2: Object Lock for the Off-Site Copy

S3-compatible object storage, at the major clouds or self-hosted, supports Object Lock: once written, an object version cannot be deleted or overwritten until its retention date. In compliance mode, nobody can shorten that retention, not even the account administrator. In governance mode, users with a special permission can. For backups against ransomware, use compliance mode, and pick a retention you are prepared to pay storage for.

Enable Object Lock when you create the bucket: some providers can add it to an existing versioned bucket, not all can. It requires versioning, and creating the bucket with the lock turns versioning on. With the standard aws command-line client, which works against any S3-compatible endpoint:

aws s3api create-bucket --bucket backups --object-lock-enabled-for-bucket \
  --endpoint-url https://s3.example.internal
aws s3api put-object-lock-configuration --bucket backups --endpoint-url https://s3.example.internal \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":30}}}'

restic then writes to it like any other repository (RESTIC_REPOSITORY=s3:https://s3.example.internal/backups/db01).

Here is the part most guides skip, which we checked in the lab. An attacker with the backup key can still run restic forget --prune, or delete the whole prefix, and both appear to succeed. Afterwards restic reports that there is no repository at all. What actually happened is that every delete added a delete marker, which hides the object, while the locked versions underneath stayed exactly where they were. A direct attempt to delete a locked version was refused.

So the data is safe, but you have to know how to bring it back. Remove the delete markers under the repository's prefix, using credentials the attacker never had:

EP=https://s3.example.internal
aws s3api list-object-versions --bucket backups --prefix db01/ --endpoint-url "$EP" \
  --query 'DeleteMarkers[?IsLatest].[Key,VersionId]' --output text |
while read -r key ver; do
  aws s3api delete-object --bucket backups --key "$key" --version-id "$ver" --endpoint-url "$EP" >/dev/null
done
restic unlock --remove-all   # stale locks come back with the markers
restic snapshots
restic check --read-data

In our test, every snapshot came back, including the one the "attacker" had forgotten, and check --read-data found no errors. Only run unlock --remove-all when you are sure no backup is running.

Two practical notes: give the backup job's key only the permissions restic needs on that one bucket, and add a lifecycle rule that expires non-current versions after the retention period, or deleted data will keep accruing storage costs.

Not sure your backups would survive an attacker with admin rights? Request a free backup and ransomware readiness assessment: we trace which credentials can delete each copy and test one restore with you.

Keep the Keys Out of Reach

Immutability fails quietly when the credentials that control it sit next to production:

• Separate identities. The backup server and the object storage account are administered with accounts that do not exist in the production directory, with MFA.
• No path from production. Production hosts reach the backup server's API port and nothing else. No SSH from production to the backup server, no shared jump host.
• The repository password, the bucket administrator's credentials and the recovery steps are kept offline as well, for example printed in a safe, so the recovery itself does not depend on systems that might be encrypted.
• Hypervisor and backup admins are different people, or at least different accounts. Whoever controls the hypervisor can reach every virtual machine's disks, and often the backup configuration too.

The Restore Test Nobody Runs

A backup job that exits with 0 proves the job ran, not that the data comes back. The only proof is a restore, so automate one. This script restores the latest snapshot of a host, loads the database dump into a throwaway PostgreSQL container, checks that a key table has rows, and publishes the result for Prometheus through node_exporter's textfile collector. Run it monthly from a machine that has no rights to delete anything:

#!/usr/bin/env bash
# Restore drill: restore the latest snapshot of one host, prove the database dump in it
# loads and holds data, and publish the result for Prometheus (node_exporter textfile
# collector). Run it from a machine whose credentials cannot delete from the repository.
#   RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE must be set; docker must be available.
#   usage: restore-drill.sh HOST DUMP_PATH TABLE
#   e.g.   restore-drill.sh db01 /var/backups/postgres/app.dump orders
set -uo pipefail
HOST="$1" DUMP="$2" TABLE="$3"
METRICS="${METRICS_DIR:-/var/lib/node_exporter/textfile_collector}"
WORK=$(mktemp -d)
PG="drill-pg-$$"
trap 'docker rm -f "$PG" >/dev/null 2>&1; rm -rf "$WORK"' EXIT

drill() {
  restic restore latest --host "$HOST" --include "$DUMP" --target "$WORK" --verify || return 1
  [ -s "$WORK$DUMP" ] || { echo "no dump at $DUMP in the latest snapshot" >&2; return 1; }
  docker run -d --name "$PG" -e POSTGRES_PASSWORD=drill -v "$WORK$DUMP:/drill.dump:ro" postgres:16-alpine >/dev/null || return 1
  for _ in $(seq 1 60); do docker exec "$PG" pg_isready -q -U postgres && break; sleep 1; done
  docker exec "$PG" pg_restore -U postgres -d postgres --no-owner --exit-on-error /drill.dump || return 1
  rows=$(docker exec "$PG" psql -U postgres -tAc "SELECT count(*) FROM $TABLE") || return 1
  [ "$rows" -gt 0 ] || { echo "table $TABLE is empty in the restored dump" >&2; return 1; }
  echo "restored $rows rows from $TABLE"
}

start=$(date +%s)
if drill; then ok=1; else ok=0; fi
end=$(date +%s)

publish() {  # file, content: write then rename, so the collector never reads half a file
  printf '%s\n' "$2" > "$METRICS/$1.tmp" && mv "$METRICS/$1.tmp" "$METRICS/$1"
}
publish "restore_drill_$HOST.prom" "backup_restore_drill_success{host=\"$HOST\"} $ok
backup_restore_drill_duration_seconds{host=\"$HOST\"} $((end - start))"
# Only a success moves this timestamp, so a run of failures shows up as an overdue drill.
[ "$ok" = 1 ] && publish "restore_drill_last_success_$HOST.prom" \
  "backup_restore_drill_last_success_timestamp_seconds{host=\"$HOST\"} $end"
exit $((1 - ok))

We ran it both ways. With a good dump it restored 500 rows and published backup_restore_drill_success 1. With a dump whose table was empty, the kind of backup that runs green for months, it failed, published 0, and left the last-success timestamp untouched. The duration metric is your measured restore time for that system; compare it with the recovery time you promised the business.

Adapt the check to what matters: a row count on the table that would hurt most to lose, a checksum list for file shares, or starting the application against the restored data.

Alert on What Matters

After each backup, publish when the newest snapshot was taken:

ts=$(restic snapshots --host "$HOST" --latest 1 --json | jq -r '.[-1].time')
printf 'backup_last_snapshot_timestamp_seconds{host="%s"} %s\n' "$HOST" "$(date -d "$ts" +%s)" \
  > "$METRICS/backup_$HOST.prom.tmp" && mv "$METRICS/backup_$HOST.prom.tmp" "$METRICS/backup_$HOST.prom"

Then three Prometheus alerts cover the failures that matter: no new backup, a failed drill, and a drill that has not succeeded for too long:

groups:
  - name: backups
    rules:
      - alert: BackupStale
        expr: time() - backup_last_snapshot_timestamp_seconds > 26 * 3600
        labels:
          severity: critical
        annotations:
          summary: "No new backup snapshot for {{ $labels.host }} in 26 hours"
      - alert: RestoreDrillFailed
        expr: backup_restore_drill_success == 0
        labels:
          severity: warning
        annotations:
          summary: "The last restore drill for {{ $labels.host }} failed"
      - alert: RestoreDrillOverdue
        expr: time() - backup_restore_drill_last_success_timestamp_seconds > 35 * 86400
        labels:
          severity: warning
        annotations:
          summary: "No successful restore drill for {{ $labels.host }} in 35 days"

We validated the rules with promtool check rules and unit-tested them with promtool test rules: the stale-backup alert fires after 26 hours without a snapshot and not before, and the overdue alert fires on day 35 after the last successful drill.

A Monthly Routine

Put this in the calendar, with a name next to it:

1. 1Check that the restore drill passed for every critical system, and read the restore durations.
2. 2Run restic check --read-data-subset 10% on each repository: each month a random tenth of the stored data is read back and verified.
3. 3Read the snapshot list of each repository before the server-side prune runs. Snapshots you did not expect are the earliest sign that someone has the backup credentials.
4. 4Once a quarter, restore one whole system on a clean machine, from nothing but the off-site copy and the offline instructions. It is the only test of the procedure you will actually use.

Our backup and disaster recovery practice sets this up and runs it, and our cybersecurity team tests whether an attacker with admin access could still reach your backups.

Free Backup and Ransomware Readiness Assessment

If you are not sure your backups would survive someone with administrator access, we will check it with you:

• Delete paths: every credential, host and admin account that can delete or overwrite each backup copy.
• Immutability: whether at least one copy is append-only or under Object Lock in compliance mode, and for how long.
• Restore proof: one real restore of a critical system, timed, with the result compared against the recovery time you have promised.

You get a written list of the gaps and the order to close them in, with no obligation attached.

Request your free assessment, or see how our backup and disaster recovery practice works.