Skip to content

Strange Duck Backup System

Last updated: 2026-09-04

Purpose

This document records the current Strange Duck backup architecture on tabor-lon.cloudhosting.uk, including the legacy backup system, the newer managed backup framework, schedules, alert routing, verification behaviour, and current migration status.

1. Current backup architecture

There are currently two backup systems running in parallel.

Legacy backup system

Path:

/home/stranged/backup-system/

Primary scripts:

/home/stranged/backup-system/scripts/backup-databases.sh
/home/stranged/backup-system/scripts/backup-files.sh

The legacy database script currently backs up:

Project Database
Backbone stranged_backbone
StrangeDuckHQ stranged_duckhq
NeuroLog stranged_neurolog
VHall stranged_vhall

The legacy system must remain enabled until each property has been explicitly migrated to, and verified under, the managed backup framework.

Managed backup system

Path:

/home/stranged/managed-backups/

Controller:

/home/stranged/managed-backups/bin/sd-backupctl

Current version observed:

0.1.3

The managed system is a newer property-based framework using per-property configuration, SSH agents, rclone, restic, per-property locks/logs/status files, database discovery modes, and profiles such as static, standard, critical, and development.

It is currently configured only for vHall properties.

Property Profile Root DB mode
app.vhall.co.uk critical /home/vhall/app.vhall.co.uk auto
dev.vhall.co.uk development /home/vhall/dev.vhall.co.uk auto
vhall.co.uk static /home/vhall/public_html none

Current managed-backup property config directory:

/home/stranged/managed-backups/config/properties/

2. Cron schedule

Current user crontab for stranged:

MAILTO="alerts@strangeduck.co.uk"

SHELL="/bin/bash"
17 * * * * /home/stranged/backup-system/scripts/backup-databases.sh hourly

SHELL="/bin/bash"
35 2 * * * /home/stranged/backup-system/scripts/backup-databases.sh nightly

SHELL="/bin/bash"
35 3 * * * /home/stranged/backup-system/scripts/backup-files.sh

SHELL="/bin/bash"
*/5 * * * * '/usr/local/bin/php' '/home/stranged/ops.strangeduck.co.uk/cron/run-all.php' > /dev/null 2>&1

# BEGIN STRANGE DUCK MANAGED BACKUPS
# Staggered while legacy backup jobs remain installed
SHELL="/bin/bash"
47 * * * * /home/stranged/managed-backups/bin/sd-backupctl run-databases hourly --all

SHELL="/bin/bash"
5 4 * * * /home/stranged/managed-backups/bin/sd-backupctl run-databases nightly --all

SHELL="/bin/bash"
35 4 * * * /home/stranged/managed-backups/bin/sd-backupctl run-files --all

# END STRANGE DUCK MANAGED BACKUPS

MAILTO="alerts@strangeduck.co.uk" is intentionally placed at the top of the crontab so it applies to all cron jobs below it.

The Ops cron is intentionally silent because it explicitly redirects both stdout and stderr:

> /dev/null 2>&1

3. Alert routing

Canonical system-alert address:

alerts@strangeduck.co.uk

alerts@strangeduck.co.uk is a cPanel forwarder rather than a mailbox. At the time of documentation it forwards to:

robbie@strangeduck.co.uk

Design:

server / cron / backup systems
        ↓
alerts@strangeduck.co.uk
        ↓
one or more monitored recipients

This allows recipients to be changed later without editing server-side cron configuration.

Alert routing was tested successfully on 2026-09-04. The test message showed:

From: stranged@tabor-lon.cloudhosting.uk
To: alerts@strangeduck.co.uk

and was successfully delivered through the forwarder.

4. Legacy database backup behaviour

Script:

/home/stranged/backup-system/scripts/backup-databases.sh

The script uses:

set -Eeuo pipefail

and an ERR trap, so failures in commands or pipelines should terminate the job and invoke the error handler.

For each configured database the script:

  1. Runs mysqldump.
  2. Compresses the dump with gzip.
  3. Streams the compressed backup directly to encrypted Google Drive storage using rclone rcat.
  4. Reads the uploaded remote file back using rclone cat.
  5. Validates the returned compressed stream using gzip -t.
  6. Only after successful validation writes a VERIFIED log entry.

Remote destination layout

Hourly backups use:

gdrive-crypt:Databases/Hourly/<Project>/<YYYY>/<MM>/

Examples:

gdrive-crypt:Databases/Hourly/Backbone/2026/09/
gdrive-crypt:Databases/Hourly/StrangeDuckHQ/2026/09/
gdrive-crypt:Databases/Hourly/NeuroLog/2026/09/
gdrive-crypt:Databases/Hourly/VHall/2026/09/

Note: the VHall directory is VHall with capital V and H.

Retention

Legacy script settings observed:

hourly  -> 336h
nightly -> 2160h

Equivalent to:

  • hourly: 14 days
  • nightly: 90 days

Expired .sql.gz files are deleted from the applicable remote tier using rclone.

5. Legacy logging

Primary database log:

/home/stranged/backup-system/logs/databases.log

Status directory:

/home/stranged/backup-system/status/

Hourly success marker:

/home/stranged/backup-system/status/databases-hourly.last-success

A successful manual run on 2026-09-04 produced:

2026-09-04T09:45:15Z

The log showed all four databases successfully completing and verifying.

6. Cron spam fix

Previously, successful hourly legacy backups generated cron email because rclone emitted routine NOTICE messages to stderr.

Typical notice:

Encrypted drive ... --checksum is in use but the source and destination have no hashes in common; falling back to --size-only

These notices were not backup failures, but cron mailed them because the job produced output.

The legacy database script was changed so normal command output is appended to the database log instead of being emitted to cron.

Relevant redirection now includes:

2>>"$log_file"

and:

>>"$log_file" 2>&1

This preserves normal logging while allowing the script's error trap to emit a real failure message to stderr.

Do not add blanket redirection such as > /dev/null 2>&1 to backup cron jobs, because that could hide genuine failures.

7. Verified backup state on 2026-09-04

A manual hourly backup run was performed and returned exit code 0.

Fresh remote backups were confirmed for:

  • Backbone
  • StrangeDuckHQ
  • NeuroLog
  • VHall

VHall remote files were confirmed at:

gdrive-crypt:Databases/Hourly/VHall/2026/09/

The newest manually generated VHall backup observed was:

stranged_vhall-2026-09-04T09-44-22Z.sql.gz

The legacy script then successfully read the uploaded copy back and validated it with gzip -t.

8. Migration status

The managed backup framework is newer, but it is not yet a complete replacement for the legacy backup system.

Property / DB Legacy backup Managed backup Current decision
Backbone Yes No Keep legacy
StrangeDuckHQ Yes No Keep legacy
NeuroLog Yes No Keep legacy
VHall production DB Yes Yes / likely overlap Keep both during migration
VHall development No confirmed legacy equivalent Yes Managed
vhall.co.uk static files No confirmed legacy equivalent Yes Managed

Migration rule

Never remove a legacy backup job merely because a managed-backup configuration exists.

A property should be considered migrated only after:

  1. managed database backup is confirmed
  2. managed file backup is confirmed where applicable
  3. remote backup location is confirmed
  4. backup freshness is confirmed
  5. restore test succeeds
  6. alerting is confirmed
  7. retention is confirmed
  8. documentation is updated
  9. legacy equivalent is explicitly retired

9. Restore-testing requirement

A backup is not fully trusted until a restore has been tested.

For each important property, periodically test:

  1. retrieval of a recent remote backup
  2. successful decryption / download
  3. gzip integrity
  4. SQL readability
  5. restore into a disposable test database
  6. application sanity check after restore

The existing legacy script proves remote readability and compressed-file integrity, but that is not the same as a full database restore test.

10. Documentation rule going forward

Infrastructure changes involving any of the following must be documented as they are made:

  • backup systems
  • cron
  • monitoring
  • DNS
  • server accounts
  • SSH keys
  • alert routing
  • retention policies
  • recovery procedures
  • restore procedures
  • production/development topology
  • migration from legacy systems

Avoid relying on memory for operational infrastructure.

11. Useful commands

View crontab:

crontab -l

Edit crontab using nano:

EDITOR=nano crontab -e

Run legacy hourly DB backup manually:

/home/stranged/backup-system/scripts/backup-databases.sh hourly
echo $?

View latest legacy DB log:

tail -30 /home/stranged/backup-system/logs/databases.log

Check hourly success marker:

cat /home/stranged/backup-system/status/databases-hourly.last-success

List current VHall hourly backups:

rclone lsl gdrive-crypt:Databases/Hourly/VHall/$(date -u +%Y)/$(date -u +%m)/

List managed backup configs:

find /home/stranged/managed-backups/config/properties -maxdepth 1 -type f -name '*.conf' -print

12. Open actions

  • Document the managed backup framework in greater detail.
  • Confirm managed-backup remote destination structure.
  • Perform a full test restore for at least one production database.
  • Decide migration order for Backbone, StrangeDuckHQ, and NeuroLog.
  • Retire legacy backup jobs only after managed equivalents pass the migration checklist.
  • Consider adding backup health/status into Strange Duck Ops.