Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Operations

This runbook covers the trusted single-node SQLite deployment. Read Security Model before operating a production instance.

Health And Readiness

Use /health or /healthz only for process liveness. Use /ready for traffic admission because it verifies that SQLite is reachable, embedded migrations are not blocked, and browser session cryptography is configured.

curl --fail --silent http://127.0.0.1:8080/health
curl --fail --silent http://127.0.0.1:8080/ready
cloud-ssh healthcheck

The container healthcheck uses the CLI command. Set CLOUD_SSH_HEALTHCHECK_URL to /ready when orchestrator health must include readiness.

Protect /metrics with its bearer token:

curl --fail --silent \
  -H "Authorization: Bearer $CLOUD_SSH_METRICS_BEARER_TOKEN" \
  https://cloud.example.com/metrics

Alert on readiness failure, journal commit failures, sustained journal queue bytes, durable output lag, room capacity exhaustion, authentication rate limiting, and unexpected abandoned generations.

Graceful Shutdown

Send SIGTERM and allow enough time for every live room to terminate and wait for its child, drain PTY output, persist a final snapshot and runtime exit event, and join the journal writer. HTTP and SSH connections each receive a bounded five-second transport drain before the server continues shared room cleanup:

kill -TERM "$(cat /run/cloud-ssh.pid)"

For Docker, use a stop timeout sized for the configured PTY and filesystem:

docker stop --time 30 cloud-ssh

Do not send SIGKILL during normal operation. If forced termination is unavoidable, the next startup marks prior nonterminal generations abandoned with reason service_restart; it does not pretend the process survived.

Backup

The supported backup path uses SQLite VACUUM INTO, which produces a consistent standalone database while the service is running. It includes identity metadata, rooms, runtime generations, events, terminal output BLOBs, snapshots, and audit events.

backup="/var/backups/cloud-ssh/cloud-ssh-$(date -u +%Y%m%dT%H%M%SZ).db"
cloud-ssh maintenance backup --output "$backup"
sha256sum "$backup" > "$backup.sha256"
chmod 0600 "$backup" "$backup.sha256"

The output path must not already exist. The command rejects failed, unknown, or checksum-mismatched migration state and missing required tables. Copy backups to separate encrypted storage and apply an operator-owned retention policy.

Always rehearse a restore after backup:

restore_check="$(mktemp --suffix=.db)"
cloud-ssh maintenance restore-check \
  --input "$backup" \
  --restore-to "$restore_check"
rm -f "$restore_check"

restore-check copies the file, opens it read-only, runs PRAGMA integrity_check, and verifies every required table. It also verifies terminal-history semantics for each generation: watermark ordering; a readable, contiguous retained-event suffix when retained events exist; every snapshot’s supported encoding and SHA-256 digest; output-segment range/byte-length consistency; and a contiguous segment tail from the latest snapshot through the durable output watermark. CI also verifies that a restored database preserves generation events, output bytes, and snapshots, and rejects logically corrupted terminal history even when SQLite file integrity remains ok.

Restore

  1. Stop Cloud SSH gracefully and confirm the process exited.
  2. Preserve the failed database, -wal, and -shm files for incident analysis.
  3. Verify the backup checksum and run maintenance restore-check against a new path on the target filesystem.
  4. Move the validated file into place atomically with service ownership and mode 0600. Do not copy a WAL or SHM file from another database generation.
  5. Run cloud-ssh migrate status. The binary must report no failed, unknown, or checksum-mismatched rows.
  6. Start the service, wait for /ready, and verify authorized room history.
  7. Inspect startup logs for expected recovery of any nonterminal generation.

Example while the service is stopped:

install -o cloudssh -g cloudssh -m 0600 \
  /var/backups/cloud-ssh/validated.db \
  /var/lib/cloud-ssh/cloud-ssh.db.next
mv /var/lib/cloud-ssh/cloud-ssh.db.next \
  /var/lib/cloud-ssh/cloud-ssh.db

Upgrade

  1. Record the current image digest or binary revision.
  2. Run make production-check for the target source revision.
  3. Create and restore-check a fresh backup.
  4. Inspect migration status with the old binary.
  5. Gracefully stop the service.
  6. Replace only the binary/image. Preserve the SQLite volume, session HMAC key, provider secrets, tenant id, and SSH host key.
  7. Start one instance. Startup applies only known pending embedded migrations and refuses failed, unknown, or checksum-mismatched history.
  8. Wait for /ready, verify browser and SSH attach, inspect metrics, and check old generation history.
  9. Keep the pre-upgrade database backup until the validation window closes.

The unreleased target currently uses one initial migration and does not promise schema downgrade compatibility. Rollback means stop, restore the matching pre-upgrade database backup, and run the matching prior binary.

Retention

Run a manual pass with the same policy as the scheduler:

cloud-ssh maintenance prune \
  --retain-room-events 10000 \
  --retain-room-snapshots 10 \
  --retain-audit-events 100000

Use --room or --tenant for scoped maintenance. At least one snapshot per generation is mandatory. The pass retains newest collaboration events, compacts snapshots per generation, then removes only output segments fully covered by the oldest retained snapshot. It reports deleted events, snapshots, output segments, and audit events.

Clients requesting removed output receive 410 history_truncated with earliest_available_output_seq, replacement_snapshot_output_seq, and a replacement snapshot URL. Treat absence of a replacement anchor as a storage incident.

Back up before materially reducing retention. Remember that typed input can be echoed by the shell and become retained output.

Runtime Cleanup

Normal idle expiry and service shutdown terminate and wait for local PTYs. If a host crash leaves operating-system processes behind, do not infer ownership from a stale SQLite running row alone. After restart:

  1. Confirm the service marked the old generation abandoned.
  2. Identify candidate processes using the dedicated service account, cgroup, or container boundary configured by the operator.
  3. Verify process start time and parentage before terminating anything.
  4. Terminate with SIGTERM, wait, then use SIGKILL only if the process does not exit.
  5. Record the incident outside the database; do not rewrite generation rows.

Cloud SSH intentionally does not scan and kill arbitrary service-account processes because the local account may own unrelated operator workloads. Running the service in a dedicated container or systemd cgroup makes host-level cleanup safer.

Disk Full

Symptoms include /ready failure, journal commit failure metrics, degraded room errors, and all live room generations stopping.

  1. Stop accepting external traffic.
  2. Preserve logs and determine which filesystem is full; check database, WAL, backups, and container logs.
  3. Free space without deleting the live database, its WAL/SHM files, or the only valid backup.
  4. Gracefully stop the service if it is still running.
  5. Create a backup when writes are possible, then run restore-check.
  6. Restart, wait for /ready, and verify startup recovery and terminal history.

The service fails closed after a terminal journal error. Do not expect existing PTYs to remain interactive without durable storage.

Database Busy Or Locked

SQLite has a bounded busy timeout. Sustained external write locks eventually fail a journal commit and trigger process-wide room shutdown.

  1. Find and stop unsupported external writers. Only Cloud SSH and its supported online backup command should write the database.
  2. Do not run manual VACUUM, schema changes, or long write transactions while serving rooms.
  3. Check free disk space and filesystem latency.
  4. Allow the service to stop rooms, then restart after the lock is gone.
  5. Verify /ready, journal failure counters, durable lag, and generation recovery.

Corrupt Or Unsupported Snapshot

A snapshot with an unsupported encoding or a SHA-256 integrity mismatch is rejected rather than sent to a client. A digest mismatch returns 503 snapshot_corrupt. Output segments remain the durability source and are not deleted unless a retained snapshot covers them. Retention fails closed before deleting snapshots or output when any selected generation has an unsupported or digest-invalid snapshot.

  1. Stop retention so it cannot advance the boundary during investigation.
  2. Back up and restore-check the database.
  3. Inspect snapshot encoding/version and neighboring output ranges in a copy, never the live database.
  4. Use an earlier retained snapshot plus segments when available.
  5. If no valid anchor exists, preserve the database and treat the generation as unavailable; do not fabricate watermarks or edit history in place.

Release Gate

Before publishing or deploying a new main image:

make ci
make docker-build
make docker-smoke

make production-check runs all three. The gate includes Rust formatting, check, Clippy, tests, frontend formatting/typecheck/unit/build/Chromium E2E, migration validation, script and repository checks, static log/metrics and frontend-distribution secret audits, bounded ordering/fanout/burst/SQLite load acceptance, docs build, production image build, and runtime smoke test. CI and the release workflow additionally run the RustSec advisory gate against the locked dependency graph; pull requests run dependency review.

A validated push to main builds, smoke-tests, and publishes only ghcr.io/wh1isper/cloud-ssh:dev. Pull requests build and smoke-test without publishing. Release versioning and multi-platform images are outside the current target.