AWS cluster teardown, 2026-10-25

The AWS Talos cluster, its KubeVirt node and all their disks were deleted on 2026-10-25 because the account's credits ran out. This page records what is gone, what was saved and where, and how to rebuild.

Why

The account ran on one credit: AWS Open Source Promotional Credits, issued $1,435.00, valid 2026-08-01 to 2028-07-31. It is not monthly. It was consumed in under three months (Aug $108, Sep $550, Oct $932 applied). October was the first month with a net bill ($66.15 by the 25th, AWS forecast $307). At the footprint before the shutdown the cost was about $41/day gross.

Earlier text in this repo calls the credits "permanent" or "$1400/mo". That was wrong. $1,400 is the migration budget's ceiling, not an allowance.

State now

ThingState
aws-migration-worker-matrix, aws-migration-controlplane-hive, aws-migration-kubevirtterminated
Their EBS volumes (200, 400, 150 GB roots, 50 GB migration-pgdata)deleted
5 blank detached volumes created 2026-10-24 for an unstarted Hive-to-EBS migrationdeleted
EBS snapshots in eu-north-1 (35)deleted
punjab (t3.large, us-east-1), its 250 GB volume, one snapshotkept, running
Elastic IPs 13.62.161.5 (the matrix.reilly.asia target) and 13.63.243.56kept, unassociated
VPC, subnet, security groups, IAM, budgets, S3 bucketkept

Matrix (reilly.asia), the Hives (school., reef., hub.tunaos.org, hive.reilly.asia), the CFP dashboard and Postgres are offline.

aws/ (OpenTofu) was not changed. The instances were removed outside it, so just aws-plan will plan to create them again. That plan is the rebuild.

Where the data is

Bucket hanthor-fleet-backups-181185361136 (eu-north-1).

Cluster state: cluster-state-2026-10-25/cluster-state.tar.gz.age

Contains a Talos etcd snapshot (2,507 keys, includes every Kubernetes Secret: the MAS and Matrix secrets generated by ESS, tokens, etc.) and a YAML dump of 58 resource types. It is encrypted with age to five SSH keys: punjab, and the james@kanpur, himachal, james@Kerala and james@Jamess-MacBook-Air.local keys. Any one private key opens it:

aws s3 cp s3://hanthor-fleet-backups-181185361136/cluster-state-2026-10-25/cluster-state.tar.gz.age .
age -d -i ~/.ssh/id_ed25519 cluster-state.tar.gz.age | tar xz   # etcd.snapshot, resources/*.yaml

Round trip checked: the decrypted archive hashes identically to the original. The private keys' own backup is not verified from here (Bitwarden was locked).

Volumes: postgres/shutdown-export-2026-10-25/

ObjectContent
ess_ess-spindle-data.tar.gzspindle store (the live homeserver)
ess_synapse-media-preseed.tar.gz, ess_spindle-v0-1-0-rollback.tar.gzmedia, rollback store
hive-reef_hive-data.tar.gz, hive_hive-data.tar.gz, hive-hanthor_hive-data.tar.gzthe three Hives' data (about 43 GiB stored)
hive_discord-reports-data, default_review-data, hive-hub_hive-hub-data, hive_hive-ops-statesmall state

tar | gzip -1, storage class GLACIER_IR (retrieval about $0.03/GB). Each archive's tar size, recomputed on the node, equals the uncompressed size in its gzip trailer. They were not restore-tested.

Postgres: postgres/2026-10-25/

Final dump, 4.1 GB: Synapse and MAS.

Permanently gone (not exported)

Contributor homes (claude-home 46 GB, pi-codex-home 43 GB, agy-home 15 GB), the spindle-rehearsal volumes, raw pgdata (the dump covers it), hive-shared-auth (provider logins, sign in again), and everything that lived on the KubeVirt node (monitoring data, VM disks, wootc). Workload replica counts and cronjob suspend flags were saved as annotations and are inside the etcd snapshot.

Rebuilding (outline, untested)

  1. just aws-apply recreates the three instances from aws/. The Elastic IPs and private addresses are unchanged, so DNS and the Talos config still match.
  2. Restore etcd onto the new control plane so the Secrets and workload definitions come back, using the snapshot from the archive: talosctl --talosconfig ~/.talos/config-aws-migration -e <eip> -n 10.20.1.10 bootstrap --recover-from=./etcd.snapshot (the machine secrets come from the OpenTofu state, which is what the nodes were built with).
  3. Recreate each volume's directory under /opt/local-path-provisioner/ on the node that holds it (the PV objects name the directories) and untar the matching archive into it. A read-write hostPath helper pod in kube-system does this; other namespaces reject hostPath.
  4. Restore Postgres from the dump (see talos-k8s/backup/), then scale the workloads back up from the saved annotations tunaos.org/pre-shutdown-replicas and tunaos.org/pre-shutdown-suspend.

Cold start of the control plane from its stopped disk was exercised once during the teardown (the API server and etcd came back intact). Nothing else here has been run end to end.