Dotfiles Fleet Handbook
Ansible-driven dotfiles and infrastructure for a personal fleet. Three distinct concerns live here:
- Workstation config — shells, packages, browser, SSH, Tailscale, kubeconfig, GNOME, and more
- Home servers — bihar and karnataka (Bluefin + k3s, linked by KubeStellar); the hives run on bihar with karnataka as standby. The Talos manifests are archived
- Hosted infrastructure — OCI spindle (Matrix, public front door, hive failover coordinator) and the AWS account; the AWS Talos cluster is terminated
Current state of every host: Fleet status.
This handbook is public. It covers architecture, decisions and runbooks. Never put secrets in it. Fleet addresses live in inventory or come from
just inventory; service runbooks may document public endpoints.
Quick Start
# Apply to local machine (most common)
just apply
# Apply specific tags only
just apply-tags kube,shell
# Apply to a remote machine
just apply-remote himachal
# Dry-run to see what would change
just check
# Lint all YAML and Ansible
just lint
Machine Fleet
Ansible-managed hosts, from inventory.yml:
| Machine | Group | Type | Role |
|---|---|---|---|
| kanpur | desktop | Laptop | Portable workstation |
| himachal | desktop | Laptop | Portable workstation; legacy hive_ops timers retired |
| dilli | desktop | Desktop | Secondary workstation |
| kerala | desktop | Mobile | postmarketOS ARM device |
| mumbai | desktop | VM | Debian VM on the phone, machine_profile: cli-only |
| punjab | server | VM | Agent box, KubeVirt VM on karnataka |
| bihar | desktop | Server | Home server: active hive host, KubeStellar hub |
| goa | server | Server | Raspberry Pi 5 ARM server |
| vm | server | VM | Local dev VM |
| termux | termux_hosts | Phone | Raw Termux/Android layer |
| test-fleet-fedora, test-fleet-node2 | test_fleet | VM | KubeVirt test VMs (vm-test) |
| matrix | vps | VPS | Retired — see the cutover runbook |
| telengana | vps | VPS | Retired — see the cutover runbook |
Outside this inventory:
- karnataka — home k3s/KubeVirt server, configured from the native-units branch. See Karnataka.
- Spindle — OCI Frankfurt VPS: Matrix, LiveKit, the Caddy front door for the hives, and the hive failover coordinator.
- The AWS Talos cluster was terminated on 2026-10-25, and the home Talos cluster was replaced by k3s (archive).
The llm group is dormant (commented out in inventory.yml).
Playbook Phases
site.yml groups roles into four commented phases. Tags cut across them — use
just apply-tags <tag> to target a subset.
| Phase | Roles | Main tags |
|---|---|---|
| 1 — System + packages + dotfiles | sudo, sshd, apk_packages, homebrew, termux_packages, bitwarden, shell_fonts, shell_dotfiles, omp, hive_ops, git, neovim | system, packages, dotfiles |
| 2 — Secrets + auth | shell_atuin, shell_ai, ssh_keys, ssh_mesh, github, tailscale, kube, forgejo_registry | secrets |
| 3 — Desktop apps | flatpak, bluefin_common, gnome, zen_browser (+ browser_fxa), pipewire_audio | desktop |
| 4 — Services | syncthing, systemd, bst_dashboard, kirocrew, proxy, tailscale_cert, server_hardening | services |
Only the secrets-tagged work needs Bitwarden. With the vault locked (or
--skip-tags secrets), phases 1, 3 and 4 run normally and BW-touching tasks
skip cleanly.
Architecture
Repository Layout
dotfiles/
├── site.yml # Main playbook — entrypoint for all machines
├── inventory.yml # Host inventory + group definitions
├── Justfile # Task runner (just apply, just lint, etc.)
├── aws/ # OpenTofu for the TunaOS AWS account and cluster
├── group_vars/
│ ├── all.yml # Shared vars: packages, flatpaks, fleet config
│ └── vps.yml # VPS-specific overrides
├── host_vars/
│ └── <machine>.yml # Per-machine settings (is_laptop, skip_*, etc.)
├── roles/
│ └── <role>/ # Each role: tasks/, files/, templates/, vars/
├── docs/ # This handbook
├── scripts/ # Utility scripts (inventory scan, BW seed)
├── talos-k8s/ # Talos Linux + Kubernetes cluster manifests
└── tools/ # Standalone tools, including the secret-broker
How It Works
Each machine manages itself via ansible-playbook --connection=local. There's no central control node. This is an Ansible pull model — the playbook lives on every machine and each host applies it locally.
Local Apply
just apply
Runs the playbook locally with --connection=local. The playbook auto-detects which host it's running on by matching /etc/dotfiles-machine or falling back to localhost.
Remote Apply
just apply-remote himachal
SSHes to the target, forwards the Bitwarden session token, pulls the latest dotfiles, then runs just apply locally on the remote machine.
Tagged Apply
just apply-tags kube,shell
Runs only roles tagged with the given tags. Useful for deploying specific configs without running the full playbook. --skip-tags works the same way in reverse — just apply-nosecrets is --skip-tags secrets under the hood (see Bitwarden).
Available Tags
Every role in site.yml carries one or more tags. Tags fall into two kinds: phase tags shared by roles in that phase, and role tags naming (or further scoping) one specific role. Phases follow the # Phase N comments in site.yml.
| Phase | Role | Tags |
|---|---|---|
| (pre/post tasks) | fact setup, platform detection, secrets summary | always — runs on every tagged apply unless --skip-tags always |
| (before Phase 1) | sudo | system, sudo |
| System + packages | sshd | system, sshd |
| System + packages | apk_packages | system, packages, apk |
| System + packages | homebrew | packages, homebrew |
| System + packages | termux_packages | system, packages, termux |
| System + packages | bitwarden | secrets, bitwarden |
| System + packages | shell_fonts | dotfiles, shell, fonts |
| System + packages | shell_dotfiles | dotfiles, shell, dotfiles |
| System + packages | omp | dotfiles, omp |
| System + packages | hive_ops | hive, hive_ops |
| System + packages | git | dotfiles, git |
| System + packages | neovim | dotfiles, neovim, editor |
| Secrets + auth | shell_atuin | dotfiles, shell, atuin, secrets |
| Secrets + auth | shell_ai | dotfiles, shell, ai, secrets |
| Secrets + auth | ssh_keys | secrets, ssh, ssh_keys |
| Secrets + auth | ssh_mesh | ssh, mesh_check |
| Secrets + auth | github | secrets, github |
| Secrets + auth | tailscale | secrets, tailscale |
| Secrets + auth | kube | secrets, kube |
| Secrets + auth | forgejo_registry | secrets, forgejo, registry |
| Desktop apps | flatpak | packages, desktop, flatpak |
| Desktop apps | bluefin_common | desktop, bluefin |
| Desktop apps | gnome | desktop, gnome |
| Desktop apps | zen_browser | desktop, browser, zen_browser |
| Desktop apps | pipewire_audio | desktop, audio, pipewire |
| Services | syncthing | services, syncthing |
| Services | systemd | services, systemd |
| Services | bst_dashboard | services, bst_dashboard |
| Services | kirocrew | services, kirocrew |
| Services | proxy | services, proxy, caddy |
| Services | tailscale_cert | services, tailscale_cert |
| Services | retired_services | system, retired_services |
| Services | server_hardening | system, hardening, security |
| Services | fleet_facts | facts, fleet_facts |
Common phase-wide selections:
--tags secrets— every BW-backed role only (bitwarden,shell_atuin,shell_ai,ssh_keys,github,tailscale,kube,forgejo_registry)--tags desktop— every desktop-app role (flatpak,bluefin_common,gnome,zen_browser,pipewire_audio)--tags services— every long-running service role (syncthing,systemd,bst_dashboard,kirocrew,proxy,tailscale_cert)--skip-tags secrets— everything except BW-backed roles. This isjust apply-nosecrets, which the dailydotfiles-update.timerand thedotsalias run;bootstrap.shandphone-apply.servicepass--skip-tags secretsdirectly
Tags do not replace the skip_* host_vars flags or inventory-group gating described below — a role can be tag-selected and still no-op if its when: condition (group membership, skip_*, machine_profile) evaluates false.
Role Structure
Each role follows the standard Ansible layout:
roles/<name>/
├── tasks/
│ └── main.yml # Role entrypoint
├── files/ # Static files to deploy
├── templates/ # Jinja2 templates (with .j2 extension)
├── vars/ # Role-specific variables
│ └── main.yml
├── meta/
│ └── main.yml # Explicit prerequisites for this role
└── handlers/ # Notify targets
└── main.yml
Role Dependencies
Prerequisites that provide required facts, files, or services are declared in
roles/<name>/meta/main.yml, not only by their position in site.yml. The
playbook still owns fleet-wide phase ordering, host conditions, and tags; the
metadata makes the prerequisite relationship part of the role interface.
| Role | Dependency | Contract |
|---|---|---|
shell_atuin | bitwarden | Reads the Atuin login item and bw_unlocked |
shell_ai | bitwarden | Reads API credentials for generated shell files |
ssh_keys | bitwarden | Recovers and mirrors the machine key |
github | ssh_keys | Registers the public key maintained by ssh_keys |
kube | bitwarden | Reads the cluster configuration notes |
forgejo_registry | bitwarden | Reads the registry login item |
tailscale | bitwarden | Reads the join key when the host is not connected |
Optional integrations keep their existing standalone behavior. For example,
browser_fxa can receive credentials directly and therefore does not declare a
Bitwarden dependency.
Conditional Execution
Roles are gated by:
- Inventory groups —
when: is_desktop | boolruns only on desktop group members - Host vars —
skip_*: trueflags inhost_vars/<machine>.ymlskip specific roles - Tags —
--tags kube,shelllimits which roles execute
Skip Flags
Every flag site.yml reads, all defaulting to false (| default(false)). Set skip_<name>: true in host_vars/<machine>.yml to skip that role on that machine. No role reads a skip_* variable itself — the gating is all in site.yml.
| Flag | Role skipped | Other when: gates on the role | Set in host_vars/ |
|---|---|---|---|
skip_sudo | sudo | — | termux |
skip_ssh_mesh | ssh_mesh | not vm-test | — |
skip_kube | kube | desktop, not vm-test | mumbai, termux |
skip_forgejo | forgejo_registry | not vm-test | — |
skip_flatpak | flatpak | desktop, not vm-test | mumbai, termux |
skip_bluefin | bluefin_common | desktop, not vm-test | mumbai, termux |
skip_gnome | gnome | desktop, not vm-test | mumbai, termux |
skip_zen_browser | zen_browser | desktop, not vm-test | mumbai, termux |
skip_pipewire_audio | pipewire_audio | desktop, not vm-test | mumbai |
skip_syncthing | syncthing | not vps, not vm-test | mumbai, termux |
skip_systemd | systemd | — | termux |
skip_proxy | proxy | not vps, not vm-test | mumbai, punjab, termux |
skip_tailscale_cert | tailscale_cert | not vps, not vm-test | mumbai, termux |
"desktop" is is_desktop (membership of the desktop group unless overridden); "not vm-test" is machine_profile != 'vm-test'. Roles without a flag either always run or are gated another way: termux_packages (termux_hosts group), omp (not Termux), kirocrew (kirocrew_enabled: true), server_hardening (server/vps groups), retired_services (not vps, not Termux, systemd hosts).
Secrets Flow
- Ansible requests a Bitwarden session (env var, cached file, or interactive unlock)
- The
bitwardenrole resolvesBW_SESSIONand caches it - Secrets-tagged roles consume the session token for API calls
- For remote applies,
scripts/bw-resolve.sh remote <host>unlocks the vault on the target and the recipe exports that session inline overssh -t(noSendEnv) — see Remote Apply
No secrets are stored in git — the repo is fully public.
Inventory & Groups
Host Groups
inventory.yml defines these groups:
desktop
Workstations and laptops with a full desktop environment (GNOME, browsers, etc.).
desktop:
hosts:
mumbai: # Debian VM on the phone (cli-only, skips desktop roles)
kerala: # postmarketOS ARM device
kanpur: # Laptop
himachal: # Laptop (runs hive_ops)
dilli: # Secondary desktop
Desktop roles: flatpak, gnome, zen_browser, bluefin_common, pipewire_audio, kube
server
Headless hosts.
server:
hosts:
punjab: # agent box, KubeVirt VM on karnataka (100.103.187.46)
vm: # Local dev VM
goa: # Raspberry Pi 5 (ARM)
Server roles: server_hardening, plus the service roles.
vps
The inventory lists two retired Hetzner VPSes, formerly migrated to the AWS Talos cluster, which was terminated on 2026-10-25. Restart nothing on these hosts; follow the Matrix cutover runbook before touching either. Active spindle is an OCI Frankfurt VPS running Matrix and standalone LiveKit; it is not in this inventory.
vps:
hosts:
matrix: # Retired — former Matrix homeserver
telengana: # Retired — hosted the Hive on k3s
vps members skip syncthing, proxy and tailscale_cert, and get
server_hardening. group_vars/vps.yml is currently empty.
test_fleet
KubeVirt VMs reached via virtctl port-forward (see the host's host_vars).
Both set machine_profile: vm-test, which skips desktop and service roles.
test_fleet:
hosts:
test-fleet-fedora:
test-fleet-node2:
termux_hosts
The raw Termux/Android layer on the phone. Gates the termux_packages role.
termux_hosts:
hosts:
termux:
llm (dormant)
Commented out in inventory.yml — karnataka, its former member, now runs k3s workloads.
Not in the inventory
bihar is in the desktop group (Bluefin, bootstrapped 2026-11-17).
karnataka is configured from the native-units branch
(#348) and is not listed here.
Neither is a Talos node any more; see Fleet status. spindle is
not in the inventory. The terminated AWS cluster's nodes were not in the inventory either.
Per-Machine Configuration
Each machine has a host_vars/<name>.yml file with common knobs:
is_laptop: true # Laptop (runs on login, not timer)
skip_flatpak: true # Skip Flatpak installs
skip_gnome: true # Skip GNOME config
skip_kube: true # Skip kubeconfig deploy
skip_proxy: true # Skip Caddy proxy
machine_profile: cli-only # Free-form profile; `vm-test` skips desktop + service roles
web_services: [] # Service links for the local dashboard
The full set of skip_* flags site.yml honours: skip_sudo, skip_ssh_mesh,
skip_kube, skip_forgejo, skip_flatpak, skip_bluefin, skip_gnome,
skip_zen_browser, skip_pipewire_audio, skip_syncthing, skip_systemd,
skip_proxy, skip_tailscale_cert.
Group Variables
group_vars/all.yml— Homebrew package lists, Flatpak remotes, Firefox Accounts config, GNOME extensions, authorized SSH keysroles/flatpak/vars/main.yml— thesystem_flatpakslistgroup_vars/vps.yml— currently empty
Secrets with Bitwarden
All secrets live in Bitwarden and are resolved at runtime.
How It Works
- The
bitwardenrole runs first in the secrets phase. - It resolves a session token in order:
BW_SESSIONenv var →/tmp/bw_session→ interactivebw unlock. - It runs
bw sync(with a 15 s hard timeout, so a flaky vault server can't hang the play) to refresh the local cached vault. - It calls
bw statusand publishes asecrets_availablehost fact:trueonly if the CLI is installed, a session was resolved, and the vault actually unlocks. - Every BW-using role gates on
when: secrets_available | default(false). The end-of-playpost_taskssummary prints one of:vault unlocked — all secret roles executed normallyBitwarden vault was LOCKED — … skipped. To enable: bw unlock …SKIPPED (--skip-tags secrets). Run dots-apply to include …
Remote Apply
just apply-remote <host> (and just apply-remote-tags <host> <tags>):
- Runs
scripts/bw-resolve.sh remote <host>, which unlocks Bitwarden on the remote and prints that host's session token. If the remote CLI is logged out it triesbw login --apikeywith thebw-api-keyitem from your local vault; to unlock it pipes the master password (itemJames Bitwarden) over stdin, falling back to an interactivebw unlockoverssh -t. - Runs
ssh -t <host> "export BW_SESSION='…'; …; just apply"— the session is exported inline in the remote command, not viaSendEnv.apply-remotealso caches it in the remote's/tmp/bw_session. - The
bitwardenrole on the remote runsbw status: if it reportsunlocked,secrets_availablebecomes true and secrets work proceeds.
Caveat — session portability is not guaranteed. A BW_SESSION token is only valid on the machine that unlocked it — that's why the recipes unlock on the remote instead of forwarding yours. If the remote unlock fails, the remote bw status returns locked, the role prints a one-line warning, and every BW-touching task is skipped cleanly — nothing fails, no files get clobbered with empty values. Unlock the vault directly on that host (bw unlock, then write the session to /tmp/bw_session) to enable secrets there.
Vault Items
Every item a role or script reads (grep bw get / bw list under roles/ and scripts/):
| Bitwarden Item | Type | Used By |
|---|---|---|
james@<machine> | SSH Key | ssh_keys role — per-machine ed25519 keys |
tailscale-apikey | Login (password) | tailscale role — key for tailscale up --authkey and stale-device cleanup via the API |
kubeconfig | Secure Note | kube role — home cluster kubeconfig |
talosconfig | Secure Note | kube role — home cluster Talos config |
kubeconfig-aws-migration | Secure Note | kube role — retained kubeconfig for the terminated AWS cluster |
talosconfig-aws-migration | Secure Note | kube role — retained Talos config for the terminated AWS cluster |
accounts.firefox.com | Login + TOTP | browser_fxa role — Firefox Account credentials |
github-token | Login (password) | github role — PAT for gh auth login |
atuin.sh | Login + custom field key | shell_atuin role — atuin sync login + mnemonic |
deepseek-api-key | Login (password) | shell_ai role — pi auth.json |
forgejo | Secure Note (API Token: line) | shell_ai role — FORGEJO_TOKEN |
tavily-api-key | Login (password) | shell_ai role — TAVILY_API_KEY |
forgejo.manatee-basking.ts.net | Login | forgejo_registry role — podman registry login |
bw-api-key | Login | scripts/bw-resolve.sh — bw login --apikey on a remote |
James Bitwarden | Login | scripts/bw-resolve.sh — non-interactive unlock on a remote |
Recovery items (not read by any role)
These items are kept so that you can rebuild a lost host by hand. No role reads them.
| Bitwarden Item | Type | Holds |
|---|---|---|
hive-github-app-hanthor-hive-agent | Secure Note | School's GitHub App: App ID, installation ID, PEM private key (the hub mounts the same key). See Hive |
hive-github-app-tuna-reef-hive | Secure Note | Reef's GitHub App: App ID, installation ID, PEM private key |
hive-spindle-env | Secure Note | The hives' .env (hub secret, dashboard token, omp broker token, provider keys). It was /srv/hives/.env on spindle and is now /var/srv/hives/.env on bihar and karnataka. The item name has not changed |
Running Without Secrets
For fast iteration that doesn't need the vault — packages, dotfiles, desktop config — use the explicit no-secrets path:
just apply-nosecrets # git pull + apply with --skip-tags secrets
dots # alias: cd to the repo + just apply-nosecrets
The daily systemd timer (dotfiles-update.service) calls apply-nosecrets for the same reason: it can't unlock the vault non-interactively, so it doesn't try.
No Session?
If Bitwarden can't unlock (first run, no BW CLI installed, vault genuinely locked on this host), every secret-touching task is skipped automatically. Non-secret roles (system, packages, dotfiles, desktop, most services) run normally and the play exits successfully — the end-of-play summary tells you exactly which roles didn't get to do their BW work.
Seeding New Secrets
# Push kubeconfig + talosconfig (home cluster) to Bitwarden from a machine that has them:
just seed-kube
seed-kube only handles the two home-cluster notes; the *-aws-migration
notes are maintained by hand.
Onboarding a New Machine
Step 0: Get on the tailnet by scanning a QR (recommended first step)
Before secrets or SSH can flow, the machine needs to be on the tailnet. Instead of copying an auth key, join it by scanning a QR with your phone — run this on the new machine:
just tailscale-qr # uses default machine name
just tailscale-qr name=custom # or provide a custom name
This runs interactive tailscale up --qr: it renders the login URL as a QR in
the terminal. Scan it with your phone and approve the device in the Tailscale
admin (passwordless via your IdP). No auth key, no Bitwarden, no master password.
Once the machine is on the tailnet, its tailnet identity becomes its first
credential — every other onboarding step (secret delivery, apply, SSH) flows
from there. This is Phase 0 of the QR secret onboarding
effort; the later phases replace the auth-key/master-password handshake below
entirely.
Requires you to be a
tagOwneroftag:fleetin the tailnet ACL (every fleet node carries this tag). Drop--advertise-tagsin the recipe to join as a personal device instead.
Then continue with Path A or B below.
Two paths depending on whether the machine is already reachable.
Path A: Machine Already Online (Tailscale + SSH)
# From any machine with the dotfiles repo:
just add-machine <name> <type>
This:
- If the machine isn't in
inventory.ymlyet, adds it (ansible_host: localhost,ansible_connection: local) and puts it in the<type>group - Creates an empty
host_vars/<name>.yml— add per-machine knobs there afterwards - Commits and pushes
- SSHes in (
ssh -t <name>) and runsbootstrap.sh --name <name> --type <type>there — the same bootstrap as Path B
Path B: Fresh Install
# From any machine with the dotfiles repo:
just onboard <name> <type>
This prints a bootstrap command to run on the new machine:
# On the new machine:
curl -fsSL https://raw.githubusercontent.com/hanthor/dotfiles/master/bootstrap.sh \
| bash -s -- --name <name> --type <type>
The bootstrap script:
- Ensures python, git, uv and Ansible, then clones the dotfiles repo from GitHub
- Registers the machine in
inventory.ymlif needed (prompts for the type when--typeisn't given) - Writes
/etc/dotfiles-machineso the playbook can self-identify, sets the hostname, sets up sudo and Tailscale - Runs
site.ymlwith--skip-tags secrets(installs Homebrew and core packages among the rest) - If the Bitwarden CLI is now available and unlocks, runs
site.yml --tags secretsand pushes any inventory changes
After Bootstrap Completes
Bootstrap ends with shell environment setup instructions. The critical next step is to start a new shell and run the first full apply:
# Start a fresh shell to load updated environment (e.g., Homebrew paths)
exec zsh
# Run the first full apply (with secrets)
just apply
Why this step matters:
- Bootstrap runs
site.yml --skip-tags secrets, which skips Bitwarden integration just applyruns the full playbook with secrets, unlocking Bitwarden and completing setup- This pull-in step distributes SSH keys across the fleet, registers with GitHub and Tailscale, and sets up daily/on-login automation
Do not edit host_vars/<name>.yml before the first just apply — bootstrap sets sensible defaults, and just apply will pull in critical secrets (SSH keys, GitHub tokens, Tailscale auth). Any per-machine customizations can follow once the first apply succeeds; the skip_* flags that turn individual roles off are listed in Architecture → Skip Flags.
The details of what happens during this first apply are covered in the Post-Onboarding section below.
Machine Types
<type> is the inventory group to join. bootstrap.sh offers desktop, server and vps:
| Type | Group | Has Desktop? | Notes |
|---|---|---|---|
desktop | desktop | Yes | Laptops are desktops too — set is_laptop: true in host_vars |
server | server | No | Gets server_hardening; desktop roles skip automatically |
vps | vps | No | Also skips syncthing, proxy, tailscale_cert |
There is no laptop type — a type that doesn't match a group leaves the host in
no group. test_fleet and termux_hosts hosts are added to inventory.yml by
hand.
Post-Onboarding
- The
ssh_keysrole generates a fresh ed25519 key on the new host, then pushes the pub key up to Bitwarden asjames@<hostname>. - The new host now needs to trust the rest of the fleet, and the rest of the fleet needs to trust it. The first apply on the new host downloads every other host's pub key into
~/.ssh/authorized_keys. The other hosts will pick up the new host's pub key on their next full apply (dots-applyon each — the timer runs without secrets, so it skipsssh_keys). - The
githubrole registers the SSH key with GitHub for both auth and commit signing. - The
tailscalerole joins the machine to the Tailscale network using the auth key in BW. - The daily
dotfiles-update.timer(servers) or on-login service (laptops) runsjust apply-nosecrets— secrets stay on the manualdots-applycadence because BW can't be unlocked non-interactively.
Run just doctor on the new host once it's bootstrapped — it'll tell you whether BW is unlocked, whether the timer is wired up, and the timestamp of the last successful apply.
See Architecture for how the playbook runs on each machine and Secrets with Bitwarden for the details of the BW handshake.
QR Secret Onboarding
Tracking issue: hanthor/dotfiles#49
Onboard (or re-align) a machine and deliver its secrets seamlessly and
securely by scanning a QR from your phone — no master password typed on the new
box, no hand-copied BW_SESSION, no chicken-and-egg SSH setup.
The core problem: bootstrap-of-trust
The hard part is delivering the first credential to a machine that has nothing.
Once the box holds any one of {tailnet membership, a single Bitwarden credential},
everything else — the full BW_SESSION, kubeconfig, SSH keys, Tailscale auth key
— already flows through existing automation (bw-resolve.sh,
the tailscale/kube/ssh_keys roles). So the whole question is: how does that
first secret arrive?
The spine: Tailscale identity as the bootstrap credential
We already run Tailscale everywhere, so the passwordless + QR experience is mostly already in the stack:
- Approve the new device on your phone.
tailscale up --qrrenders the login URL as a QR; scan it and approve in the Tailscale admin (passwordless via the IdP). The box joins the tailnet, taggedtag:fleet. - The machine's tailnet identity is then its first credential. A small tailnet-only secrets broker sees the authenticated node identity and hands it exactly the secrets it needs, read from Bitwarden server-side.
- No secret ever rides in the QR — the QR carries a login URL; the phone approves; the identity does the rest.
End-to-end flow
sequenceDiagram
participant N as New machine
participant P as Your phone
participant TS as Tailscale admin
participant B as secret-broker (tailnet)
participant BW as Bitwarden Secrets Manager
Note over N,TS: Phase 0 — join the tailnet
N->>N: just tailscale-qr (renders login QR)
P->>TS: scan QR + approve device
TS-->>N: authenticated; tagged tag:fleet, gets a StableNodeID
Note over N,BW: Phase 1 — first secrets, over the tailnet
N->>B: GET /v1/secrets (identity = WhoIs, unforgeable)
B-->>N: 202 pending + fingerprint
P->>B: just broker-pending (see stableID + fingerprint)
P->>B: just broker-approve <stableID> (match fingerprint)
N->>B: GET /v1/secrets (retry)
B->>BW: bws get (only this node's policy refs)
BW-->>B: secret values
B-->>N: 200 + least-privilege secrets
Bitwarden stays the source of truth — we're replacing the bootstrap, not the
vault. (The bw CLI cannot do passwordless "log in with device"; it only supports
password / --apikey / --sso, so we don't build on that.)
Bonus: adopting Tailscale identity here also opens the door to Tailscale SSH
(auth by tailnet identity, ACL-gated), which would have prevented the "kerala
couldn't SSH anywhere" incident — there'd be no authorized_keys to fall out of.
Phases
| Phase | What | Status |
|---|---|---|
| 0 | QR tailnet join — just tailscale-qr renders the tailscale up login URL as a console QR; scan + approve on phone. | Done |
| 1 | Identity broker — a tailnet-only service (tools/secret-broker) that issues per-node least-privilege secrets from Bitwarden Secrets Manager, gated on unforgeable StableNodeID + operator approval (mirrors Tailscale device approve). | Code-complete — deny-path unit-tested; live run pending (needs TS authkey + BWS token per the runbook) |
| 2 | Hardening — ACL restricting tag:fleet→broker port, wiring the client into onboarding once run live, richer per-node policy config. | Planned |
Phase 0 (implemented)
# On the new machine:
just tailscale-qr <name>
Runs interactive tailscale up --qr --hostname=<name> --advertise-tags=tag:fleet --accept-routes. Uses the Tailscale CLI's native --qr flag (confirmed on the
fleet's 1.98.x) — no qrencode dependency. See the tailscale-qr recipe in the
Justfile; if tailscale isn't installed yet, the recipe installs it first.
Phase 1 (code-complete, live run pending)
Once the node is on the tailnet, it fetches its first secrets from the broker, gated on its unforgeable tailnet identity plus a one-time operator approval:
just broker-request # on the NEW node — prints its fingerprint + "pending"
just broker-pending # on your ADMIN node — see the queued request
just broker-approve <stableID> # approve that specific node (verify the fingerprint)
just broker-request # on the NEW node — now returns its secrets
Authorization is scoped to the StableNodeID (unforgeable), never hostname;
approval is per-node (no time-window race); secrets come per-node least-privilege
from Bitwarden Secrets Manager (read-scoped machine-account token, no vault
master password). Full design, security model, blast-radius note, and the
operator runbook for the live setup (mint TS authkey, provision the BWS
project/token, systemd unit, ACL) live in
tools/secret-broker/README.md.
The client recipes are manual/opt-in — deliberately not wired into
just apply/onboarding until the flow has been run live once.
Security requirements (all phases)
- No long-lived secret encoded in a QR that persists on screen/scrollback.
- Tokens are short-TTL and single-use; pairing nonces expire fast.
- Out-of-band verification fingerprint on both phone and machine (anti-MITM).
- Master password stays on the phone; never keyed into the onboarding box.
- Transport is the tailnet (WireGuard) or otherwise E2E-encrypted.
- Repo stays public — code and docs only, never secrets.
References
- Onboarding a New Machine — practical steps
- Secrets with Bitwarden — vault structure and the BW handshake
scripts/bw-resolve.sh— current session resolution- Tracking issue #49
Operational runbook
Recipes for the questions that recur. If you find yourself answering one of these from memory or commit logs more than twice, add it here.
I rotated my SSH key on host X — how do I propagate it?
An intentional local rotation is explicit: when the local public key differs
from the Bitwarden item, the ssh_keys role leaves Bitwarden unchanged and
warns. This protects the stored key if a new key was generated while the vault
was locked. To publish an intentional rotation, on host X with BW unlocked:
# `dots-apply` forwards these arguments to the Ansible playbook.
dots-apply -e ssh_keys_force_push=true
That updates the james@<hostname> item in Bitwarden and refreshes this host's
authorized_keys while it is at it. Then on every other host:
dots-apply # picks up the fresh pubkey from BW into authorized_keys
If the mismatch was not intentional, do not use the force flag: move the local key aside and re-apply so the host fetches the Bitwarden copy.
If you can't reach an "other" host because of the stale key situation, you can also push the new key directly first:
# From a host that can still reach the broken one
ssh-copy-id <broken-host> # adds the rotating host's pub key by hand
# then `dots-apply` on broken-host as above
BW is locked on a remote — how do I unlock it there?
just apply-remote / apply-remote-tags try to unlock the vault on the remote
themselves (scripts/bw-resolve.sh remote), but that can fail — e.g. no
bw-api-key item, or the remote CLI was never logged in. To unlock directly on
that host:
# From your laptop, let bw-resolve.sh handle unlock internally:
just apply-remote-tags <host> bitwarden,ssh_keys
The scripts/bw-resolve.sh remote <host> script (invoked internally) will
SSH to the remote, ensure Bitwarden is logged in and unlocked (either via
stored API key, master password from your local vault, or interactive prompt),
and return the session token. The bitwarden role will then confirm the vault
is unlocked, and downstream secrets roles will run.
The timer apply failed last night — what do I do?
The user-level dotfiles-update.service runs just apply-nosecrets daily,
which now writes ~/.cache/dotfiles/last-apply.json. Quickest signal:
just doctor # `→ Last apply` line shows status + age
If rc != 0, see what blew up:
journalctl --user -u dotfiles-update.service -e --no-pager | tail -80
The most common causes:
- git pull failure (DNS / network blip): the recipe tolerates it with
git pull --ff-only || true; the ansible run continues with whatever's on disk. If the working tree drifted from main, fix manually. - a role started failing after a config change: re-run interactively
with
just apply-nosecretsand read the failure. mkswap: /swapfile is mounted(resolved):server_hardeningnow gatesmkswap/swapononswapon --show, so an already-active swapfile is left alone. If you still see this, the host is running an old checkout.
BW master password just returned "decryption failed" — vault corrupt?
Two BW CLI processes can corrupt the local encrypted vault if they sync at the
same time. Symptom: bw unlock --raw returns
Cryptography error, The decryption operation failed.
Fix is to re-login (downloads a clean vault):
bw logout
bw login # email + master password + 2FA
export BW_SESSION=$(scripts/bw-resolve.sh local)
The bitwarden role now has a 15 s timeout on bw sync, which prevents the
race in most cases going forward.
I want to onboard a new machine — full sequence
# 1. From an existing fleet machine (with BW unlocked):
just onboard <newhost> desktop # or server / vps
# 2. Copy the printed bootstrap command, then on the new machine:
curl -fsSL https://raw.githubusercontent.com/hanthor/dotfiles/master/bootstrap.sh \
| bash -s -- --name <newhost> --type desktop
# 3. After bootstrap finishes, on the new machine, get BW into a usable state:
bw login # if not logged in
export BW_SESSION=$(scripts/bw-resolve.sh local) # unlocks vault
# 4. Run the full apply (this pushes the new machine's pubkey to BW):
dots-apply
# 5. Back on each existing host (the no-secrets timer skips ssh_keys):
dots-apply # picks up the new host's pubkey
Verify with just doctor on the new host — should be all green and report
"last apply
I added a flatpak / brew package — how do I roll it out?
Edit the package list — system_flatpaks in roles/flatpak/vars/main.yml,
or the brew lists (core_brews, desktop_brews, …) in group_vars/all.yml,
which render the Brewfile — then commit, push. Every host picks it up on the next
dots-apply or the next timer fire.
To force-apply immediately on the local machine:
just apply-tags packages
I want to disable a role on one host
In that host's host_vars/<name>.yml, add a skip_<role>: true line. The
flags site.yml actually checks:
skip_sudo, skip_ssh_mesh, skip_kube, skip_forgejo, skip_flatpak,
skip_bluefin, skip_gnome, skip_zen_browser, skip_pipewire_audio,
skip_syncthing, skip_systemd, skip_proxy, skip_tailscale_cert.
Roles without a flag are gated some other way — server_hardening on
server/vps membership, termux_packages on termux_hosts, many roles on
machine_profile != 'vm-test'. Grep site.yml.
How do I verify a change made it to every host?
just doctor-fleet
Runs just doctor on every online host in parallel. The → Last apply
line tells you which hosts have an old or failed convergence — those are
the ones that need attention.
To roll a single change out manually across online hosts instead of waiting for the timer:
# Check online tailnet hosts in fleet inventory
just online
# Apply specific tags to all online fleet machines in parallel
just apply-online-tags <tag-or-tags>
# Or perform a full apply across all online fleet machines
just apply-all
Self-improving repo: bots, agents and guardrails
This repo is edited by bots, and every fleet machine pulls master daily and
applies it with sudo. A bad merge therefore reaches every machine within
24h. The automation below is set up so that routine improvements land without
you, and anything that executes on the fleet still waits for you.
Who changes what
| Actor | What it does | Merges by itself? |
|---|---|---|
| Renovate | minor/patch/digest bumps (pi npm deps, Go, GitHub Actions, OpenTofu providers), weekly lockfile maintenance | yes, once CI is green |
| Renovate (cluster) | image/chart bumps in talos-k8s/, grouped weekly, label deploy-needed | no — manifests are applied by hand, so a merge would only make git lie about the cluster |
Hive (school and reef spokes on bihar, with karnataka as standby; Apps hanthor-hive-agent and tuna-reef-hive) | its agents open PRs labelled hold; the Hive itself only merges PRs labelled lgtm | only if the PR stays inside the carve-outs below. Neither hive's roster includes this repo today (both cover only tuna-os/*), so no Hive PRs are expected here unless a hanthor spoke is rebuilt |
| You | direct pushes, just onboard, etc. | admin bypass — unaffected |
The gate
Three pieces, and none of them trusts the PR itself:
- Branch protection on
master: required checkslint,python,tests,tofu,justfile,broker,deps-guard, plus code-owner review. Admins can bypass, so your own pushes still work. .github/CODEOWNERSis deny-by-default (* @hanthor). The paths carved out are the ones that never run on a fleet machine:docs/,*.md,tests/,talos-k8s/, pi skill text, and dependency manifests..github/is always owned, so nobody can loosen the gate from inside a PR.agent-automerge.ymlqueues GitHub auto-merge on every Hive PR. It runs frommaster's copy (pull_request_target), never checks out PR code, and labels PRs that touch owned pathsneeds-human, with a comment listing the paths.
deps-guard lets Renovate change versions in roles/pi/files/package.json
but fails if a PR adds or removes a dependency. The manifest is tracked by
repository automation; the omp role copies roles/pi/files/skills/ and
installs OMP plugins separately. It does not run npm install from this
manifest on fleet machines.
Day to day
- Nothing to do for docs, tests or dependency bumps. They merge once green.
- Review anything labelled
needs-human(Hive) ordeploy-needed(Renovate, cluster). The Discord daily digest lists them under Needs you. - To let the Hive merge more, carve the path out in CODEOWNERS (and mirror it
in
agent-automerge.yml's comment logic). Only do that for paths that never execute on the fleet.
Hive config lives on its volume
Each live spoke persists its configuration to data/hive.yaml.runtime next to
the seed config/hive.yaml, and the runtime file wins on restart, so roster or
level changes made in the dashboard stick. hive.reilly.asia is the hanthor
spoke on bihar (rebuilt 2026-11-19); its seed is
hives/hanthor/hive.yaml.
The AWS-era ConfigMap in talos-k8s/hive/spokes/hanthor/manifest.yaml is
history. For the live files, read-only inspection and the safe edit procedure,
see the
Hive operations guide:
ssh james@bihar 'sudo podman ps --filter name=hives-; sudo cat /var/srv/hives/hanthor/data/hive.yaml.runtime'
Utility Scripts
The scripts/ directory contains helper utilities for inventory management, Bitwarden integration, and operational tasks. They are not deployed to machines — they run on your local development machine or during CI.
Inventory Management
register-machine.py
Register a new machine in inventory.yml and create its host_vars file.
Usage:
python3 scripts/register-machine.py <name> [type] [path-to-inventory.yml]
Arguments:
<name>— machine name (e.g.,kanpur,goa)[type]— group/machine type:desktop,server,vps,termux_hosts(default:desktop)[path-to-inventory.yml]— inventory file path (default:inventory.yml)
What it does:
- Adds the machine to the
all.hostssection - Adds it to the specified group under
all.children - Sets
ansible_host: localhostandansible_connection: local - Creates
host_vars/<name>.ymlwith a template
Example:
python3 scripts/register-machine.py goa server
purge-machine.py
Remove a machine from inventory.yml and its host_vars file. Mirror of register-machine.py.
Usage:
python3 scripts/purge-machine.py <name> [path-to-inventory.yml]
Important: Does NOT delete the corresponding Bitwarden SSH key item — you can re-onboard the machine later using the same name.
Example:
python3 scripts/purge-machine.py matrix
nmap-inventory.sh
Scan the LAN using nmap and cross-reference with Tailscale to show network status.
Usage:
./scripts/nmap-inventory.sh [CIDR]
Arguments:
[CIDR]— network range to scan (default:192.168.0.0/24). Override with a different subnet.
Output columns:
- IP address
- MAC address
- Hostname (reverse DNS or nmap service detection)
- Tailscale status (online/offline)
Prerequisites:
nmapinstalled- Tailscale running and authenticated
- Network access to the scanned range
Example:
./scripts/nmap-inventory.sh 10.0.0.0/24
Bitwarden Integration
bw-resolve.sh
Resolve a usable BW_SESSION for local and remote operations. Tries multiple fallback tiers.
Usage:
./scripts/bw-resolve.sh local
./scripts/bw-resolve.sh remote <host>
Modes:
-
local— Resolve a session on your local machine. Returnsexport BW_SESSION=...on stdout or exits 1 if unlock fails. -
remote <host>— Unlock Bitwarden on a remote machine, export that session inline overssh -t, and return it. Used byjust apply-remoteto drive secret-dependent tasks on the target.
Session resolution order:
- Existing
BW_SESSIONenvironment variable (if still valid) - Cached session in
/tmp/bw_session(if not expired) - Interactive
bw unlockprompt (asks for master password)
Example — local:
eval "$(./scripts/bw-resolve.sh local)"
bw list items # Now BW_SESSION is set
Example — remote:
./scripts/bw-resolve.sh remote himachal
# Executes commands on himachal with Bitwarden access
bw-unlock.sh
Simpler variant of bw-resolve.sh — resolves a session on the local machine only.
Usage:
./scripts/bw-unlock.sh
Output: Prints <session-token> on stdout, or exits 1 if all tiers fail.
bw-seed-ssh-keys.sh
Pull SSH public keys from all online machines and seed them into a Bitwarden collection.
Usage:
./scripts/bw-seed-ssh-keys.sh
What it does:
- Connects to each machine in the online fleet
- Retrieves their
~/.ssh/id_ed25519.pub - Creates Bitwarden login items named
james@<hostname>with the public key in the password field - Updates existing items in place (safe to re-run)
Prerequisites:
- SSH access to all machines (keys already in
~/.ssh/authorized_keys) - Bitwarden CLI (
bw) authenticated - Tailscale or direct network access to all machines
Run once: Before onboarding a new machine, so their SSH keys are available in Bitwarden during setup.
bw-seed-kube.sh
Seed kubeconfig and talosconfig from this machine into Bitwarden.
Usage:
./scripts/bw-seed-kube.sh
What it does:
- Exports your local
~/.kube/configto Bitwarden as a secure note - Exports your local
~/.talos/configto Bitwarden - Updates existing items in place
Prerequisites:
- kubectl configured with cluster access
- talosctl configured with cluster access
- Bitwarden CLI (
bw) authenticated
Use when: You've set up cluster access locally and want to share it to other machines.
Apply and Status
record-apply.py
Record the outcome of a just apply run for tracking and status checks.
Usage:
python3 scripts/record-apply.py <exit-code> [label] [skip-tags]
Arguments:
<exit-code>— exit code from the Ansible apply (0 = success, non-zero = failure)[label]— optional descriptive label (default: inferred from context)[skip-tags]— optional comma-separated skip tags (for partial applies)
Output: Writes to ~/.cache/dotfiles/last-apply.json with:
- Timestamp
- Exit code (0 = success, 1 = failure)
- Label and skip tags (if provided)
Used by: just doctor to check health status and flag old or failed applies.
last-apply-status.py
Pretty-print the last recorded apply status. Used by just doctor.
Usage:
python3 scripts/last-apply-status.py
Exit codes:
0— apply succeeded and timestamp is recent (< 48 hours)1— apply failed OR timestamp is stale (≥ 48 hours)
Output: Human-readable status with timestamp, result, and any tags.
online_hosts.py
Print the online fleet hosts: inventory hosts whose tailnet name is online, excluding the vps group and this machine. A host's tailnet name is its tailnet_name in inventory.yml (e.g. termux → pixel-9a), otherwise its inventory name. A peer's name is its MagicDNS short name; the OS HostName is used only when a peer has no DNSName, because HostNames aren't unique (several phones report localhost).
Usage:
python3 scripts/online_hosts.py # inventory names, space-separated
python3 scripts/online_hosts.py --peers # raw online tailnet names, one per line
Used by: just _online_hosts (bulk applies, doctor-fleet, push-terminfo), and roles/ssh_mesh, which runs --peers on the target via the script module.
Silently fails: If Tailscale is not running, returns empty output (so scripts gracefully degrade).
Cluster and Infrastructure
kubevirt-node
Operations on KubeVirt nodes for the test cluster.
Usage:
./scripts/kubevirt-node [command] [node-name]
Supported commands:
list— List all KubeVirt test nodesstart <node>— Power on a nodestop <node>— Power off a nodestatus <node>— Show node status
Prerequisites:
kubectlconfigured with cluster access- Permission to manage VirtualMachine resources
sync-brews.sh
Sync the Homebrew package state: fetch the current installed packages from Homebrew and update group_vars/all.yml with the new list.
Usage:
./scripts/sync-brews.sh
What it does:
- Runs
brew list --formulaeandbrew list --casks - Extracts the current package list
- Updates the
homebrew_packagesandhomebrew_caskssections ingroup_vars/all.yml - Prints a diff so you can review changes
Use when: You've manually installed packages with brew install and want to record them in the inventory for automated management.
ess-render-values.py
Historical helper for rendering ESS Helm values for the terminated AWS Talos cluster from Bitwarden. It does not configure the standalone OCI deployment.
Usage:
python3 scripts/ess-render-values.py < ess-helm-values-template.yml > values.yml
What it does:
- Reads the
ess-helm-valuessecure note from Bitwarden - Applies environment and variable substitutions (e.g.,
${DOMAIN}) - Outputs the rendered Helm values to stdout
Previously used by: AWS cluster deployment pipelines to generate Helm configurations.
Prerequisites:
bwCLI authenticated with Bitwarden vault access- The
ess-helm-valuessecure note exists in Bitwarden
Prerequisites and Setup
All scripts assume:
- Python 3 (for
.pyscripts) - Bash (for
.shscripts) - Bitwarden CLI (
bw) installed and configured when BW scripts are used - SSH access to target machines (for remote operations)
- Tailscale installed for machine discovery (for inventory scripts)
Set up Bitwarden CLI once:
brew install bitwarden-cli # macOS
# or: apt-get install bw # Linux
bw login <email>
Session tokens are cached locally in /tmp/ for convenience; they expire and are renewed automatically.
Fleet status
Snapshot of 2026-11-18, made from tailscale status and the live hosts.
Update this page when a host changes role. To refresh it, run
tailscale status --json | jq -r '.Peer[] | [.HostName,.Online,.LastSeen] | @tsv'.
How it fits together
Internet ─▶ Cloudflare ─▶ spindle (Oracle, Frankfurt): Caddy, Matrix, LiveKit, OMP auth broker, hive failover coordinator
│ tailnet
┌───────────────────┴────────────────────┐
▼ ▼
bihar (home, WiFi) karnataka (home)
k3s + KubeStellar hub k3s + KubeVirt + ARC runners + LMS
hives: ACTIVE ───── hive-sync 15 min ───▶ hives: cold standby
└─ VM: punjab (agent box, Ansible host)
Live hosts
| Host | Kind | OS | Role | Docs |
|---|---|---|---|---|
| bihar | Home server (Intel, 62 GB) | Bluefin utah:testing | Active hive host; k3s + KubeStellar hub | Bihar |
| karnataka | Home server (AMD Strix Halo, 125 GB) | Bluefin utah:testing | k3s: KubeVirt VMs, CI runners, LMS, Corral; hive standby | Karnataka |
| spindle | Oracle A1 VM, eu-frankfurt-1 (2 ARM cores, 12 GB) | Ubuntu 24.04 | Public front door (Caddy, 92.5.82.232), Matrix (reilly.asia), LiveKit, OMP auth broker, hive failover coordinator | Spindle VPS |
| punjab | KubeVirt VM on karnataka (12 vCPU, 23 GB, 500 GB) | Ubuntu 24.04 | Agent box and fleet Ansible host. Disk 96% full. | Punjab |
| corral | Tailscale sidecar pod on karnataka | n/a | Tailnet name for corral-web | Karnataka |
pixel-9a (termux) | Phone | Android + Termux | Ansible termux host | Inventory |
Cross-host services:
- The hives (school, reef, hanthor, hub) are on bihar. If bihar fails, they move to karnataka. See Hives hosting.
- KubeStellar is described in Home k3s and KubeStellar.
Intermittent (desktops and laptops)
| Host | Last seen | Notes | Docs |
|---|---|---|---|
kerala (kerala-1) | 2026-11-17 | Desktop. Earlier tailnet node kerala last seen 2026-08-29 | Kerala |
| himachal / himachal-1 | 2026-11-09 / 2026-11-10 | Desktop. Two tailnet nodes; remove the earlier one | Himachal |
| dilli | 2026-10-21 | Desktop | Dilli |
| kanpur | 2026-07-21 | Desktop or Windows test machine | Kanpur |
| mumbai | 2026-07-20 | Debian VM on the phone | Inventory |
| goa | 2026-07-08 | Server | Goa |
| MacBook Air (2 nodes), CMF Phone 1 | 2026-10 | Personal devices, not managed | none |
Retired
| Host | Retired | Notes | Docs |
|---|---|---|---|
| Home Talos cluster (bihar and karnataka on Talos) | 2026-10/11 | Replaced by Bluefin + k3s on the same hardware | Archive |
| AWS Talos cluster (eu-north-1) | 2026-10-25 | Terminated. Configurations kept as recovery material | Archive |
punjab EC2 (punjab-aws-retired, 100.78.73.8) | 2026-10-28 | Replaced by the punjab VM on karnataka | Punjab |
matrix (reilly-asia-matrix), telengana | 2026-09 | Hetzner VPSes | Archive |
Hives on spindle (/srv/hives) | 2026-11-18 | Moved to bihar. The copy is stopped and kept for rollback | Hives hosting |
Tailnet nodes to remove
The tagged devices from the retired clusters are offline and can be deleted from the Tailscale admin console. On 2026-11-18 these were:
- since 2026-07:
appflowy*,argocd,auth,forgejo,grafana,loki,prometheus,n8n,lemonade,home,kubevirt-manager*,default-deepseek-14b,repro-ctr-vm-vm,gate-*-vm - since 2026-10-25:
hive,hive-auth-egress-0/1,review,search,spindle-grafana,aws-migration-operator,tuna-v1-preflight-* - since 2026-09:
tailscale-operator*,secret-broker - since 2026-04:
uisp,chuppvm - since 2026-07 and 2026-08:
vm,WIN-G9NAVN79GN0
hanthor/dotfiles#333 (scripts/tailnet.py) automates this cleanup.
Network
The home LAN is a single flat private /24 behind a consumer router (DNS,
DHCP). Most fleet traffic goes over Tailscale instead. Per-host addresses are
not published here — run just inventory for the live LAN + Tailscale
crosswalk.
Devices
| Name | Role |
|---|---|
| gateway | Router (DNS, DHCP) — see Gateway |
| karnataka | Home server: k3s, KubeVirt, AMD GPU — reserved address |
| goa | Fleet control, Debian ARM (Raspberry Pi) — reserved address |
| phone | Android phone (reserved address) |
| printer | Network printer (DHCP) |
| macbook | MacBook Air (DHCP) |
| kvm | NanoKVM (reserved address) — see KVM |
| esp32 | Unidentified IoT device (static address) |
| dilli | Laptop (WiFi, when online) |
Fleet nodes (Tailscale MagicDNS)
Hosts are reached by MagicDNS name; ansible_host in inventory.yml holds the
tailnet address where one is needed.
| Hostname | LAN | Group |
|---|---|---|
| goa | reserved | server (applies locally) |
| punjab | — (VM on karnataka) | server |
| himachal | DHCP | desktop |
| dilli | DHCP | desktop |
| kanpur | DHCP | desktop |
| kerala | cellular | desktop |
mumbai (ssh :2222) | phone | desktop (phone VM) |
termux (ssh :8022) | phone | termux_hosts |
| matrix | remote | vps — retired |
| telengana | remote | vps — retired |
| bihar | — (separate site, WiFi) | desktop — home server, hive host |
| karnataka | reserved | not in inventory (native units) — k3s host |
Gateway
The home router: a consumer WiFi 6 router doing DNS and DHCP for the flat LAN.
Network
- One private /24, DHCP pool covering most of the range, short lease time
- Fixed addresses for infrastructure (bihar, karnataka, goa, kvm) come from the router's address-reservation table, not from static config on the hosts
- ISP fiber, ONT in bridge mode, router holds the WAN address directly
- Nothing is port-forwarded; remote access to the fleet is via Tailscale
Third-party firmware
The router's chipset (MediaTek Filogic 820) has only snapshot-level OpenWrt support so far, so it runs stock firmware. Revisit once a stable OpenWrt release supports it.
IPv6
The ISP delegates a single /64 via SLAAC only. The stock firmware has no LAN DHCPv6 server and no configurable IPv6 firewall. Hosts use privacy extensions, so their addresses rotate — impractical for hosting or stable SSH. Options if stable v6 addresses are ever wanted:
- Stable SLAAC token per machine:
# /etc/systemd/network/eth0.network [IPv6AcceptRA] Token=::1 # becomes <prefix>::1 - Disable privacy extensions (EUI-64, based on MAC):
net.ipv6.conf.eth0.use_tempaddr = 0 - DHCPv6 — not available on the stock firmware.
Tailscale remains the default path for fleet SSH.
NanoKVM
IP KVM for the fleet: a Sipeed NanoKVM running OpenWrt.
Notes
- Plugged into Karnataka; provides remote keyboard/video/mouse for it
- Web UI on its reserved LAN address (find it with
just inventory)
Dilli
Secondary workstation in the hanthor fleet.
Hardware
- Arch: x86_64
- WiFi on the home LAN (DHCP)
- Tailscale:
dilli(MagicDNS)
OS
Bluefin (Fedora Atomic)
Services
- BuildStream Dashboard — build monitor
Notes
- PaperWM, Papershell, Copyous extensions synced from fleet
- Brew packages managed via dotfiles playbook
Himachal
Laptop workstation in the hanthor fleet.
Hardware and OS
- Architecture: x86_64
- OS: Bluefin
- The reachable Tailscale SSH name used for the 2026-10-27 inspection was
james@himachal-1(hostnamehimachal). The inventory entry forhimachalstill specifies100.73.3.51; it should not be assumed to identify the reachable node without checking.
Services
- Dotfiles updates:
host_vars/himachal.ymlsetsis_laptop: trueanddotfiles_auto_update: false. On 2026-10-27,dotfiles-update.servicewas not found onhimachal-1. The configured 90-second delay applies only if login-time automatic updates are enabled; it is not evidence that updates currently run. The host-vars comment suggestsjust apply-nosecretsfor a manual update. - Hive operations:
hive_ops_enabled: falsein host vars. Read-only inspection on 2026-10-27 found the seven former Hive and pi peak timersnot-foundand inactive onhimachal-1. They previously targeted AWS Kubernetes, which was terminated on 2026-10-25; the host-vars comment saying those jobs now run in-cluster is stale. See hive_ops and the current Hive deployment.
The fleet's GNOME role includes PaperWM, PaperShell and Copyous; their enabled state on this host was not checked.
Kanpur
Laptop workstation in the hanthor fleet.
Hardware
- Arch: x86_64
- Laptop (runs on login, not periodic timer)
- Tailscale:
kanpur(MagicDNS)
OS
Bluefin (Fedora Atomic)
Notes
dotfiles_update_delay_seconds: 90— applies dotfiles 90s after login
Kerala
PostMarketOS mobile device in the hanthor fleet.
Hardware
- Arch: aarch64 (ARM)
- Mobile device (laptop power profile)
- Tailscale:
kerala(MagicDNS)
OS
PostMarketOS (Alpine/musl-based)
Notes
- Uses
apkfor package management (no Homebrew) - Packages installed via
apk_packagesrole - Musl libc — Homebrew incompatible
Bihar
Home server, the primary host for the hives, and the KubeStellar hub. Since 2026-11-17 it runs Bluefin, not Talos. The Talos page is archived.
| Hardware | ASRock B660M-STX, Intel, 24 threads, 62 GB of RAM |
| Disk | 1.9 TB NVMe (/var) |
| OS | Bluefin ghcr.io/projectbluefin/utah:testing (bootc: read-only /usr, writable /etc and /var) |
| Network | WiFi (wlp3s0). Not on the same LAN as karnataka. |
| Tailnet | bihar / 100.103.44.48 (bihar.manatee-basking.ts.net) |
| Access | ssh james@bihar (passwordless sudo through /etc/sudoers.d/zz-james) |
| Ansible | Yes: host_vars/bihar.yml, skip_pipewire_audio: true |
Roles
- Active hive host. The hive containers (hub, hub-front, school and reef) run under rootful podman compose, and spindle's Caddy sends public traffic to them. See Hives hosting and failover.
- k3s and KubeStellar hub. A single-node k3s runs KubeStellar 0.30.2, with
ITS
its1and WDSwds1. bihar and karnataka are both workload clusters. See Home k3s and KubeStellar.
Host settings (applied 2026-11-17)
- No sleep: the sleep, suspend and hibernate targets are masked, the
90-nosleep.confdrop-ins are in/etc/systemd/{sleep,logind}.conf.d/, and the GDM dconfsleep-inactive-ac-type='nothing'is set. Before this change, GNOME suspended the host after 15 min idle, and the host went offline. - Default target is
multi-user.target: no graphical login. Usesystemctl isolate graphical.targetto start one. - Wake-on-LAN is set to
magiconenp0s31f6. The host uses WiFi, so wake-on-LAN works only if you connect a cable. - To do: in the firmware, set "Restore on AC/Power Loss" to Power On. This setting is necessary because the hives depend on this host.
- Dotfiles bootstrap fixes for a new bootc host are in hanthor/dotfiles#385.
- Bitwarden phase 2 is not done yet. Run
just apply -e skip_pipewire_audio=truefrom an interactive terminal.
Timers
| Unit | Purpose |
|---|---|
hive-reconcile.timer | Follows the hive lease on spindle (every 1 min) |
hive-sync.timer | Pushes the hive data to karnataka while bihar is active (every 15 min) |
fleet-osquery-report.timer | Fleet facts |
tailscale-cert-renew.timer | Tailscale TLS certificate |
Gotchas
- Do not use
pkill -f <pattern>over SSH if the pattern also occurs in the SSH command line. The command stops your own session. gsettingsover SSH needsDBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus.semanagecannot write on bootc. Usechconfor SELinux labels.
Karnataka
The main home workload server: k3s with KubeVirt (punjab and the test VMs run here), GitHub Actions runners, Frappe LMS, Corral, and the cold standby for the hives. Since late 2026-10 it runs Bluefin, not Talos. The Talos page is archived.
| Hardware | AMD Strix Halo APU (32 threads, integrated GPU), 125 GB of RAM |
| Disk | 912 GB NVMe (/var), 88% used on 2026-11-18 |
| OS | Bluefin ghcr.io/projectbluefin/utah:testing |
| Tailnet | karnataka / 100.66.158.127 |
| Access | ssh james@karnataka (passwordless sudo). kubectl from punjab uses ~/.kube/config |
| Ansible | Yes: host_vars/karnataka.yml |
k3s
k3s v1.36.5+k3s1 runs as a binary in /var/usrlocal/bin. It binds to the tailnet IP
only, uses flannel on tailscale0, and has traefik and servicelb disabled. The kit is in
home-k3s/karnataka/:
install-host.shinstalls k3s.apply-stack.shinstalls KubeVirt v1.9.1, CDI v1.66.1, corral-web, the AMD GPU device plugin and Lemonade.
See Home k3s and KubeStellar for the shared design and the firewalld rules.
Workloads (2026-11-18)
| Namespace | What | Exposed as |
|---|---|---|
corral-vms | KubeVirt VMs: punjab (agent box, 500 Gi root), messages and the macos-beeper* disks (macOS for Beeper/iMessage), tuna-q test VMs, mandelbrot-visual (stopped) | punjab is on the tailnet |
corral | corral-web (VM manager UI) and its Tailscale sidecar | https://karnataka.manatee-basking.ts.net/, tailnet node corral |
front-door | Caddy link page and redirects (/course, /learn, /frappe, /books, /corral) | http://karnataka/ |
lms | Frappe Learning v16.1.0 (frappe, db, redis), local image from frappe-lms-image/ | https://karnataka.manatee-basking.ts.net:8443 |
beale | beale-course (course site) | beale.karnataka.ts.net |
beeper-bridges | rustpush (iMessage bridge) | none |
arc-systems, tuna-ci, tuna-ci-storage | GitHub Actions runner scale sets tuna-ci-24.04 (1-6), tuna-ci-26.04 (0-2), tuna-iso-gpu (0-1), with scratch storage | none |
lemonade | Lemonade AMD LLM server, scaled to 0 | :13305 (tailscale serve) |
open-cluster-management-agent* | KubeStellar WEC agent for the hub on bihar | none |
tailscale serve (tailnet only):
:80goes to front-door.:443goes to corral-web.:8443goes to LMS.:8444goes to127.0.0.1:8080(RaveBSD books).:13305goes to Lemonade.
Hive standby
/var/srv/hives is a copy of the hive data. When bihar is active, bihar pushes it
here every 15 min. The containers stay stopped until the lease on spindle names
karnataka. See Hives hosting and failover.
Notes
- Disk:
/varis 88% full. The ARC runner PVCs and the macOS VM disks use most of the space. Check this before you add VMs. - punjab is a VM on this host. If karnataka restarts, punjab also restarts.
Home k3s and KubeStellar
bihar and karnataka each run a separate single-node k3s on Bluefin. They replace the two-node Talos cluster (archive). KubeStellar on bihar connects the two clusters. The hosts are not on the same LAN, so all cluster traffic uses the tailnet.
The kits are in home-k3s/:
bihar/ and karnataka/ each contain install-host.sh, config.yaml,
k3s.service, wait-tailnet.sh and resolv.conf.
Shared design
- The k3s binary is in
/var/usrlocal/bin/k3s./usris read-only on bootc.k3s.servicewaits for the tailnet (k3s-wait-tailnet). - The API server, kubelet and node IP bind to the tailnet IP only.
Flannel uses
tailscale0. - traefik and servicelb are disabled. Services are exposed with
tailscale serve(karnataka) or ingress-nginx (bihar, KubeStellar only). - Pod DNS uses
resolv.conf(1.1.1.1 and 1.0.0.1). - SELinux on the host stays enforcing. The k3s SELinux module cannot be installed on bootc.
firewalld (required: without these rules metrics-server fails and pods cannot resolve DNS)
# cni0 goes in the internal zone; pods can reach the kubelet
firewall-cmd --permanent --zone=internal --change-interface=cni0
firewall-cmd --permanent --zone=internal --add-rich-rule='rule family=ipv4 source address=10.42.0.0/16 destination address=<tailnet-ip>/32 port port=10250 protocol=tcp accept'
# pods can reach the internet
firewall-cmd --permanent --new-policy k3s-pod-egress
firewall-cmd --permanent --policy k3s-pod-egress --set-target CONTINUE
firewall-cmd --permanent --policy k3s-pod-egress --add-ingress-zone internal
firewall-cmd --permanent --policy k3s-pod-egress --add-egress-zone public
firewall-cmd --permanent --policy k3s-pod-egress --add-rich-rule='rule family=ipv4 source address=10.42.0.0/16 accept'
# API (and KubeStellar on bihar) from the tailnet only
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=100.64.0.0/10 destination address=<tailnet-ip>/32 port port=6443 protocol=tcp accept'
bihar's install-host.sh applies these rules. On karnataka the rules were applied
by hand.
Differences
| bihar | karnataka | |
|---|---|---|
| service CIDR | 10.96.0.0/12 (KubeFlex's vcluster hard-codes the kind range) | k3s default 10.43.0.0/16 |
| Extras | ingress-nginx (SSL passthrough), KubeFlex, KubeStellar | KubeVirt, CDI, ARC, workloads (see Karnataka) |
| Reset | reset-k3s.sh | none |
KubeStellar (hub on bihar)
KubeStellar 0.30.2 core chart (oci://ghcr.io/kubestellar/kubestellar/core-chart):
- ITS
its1: vcluster with the OCM cluster-manager. - WDS
wds1: where you apply workloads. - Both are served at
https://<name>.100.103.44.48.sslip.io:9443. Port 9443 is open to the tailnet only. - kubectl contexts on bihar:
bihar,its1,wds1(in~/.kube/config). - The join command is in
bihar:~/.kube/its1-join.cmd. It is a secret.
Workload execution clusters (joined with clusteradm 1.0.0 --singleton):
| Cluster | Labels |
|---|---|
| bihar | location-group=home name=bihar cpu=intel role=hub |
| karnataka | location-group=home name=karnataka cpu=amd gpu=amd kubevirt=true |
Use it like this: apply manifests with kubectl --context wds1, and add a
BindingPolicy with a clusterSelectors label match. Set
wantSingletonReportedState: true to get status back from one cluster. Notes are
in home-k3s/bihar/KUBESTELLAR.md.
Problems found during installation (2026-11-17)
- If metrics-server is not ready, the KubeFlex hooks fail with "stale GroupVersion
discovery". Correct the firewall, then restart
kubeflex-controller-manager. - A failed
ks-corerelease stopped on the uninstall-postgresql hook. To recover, runhelm uninstall ks-core --no-hooks, delete thekubeflex-system,its1-systemandwds1-systemnamespaces, then install again.
Hives hosting and failover
TunaOS infrastructure. This page describes where and how the hives run. For the hives themselves (agents, configuration, health), see the Hive operations guide.
Last updated: 2026-11-19 (hanthor added). The source for the scripts, compose layers and systemd
units is hives/ in this
repository. Each host also has a copy of this runbook at
/var/srv/hives-host/RUNBOOK.md (on spindle: /var/lib/hive-failover/RUNBOOK.md).
Overview
The hives run as one compose project named hives, with five containers:
| Service | Purpose | Public URL |
|---|---|---|
hub | Hub API that all spokes report to | https://hub.tunaos.org/api/*, /login/* |
hub-front | Static hub page (nginx) | https://hub.tunaos.org/ |
school | Spoke for the tuna-os org (GitHub App 3942065) | https://school.tunaos.org |
reef | Spoke (GitHub App 5546574) | https://reef.tunaos.org |
hanthor | Spoke for the hanthor/* personal repositories (App 3942065, installation 158987150). Added 2026-11-19 | https://hive.reilly.asia |
Until 2026-11-18 the hives ran on spindle (Oracle ARM VM, Frankfurt, 2 cores and 11 GB of RAM). They now run on bihar (x86, 24 cores, 62 GB of RAM). karnataka is a cold standby. spindle keeps the public IP and Caddy, and it coordinates failover.
Internet ─▶ Cloudflare ─▶ spindle Caddy (92.5.82.232)
│ tailnet, ports 3100-3104
┌──────────────┴──────────────┐
▼ (active, holds the lease) ▼ (standby, stopped)
bihar 100.103.44.48 karnataka 100.66.158.127
└── hive-sync every 15 min ──▶
Outbound traffic from the hives (to GitHub and the model APIs) leaves through the active host's own internet connection. It does not go through spindle.
The OMP auth broker still runs on spindle at http://100.113.75.88:8765, reached over the tailnet.
Host layout (bihar and karnataka are identical)
| Path | Contents |
|---|---|
/var/srv/hives/ | Hive data and configuration, copied from spindle's /srv/hives: compose.yaml, compose.override.yaml, .env, {hub,school,reef,hanthor}/{config,data,secrets}, hub-front/, proxy/, bin/dashboard-*.html |
/var/srv/hives-host/compose.<host>.yaml | Host layer: ports, memory limits, broker URL, image digests, and the amd64 binary mount |
/var/srv/hives-host/bin/hive-2ef7dc7c0-amd64 | tuna-os/hive at 2ef7dc7c0 (branch deploy/upstream-v6-20261113) built with CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath ./cmd/hive |
/var/srv/hives-host/hives | Wrapper that runs docker-compose against rootful podman with the three compose files |
/var/srv/hives-host/hive-reconcile | Makes the host follow the lease (timer, every 1 min) |
/var/srv/hives-host/hive-sync | Pushes the data to the peer host (timer, every 15 min) |
/var/srv/hives-host/keys/ | lease (read-only access to the lease on spindle) and sync (rsync to the peer host) |
/var/srv/hives-host/maintenance | If this file exists, reconcile does not start the hives |
Each host publishes its ports on its own tailnet IP only:
| Port | Service |
|---|---|
| 3100 | hub-front |
| 3101 | hub |
| 3102 | school |
| 3103 | reef |
| 3104 | hanthor |
A firewalld rich rule in the public zone accepts these ports only from spindle (100.113.75.88). Inside the compose network the services reach each other by name, for example http://hub:3001. The podman network is named spindle_default so that compose.yaml works unchanged.
Images are pinned by digest, and the pin must match the binary:
ghcr.io/hivecommons/hive@sha256:d56425fb…(school, reef and hanthor)ghcr.io/hivecommons/hive-hub@sha256:3c0162eb…(hub)
Do not change to :v6-latest. On 2026-11-18 a newer image made the hub log "failed to write token cache". Upgrade the binary and the images together.
Daily operations (on the active host)
sudo /var/srv/hives-host/hives ps
sudo /var/srv/hives-host/hives logs -f --tail 50 school
sudo /var/srv/hives-host/hives restart reef
sudo podman stats --no-stream
To find the active host, run this on spindle:
sudo hive-failover status
The command shows the lease, the health and container count of each host, and the targets in Caddy.
Change the hive configuration
- Edit the files under
/var/srv/hives/on the active host only. - Run
sudo /var/srv/hives-host/hives up -d.
The next hive-sync copies the change to the standby.
Change a host-specific setting
- Edit
/var/srv/hives-host/compose.<host>.yamlon both hosts. This covers ports, memory and image pins. - Run
sudo chcon -t container_file_ton the file.
Upgrade the hive binary
- Build the amd64 binary from the fork commit.
- Copy it to
/var/srv/hives-host/bin/on both hosts. - Change the binary mount and the image digests in both host files.
- Run
hives up -don the active host.
Failover
The lease
The file spindle:/var/lib/hive-failover/active contains bihar, karnataka or none. Only the host named in the lease may run the hives. This rule prevents two hosts from running the same GitHub Apps at once.
| Component | Where | Schedule | What it does |
|---|---|---|---|
hive-failover check | spindle, hive-failover.timer | every 1 min | Checks the active host on ports 3101-3103. After 5 min of failures it sets the lease to none, stops the old host if it can, and waits. When the old host has 0 containers, or after 10 min, it starts the standby, sets the lease, changes Caddy and sends a ntfy notification on $NTFY_TOPIC. |
hive-reconcile | both hosts, hive-reconcile.timer | every 1 min | If the lease names this host and fewer than 4 containers run, it starts them (unless maintenance exists). If the lease names a different host, it stops them. If it cannot read the lease for 8 min, it stops them (self-fence). |
hive-sync | both hosts, hive-sync.timer | every 15 min | If the lease names this host, it uses rsync to push /var/srv/hives to the peer, without caches and toolchains. |
hive-cleanup | both hosts, hive-cleanup.timer | every 1 h | On running spokes only: removes idle Claude session scratch and stale work dirs in /tmp inside the containers, removes dangling images and old backups, and warns when /var has less than 10 GB free. Ported from spindle, where the writable layers of the hive containers had reached 16.8 GB. |
Logs:
journalctl -t hive-failover # on spindle
journalctl -u hive-reconcile -u hive-sync # on the hosts
The podman containers have restart policy unless-stopped and podman-restart.service is disabled. As a result, a host that boots does not start the hives itself. Only reconcile starts them, and only when the lease names that host.
Commands (on spindle)
sudo hive-failover status
sudo hive-failover move bihar # planned move or failback: stop, sync, start, Caddy
sudo hive-failover move karnataka
sudo hive-failover hold on # pause automatic failover (for example during maintenance)
sudo hive-failover hold off
move sets hold on while it runs, and clears it when it succeeds. If move fails, the lease stays at none and hold stays on. Correct the problem and run move again.
After an automatic failover
The hives run on karnataka. Failback is manual, so that an unstable WiFi connection cannot move the hives back and forth.
- Make sure bihar is up:
ssh james@bihar uptime. - Run
sudo hive-failover move biharon spindle. It takes about 1 min.
Planned maintenance on bihar
Use one of these methods:
- Run
sudo hive-failover move karnatakafirst. - Or run
sudo hive-failover hold onon spindle, thensudo touch /var/srv/hives-host/maintenanceandhives stopon bihar. When you finish, remove the file and runhold off.
Limits
- The data on the standby can be up to 15 min old. Rsync copies files while they are in use, so a file that changes during the copy can arrive in an inconsistent state.
- The self-fence after 8 min without spindle was tested by hand only. A real network partition has not been tested.
- If spindle is down, there is no public access and no failover. The hosts stop the hives after 8 min, because they cannot read the lease.
Drill on 2026-11-18
- Hives stopped on bihar at 13:55:44Z.
- Spindle revoked the lease at 14:01:29Z.
- Karnataka was healthy and Caddy changed at 14:01:56Z.
move biharcompleted in 50 s, including the sync back.
Gotchas
- SELinux (bihar and karnataka are bootc and enforcing):
- Files mounted into containers must have the type
container_file_t.semanagecannot write on bootc, sohive-syncrunschcon -Rafter each copy. Do not copy SELinux attributes with rsync (-X). - systemd cannot run scripts that have the label
var_t(the result is203/EXEC Permission denied). For this reason the units start the scripts as/usr/bin/bash /var/srv/hives-host/<script>. Do not usechconon the scripts. - Under systemd, rsync changes to the confined
rsync_tdomain, and its ssh cannot read the key.hive-sync.servicetherefore runs asSELinuxContext=system_u:system_r:unconfined_t:s0.setsebool -Phas no effect on bootc. - Until 2026-11-18 18:50Z, the reconcile and sync timers did not run because of these two problems. The failover drill passed only because spindle sent the commands directly over SSH. After you change a unit, use
systemctl show <unit> -p Resultto make sure that it ran.
- Files mounted into containers must have the type
- The Caddyfile on spindle is a single-file bind mount:
sed -imakes a new inode, and Caddy continues to see the old file. Write in place (cat new > Caddyfile), or rundocker cp Caddyfile spindle-caddy-1:/tmp/Caddyfilethendocker exec spindle-caddy-1 caddy reload --adapter caddyfile --config /tmp/Caddyfile.hive-failoverdoes both. - The spindle-to-host path uses a relay: it goes through Tailscale DERP (Frankfurt), with approximately 170 ms of latency and 1 MB/s. This is sufficient for dashboard and API traffic.
- Disk: the caches (Go build, rustup and others) are not synced. After a move, the first builds on the new host are slower.
- Use separate SSH keys:
spindle:/root/.ssh/hive_failovergives access tojames@on both hosts.- The
leasekey of each host givesubuntu@spindlewith a forced command:cat /var/lib/hive-failover/active. - The
synckey of each host givesjames@on the peer host.
Install or rebuild a host
The files in hives/ go to these locations:
| Repository file | Destination |
|---|---|
hives, hive-reconcile, hive-sync | /var/srv/hives-host/ on bihar and karnataka (mode 0755) |
compose.<host>.yaml | /var/srv/hives-host/ on that host |
hive-cleanup | /var/srv/hives-host/ on bihar and karnataka (mode 0755) |
systemd/hive-{reconcile,sync,cleanup}.* | /etc/systemd/system/ on bihar and karnataka |
hive-failover | /usr/local/sbin/ on spindle |
systemd/hive-failover.* | /etc/systemd/system/ on spindle |
On a new host, also do these steps:
- Install
docker-composewith brew. - Run
systemctl enable --now podman.socket. - Create the podman network
spindle_default. - Pull the pinned images.
- Copy
bin/hive-2ef7dc7c0-amd64from the other host. If neither host has it, build it fromtuna-os/hiveat2ef7dc7c0. - Create the SSH keys in
/var/srv/hives-host/keys/, and add the firewalld rule for ports 3100-3104 from spindle. Also install/etc/sysctl.d/90-hives-inotify.conf(fs.inotify.max_user_instances = 1024,fs.inotify.max_user_watches = 1048576): with five containers the per-agent permission watchers exhaust the default of 128 instances. - Label the data and compose files (not the scripts):
chcon -R -t container_file_t /var/srv/hives /var/srv/hives-host/compose.*.yaml /var/srv/hives-host/bin. - Enable
hive-reconcile.timer,hive-sync.timerandhive-cleanup.timer. Start each service one time and make sure that itsResultissuccess.
Rollback to spindle (emergency only)
On 2026-11-18 the cleanup removed the stale data copies, the arm64 binaries, and the hive containers and images on spindle. /srv/hives contains only the compose files, .env, <svc>/config, <svc>/secrets and backups/. karnataka is the fallback. Use spindle only if both hosts are lost:
- On spindle, run
sudo hive-failover hold on. - Build an arm64 binary from
tuna-os/hiveat2ef7dc7c0(GOARCH=arm64) and copy it to/srv/hives/bin/. - Copy the newest
/var/srv/hivesfrom bihar or karnataka to spindle (<svc>/data, without.cache). - Change the hive blocks in Caddy back to
school:3001,reef:3001,hub:3001andhub-front:80. - Run
docker compose up -din/srv/hives. Spindle has 2 ARM cores, so reduceHIVE_SHARED_AGENT_POOL_MAX_SESSIONSto 2.
Spindle VPS (Matrix and the public front door)
The Oracle Cloud VPS in Frankfurt runs the Matrix homeserver that was on the AWS Talos cluster,
terminated 2026-10-25. Spindle, Postgres, standalone LiveKit and lk-jwt-service
(MatrixRTC; no ESS chart/runtime), synctap and Caddy run as Docker Compose services under
/srv/spindle (state on 2026-11-18). On 2026-11-16, Spindle's built-in OIDC replaced MAS
(auth.reilly.asia now goes to Spindle). The MAS sections below are history.
The hives moved off this host on 2026-11-18. School, Reef, Hub and the hub
frontend now run on bihar, with karnataka as a cold
standby. The hanthor spoke (hive.reilly.asia) joined them on 2026-11-19. Spindle keeps three hive roles:
- Caddy front door:
school,reef,hub.tunaos.organdhive.reilly.asiago to the active host's tailnet IP on ports 3100-3104 (hive.reilly.asiaon 3104).hive-failoverrewrites these upstreams. - Failover coordinator: the lease in
/var/lib/hive-failover/active,/usr/local/sbin/hive-failover, andhive-failover.timer. - OMP auth broker: the
omp-auth-brokeruser service on100.113.75.88:8765. The hives connect to it over the tailnet. Theomp-auth-broker-docker-proxysocket (172.18.0.1:18765) was used only by the hives when they ran on this host.
See Hives hosting and failover. The stopped copy in /srv/hives is
kept for rollback only (MOVED-TO-BIHAR.txt). This removes the 2560 MB per-spoke
memory budget from this host.
| Piece | Source |
|---|---|
| Host | Oracle Cloud Always Free, VM.Standard.A1.Flex, Ubuntu 24.04 aarch64, eu-frankfurt-1 AD-1; public IPv4 92.5.82.232, Tailscale 100.113.75.88 |
| Spindle | ghcr.io/tuna-os/spindle:v0.1.6 running on 2026-11-18 (v0.1.1 deployed 2026-11-07; arm64 release images). The upgrade from the outbox-hotfix-arm64 build and its rehearsal are in upgrade-to-main.md |
| MatrixRTC | livekit-server:v1.13.5 + lk-jwt-service:0.6.0 as standalone containers |
| Data | ess_ess-spindle-data.tar.gz (the store, taken 15:33 UTC) and postgres/2026-10-25/mas.dump (15:22 UTC), from ~/backups/tunaos-aws-2026-10-25 on himachal-1 |
| Config | production runtime + delegated-auth block, MAS under/overrides and LiveKit template; deploy.sh renders credentials from cluster-state |
MatrixRTC is enabled by the standalone services above. element-admin (matrixadmin.reilly.asia)
remains unmigrated.
1. Create the instance (Oracle console)
- Shape
VM.Standard.A1.Flex.spindleis ineu-frankfurt-1, AD-1, with 2 OCPU / 12 GB; stay within that Always Free ceiling. Spindle is capped at 7 GiB, LiveKit at 2 GiB / 1 OCPU. - Image: Canonical Ubuntu 24.04 (aarch64); 100-GB boot volume.
- Assign and reserve a public IPv4. Paste
cloud-init.yaml; it installs Docker, opens direct host 80/443, and authorizes the punjab and himachal-1 SSH keys. - In the OCI security list allow TCP 80, 443, 30001 and UDP 32700–32767 from
0.0.0.0/0. Remove the default public SSH rule after tailnet access works. - Oracle may reclaim idle Always Free instances. We accept that risk and will not upgrade this
tenancy to a paid account. Cloud-init package installation failed on the existing instance and
required manual installation; check
cloud-init status --longand installed packages before deploy.
A1 capacity is often "out of host capacity"; retry another availability domain or region later.
2. Deploy (on himachal-1)
cd ~/.local/share/dotfiles/docs/src/servers/spindle-vps # a checkout of this branch
./deploy.sh <vps-tailnet-ip-or-host> # SSH destination; use the tailnet address after lockdown
./deploy.sh --render-only /tmp/r # just render the configs (contains secrets; delete after)
It renders the secrets from cluster-state.himachal.tar.gz.age, copies the kit to /srv/spindle,
streams the store (excluding store.pre723), restores mas.dump, runs mas database migrate and
starts the stack. Restore markers prevent re-importing the original snapshots on a normal rerun;
do not remove those markers or restore old data over live data. The script restarts Spindle and
Caddy, causing an interruption. Nothing secret is printed or committed.
The supplied Caddyfile includes the Hive routes. Do not deploy a Matrix-only Caddyfile over the shared live configuration: doing so removes Hive TLS sites and causes Cloudflare 525 errors.
2b. Join the tailnet and lock down
ssh ubuntu@<ip> 'sudo tailscale up --hostname=spindle --ssh=false' # prints a login URL; approve it
ssh ubuntu@spindle true # confirm SSH over the tailnet
scp lockdown.sh ubuntu@spindle: && ssh ubuntu@spindle 'sudo ./lockdown.sh'
Then delete the public SSH rule from the OCI security list. Docker publishes the listed LiveKit
ports; keep the OCI ingress list in sync with all public ports in compose.yaml.
3. DNS cutover
Cloudflare DNS-only A records for matrix.reilly.asia, auth.reilly.asia, and
call.reilly.asia point to the VPS at 92.5.82.232 with a 60-second TTL. call must remain
DNS-only: its UDP media ports do not pass through Cloudflare's HTTP proxy.
Verify the public endpoints:
curl https://matrix.reilly.asia/_matrix/client/versions
curl https://auth.reilly.asia/.well-known/openid-configuration
curl https://reilly.asia/.well-known/matrix/client
curl -i https://call.reilly.asia/sfu/get # GET should return 405
curl https://school.tunaos.org/api/health
curl https://reef.tunaos.org/api/health
curl https://hub.tunaos.org/api/fleet-stats
The apex reilly.asia is a Cloudflare Worker custom domain (reilly-asia) serving the personal
portfolio and active Matrix client .well-known document. Keep it on the Worker; pointing the apex
at this VPS or removing the Worker domain would take the homepage offline. The Worker advertises the
same https://call.reilly.asia MatrixRTC service as this deployment.
call.reilly.asia routes JWT requests to the standalone lk-jwt-service and LiveKit signaling to
the standalone SFU. Since v0.1.1 Spindle's built-in minter also serves federated participants;
moving to it is the consolidation step, not yet applied here.
This is manual DNS cutover, not automatic failover. The zone is on Cloudflare Free; health-checked Load Balancing is an add-on, and no second Matrix origin is available. A 60-second TTL only limits resolver caching; it does not detect failures or switch records automatically.
The authorizer resolves matrix.reilly.asia and call.reilly.asia through Docker's host gateway
to avoid public-IP hairpin routing; reilly.asia remains on the portfolio Worker.
Incidents
- 2026-11-16/17: slow sync, missing avatars, Element X image errors: causes, fixes, the live state left behind, and how to diagnose on this host.
Caveats
- The MAS dump is from 15:22 UTC. The Spindle store's last write was 15:25 UTC and its archive was taken at 15:33 UTC; the old cluster continued until 17:16 UTC. Writes after the store archive are only in the encrypted cluster-state backup and are not in this VPS store. A real login and joined room read passed against the VPS, but that does not prove the later writes were restored.
- The AWS cluster is terminated. The S3 and himachal-1 backups are untouched, but stopping the VPS stack is an outage, not a rollback; recovery requires provisioning and restoring from backup.
v0.1.1contains #723, #736 and #794, so theops/arm64-perf-723,ops/legacy-media-compatandops/outbox-hotfixbranches can be deleted once production has run it for a week.- LiveKit is capped at 2 GiB and 1 OCPU to leave room for Spindle and the Hives on this 2-OCPU host; suitable for light personal calls, not high-concurrency conferencing. No TURN relay is configured, so clients on networks that block UDP may fail to connect.
Upgrading the VPS Spindle from the hotfix image to main
Status: done 2026-11-07. Production runs ghcr.io/tuna-os/spindle:v0.1.1 (main 1d8dfd99,
arm64 release image). The rehearsal below was run first, against a copy of production; both
blockers it found were cleared before the upgrade (see Blockers).
Before the upgrade production ran ghcr.io/tuna-os/spindle:outbox-hotfix-arm64, built from
branch ops/outbox-hotfix. That branch is the Oct 22 main (79f00253) plus #723, the
legacy-media patch and the concurrent outbox (#794).
The production upgrade (2026-11-07)
The procedure under Production procedure
was followed as written, with the release tag v0.1.1:
| Step | Result |
|---|---|
docker pull | v0.1.1, arm64 linux |
Stop (05:22:53 UTC) and cp -a of the store | data/store.pre-v0.1.1, 3.4 GB |
migrate --list / --dry-run | store and binary at schema 1/1/3; intern-state-keys "not needed"; "already at this binary's schema" |
| Start | listening at 05:24:58 (outage about 2 minutes); room warm-up finished, 118 rooms in 21.2 s |
/_matrix/client/versions, .well-known/matrix/client, MAS discovery, call.reilly.asia/sfu/get | 200, 200, 200, 405 |
| First 8 minutes | 324 inbound /send answered 200, 156 PDUs accepted, 0 ERROR log lines |
The compose file before the upgrade is /srv/spindle/compose.yaml.bak-20261107T052253.
What the rehearsal showed
The rehearsal compared main 1e8f6973 with hotfix fe76489a. Both were x86_64 release builds,
run on punjab, pinned to two cores (taskset -c 0,1, TOKIO_WORKER_THREADS=2 as in
compose.yaml). The config was production's spindle.toml with these changes: scratch store path,
localhost binds, federation and push disabled, and no [auth.delegated] block. A scratch password
replaced MAS login for one account, set on the copy only.
The store was the live production store (2.7 GB including 2,371 media blobs). It was copied
read-only over Tailscale: sudo nice -n 19 ionice -c3 tar, then rsync, then a final rsync pass at
01:58 UTC. That final pass changed only the active journal (89.jnl). Nothing was written on the VPS.
| Check | Hotfix (baseline) | Main |
|---|---|---|
Opens the copied store; verify-media | 2,371 blobs, all present (0.5 s) | 2,371 blobs, all present (0.4 s) |
migrate --list / --dry-run | n/a | store at 1/1/3, only step intern-state-keys is "not needed"; dry-run and migrate: "already at this binary's schema" (0.5–0.6 s, writes nothing) |
| Rooms (admin API) | 118; 70,919 joined members; 105,020 state events | identical, same room-ID hash |
| State of the 15 largest rooms (up to 51,884 events) | event-ID hashes | identical hashes |
Last 100 log entries of those rooms; /messages in 5 joined rooms | event-ID hashes | identical hashes |
| Accounts (admin API) | 1,360 | 1,360, same hash |
Initial /sync for a local user (115 joined rooms, 2,103 timeline events) | 200 in 9.7–10.1 s | 200 in 9.1–11.0 s; same counts |
Incremental /sync | 0.03 s | 0.03–0.06 s |
| Cold start (page cache dropped): listening / warm-up done (118 rooms) | 1.0 s / 14.2 s | 0.4 s / 12.0 s |
| Largest room cold load (≥1M bucket) | 1.04 s cold, 5.37 s on first run | 1.06 s cold, 1.70 s on first run |
| RSS after probe | 2.9 GiB | 2.8 GiB |
| WARN/ERROR log lines | 0 | 0 |
| Write path: send message, upload media, restart, read back | n/a | message found, media 200, verify-media 2,372/2,372 |
| Rollback: hotfix binary on the store after main ran and wrote | n/a | opens; same room/state/timeline/account hashes; 0 WARN/ERROR |
Legacy GET /_matrix/media/v3/download and /thumbnail | 200 | 404 (see Blockers) |
Other measurements on the copy (main binary, offline):
spindle backuptook 15.7 s and wrote 16.7 M rows into a 4.0 GB file. The backup holds rows, not media.spindle gc --dry-runtook 64.5 s and found 364,697 unreachable state nodes (290 MB) that could be swept.
Production warm-up is slower than on punjab. The 2026-11-06 restarts logged 19.2 s and 59.9 s for the same 118 rooms on the hotfix image. Budget about a minute of degraded service after the container starts.
No data migration runs. The store's schema (key 1, record 1, content digest 3, fjall 3) is the
same in both builds, so migrate is a no-op. The rollback is a plain image swap, and the rehearsal
exercised it.
Behaviour that changes with main and is not a data question:
- Per-user rate limits now default to Synapse's numbers: messages 0.2/s with burst 10, local joins 0.1/s with burst 10, room creation 0.016/s with burst 10, media uploads 10/s with burst 50. Production's config sets none of these. There are no appservices, so no bridge is affected.
- The MAS introspection verdict is cached for 120 s
(
[auth.delegated] introspection_cache_ttl_secs). A revoked MAS session can work for up to two minutes. - Production's
spindle.tomlparses unchanged under main. The rehearsal checked this by starting main with that file, delegated block included.
Blockers
Both were cleared before the upgrade:
- Legacy media. Production depends on the frozen unauthenticated media endpoints
(
/_matrix/media/v3/download|thumbnail), which pre-MSC3916 clients use for avatars and thumbnails. The rehearsal foundmainreturned 404 there.tuna-os/spindle#736merged beforev0.1.1; it serves local media and remote media already cached, and never fetches uncached remote media without a token. - No arm64 image of main.
release.ymlbuilds amd64 and arm64 images for everyv*tag;v0.1.1was cut frommain1d8dfd99after #736, #800, #804 and #805 merged.
Re-running the rehearsal (punjab)
The rehearsal needs about 3 GB for the copy, 3 GB for a working copy and 8 GB more if a backup is taken. Everything stays in one mode-700 directory, which is deleted at the end because it holds user data.
S=~/rehearsal-scratch; mkdir -p "$S/prod-copy/store" && chmod 700 "$S"
# Read-only copy of the live store. Run it twice; the second pass should only touch the journal.
rsync -a --delete --partial -z --rsync-path='sudo -n nice -n 19 ionice -c3 rsync' \
ubuntu@100.113.75.88:/srv/spindle/data/store/ "$S/prod-copy/store/"
cp -a "$S/prod-copy/store" "$S/store"
# Binaries: cargo build --release -p spindle-server --bin spindle --locked at each commit.
OLD=spindle-hotfix; NEW=spindle-<tag>
CFG=$S/cfg/prodlike.toml # production spindle.toml: storage path -> $S/store, binds on
# 127.0.0.1, federation/push disabled, [auth.delegated] removed
$OLD verify-media "$CFG"
$NEW promote-admin "$CFG" james
$NEW set-password-hash "$CFG" james < pw.hash # argon2id of a throwaway password
$NEW migrate --list "$CFG"; $NEW migrate "$CFG" --dry-run
$NEW migrate "$CFG"; $NEW verify-media "$CFG"
TOKIO_WORKER_THREADS=2 taskset -c 0,1 $NEW "$CFG"
# Probe the server: admin API /_spindle/admin/v1/rooms, rooms/{id}/state and
# rooms/{id}/timeline, users; client /joined_rooms, /messages and /sync;
# metrics spindle_room_cold_* and spindle_room_warmup_*. Compare the results
# with a run of $OLD on a fresh copy.
rm -rf "$S"
The rehearsal compared runs by hashing event and room IDs, so the comparison files never held message content. An incoming message on production during the copy window would show up as a difference between the copy and production. It would not show up as a difference between the two binaries.
The run on 2026-11-07 took about 40 minutes. Most of that was the copy: 2.7 GB over a DERP-relayed Tailscale path at about 0.7 MB/s. All the checks together took under 5 minutes.
Production procedure (once the blockers are cleared)
Run on the VPS in /srv/spindle. <tag> is the release tag, for example v0.1.1. Expected
outage: the time to stop and copy the store (seconds; 2.7 GB on local disk), plus about a minute
of warm-up.
cd /srv/spindle
# 1. Pull ahead of time; no outage.
docker pull ghcr.io/tuna-os/spindle:<tag>
cp compose.yaml compose.yaml.bak-$(date -u +%Y%m%dT%H%M%S)
sed -i 's#ghcr.io/tuna-os/spindle:outbox-hotfix-arm64#ghcr.io/tuna-os/spindle:<tag>#' compose.yaml
# 2. Stop Spindle and keep a byte-for-byte copy of the stopped store (includes media).
docker compose stop spindle
sudo cp -a data/store data/store.pre-<tag>
# 3. Migration plan; must say "already at this binary's schema" (rehearsed: no steps).
docker compose run --rm --no-deps spindle migrate --list /etc/spindle/spindle.toml
docker compose run --rm --no-deps spindle migrate /etc/spindle/spindle.toml --dry-run
docker compose run --rm --no-deps spindle verify-media /etc/spindle/spindle.toml
# 4. Start and watch warm-up.
docker compose up -d spindle
docker compose logs -f spindle | grep -m1 'room warm-up finished'
If the dry run lists any step other than "not needed", stop. Swap the image back and investigate: the rehearsal did not cover that step.
Checks after the start:
curl -s https://matrix.reilly.asia/_matrix/client/versions | jq '.versions[-1]'
curl -s -o /dev/null -w '%{http_code}\n' \
https://matrix.reilly.asia/_matrix/media/v3/download/reilly.asia/<known-media-id> # 200 (#736)
curl -i https://call.reilly.asia/sfu/get # 405
Then:
- Log in through MAS in Element and confirm that sync completes.
- Send a message in a federated room and confirm a remote server receives it.
- Start a call.
Rollback
The schema did not change, so rolling back is an image swap:
docker compose stop spindle
cp compose.yaml.bak-<stamp> compose.yaml
docker compose up -d spindle
The rehearsal opened a store that main had already written with the hotfix binary: same rooms, state and timelines, no errors. Data written by features only main has (new keyspaces) is ignored by the old binary.
Restore data/store.pre-<tag> only if the store itself is damaged:
sudo mv data/store data/store.bad && sudo mv data/store.pre-<tag> data/store
That restore loses everything written since the stop.
Afterwards
- Delete
data/store.pre-<tag>after a week without problems, and remove the old images (outbox-hotfix-arm64,legacy-media-compat-arm64,perf-723-de17805-arm64). - Optional, offline, with a fresh copy taken first:
spindle gcreclaims about 290 MB of unreachable state nodes. It took 64 s on the copy. - Update the image row in README.md and delete the
ops/outbox-hotfixbranch.
2026-11-16/17: slow sync, missing avatars, Element X image errors
Symptoms on matrix.reilly.asia: sync felt slow, profile and room pictures were missing, and Element X
showed "Sorry, an error occurred" for every image. Four separate faults, plus host load.
The general fault guide (checks and fixes for any Spindle operator) is in the Spindle repo:
docs/troubleshooting.md.
What was wrong and where it was fixed
| Fault | Symptom | Fix |
|---|---|---|
Element X 26.11 requests media at /_matrix/client/unstable/org.matrix.msc3916/media/…; Spindle served only /v1/media | Every image and avatar 404s (route="unmatched" in metrics); Element X error dialog | Caddy rewrite (live now, also in this kit's Caddyfile); Spindle #940 serves the prefix itself |
Federation media answered with a Location header was cached as 0 bytes | Beeper avatars blank forever | #933, in v0.1.5 |
CDN labels images application/octet-stream | M_UNSUPPORTED: no thumbnails for application/octet-stream | #937, in v0.1.6 |
| Partial-state (faster-join) rooms never finished resync: one bad member signature, or one auth event the resident refuses with 403, failed the whole room; keys were fetched one server at a time | Newly joined big rooms have no member list or member avatars; /joined_members hangs | #935 (v0.1.5), #938 (v0.1.6) |
Not Spindle bugs, still open:
- A browser tab looping on a dead token. Firefox at
https://hanthor.github.iosent sync/whoami/keys ~20 req/s, all401 M_UNKNOWN_TOKEN, after the MAS → built-in OIDC switch on 2026-11-16. Sign out and back in (or close it); the client should have logged itself out. - Host contention. 2 OCPU / 12 GB, no swap; the School and Reef Hives use ~2.6 GB and heavy CPU/disk. Spindle's first sync after a restart takes up to a minute while caches warm. Moving the Hives off this host is the biggest remaining speed-up.
synctapstill proxies sliding sync for@james(0.5 CPU cap). Remove the service and the@ssyncCaddy handle when that diagnosis is done.
Live state left behind (as of 2026-11-17)
compose.yamlrunsghcr.io/tuna-os/spindle:v0.1.5(backupcompose.yaml.bak-pre-v0.1.5-*). Bump to the release that contains #940, then delete the@msc3916rewrite.Caddyfilehas two temporary additions, each with a comment:- the
@msc3916rewrite (keep until #940 is deployed); - an access log to
/data/access-diag.loginside the Caddy container (./caddy/access-diag.logon the host). Headers andaccess_tokenare stripped. Remove thelogblock when done; the file grows.
- the
@claude-test:reilly.asiais a non-admin test account, joined to#fedora:fedoraproject.org. Its password is in~/.spindle-testuser-pass(mode 600) on the VPS. Deactivate it when it is no longer needed.
Diagnosing on this host
- Metrics:
docker exec spindle-caddy-1 wget -qO- http://spindle:9100/metrics.spindle_http_requests_total{route="unmatched"}counts paths Spindle does not serve. docker logs spindle-caddy-1shows nothing new: the container's json log file was deleted from disk. Log to a file in/datainstead (above).- Editing
Caddyfileorspindle.toml: both are single-file bind mounts.sed -iand most editors replace the inode, and the container keeps reading the old file. Write in place (cat new > Caddyfile, or Pythonopen(p, "r+")+truncate), or restart the container. Check withdocker exec spindle-caddy-1 cat /etc/caddy/Caddyfile. - Seeing what a client requests:
sudo tcpdump -i any -A "tcp port 8008 and host <spindle container IP>"shows plaintext proxy→Spindle traffic (it includes bearer tokens; do not save it). - Partial-state rooms:
docker logs spindle-spindle-1 | grep -E "resync|partial-state". - Test account while registration is closed: add
shared_secretunder[registration], restart, register via/_synapse/admin/v1/register(Synapse HMAC), remove the secret, restart. Details in the Spindle fault guide.
Releasing a Spindle fix to this host
release.yml (dispatch with bump=patch) tags main only after every check on the commit has finished,
including the Matrix compliance suite (~1 h after merge). Dispatching earlier logs "CI is still running" or
"The Rust quality gate has not passed" and builds nothing. Then:
cd /srv/spindle
cp compose.yaml compose.yaml.bak-pre-vX.Y.Z-$(date -u +%Y%m%dT%H%M%S)
sed -i 's#tuna-os/spindle:vOLD#tuna-os/spindle:vNEW#' compose.yaml # compose.yaml is not bind-mounted
docker compose pull spindle && docker compose up -d spindle
Punjab
Headless agent and development box, and the fleet's Ansible host. Since
2026-10-27 it is a KubeVirt VM on karnataka. The EC2
instance it replaced (tailnet node punjab-aws-retired, 100.78.73.8) was last seen on 2026-10-28.
See EC2 history.
| Runs on | karnataka k3s, corral-vms/punjab (VM created 2026-10-27) |
| Resources | 12 cores, 24 Gi of RAM, 500 Gi root PVC (punjab-root, local-path on karnataka) |
| OS | Ubuntu 24.04 LTS |
| Tailnet | punjab / 100.103.187.46 |
| Access | ssh ubuntu@punjab (there is no james user). For the VM console, run virtctl console -n corral-vms punjab from a host that has karnataka's kubeconfig |
Role
- Agent host: Claude Code, OMP and Codex sessions in tmux. Most git worktrees
are in
~/dev. kubectlagainst karnataka's k3s:~/.kube/config.- Hive fork builds:
~/dev/hive-v6-port(tuna-os/hive, deploy branches). The amd64 binary that bihar and karnataka run was built here. See Hives hosting. sccache.service(user) is a shared compiler cache for cargo builds.- The Kiro Crew gateway is retired:
kirocrew.serviceis masked.
Disk
On 2026-11-18 / was 96% full (19 GB free). ~/dev used 156 GB. Delete
stale worktrees under ~/dev and ~/tuna-wt before you make the PVC larger.
Dependencies
punjab is a VM on karnataka. When karnataka restarts or k3s restarts its VMs, punjab also restarts, and the agent tmux sessions stop.
Ansible notes (host_vars/punjab.yml)
skip_proxy: true: there is no podman on stock Ubuntu, so there is no Caddy quadlet.- Not a desktop: no GNOME, flatpak or browser roles.
syncthingruns only if its brew binary exists.
EC2 history (retired)
From 2026-08 to 2026-10, punjab was an EC2 t3.large in us-east-1, the
Ansible host of the TunaOS AWS account (AWS account). It had
SSM break-glass access, daily DLM snapshots and Kiro Crew. It is described in
aws/punjab.tf.
Its tailnet node is now punjab-aws-retired. Before you delete the instance and
its snapshots, confirm that the VM has all the data.
Goa
Home server / Ansible control node for the hanthor fleet.
Hardware
- Arch: aarch64 (ARM Cortex-A76)
- Board: Raspberry Pi 5
- Tailscale:
goa(MagicDNS)
OS
Debian GNU/Linux 13 (trixie), kernel 6.18.29
Role
- Fleet control plane — runs Ansible to manage all nodes
- Primary pi-coding-agent development host
- Inventory + health check hub (
just inventory,just doctor)
Notes
- Homebrew at
/home/linuxbrew/.linuxbrew(ARM — some bottles unavailable) - System packages via
apt, userland tools viabrew - Hosts the dotfiles git repo at
~/.local/share/dotfiles - Pi agent config at
~/.pi/agent/
Home Kubernetes Server Images
Not deployed (2026-11-18). bihar and karnataka run stock Bluefin (
ghcr.io/projectbluefin/utah:testing), not these images; k3s is installed on the host fromhome-k3s/. Theserver-imagesCI still builds them. Treat this page as a design for a future rebuild.
Bootable container images for the home K8s cluster (bihar, karnataka, starter). Built from Fedora Hummingbird + GNOME desktop + K3s, with graphical target disabled by default.
Image Variants
| Variant | Role | CPU/GPU | Image URI | Installer ISO |
|---|---|---|---|---|
| bihar | K3s control plane | Intel | ghcr.io/hanthor/home-k8s-bihar:latest | Release |
| karnataka | K3s worker (AMD APU + GPU) | AMD Strix Halo | ghcr.io/hanthor/home-k8s-karnataka:latest | Release |
| starter | Disposable test/staging node | Any | ghcr.io/hanthor/home-k8s-starter:latest | Release |
Image Composition
Each image includes:
- Base:
projectbluefin/utah(Hummingbird + GNOME 51 desktop + Bluefin theming) - K3s: Kubernetes v1.36.1 + SELinux policy
- Storage: iSCSI utils, NFS utils, Longhorn tools
- Networking: NetworkManager, Tailscale, iptables-nft
- GPU: AMD firmware + amdgpu kernel driver (built into boot image via Talos Image Factory)
- Filesystems: ext4, xfs, btrfs (all tools included)
- Default state: Multi-user target (text console), GDM masked
Obtaining Images
Latest Release (Recommended)
Download dated ISO releases from GitHub Releases:
{bihar,karnataka,starter}-installer.iso— offline bootc-installer-tui ISOs{bihar,karnataka,starter}.oci.tar— OCI archives forbootc switchSHA256SUMS— checksums for all artifactsprovenance.json— builder details (source commits, acceptance proofs)
Development/Testing
Build locally:
just server-images # Builds all three variants to localhost/home-k8s-*:testing
just server-installer bihar ghcr.io/hanthor/home-k8s-bihar:testing artifacts/bihar.iso
Container Registry
- Tracking tags (mutable):
ghcr.io/hanthor/home-k8s-{bihar,karnataka,starter}:testing - Immutable tags (dated):
ghcr.io/hanthor/home-k8s-{bihar,karnataka,starter}:YYYYMMDD
Both are public (no login required).
Installation
From Installer ISO
- Boot the ISO from USB stick (UEFI, Secure Boot must be disabled)
- At root console:
bootc-installer-tui - Select image: Custom URL →
oci:/usr/share/home-k8s/payload - Choose filesystem: ext4 (default), xfs, or btrfs
- Set hostname:
bihar,karnataka, orstarter(enables K3s role) - Create admin user:
nodeadmin(or any name) - Confirm and install
The TUI automatically configures:
- Disk partitioning (EFI + /boot + root)
- K3s service enablement (based on hostname)
- Tailscale daemon (unenrolled at install time)
- SELinux enforcing mode
After reboot, the node boots to text console (multi-user.target). K3s is gated (disabled) until you manually enable it.
From OCI Archive (bootc switch)
To upgrade an existing machine from a previous image:
# Download the OCI archive
wget https://github.com/hanthor/dotfiles/releases/download/home-k8s-20261021/home-k8s-bihar.oci.tar
# Extract locally
podman load -i home-k8s-bihar.oci.tar
# → Loads as ghcr.io/hanthor/home-k8s-bihar:testing
# Switch to new image
sudo bootc switch ghcr.io/hanthor/home-k8s-bihar:testing
# Reboot to apply
sudo systemctl reboot
Filesystem Options
All three mkfs tools are included. Choose at install time or format a custom layout:
| Filesystem | Use case | Performance | Notes |
|---|---|---|---|
| ext4 | Default, safe | Stable, widely supported | Verity-enabled for composefs |
| xfs | High-throughput, servers | Fast journal | Requires mkfs.xfs (included) |
| btrfs | Snapshots, COW | Modern, complex | Requires mkfs.btrfs + btrfs CLI |
Deployed images support all three without additional tools. If a filesystem option fails to appear in the TUI, verify the mkfs tool exists: which mkfs.xfs etc.
Graphical Target (Desktop)
Utah ships with GNOME desktop, but the images disable it by default (multi-user.target) for server use.
Enable Graphical Target
# Simple toggle script (on the machine)
sudo toggle-graphical --reboot
# Or manually
sudo systemctl set-default graphical.target
sudo systemctl reboot
What happens:
- GDM (GNOME Display Manager) starts
- Full GNOME 51 desktop available
- Tailscale MagicDNS in system tray
- K3s still running (if enabled)
Disable Graphical Target
sudo toggle-graphical --reboot
K3s Enablement
K3s is installed but gated (disabled) at boot. Roles are determined by hostname:
| Hostname | K3s Role | Service | Enabled at boot |
|---|---|---|---|
bihar | server (control plane) | k3s@server.service | No (gated) |
karnataka | agent (worker) | k3s@agent.service | No (gated) |
starter | (none) | — | — |
To enable K3s on bihar (control plane):
sudo systemctl enable k3s@server.service
sudo systemctl start k3s@server.service
# Wait for API server
sudo systemctl status k3s@server.service
To enable K3s on karnataka (worker):
sudo systemctl enable k3s@agent.service
sudo systemctl start k3s@agent.service
Check that nodes joined the cluster:
kubectl get nodes -o wide
Networking & Access
- SSH:
nodeadmin@{hostname}.manatee-basking.ts.net(via Tailscale) - Kubeconfig: At
/etc/rancher/k3s/k3s.yaml(after K3s start) - Tailscale: Daemon running, identity unenrolled (register via web UI or
sudo tailscale up)
Storage & Persistence
iSCSI (Longhorn)
Both nodes are configured for Longhorn iSCSI workloads:
iscsid.socketrunningiscsi-initiator-utilsinstalled- Unique initiator name generated at first boot
/var/lib/longhorn/mounted to second NVMe (karnataka only)
Temporary Disks
# View all mounts
findmnt /
# Root filesystem type
lsblk -f /dev/sda3 # or appropriate root partition
Image History & Migration
Changes in Latest Release
home-k8s-20261024:
- Base switched from
os-minimaltoutah(added GNOME desktop) - Filesystem detection in TUI (only shows available mkfs tools)
toggle-graphicalscript for easy desktop on/off- xfsprogs + btrfs-progs included (all filesystem tools)
Previous release (home-k8s-20261021):
- Hummingbird base (minimal, no desktop)
- ext4 only (xfs/btrfs not available)
- No toggle script
Upgrading from Old Images
If you're running home-k8s-20261021 or earlier, you can upgrade:
# Get the new OCI archive
wget https://github.com/hanthor/dotfiles/releases/download/home-k8s-20261024/home-k8s-bihar.oci.tar
podman load -i home-k8s-bihar.oci.tar
sudo bootc switch ghcr.io/hanthor/home-k8s-bihar:testing
sudo systemctl reboot
Note: Old images do not have toggle-graphical. Update to get the script.
Troubleshooting
TUI Filesystem Option Missing
If xfs or btrfs don't appear in the installer TUI filesystem picker:
# On the live ISO (at root console)
which mkfs.xfs # Check if tool exists
which mkfs.btrfs
If missing, the image was built without those packages. Rebuild or wait for next release.
K3s Won't Start
# Check service status
sudo systemctl status k3s@server.service
sudo journalctl -u k3s@server.service -n 50
# Verify kubeconfig exists
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml get nodes
Can't SSH After Graphical Enable
If you SSH while the system is switching targets, the SSH session may hang:
# On local machine, from another terminal
ssh nodeadmin@karnataka.manatee-basking.ts.net "sudo systemctl is-active graphical.target"
# Or just reboot the graphical machine and reconnect
Tailscale Not Enrolled
Machines boot with Tailscale daemon running but unenrolled:
sudo tailscale up
# or web UI at: https://login.tailscale.com/admin/machines
See Also
- Talos K8s Cluster Handbook — Bihar & karnataka hardware, networking, advanced operations
- AWS K8s Cluster Handbook — Production cluster migration info
- Roles Documentation — Individual role configurations
- GitHub Releases — Download ISOs and archives
TunaOS AWS Account
Cluster terminated 2026-10-25; IaC is not reconciled. The AWS Talos cluster is historical. Matrix and standalone LiveKit are deployed on the OCI spindle VPS. S3 backups and a mirror on himachal-1 were retained; do not assume other AWS resources have been removed or retained without checking the account.
Do not blindly run
just aws-planorjust aws-apply.aws/still describes removed cluster resources. A plan can propose recreating them; applying it can recreate infrastructure and incur costs. Review live state, configuration and every proposed action before any apply.
This TunaOS account's OpenTofu configuration is in
aws/; runs-on was
managed separately by CloudFormation. The table describes the configuration
and prior deployment, not a verified live resource inventory.
| Region | Configured or historical resource | Configuration |
|---|---|---|
eu-north-1 | Terminated Talos cluster — VPC, nodes, Elastic IPs, security groups, pgdata volume | aws/cluster.tf |
eu-north-1 | Backup configuration — DLM policy and retained fleet-backups bucket | aws/backups.tf |
us-east-1 | punjab EC2 agent box; running as of 2026-10-27 | aws/punjab.tf |
us-east-2 | runs-on GitHub Actions runners; current stack status unverified | CloudFormation stack runs-on |
| global | IAM, backup, account and budget configuration | aws/iam.tf, aws/backups.tf, aws/account.tf, aws/budgets.tf |
OpenTofu layout
aws/
├── versions.tf # provider (eu-north-1 default, us_east_1 alias) + S3 backend
├── cluster.tf # VPC/subnet/IGW/routes, 3 SGs, 2 Talos nodes, EIPs, pgdata
├── backups.tf # DLM role + policy, fleet-backups bucket (+ versioning/SSE/lifecycle)
├── punjab.tf # key pair, SG, instance, EIP, SSM role, DLM
├── account.tf # EBS encryption by default (both regions)
├── iam.tf # scoped admin user + its policies
├── budgets.tf # 5 budgets, one for_each
├── imports.tf # one-time adoption record (no-op once imported)
├── variables.tf
└── terraform.tfvars.example
- State: in the fleet-backups bucket under
tofu/aws/(versioned, SSE-S3, lockfile). It contains the Talos nodes'user_data— i.e. cluster PKI — so it never goes in git. - Variables:
aws/terraform.tfvarsholds the alert email and the admin ingress allowlists. The repo is public, so it is gitignored and stored in the Bitwarden secure noteaws-tofu-tfvars. - Talos user_data is not in the config. The node resources ignore
user_dataandami;prevent_destroyis configured on the nodes, punjab, worker EIP, pgdata volume and backup bucket. It does not prevent creation of resources missing from the account.
Usage and drift
The former expectation of a no-change plan no longer applies after termination.
Do not treat drift as a reason to restore resources. Before using just aws-plan,
inspect the account and configuration; review the resulting plan carefully and
reconcile the intended resource lifecycle before any just aws-apply. Planning
does not itself create resources, but applying proposed creates can restore
infrastructure and incur charges. aws/terraform.tfvars remains sensitive;
just aws-seed-tfvars stores it in Bitwarden after an intentional edit.
IAM & access
- Root historically had MFA and no access keys. Verify current access configuration before relying on it.
- A scoped admin user was the day-to-day identity, with EC2, S3, budgets,
DLM, SSM and read-only IAM access. The historical key item is
aws-james-admin; verify its current permissions and status before use. - A backup user was configured for put/get under
postgres/in the backup bucket, with no delete. Its key was stored in Bitwarden aspostgres-backup-s3and in the former cluster'spostgres/postgres-backup-s3Secret, not tofu state. - punjab's configured instance role has
AmazonSSMManagedInstanceCore. - DNS was managed on Cloudflare. Verify current AWS security and DNS settings before relying on this historical inventory.
Backups
The S3 fleet-backups bucket was retained, as was a mirror on himachal-1 at
~/backups/tunaos-aws-2026-10-25. Protect these backups and the OpenTofu state;
do not assume their presence means every former cluster volume or snapshot
survived. The former cluster uploaded verified Postgres dumps to
postgres/<date>/; configured S3 lifecycle rules moved them to IA after 30
days, Glacier IR after 90, and expired them after a year. Check live bucket
settings before relying on that schedule.
The configuration also describes DLM schedules for the former Talos root volumes and punjab. Those schedules and the availability of individual snapshots are not a current backup inventory.
Cost
The AWS credits were exhausted. Remaining resources can incur charges, and
recreating cluster infrastructure would add costs. aws/budgets.tf describes
budget alerts; verify current budgets and actual charges in the AWS account.
Hive — AI Agent Supervisor
Current deployment: bihar (karnataka standby)
Three Hive spokes, school, reef and hanthor, and their hub run as compose
services under rootful podman on bihar, in
/var/srv/hives (ssh james@bihar, commands need sudo). Until 2026-11-18
they ran on the Spindle VPS under
/srv/hives. Spindle's Caddy still routes
school.tunaos.org,
reef.tunaos.org,
hive.reilly.asia and
hub.tunaos.org, over the tailnet to the active host.
karnataka is a cold standby. A lease on spindle
selects the active host. Run every command below on the host the lease
names (ssh spindle sudo hive-failover status). Hosting, failover and the
host-specific compose layer are described in
Hives hosting and failover.
hanthor (hive.reilly.asia)
The personal-account spoke from the old AWS cluster was rebuilt on bihar on
2026-11-19. It is the hanthor service in compose.override.yaml (port 3104),
with its data in /var/srv/hives/hanthor. It uses the same GitHub App as
school (hanthor-hive-agent, App ID 3942065) through the App's installation on
the hanthor account (158987150), and the same key file. Its seed is
hives/hanthor/hive.yaml:
ACMM level 5, four agents (supervisor, scanner, quality, guide) on the claude
backend, and the repositories dotfiles, reilly.asia, homebrew-tap,
shrimply and indiafoss-chat-android. Compared with the AWS seed it drops
indiafoss-companion (archived), hummingbird-github (deleted) and
rust-wayland-desktop (now tuna-os/tuna-desktop, which school covers). The
AWS data volume was not kept, so the hive started with empty state.
The layout and roster below were read on spindle on 2026-11-13. The move changed the paths and the commands, but not the hive configuration. Container state alone does not prove that agents are healthy. See Checking health.
This section is the current operations guide. The archived cluster guide after it describes the retired Kubernetes deployment and must not be applied here.
Layout
/var/srv/hives/ # on the active host; the host layer is /var/srv/hives-host/compose.<host>.yaml
├── compose.yaml # services: hub, hub-front, school, reef (network spindle_default)
├── compose.override.yaml # pins the running binary/dashboard, memory limits, session pool;
│ # also defines the hanthor spoke (added 2026-11-19)
├── .env # secrets and NTFY topic (mode 600; never print or commit)
├── bin/hive-<sha> # fork builds; compose.yaml mounts bin/hive, the override re-pins one
├── <spoke>/config/hive.yaml # seed, mounted read-only at /etc/hive
├── <spoke>/data/hive.yaml.runtime # live config, in the spoke's /data volume
├── hub/ hub-front/ # hub config/data/secrets; static front page
└── backups/ # timestamped copies of every config and data snapshot
- Seed vs runtime. The spoke starts from the seed and persists its live
configuration to
data/hive.yaml.runtime, which then wins. Hive can rewrite the runtime file (dashboard changes, ACMM level), so the two files drift: reef's runtime is ACMM 6 while its seed says 5. Edit the seed and the runtime for anything you want to stick, then verify the parsed result. - Binary.
compose.yamlbind-mountsbin/hiveover/usr/local/bin/hive; on 2026-11-13compose.override.yamlre-pins hub, school and reef tobin/hive-2ef7dc7c0(atuna-os/hivefork build on upstream v6) and mounts a matching dashboard HTML on the spokes. The image tag (ghcr.io/hivecommons/hive:v6-latest, hubhive-hub:v6-latest) does not identify the running code; the header of the override file is the change log and rollback note. - Agents. All agents use the
claudebackend (sonnet, opus, haiku and fable tiers) on both spokes; the seed'sbackends.allowalso listsomp(andmuseon school). Runtime model assignments differ from the seed's. - Dashboard.
dashboard.authorized_usersishanthor:owner. Thedashboard.auth_tokenin the runtime file and the GitHub App key are secrets: never print, paste or commit them.
Roster
Each non-empty tuna-os repo is in exactly one hive, except the deliberate
exclusions below (state of 2026-11-13; xfce-linux was renamed brussels).
| Hive | Repos |
|---|---|
| school (21) | tunaos, tromso, tacklebox, brussels, tunaos-packages, wootc, fisherman, bootc-installer, bootc-migrate, bootsahi, remora, bst-ci, iso-builder, branding, homebrew-tap, scoop-bucket, athens, docs, .github, bluefin-cli, tuna-desktop |
| reef (22) | Tavern, corral, flatpak-index, gtk-office-suite, blueshell, dualcut, gnome-hive-monitor, hive, mandelbrot, mariner, protota, spindle, PaperWM, compass, ccleft, hive-operator, quickcast, postcard, lifafa, tuna-mail, proton-unofficial-sdk, tuta-unofficial-sdk |
| excluded | iced, libcosmic, rust-atomicwrites, rdocx, changelog-action (patch-carrier forks, no issues); flatpark (excluded from automation); publish-proof (empty) |
The roster is two lists, project.repos and dashboard.matrix.repos, and both
must match in both files of the spoke (four edits per change). The seed uses
either bare names or tuna-os/<name> in project.repos (reef bare, school full);
the runtime file uses bare names there and tuna-os/<name> in the matrix. Keep
whatever style the file already has. school also has a separate
revert_sheriff.repos list (shadow mode); it is not part of the roster.
As of 13:01Z on 2026-11-13 the reef seed lacked postcard (21 repos) while
the runtime listed it (22); the intent of that seed edit is unverified.
Identity
| Hive | GitHub App | App id |
|---|---|---|
| school | hanthor-hive-agent | 3942065 |
| reef | tuna-reef-hive | 5546574 |
Both set github.app_authored_prs: true and project.ai_author: "", so PRs and
comments come from the App's [bot] account, not from a human or AI-author
trailer. Each hive lists the other App in github.additional_bot_logins. Both
Apps are installed on every org repo (per the maintainer; not checkable from
the host), so adding a repo needs only a roster edit, no App change.
Both hives set sentinel.exempt_logins: [renovate[bot]]. Sentinel's
sensitive_path rule flagged every dependency bump and re-flagged after each
rebase. The alternative, auto_merge.trusted_bot_authors, would also let Hive
auto-merge Renovate major bumps, which the org forbids.
Adding or moving a repo
- Pick the hive: school for the OS, packaging, installer and org-level repos; reef for apps, libraries and the Matrix/Hive tooling (see the table). Check the repo is in no other roster.
- Apply the safe edit procedure with
add <repo>on the target hive's two files. To move a repo, also runremove <repo>on the old hive's two files, in the same maintenance window, so it is never in both or neither. - Re-parse both files and check the list lengths and that the new repo shows up on the hive's dashboard; no App installation step is needed.
GitHub Apps
Since 2026-10-30 School and Reef use separate GitHub Apps, so each has its own installation rate limit (6,150 requests/hour across the 43 org repositories). Before that change they shared one installation and used all of its limit every hour.
| Hive | App (bot login) | App ID | Installation | Key in Bitwarden |
|---|---|---|---|---|
| school (the hub mounts the same key) | hanthor-hive-agent (hanthor-hive-agent[bot]) | 3942065 | 137498420 | hive-github-app-hanthor-hive-agent |
| reef | tuna-reef-hive (tuna-reef-hive[bot]) | 5546574 | 177007234 | hive-github-app-tuna-reef-hive |
The IDs occur in three places for each hive:
GH_APP_IDandGH_APP_INSTALLATION_IDincompose.yaml;github.app_idandinstallation_idin the seed;- the same keys in
hive.yaml.runtime.
Both hives manage .github, Tavern, bluefin-cli, corral, docs and
flatpak-index. Each hive lists the bot of the other hive in
github.additional_bot_logins, so that the issues and PRs of the other hive count as
hive-authored.
To restore a key:
- Write the key block of the Bitwarden note to
/var/srv/hives/<svc>/secrets/gh-app-key.pemon the active host (mode 600). - Run
chcon -t container_file_ton the file. - Recreate the service with
sudo /var/srv/hives-host/hives up -d --no-deps <svc>.
The next hive-sync copies the key to the standby host.
To make sure that each hive has its own rate limit, run:
ssh james@bihar 'for c in hives-school-1 hives-reef-1; do sudo podman exec $c sh -c \
"curl -s -H \"Authorization: Bearer \$(cat /var/run/hive-metrics/gh-app-token.cache)\" https://api.github.com/rate_limit" \
| jq -c .resources.core; done'
Different reset times show separate installations.
Safe edit procedure
More than one operator edits /var/srv/hives (including agents). Re-read the files
immediately before editing and never overwrite a file you read earlier.
Do not hives up a spoke that is running to apply config: it
triggered a container-rename bug. Remove the container first:
sudo -i # the files are root-owned on bihar/karnataka
cd /var/srv/hives
H=/var/srv/hives-host/hives # docker-compose wrapper: podman + host layer
$H rm -f -s reef # stop and remove the spoke container only
python3 hive-roster.py add NAME reef/config/hive.yaml reef/data/hive.yaml.runtime
$H up -d --no-deps reef # recreate only this spoke
podman ps --filter name=hives- # verify: Up, and the right name
The edit step must (a) write a timestamped copy of each file into
/var/srv/hives/backups/ first, (b) parse the YAML before and after, and (c) abort,
writing nothing, if anything other than the intended change would differ. This
helper does that for roster edits (save it as /var/srv/hives/hive-roster.py and
run it from /var/srv/hives; it needs PyYAML):
#!/usr/bin/env python3
"""hive-roster.py add|remove REPO FILE... (run with the spoke container removed)"""
import datetime, shutil, sys, yaml
ORG = "tuna-os"
BACKUPS = "/var/srv/hives/backups"
PATHS = (("project", "repos"), ("dashboard", "matrix", "repos"))
def seq_node(root, path):
node = root
for key in path:
node = next(v for k, v in node.value if k.value == key)
return node
def edit(text, op, repo):
lines = text.split("\n")
root = yaml.compose(text)
# bottom-most list first so earlier line numbers stay valid
seqs = sorted((seq_node(root, p) for p in PATHS), key=lambda n: -n.start_mark.line)
for seq in seqs:
full = seq.value[0].value.startswith(ORG + "/")
item = f"{ORG}/{repo}" if full else repo
hit = [n for n in seq.value if n.value in (repo, f"{ORG}/{repo}")]
col, last = seq.value[0].start_mark.column, seq.value[-1].start_mark.line
if op == "add" and not hit:
lines.insert(last + 1, " " * (col - 2) + "- " + item)
elif op == "remove":
for n in sorted(hit, key=lambda n: -n.start_mark.line):
del lines[n.start_mark.line]
return "\n".join(lines)
def main(op, repo, *files):
assert op in ("add", "remove") and files
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ")
new = {}
for f in files:
old = open(f).read()
want = yaml.safe_load(old)
for path in PATHS:
d = want
for k in path[:-1]:
d = d[k]
style = d[path[-1]][0].startswith(ORG + "/")
item = f"{ORG}/{repo}" if style else repo
d[path[-1]] = [x for x in d[path[-1]] if x not in (repo, f"{ORG}/{repo}")]
if op == "add":
d[path[-1]].append(item)
text = edit(old, op, repo)
if yaml.safe_load(text) != want:
sys.exit(f"{f}: unrelated change detected, aborting (nothing written)")
new[f] = text
for f, text in new.items():
shutil.copy2(f, f"{BACKUPS}/{f.replace('/', '-')}.{stamp}")
open(f, "w").write(text)
main(*sys.argv[1:])
The script was exercised only on synthetic copies of the two file styles, not
on the live files. Backups from the 2026-11-13 edits are
backups/*.20261113T040153Z (sentinel exemption) and
backups/*.roster.20261113T125525Z (roster); to roll back, remove the
container, restore both files of the spoke from the same stamp, and up it.
Rolling back the binary is described in the header of compose.override.yaml.
Operating from GitHub
You rarely need the host. On a hive-managed issue or PR:
/hive decision <instruction>records a maintainer decision for the agents;/hive approvelifts Hive's ownneeds-humanhold on a PR;/fixedtells Hive an issue is resolved.- On PRs that Hive did not author, a GitHub review (approve or request changes) is honoured.
- When Hive un-parks an issue it removes the
blockedlabel itself. Decision comments written by the maintainer's agent start withDecision (for @hanthor):.
(The command names are present in the running binary and the usage comes from the maintainer; their exact semantics were not exercised for this page.)
Checking health
curl -s -o /dev/null -w '%{http_code}\n' https://school.tunaos.org/api/health # 200
curl -s -o /dev/null -w '%{http_code}\n' https://reef.tunaos.org/api/health # 200
curl -s https://hub.tunaos.org/api/fleet-stats # hub: /api/health is 404
ssh james@bihar 'sudo podman ps --filter name=hives-' # or karnataka after a failover
For authenticated status from the host, call the API inside the container with the container's own token, so it is never printed:
ssh james@bihar 'sudo podman exec hives-school-1 sh -c \
"curl -sS -H \"X-Hive-Internal: \$HIVE_DASHBOARD_TOKEN\" http://127.0.0.1:3002/api/status"' \
| jq '.agents[] | {name, paused, state}'
Agents that show stopped are normal between governor kicks; check authored
PRs and issues (gh search prs --author "hanthor-hive-agent[bot]" and
"tuna-reef-hive[bot]") to see whether a hive is producing.
Known limits
- Memory. On bihar and karnataka, each spoke can use 8 GB and the hub allows 4
concurrent shared-pool sessions (
compose.<host>.yaml). On spindle the limits were 2560 MB per spoke and 2 sessions, and spokes were OOM-killed during rustc/govulncheck builds. - Pinned images. The hive and hive-hub images are pinned by digest with the
amd64 fork binary
hive-2ef7dc7c0-amd64. Do not change to:v6-latest. - Hive's own CI for
tuna-os/hiveis blocked by GitHub ("Actions has been disabled"; a support ticket is pending), so fork builds are not CI-verified. - Reply spam: upstream hivecommons/hive#17512 (draft) fixes Hive repeating its un-park replies because it does not recognise its own earlier reply when posting as an App login.
- Concurrent edits and drift: see the layout notes above; verify after every restart that the parsed seed and runtime agree on the roster.
- The spokes call the
omp-auth-brokeron spindle (OMP_AUTH_BROKER_URL,http://100.113.75.88:8765). Caddy routing and failover are described in Hives hosting and failover. This page does not describe the hub's own configuration. The hive_ops role targets the retired cluster, not this deployment.
Archived cluster guide
Historical reference only — do not apply or run the Kubernetes, image-build, secret, restart, or state-reset instructions below. The AWS Talos cluster was terminated on 2026-10-25; its Kubernetes resources and CronJobs are not a current deployment. The home Talos cluster (bihar and karnataka) is powered down temporarily, not decommissioned. Preserve its configurations. The sections below describe older home-cluster and AWS configurations at different points in time, not the current Spindle service. See the AWS cluster history and historical cluster manifests. Examples are retained for context, not as instructions for either cluster.
The earlier home-cluster Hive used pi with DeepSeek, tmux agents, a governor, and a GitHub App. The architecture and examples below are snapshots of that period; later AWS configurations and the current Compose deployment differ.
1. Architecture (historical)
┌──────────────────────────────────────────────────────────────┐
│ Namespace: hive (node: bihar) │
│ │
│ ┌─────────┐ ┌──────────┐ ┌────────────────────────────┐ │
│ │ hive │ │ proxy │ │ 9x pi agents (tmux sessions)│ │
│ │ Go bin │ │ node.js │ │ │ │
│ │ :3002 │ │ :3001 │ │ supervisor ADVISORY │ │
│ │ (API) │ │ (web UI) │ │ scanner ISSUES_AND_PRS │ │
│ └────┬────┘ └────┬─────┘ │ ci-maintain ISSUES_AND_PRS │ │
│ │ │ │ quality ISSUES_AND_PRS │ │
│ └────────────┘ │ sec-check ISSUES_AND_PRS │ │
│ │ guide ISSUES_AND_PRS │ │
│ ┌──────────────────────────│ architect ISSUES_AND_PRS │ │
│ │ pi → DeepSeek API │ strategist ISSUES_AND_PRS │ │
│ │ (native, no proxy) │ brainstorm on-demand │ │
│ │ GitHub App → gh CLI └────────────┬───────────────┘ │
│ └───────────────────────────────────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ ConfigMap│ │ Secret │ │ PVC │ │
│ │ hive.yaml│ │ DEEPSEEK │ │ hive-data│ │
│ │ backends │ │ GH App │ │ (10Gi) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────────────────────────────────────────┘
│
┌────▼────┐ ┌──────────────────┐
│ GitHub │ │ Tailscale Ingress │
│ tuna-os │ │ hive.manatee- │
│ repos │ │ basking.ts.net │
└─────────┘ └──────────────────┘
Components
| Component | Language | Purpose |
|---|---|---|
| hive (Go binary) | Go | Agent manager, governor, scheduler, dashboard API |
| proxy (Node.js) | JS | Dashboard web UI + SSE, proxies to Go API |
| pi (coding agent) | Node.js | AI agent CLI running in tmux sessions |
| pi-wrapper.sh | Shell | Tmux interface adapter — restart loop, ready markers |
How agents work
Each agent runs as a long-lived pi process inside a tmux session. The Go binary:
- Creates a tmux session per agent (
hive-supervisor,hive-scanner, etc.) - Writes a boot prompt (agent role, authorized repos, instructions) via
send-keys - Detects agent readiness from CLI markers in the pane output
- Governor evaluates the issue queue every 5 minutes and sends kick prompts to due agents
clear_on_kick: trueagents receive^Cbefore each kick to reset their session
Agents share a persistent /data volume (10Gi PVC) for beads, session state, and agent configs.
2. What Hive did (historical)
Advisory digest
The dashboard shows agent findings — CI failures, test coverage gaps, security issues, documentation problems. Each agent produces beads (structured findings) that appear in the advisory digest.
Issue creation
At ACMM L3+, agents create GitHub issues. At L4+, they can create hold-gated PRs. The sec-check agent scans dependencies for CVEs. The ci-maintainer agent monitors CI failures.
Strategy Lab
Inception feature: describe what you want to build, and Hive scaffolds a new project with knowledge base facts and spec-kit specifications. Powered by the brainstorm agent (on-demand).
Knowledge Base
Agents can contribute facts to a shared knowledge base. Facts are tagged by type (pattern, gotcha, regression, decision, etc.) and searchable from the dashboard.
3. Deployment (historical)
Namespace and resources
The home-cluster deployment used the hive namespace on bihar:
hive (namespace)
├── deploy/hive — main controller pod
├── svc/hive — ClusterIP :3001 (dashboard), :3002 (API), :7681 (ttyd)
├── ingress/hive — Tailscale ingress → hive.manatee-basking.ts.net
├── configmap/hive-config — hive.yaml (project config, agents, governor)
├── secret/hive-secrets — DEEPSEEK_API_KEY, GH App credentials
├── pvc/hive-data — 10Gi persistent state
└── secret/ghcr-auth — GHCR pull credentials
Historical files (not the current repository layout)
talos-k8s/hive/
├── hive.yaml — All K8s resources (namespace, deploy, svc, ingress, configmap)
├── Dockerfile — Custom image (hive v2 + pi + fd/ripgrep)
├── pi-wrapper.sh — Wrapper: Hive tmux interface → pi interactive TUI
├── pi-marker.patch — Adds "pi" to hive's cliPaneMarkers (source patch)
├── build-job.yaml — Kaniko build job (deprecated, use CI)
├── deepseek-chat.py — Python fallback (unused with pi)
├── goose-wrapper.sh — Legacy goose wrapper (replaced by pi-wrapper.sh)
├── goose-marker.patch — Legacy goose patch (replaced by pi-marker.patch)
├── SKILL.md — Skill file for pi coding agent
├── README.md — Quick reference
└── HANDOFF.md — Debugging session notes
CI workflow
The former .github/workflows/hive-build.yml workflow did the following:
- Clones
kubestellar/hivev2 source - Applies
pi-marker.patchand three sed patches to the Go source:- Adds
"pi"tocliPaneMarkers(so agents aren't detected as crashed) - Adds
"pi": truetovalidBackends(config validation) - Adds
"pi": "pi"tobackendBinary()(binary resolution)
- Adds
- Cross-compiles Go binary for amd64
- Builds Docker image with pi + pi-wrapper.sh + fd/ripgrep
- Pushes to
ghcr.io/hanthor/hive:latest
Secrets and deployment (historical)
The home-cluster pod used a Kubernetes Secret for DeepSeek and GitHub App
credentials. Later, the AWS deployment exported spoke manifests under
talos-k8s/hive/spokes/ and used the hive-operator to manage images. These
manifests are historical records, not deployment instructions. The home
cluster's former Tailscale ingress was hive.manatee-basking.ts.net.
4. AI backend: pi + DeepSeek (historical)
Why pi (not goose)
Hive originally used goose (v1.36 from AAIF) as the AI agent CLI. Goose does not natively support DeepSeek, requiring a Python proxy (deepseek-proxy.py) to:
- Strip
reasoning_contentfrom message history - Inject
thinking:{type:disabled}to prevent thinking block generation - Route through
litellmpointing at a local proxy
This was fragile. pi natively supports DeepSeek via its deepseek provider — no proxy, no litellm, no workaround.
Why pi works for unattended agents
Pi intentionally has no permission popups — tools (read, write, edit, bash) auto-execute without user confirmation. This makes it ideal for Hive's unattended agent workflow where no human is available to approve tool calls.
pi-wrapper.sh
The wrapper translates Hive's backend CLI interface to pi:
# Hive calls: pi --no-confirm --model deepseek-v4-pro
# Wrapper strips --no-confirm (pi doesn't need it), passes --model to pi
# Sets up ~/.pi/agent/settings.json with DeepSeek config
# Runs pi in interactive TUI mode inside the tmux pane
# Restart loop: ^C from clear_on_kick kills pi-real, wrapper re-emits ready markers
pi configuration
Written at runtime by the wrapper:
{
"defaultProvider": "deepseek",
"defaultModel": "deepseek-v4-pro",
"quietStartup": true,
"hideThinkingBlock": true,
"enableInstallTelemetry": false
}
Helper tools
fd (fd-find) and ripgrep are pre-installed via apt in the Docker image so pi doesn't need to download them at runtime (the Hive MITM proxy would corrupt the downloads).
5. Agents (historical)
Historical agents (ACMM L5)
| Agent | Mode | Cadence | Role |
|---|---|---|---|
| supervisor | ADVISORY | 5m | Agent health monitoring, sweep analysis, stall detection |
| scanner | ISSUES_AND_PRS | 4h | Issue triage, PR dispatch, merge management |
| ci-maintainer | ISSUES_AND_PRS | 4h | CI/CD health, workflow fixes, build monitoring |
| quality | ISSUES_AND_PRS | 2h | Test coverage, integration tests, quality gates |
| sec-check | ISSUES_AND_PRS | 4h | Security scanning, dependency audit, CVE monitoring |
| guide | ISSUES_AND_PRS | 4h | Documentation gaps, onboarding analysis |
| architect | ISSUES_AND_PRS | 4h | Cross-cutting RFCs, refactors, new features |
| strategist | ISSUES_AND_PRS | 4h | Experiment design, A/B testing, strategy lab |
| brainstorm | on-demand | — | Ideation via Inception; proposes features and architecture |
Agent modes
| Mode | Can create issues | Can create PRs | Can merge |
|---|---|---|---|
| ADVISORY | ✗ | ✗ | ✗ |
| ISSUES_ONLY | ✓ | ✗ | ✗ |
| ISSUES_AND_PRS | ✓ | ✓ (hold-gated) | ✗ |
Mode is determined by ACMM level + agent config. At L5, most agents run in ISSUES_AND_PRS.
Agent lifecycle
- Boot: Agent starts in tmux session, receives role prompt + authorized repos
- Kick: Governor sends a task prompt (e.g., "scan for open issues", "check CI health")
- Work: pi processes the prompt, runs tools, produces beads or GitHub issues/PRs
- Idle: Agent waits for next kick
- Clear on kick (
clear_on_kick: true):^Ckills the pi process before the next kick, wrapper restarts it fresh
Adding a new agent
# In hive.yaml ConfigMap, under agents:
my-agent:
enabled: true
backend: pi
model: deepseek-v4-pro
mode: ISSUES_AND_PRS
beads_dir: /data/beads/my-agent
clear_on_kick: true
Also add the agent to governor cadences for each mode:
governor:
modes:
idle:
my-agent: 4h
quiet:
my-agent: 4h
busy:
my-agent: 1h
surge:
my-agent: pause
6. ACMM levels (historical)
The ACMM (Autonomous Code Maintenance Maturity) framework defines 6 levels:
| Level | Name | Behavior |
|---|---|---|
| L1 | Advisory | Advisory-only reports, no writes |
| L2 | Issues | Can create issues, no PRs |
| L3 | CI/CD | Issues + hold-gated PRs, merge requires human LGTM |
| L4 | Guarded | Broader write access, more agents active |
| L5 | Guarded Auto-Merge | Auto-merge on green CI with guardrails |
| L6 | Full Autonomy | Unrestricted agent actions |
That historical configuration used L5; this is not a statement about the current spokes.
Changing ACMM level
# In hive.yaml ConfigMap
acmm_level: 5
The old procedure reset persistent state and restarted the pod; do not use it on the current deployment or on retained backups.
7. Governor (historical)
The governor evaluates the issue queue every 5 minutes (eval_interval_s: 300) and switches between four modes:
| Mode | Threshold | Behavior |
|---|---|---|
| idle | 0 issues | Slow cadences, minimal activity |
| quiet | ≤2 issues | Normal cadences |
| busy | ≤50 issues | Faster scanner, ci-maintainer still active |
| surge | >50 issues | Scanner every 15m, ci-maintainer paused |
Historical cadences
governor:
eval_interval_s: 300
modes:
surge:
threshold: 50
scanner: 15m
ci-maintainer: pause
quality: pause
sec-check: pause
guide: pause
architect: pause
strategist: pause
busy:
threshold: 10
scanner: 15m
ci-maintainer: 1h
quiet:
threshold: 2
scanner: 15m
ci-maintainer: 45m
idle:
threshold: 0
scanner: 4h
ci-maintainer: 4h
The supervisor always runs at 5m regardless of mode.
8. Configuration (historical)
Backends (backends.conf)
The backends.conf in the ConfigMap defines supported AI backends. Adding a new backend requires entries in three places:
KNOWN_BACKENDSlistbackend_binary()— maps backend name to binary in PATHbackend_perm_flag()— permission flag (pi returns empty — no permission popups)
The old supported list was claude copilot bob gemini codex amazonq goose aider pi.
Agent configuration (hive.yaml)
agents:
supervisor:
enabled: true
backend: pi
model: deepseek-v4-pro
beads_dir: /data/beads/supervisor
clear_on_kick: true
Persistent state
Two locations store state that survives pod restarts:
/data/hive-state.json— Governor mode, agent overrides, kick history, cadence overrides/data/agent-configs/*.yaml— Per-agent persistent config (backend, model, mode, enabled, display name)
Persistent overrides could cause stale values in that deployment. Do not copy its old state-reset and backend-rewrite commands to the current deployment.
9. Building the image (historical)
The former CI workflow built ghcr.io/hanthor/hive:latest; an older Kaniko
job was also used. Neither is a current deployment procedure.
What's in the image
- Base:
node:24-slim - Runtime deps: bash, curl, git, gosu, iptables, jq, procps, python3, tmux,
fd-find,ripgrep - Go binaries: hive (agent manager), bd (bead CLI) — cross-compiled from
kubestellar/hivev2 - Node.js proxy: Dashboard UI + SSE, serves on :3001
- pi:
@earendil-works/pi-coding-agentinstalled via npm - pi-wrapper.sh: Replaces
/usr/local/bin/piwith the tmux interface wrapper - gh CLI: GitHub CLI with wrapper for App token auth
- Entrypoint: UID isolation, tmux setup, config backup, App token generation
10. Historical operations
Operators inspected the old pod's health, tmux panes, pi processes and logs
with kubectl. Those commands targeted the retired Kubernetes deployment;
the read-only Compose inventory at the top of this page is the current starting
point instead.
Historical restarts and state resets
The former procedure restarted the pod and sometimes reset persistent state. It is not a recovery procedure for Spindle; do not reset retained data.
11. Historical troubleshooting
| Symptom | Check |
|---|---|
| ImagePullBackOff | The home cluster once encountered IPv6 timeouts to ghcr.io |
| Agent CLI crashed | Check ps aux | grep pi — should see 9 pi processes |
| Agent stuck in ADVISORY mode | Historical state-file ACMM overrides |
| Dashboard shows old backend/model | Historical /data/agent-configs/*.yaml overrides |
| No kicks sent | Governor eval cycle: 5min. Agent cadences vary per mode |
| Issues not created | GitHub App installation and authentication were relevant |
| "CLI did not reach input prompt" | pi is mid-response when kick arrives. Normal — agent will pick up next kick |
| fd/ripgrep download errors | pi can't download tools through MITM proxy. Pre-installed in Docker image |
| "backend binary not found" | Missing binary in PATH. Check backends.conf and verify wrapper is at /usr/local/bin/pi |
| "failed to persist config" | Read-only ConfigMap mount. Cosmetic — config is cached in memory |
Historical agent issues
Agent shows bare bash shell:
- pi process died or never started
- Check:
tmux capture-pane -t hive-scanner -p | tail -5 - Should show pi's footer (
deepseek-v4-pro • high), not$bash prompt
Agent stuck at /clear:
- Hive sent
/clearto reset the agent before a kick - Agent processed it but new kick hasn't arrived yet
- Wait for next governor cycle
Agent mode mismatch:
- The state file held persistent mode overrides in that deployment.
12. GitHub App (historical)
Hive authenticates to GitHub via a GitHub App installed on tuna-os (App and installation IDs are in the hive-secrets Secret).
Token generation
The entrypoint generates an installation token on startup and caches it at /var/run/hive-metrics/gh-app-token.cache. Agents access it via $GH_TOKEN.
Required permissions
| Permission | Scope | Why |
|---|---|---|
contents: read | Repo | Read code, files |
contents: write | Repo | Create branches, push commits for PRs |
issues: write | Repo | Create and update issues |
pull_requests: write | Repo | Create PRs |
metadata: read | Repo | Basic repo access |
security_events: read | Repo | Dependabot alerts (sec-check agent) |
Rate limits
- GitHub App: 15,000 req/hr (shared across all installations)
- Personal Access Token: 5,000 req/hr
The App gives Hive its own rate limit pool, separate from personal API usage.
Permission changes (historical)
GitHub App permission changes required re-acceptance by the installation. Do not use the old Kubernetes restart procedure to apply them.
Images and upgrades on the former AWS cluster
On 2026-09-24 the AWS Hives used stock upstream images
(ghcr.io/hivecommons/hive, hive-hub), not the tuna-os/hive fork.
The configuration described below was subsequently superseded by the
HiveRelease operator controller before the cluster was terminated.
| Hive | Namespace | Branding |
|---|---|---|
| hive.reilly.asia (canary) | hive-hanthor | none |
| reef.tunaos.org | hive-reef | REEF |
| school.tunaos.org (legacy alias hive.tunaos.org) | hive | SCHOOL |
| hub.tunaos.org | hive-hub | — |
At that time the hive-upgrade CronJob checked upstream v5 releases daily,
upgrading one Hive at a time with a soak and rollback. It was later deleted in
favor of the operator's HiveRelease controller. The retained
upgrade history
is not an active upgrade runbook.
The old hive-fork-{switch,verify,drift,ai-check} CronJobs had been suspended;
they do not run on the terminated AWS cluster.
sshd
Tags: system, sshd
Secrets needed: No
Runs on: All machines
Configures the SSH daemon to accept Bitwarden environment variables from
clients that explicitly use SSH environment forwarding. The current
just apply-remote recipe does not depend on this drop-in: it resolves the
session on the target and exports it inside the remote command.
What It Does
Drops a config file at /etc/ssh/sshd_config.d/99-bw-env.conf:
AcceptEnv BW_SESSION BW_CLIENTID BW_CLIENTSECRET
This drop-in is compatibility for explicit SendEnv/AcceptEnv workflows. It
is not required by just apply-remote, which uses target-side resolution
instead of SSH environment forwarding.
Handlers
- Reload sshd — restarts the SSH service after config changes
Notes
- Only runs if
/etc/ssh/sshd_config.d/exists (Debian/Ubuntu/Fedora convention) - On Alpine/musl systems this directory may not exist — the role skips gracefully
sudo
Tags: system, sudo
Secrets needed: No
Runs on: All machines
Grants passwordless sudo to the current user and cleans up old sudoers entries.
What It Does
- Writes a sudoers drop-in at
/etc/sudoers.d/zz-<user>grantingNOPASSWD:ALL - Validates the file with
visudo -cfbefore deploying - Removes old/conflicting sudoers files from previous naming conventions
Notes
- Required for unattended playbook runs — many tasks use
become: true - The
zz-prefix ensures this file loads last, overriding any earlier restrictions
apk_packages
Tags: system, packages, apk
Secrets needed: No
Runs on: All machines (skips if not musl/Alpine)
Installs core packages on Alpine/musl-based systems where Homebrew is unavailable.
What It Does
- Detects musl libc (Alpine, postmarketOS) — skips entirely on glibc systems
- Installs core CLI tools via
apk: bash, fish, zsh, git, neovim, tmux, ripgrep, fzf, jq, podman, etc. - Installs Bitwarden CLI via npm (not available in apk repos)
- Enables podman and podman.socket system services
- Adds the user to the
podmangroup
When It's Skipped
On glibc systems (Fedora, Debian, Ubuntu, Bluefin), this role does nothing — Homebrew handles all package management.
server_hardening
Tags: system, hardening, security
Secrets needed: No
Runs on: server and vps groups (site.yml gates on group membership)
Applies reliability and light security hardening for long-running server hosts.
What It Does
- journald: sets
SystemMaxUse=500Mand vacuums journals to that size - swap: creates a 2G
/swapfileif missing, sets 0600, runsmkswap+swapononly when it isn't already active, adds it to/etc/fstab - NTP: enables and starts
systemd-timesyncd - SSH rate limits:
MaxStartups 3:50:10,MaxSessions 10,ClientAliveCountMax 3insshd_config(reloads sshd) - UFW: turns logging on if UFW is already active (doesn't enable UFW or add rules)
- Livepatch: runs
pro enable livepatchif Ubuntu Pro is attached (best-effort) - Reboot check: prints a warning if
/var/run/reboot-requiredexists
Notes
- Does not install fail2ban, change SSH auth/cipher settings, set sysctls, or configure automatic updates
- No
skip_flag — to keep a host out, take it out ofserver/vps - The retired VPSes (
matrix,telengana) are still invps; don't apply to them (see the cutover runbook) — this role reloads sshd and restarts journald
fleet_facts
Publishes each machine's public device facts for the live fleet view at reilly.asia/infra.
- Runs on: every host (opt out with
fleet_facts_publish: false), last in the play, including the no-secrets daily timer - Tags:
facts,fleet_facts - Needs: a logged-in
ghCLI (skipped quietly otherwise)
Each run writes facts/<host>.json to the fleet-facts branch of this
repo through the GitHub contents API. There's no clone and no merge, and
master isn't touched. The Worker behind reilly.asia/infra reads those files.
| Field | Source |
|---|---|
os.{name,version,codename,kernel} | Ansible distribution facts |
hardware.model | Android getprop, else device-tree model (Pi), else DMI product name (placeholders like "System Product Name" dropped) |
hardware.{vendor,form_factor,arch,cpu,cpu_threads,memory_gb} | Ansible hardware facts |
hardware.gpus | lspci -mm display devices (unnamed virtual adapters dropped) |
virtualization, uptime_days, last_apply | Ansible facts, ~/.cache/dotfiles/last-apply.json |
Never published: IP/MAC addresses, serial numbers, usernames, anything from Bitwarden.
Private drift monitoring (osquery)
The separate osquery role watches changes that must not go to the public
fleet-facts branch. It installs checksum-pinned upstream osquery 5.23.1 under
/opt, including on immutable desktops: no package layering, writable /usr,
reboot, Fleet server, enrollment token, or new network listener.
Tags: osquery, services, security. Requires root, glibc Linux, systemd,
and x86_64 or aarch64. Nura/Alpine musl (Kerala) and unrooted Android/Termux are
unsupported; osquery_supported: false records that exclusion. Offline hosts
install on their next successful normal apply. skip_osquery: true skips the
role; it does not uninstall an already-running daemon. The retired VPS group is
never modified, and Talos nodes are not Ansible-managed.
What's watched
- TCP listeners and bound UDP ports: address, port, protocol, executable path. PIDs are excluded so a process restart alone does not create drift. UDP client sockets and temporary development listeners can legitimately change.
- Enabled system startup triggers: service, timer, socket and path symlinks in
/etc/systemd/system/*.wants,*.requiresand their/runequivalents. This includes enabled-but-not-started units, without mistaking ordinary active/inactive transitions for startup changes. User units and vendor dependencies compiled into unit files are outside this query. - Kernel, OS and osquery versions.
osqueryd.service samples every five minutes, writing differential JSON logs
only under /var/log/osquery (10 MiB rotation, three retained archives).
fleet-osquery-report.timer refreshes a private baseline comparison every
15 minutes. Both have bounded CPU/memory use; audit/event capture and distributed
queries are disabled. Sampling is not real-time intrusion detection or unit-file
content integrity checking.
Review the fleet
From an updated checkout:
just apply-tags osquery
just osquery-fleet
just osquery-fleet --host punjab --host dilli
just osquery-fleet --json
The controller uses effective Ansible inventory, Tailscale online status and
existing authenticated SSH plus noninteractive sudo. Host-key verification stays
enabled. It shows ok, drift, offline, unsupported, excluded, stale or
error; missing/bad reports and failed reads never become ok. Reports older
than 30 minutes are stale. Stale/errors return nonzero; ordinary drift and offline
coverage do not. Each result shows its collection time, not a claim of live state.
--inventory PATH (or DOTFILES_INVENTORY) selects another inventory.
Always-on poller: Punjab
Punjab has osquery_poller_enabled: true. Its system-level
fleet-osquery-poll.timer runs every 15 minutes at :02/:17/:32/:47, after the
hosts' report-refresh window, and two minutes after boot. It runs as ubuntu,
using that account's existing SSH keys, known hosts and noninteractive sudo.
It does not depend on an interactive SSH agent or copy credentials into Kubernetes.
The role installs a root-managed collector and a minimal inventory under
/opt/fleet-osquery and /etc/fleet-osquery/inventory.yml. Only connection and
exclusion metadata is copied—not the full effective inventory or Bitwarden facts.
Apply the osquery role on Punjab after changing fleet membership or connection
details to refresh this inventory snapshot.
Each poll atomically replaces one aggregate:
/home/ubuntu/.local/state/fleet-osquery/latest.json (0600, inside a 0700 directory,
owned by ubuntu). No unbounded history is kept. Journald gets coverage counts,
not raw addresses or executable paths. Polling never accepts host baselines.
ssh ubuntu@punjab 'cat ~/.local/state/fleet-osquery/latest.json'
ssh ubuntu@punjab 'systemctl status fleet-osquery-poll.timer --no-pager'
ssh ubuntu@punjab 'sudo -n systemctl start fleet-osquery-poll.service'
Offline machines remain explicitly offline. Failed reads/stale reports are saved as degraded coverage and fail the service; an unusable or empty inventory saves a timestamped error instead of retaining a supposedly healthy snapshot. If the process itself cannot run or write, check the service journal and the snapshot's collection time—an old file is not current health.
osquery_poller_enabled defaults to false on other hosts. Setting it false on a
previous poller and applying the role stops its timer and in-flight service;
the last private snapshot remains for inspection. The manual fleet CLI still works
from other machines and supports --output PATH for explicit private snapshots.
Baseline and local investigation
First apply records the machine's existing state, not an audited known-safe state. Later applies never accept new drift automatically. Changes remain pending against that baseline until reviewed and explicitly accepted:
sudo systemctl start fleet-osquery-report.service
sudo /opt/fleet-osquery/fleet-osquery report
sudo journalctl -u fleet-osquery-report.service
# Only after reviewing the changes:
sudo /opt/fleet-osquery/fleet-osquery baseline --accept
sudo systemctl start fleet-osquery-report.service
Baseline/report/lock files are root-owned mode 0600 in
/var/lib/fleet-osquery (0700). Failed collection does not replace the last
successful report or baseline; inspect the failed service and its journal, and
treat an aging cached sample accordingly. Manual CLI runs do not save fetched
reports unless --output is supplied; Punjab's scheduled aggregate is described
above. Keep all report output private: addresses and executable paths are
sensitive; do not attach raw JSON to public issues or publish it through fleet_facts.
No command lines, process environments, credentials or serial numbers are queried.
Check mode reports platform support only; actual installation and baseline
collection require a normal apply. CI exercises the actual role and
sudo python3 tests/osquery_smoke.py on x86_64 and aarch64 Ubuntu runners.
That smoke creates a temporary loopback listener and an enabled, never-started
unit, checks pending/accepted/removed drift, and compares a synthetic prior kernel
against the real current kernel. It uses isolated state and removes its fixtures;
the fleet baseline is untouched.
The native CI jobs also enable the persistent poller and run
tests/osquery_poller_smoke.py: a real temporary listener must appear in the
aggregate, disappear after removal, and leave the host baseline unchanged.
retired_services
Tags: system, retired_services
Secrets needed: No
Runs on: every systemd host except the vps group and Termux (site.yml gates on vps membership, is_termux and service_mgr)
Deleting a role from the repo stops the fleet from deploying its services, but nothing ever removed what it had already installed — so retired podman quadlets (Prometheus, Grafana, Cockpit, …) kept running on laptops long after their roles were gone (#170). This role removes those leftover unit files.
What It Does
For every unit file in its lists that exists on the machine:
- stops the service generated from it — quadlet service names are the file stem
(
foo.container→foo.service,foo.volume→foo-volume.service,foo.network→foo-network.service); plain.service/.timerfiles are also disabled - deletes the unit file
- runs one
daemon-reload(system and/or user, whichever scope had leftovers)
On a machine with none of the files, everything after one find over the unit
directories is skipped — the role reports no changes.
Only unit files are removed. Podman volumes (the data of Prometheus,
Grafana, NetBox's Postgres, …), images, and config directories such as
/etc/cockpit, /etc/homepage, /etc/netbox and ~/.config/monitoring are
left alone; delete them by hand once you're sure nothing is wanted
(podman volume ls, sudo podman volume ls).
Units
| Role | Removed in | Scope | Unit files |
|---|---|---|---|
cockpit | 2af694a | system (/etc/containers/systemd/) | cockpit.container |
homepage | f0143a6 | system | homepage.container |
netbox | 7f3c122 | system | netbox.container, netbox-worker.container, netbox-redis.container, netbox-postgres.container, netbox-postgres.volume, netbox-media.volume, netbox.network |
netbox_agent | 7f3c122 | system (/etc/systemd/system/) | netbox-agent.timer, netbox-agent.service |
lima | f0143a6 | user (~/.config/containers/systemd/) | lima-bootc.container |
monitoring | 8c9b171 | user | node-exporter, podman-exporter, alloy, promtail, prometheus (also the hub variant), perses, grafana, loki, alertmanager, pve-exporter, matrix-notifier .container; alloy-data, prometheus-data, perses-data, grafana-data, loki-data, alertmanager-data .volume; monitoring.network |
appflowy | f7f2a2a | user | appflowy-cloud, appflowy-gotrue, appflowy-s3, appflowy-minio, appflowy-redis, appflowy-db .container; appflowy-s3-data, appflowy-minio-data, appflowy-db-data .volume; appflowy.network |
authentik | f7f2a2a | user | authentik-server, authentik-worker, authentik-redis, authentik-db .container; authentik-db-data, authentik-media, authentik-certs, authentik-custom-templates .volume; authentik.network |
n8n | f7f2a2a | user | n8n-server, n8n-db .container; n8n-data, n8n-db-data .volume; n8n.network |
caddy.container is not on the list — the proxy role still deploys it.
Variables
| Variable | Purpose |
|---|---|
retired_services_system_units | Absolute paths of rootful quadlets / system units to remove |
retired_services_user_units | File names under ~/.config/containers/systemd/ to remove |
Notes
- When you delete a role that installed units, add them here in the same PR.
- User-scope
systemctlgetsXDG_RUNTIME_DIR=/run/user/<uid>pinned so it works from shells that don't export it. - No
skip_flag — the role is a no-op wherever the files are absent.
homebrew
Tags: packages, homebrew
Secrets needed: No
Runs on: All glibc machines (skipped on musl/Alpine)
Installs and manages Homebrew packages on Linux.
What It Does
- Installs Homebrew to
/home/linuxbrew/.linuxbrewif missing - Generates a
Brewfilefromgroup_vars/all.ymlpackage lists - Runs
brew bundleto install/update all packages - Ensures Python
packagingmodule is available (required by Ansible's systemd module)
Package Categories
| Variable | Where Defined | Scope |
|---|---|---|
core_brews | group_vars/all.yml | All machines |
core_tap_brews | group_vars/all.yml | All machines (tapped) |
desktop_brews | group_vars/all.yml | Desktop group only |
Key Packages
- Shells: fish, zsh, bash-preexec
- CLI tools: neovim, tmux, direnv, just, make, ripgrep, fzf, eza, bat, fd, zoxide
- Bluefin extras: atuin, starship, carapace, tealdeer (via
bluefin-cli) - Dev tools: go, kubernetes-cli, helm, k9s, podman-compose, ansible, uv
- AI/ML: ollama, ramalama, openai-whisper, pi-coding-agent
Notes
- Homebrew on Linux installs to
/home/linuxbrew/.linuxbrew— no root needed - The playbook adds
brew_bintoPATHfor all subsequent tasks
flatpak
Tags: packages, desktop, flatpak
Secrets needed: No
Runs on: Desktop group only (skip_flatpak: false)
Installs and manages Flatpak applications from Flathub.
What It Does
- Fixes broken Flatpak symlinks if present
- Adds the Flathub remote
- Removes unwanted system flatpaks (Firefox, Thunderbird — replaced by Zen)
- Installs missing flatpaks in batch (fast path), falling back to individual installs
- Updates all flatpaks when
upgrade: trueis set - Creates Chromium wrapper scripts for CLI compatibility
Package List
Defined in roles/flatpak/vars/main.yml as system_flatpaks (the Flathub remote is in flatpak_remotes, also set in group_vars/all.yml). Key apps:
| App | Flatpak ID |
|---|---|
| Zen Browser | app.zen_browser.zen |
| VS Code | com.visualstudio.code |
| Ungoogled Chromium | com.github.Eloston.UngoogledChromium |
| Flatseal | com.github.tchx84.Flatseal |
| GNOME apps | Calculator, Calendar, Clocks, Contacts, Loupe, etc. |
| Vesktop | dev.vencord.Vesktop |
| Obsidian | md.obsidian.Obsidian |
Wrapper Scripts
Creates ~/.local/bin/chromium, ~/.local/bin/google-chrome, and ~/.local/bin/chromium-browser that exec into the Ungoogled Chromium flatpak — tools that hardcode google-chrome or chromium-browser still work.
Notes
- Installs system-wide (
--system) withbecome: true - Batch install tolerates partial failures — individual retries catch stragglers
termux_packages
Tags: system, packages, termux
Secrets needed: No
Runs on: termux_hosts group only (the raw Termux/Android layer on the phone)
Package and sshd management for Termux, which has no root, no systemd, no Homebrew and bionic libc.
What It Does
pkg update -y(retried 3×), thenpkg install -ythetermux_pkgslistnpm install -gthetermux_npm_globallist (the pi coding agent)- Deploys
~/.termux/boot/start-sshd(termux-wake-lock+sshd) for Termux:Boot - Starts
sshdif it isn't running - Schedules the same script as a persisted periodic job via
termux-job-scheduler(job id 1), so sshd comes back after Android kills Termux
Configuration
roles/termux_packages/defaults/main.yml:
termux_pkgs— openssh, git, python, nodejs, fish, vim, nano, termux-api, curl, unzip, plusfileande2fsprogs(Ansible's stat/copy modules need them)termux_npm_global—@earendil-works/pi-coding-agenttermux_keepalive_period_ms—1800000(30 min; Android's floor is 15)
Notes
- Requires the Termux:Boot and Termux:API companion apps, installed separately (F-Droid/GitHub)
- Claude Code, Codex etc. ship no android-arm64 binary — they live on the
mumbaiphone VM instead - Connection quirks (remote tmp, scp, bash as shell executable) are pinned in
host_vars/termux.yml
shell_fonts
Tags: dotfiles, shell, fonts
Secrets needed: No
Runs on: All machines (font install only when is_desktop)
Installs the JetBrains Mono Nerd Font for terminals and the prompt.
What It Does
- Ensures
~/.local/share/fonts/exists (every host) - On desktops, if
JetBrainsMono/JetBrainsMonoNerdFont-Regular.ttfis missing: downloads the latestJetBrainsMono.ziprelease, unpacks it into~/.local/share/fonts/JetBrainsMono/, and runsfc-cache -f
Notes
- Idempotent on the presence of the Regular
.ttf— delete it to force a re-download - The
gnomerole sets Ptyxis toJetBrainsMono Nerd Font 12, which depends on this
shell_dotfiles
Tags: dotfiles, shell
Secrets needed: No
Runs on: All machines
Deploys the shell, prompt, multiplexer and terminal configs.
What It Does
~/.config/shell/aliases.sh(bash/zsh) and~/.config/fish/conf.d/aliases.fish— shared aliases, templated~/.bashrc,~/.zshrc,~/.zprofile— templated~/.config/fish/config.fish,~/.tmux.conf,~/.inputrc,~/.config/starship.toml— copied as-is- Ghostty terminfo: ships
xterm-ghostty.terminfoand compiles it into~/.terminfowithtic, so SSH sessions from Ghostty don't break ncurses apps. Warns and skips ifticisn't installed. ~/.config/ghostty/config— auto light/dark following GNOME (inert on headless hosts)
Fleet aliases
| Alias | Expands to |
|---|---|
dots | cd ~/.local/share/dotfiles && just apply-nosecrets (git pull + --skip-tags secrets) |
dots-apply | cd ~/.local/share/dotfiles && git pull --ff-only && just apply (full, with BW unlock) |
dots-edit | open the repo in $EDITOR |
In fish these are abbreviations (abbr), not aliases.
Notes
- Shell secrets (
~/.config/shell/secrets.sh,~/.config/fish/conf.d/secrets.fish) are written byshell_ai, not here - Source files:
roles/shell_dotfiles/files/androles/shell_dotfiles/templates/
shell_atuin
Tags: dotfiles, shell, atuin, secrets
Secrets needed: Yes, for login (atuin.sh from Bitwarden)
Runs on: All machines
Configures Atuin shell history and logs it in to sync.
What It Does
- Writes
~/.config/atuin/config.toml— sync toapi.atuin.shevery 5m, fuzzy search,filter_mode = "global"(including the Up-arrow binding) - Checks
atuin status; if already logged in, skips the Bitwarden round-trip - Otherwise, when
secrets_available, reads theatuin.shitem (username, password, and a custom fieldkeyholding the mnemonic) and runsatuin login -u … -p … -k … - Runs a best-effort
atuin sync - Imports existing shell history once (
atuin import auto, sentinel~/.local/share/atuin/.history_imported)
Notes
- Skips login with a warning if the
atuinbinary is missing (it's incore_brews) - The mnemonic is passed to
atuin login -k, never written to~/.local/share/atuin/keydirectly — atuin expects that file to be a binary key - Don't set the Up-arrow filter to
session: it looks broken in fresh terminals
shell_ai
Tags: dotfiles, shell, ai, secrets
Secrets needed: Yes (deepseek-api-key, forgejo, tavily-api-key)
Runs on: All machines
Deploys API credentials for the AI coding tools and shell.
What It Does
- When
secrets_available:bw get password deepseek-api-key→DEEPSEEK_API_KEYin shell secret filesbw get notes forgejo(theAPI Token:line) →FORGEJO_TOKENbw get password tavily-api-key→TAVILY_API_KEY- Writes these exports to
~/.config/shell/secrets.sh(bash/zsh) and~/.config/fish/conf.d/secrets.fish(fish), both 0600
- Warns if
deepseek-api-keyis missing; Kiro OAuth remains a per-user OMP sign-in
Notes
- A locked vault leaves any previously written secret files untouched — nothing is overwritten with empty values
TAVILY_API_KEYis read by thetavily-searchskill deployed by theomprole
omp
Tags: dotfiles, omp
Secrets needed: No for OMP/Kiro installation; shell_ai optionally provides the DeepSeek API key
Runs on: All supported hosts except raw Termux
Installs Oh My Pi and the Kiro provider on each supported computer. Homebrew hosts install OMP and Bun from the can1357/tap tap. musl hosts use OMP's checksum-verified binary installer and Bun's installer. The Kiro provider is installed from github:elkibalion/omp-kiro-provider.
What It Does
- Installs and verifies
omp. - Installs the Kiro provider plugin for the current user; it is available as
kiro/*models. - Deploys personal skills to
~/.omp/agent/skills/james/and symlinks them into~/.claude/skills/. - Exposes the Bitwarden DeepSeek key as
DEEPSEEK_API_KEYthrough the shell secret files, not an agent-specific credential file. - Removes the legacy Pi CLI from raw Termux. OMP has no native Android/Bionic build; the Debian VM (
mumbai) is supported normally.
Skills and the package manifest
The playbook copies roles/pi/files/skills/ to
~/.omp/agent/skills/james/ and links those skill directories into
~/.claude/skills/. The roles/pi path is skill source data, not a separate
Ansible role.
roles/pi/files/package.json is tracked by Renovate and protected by
deps-guard; it is not installed by this role. OMP plugin installation is
handled separately by the role's explicit plugin tasks.
Kiro authentication
Plugin installation is unattended. OAuth is per user and must be completed on each computer:
omp
/login kiro
Complete the browser sign-in. Then select kiro/auto with /model kiro/auto. The playbook does not set Kiro as the default model because the OAuth session is not present until a user signs in.
Package source
The OMP and Bun Homebrew formulas are in can1357/tap; both are core packages for Homebrew-supported hosts. On musl, the official OMP binary installer verifies its SHA-256 and requires libgcc and libstdc++. Raw Termux receives neither OMP nor the obsolete Pi npm package.
hive_ops
Tags: hive, hive_ops
Default: hive_ops_enabled: false
Credentials: no Bitwarden lookup in this role; its Kubernetes kubeconfig contains credentials.
Retired AWS-cluster integration, not the current Hive operations guide. The AWS Talos cluster was terminated on 2026-10-25. The current school and reef spokes and hub run as Docker Compose services on the bihar (karnataka standby). Do not enable this role or run its old Kubernetes helpers against that deployment.
The role still exists in the repository. When enabled it installs host user
units and scripts that use kubectl against ~/.kube/config-aws-migration by
default; they do not manage the Compose services. Historically these units ran
on telengana against its local k3s, then on himachal against AWS. The cluster
CronJobs and operator controllers that later replaced parts of this work are
also historical: the AWS cluster no longer exists.
Observed and configured state
host_vars/himachal.yml sets hive_ops_enabled: false. On 2026-10-27,
read-only SSH to james@himachal-1 found all seven role timer units
not-found and inactive. This observation is not a fleet-wide timer audit;
a host-vars value alone does not establish a host's live state.
When disabled, the role's cleanup runs only if
~/.config/systemd/user/hive-watchdog.timer exists. In that case it attempts
to stop/disable the seven timers and remove their units, drop-ins and helper
scripts. If the sentinel is absent, the cleanup block does not run; do not
assume that it removed every possible leftover. The role was not applied as
part of this documentation update.
Historical unit inventory
The checked-in unit files describe the old DeepSeek-era setup, not the current
claude-backend configuration on Spindle. Their schedules are:
| Timer | Checked-in schedule | Historical purpose |
|---|---|---|
hive-rotate | 10 min after boot, then 20 min after activation | Rotate backends based on provider headroom |
hive-watchdog | 3 min after boot, then every 5 min | Probe and heal agents |
hive-tiers | Monday 04:30 UTC, up to 30 min random delay | Refresh model tiers |
hive-peak-pause / hive-peak-resume | Weekdays 01:00 and 06:00 / 04:00 and 10:00 UTC | Pause/resume around DeepSeek peak windows |
pi-peak-stop / pi-peak-start | Weekdays 01:00 and 06:00 / 04:00 and 10:00 UTC | Stop/start local pi.service around those windows |
When enabled, the role installs hive-peak.sh, hive-rotate.sh, and
hive-tiers.sh and writes service drop-ins for KUBECONFIG and PATH.
The drop-ins made Kubernetes and Homebrew tools available to the systemd user
manager; they do not make the legacy scripts suitable for the VPS.
Variables retained in the role
| Variable | Default | Historical purpose |
|---|---|---|
hive_ops_enabled | false | Opt a host in; multiple operators could conflict |
hive_ops_kubeconfig | ~/.kube/config-aws-migration | AWS cluster credentials |
hive_ops_path | Homebrew plus system paths | Find kubectl and jq in user services |
hive_ops_timers | Seven timers above | Units deployed or cleaned up |
git
Tags: dotfiles, git
Secrets needed: No
Runs on: All machines
Deploys Git configuration with SSH commit signing.
What It Does
- Deploys
~/.gitconfigwith name, email, and signing preferences - Configures SSH commit signing using
~/.ssh/id_ed25519 - Deploys
~/.ssh/allowed_signersfor signature verification - Deploys GitHub CLI config at
~/.config/gh/config.yml
Key Settings
- Signing: SSH (
ssh-ed25519key) for all commits - Allowed signers: All fleet machine keys from Bitwarden
- Delta: Git diff viewer (
git-delta) for side-by-side diffs
neovim
Tags: dotfiles, neovim, editor
Secrets needed: No
Runs on: All machines
Deploys Neovim configuration.
What It Does
- Creates
~/.config/nvim/directory structure - Deploys init.lua and plugin configuration
- Installs Lazy.nvim plugin manager and configured plugins
Key Plugins
- LSP: nvim-lspconfig, mason.nvim (language server management)
- Completion: nvim-cmp with LuaSnip snippets
- UI: Catppuccin theme, lualine statusline, nvim-tree file explorer
- Editing: treesitter, comment.nvim, autopairs
- Git: gitsigns (inline git blame/diff)
- Navigation: telescope.nvim (fuzzy finder)
bitwarden
Tags: secrets, bitwarden
Secrets needed: Yes (provides secrets to other roles)
Runs on: All machines
Resolves a Bitwarden session, refreshes the local vault cache, and publishes a single secrets_available host fact that every other BW-using role consumes. Failure to unlock is never fatal to the play; it's reported once at the end and downstream BW operations are skipped cleanly.
What it does
- Resolves
BW_SESSIONin this order: environment →/tmp/bw_sessioncache → interactivebw unlock. - Runs
bw syncwith a hard 15 s timeout (so a flaky vault server can't hang the whole play). - Calls
bw statusand setssecrets_available(boolean host fact) —trueonly ifbwis installed, a session was resolved, and the vault actually reportsunlocked. - If a session is set but the vault is still locked, emits a one-line debug warning so the operator knows secrets work is being skipped here. On a remote apply, this means target-side session resolution did not unlock the target vault.
Every BW-using role downstream gates on when: secrets_available | default(false). The end-of-play post_tasks summary then prints one of three messages:
| State | Message |
|---|---|
| unlocked | vault unlocked — all secret roles executed normally |
| locked | Bitwarden vault was LOCKED — … skipped. To enable: bw unlock … |
--skip-tags secrets | SKIPPED (--skip-tags secrets). Run dots-apply to include … |
Remote applies
just apply-remote <host> (and just apply-remote-tags <host> <tags>) resolve
the Bitwarden session on the target host; they do not forward the local
host's BW_SESSION:
scripts/bw-resolve.sh remote <host>unlocks Bitwarden on the target and prints that host's session token. If the remote CLI is logged out, it tries the configured API-key and master-password paths, then falls back to an interactive unlock overssh -t.- The recipe exports the returned
BW_SESSIONinline in the remote command and caches it in the target's/tmp/bw_session. - The remote
bitwardenrole runsbw status; when it reportsunlocked,secrets_availablebecomes true and secrets work proceeds.
A session token is valid only on the host that unlocked it. If target-side resolution fails, the remote vault remains locked and BW-using tasks skip cleanly. See the full remote apply flow.
How to verify it ran correctly
just doctorwill showvault unlockedif BW is good on this host.- Check the end-of-play summary line printed by every
just apply*. - Look for any task in
ansible-playbookoutput with a[bitwarden : …]prefix that ended infailed:— should never happen; only "skipping" is acceptable.
Notes
- Must run before any other secrets role (it does, by position in
site.yml). - The cached
/tmp/bw_sessionpersists for the user across reboots only as long as/tmpsurvives (most modern systemd boots clear it). The role re-resolves the session on every play. bw syncfailure is non-fatal; downstream operations use the locally cached vault, which may be stale.
ssh_keys
Tags: secrets, ssh, ssh_keys
Secrets needed: Yes
Runs on: All machines
Manages per-machine ed25519 SSH keys, distributes the fleet's public keys via authorized_keys, and maintains the allowed_signers list for git commit verification.
Source of truth is the local disk. Bitwarden is the durable mirror that lets new/rebuilt machines recover and lets the rest of the fleet stay in sync.
What it does
For the local machine's key (james@<hostname> in Bitwarden):
- No key on disk → fetch from BW; if absent there too,
ssh-keygena fresh ed25519 pair. - Local pub key matches BW → no-op.
- Local pub key differs from BW → leave the Bitwarden item unchanged and warn by default. An intentional rotation must opt in with
dots-apply -e ssh_keys_force_push=true; only then is the BW item replaced so every other host sees the current key on its next apply. This gate prevents a key generated while the vault was locked from clobbering the stored key.
Then, regardless of whether the local item changed:
- Slurps every other
james@<machine>SSH item from BW in a singlebw listcall. - Rewrites
~/.ssh/authorized_keyswith the union of those +extra_authorized_keys. - Refreshes
~/.ssh/known_hostsfor every fleet host viassh-keyscan, removing stale entries first. - Updates
~/.ssh/allowed_signerswith this host's pub key for git commit signing.
If secrets_available is false (vault locked, BW not installed, etc.), every BW-touching step is skipped and authorized_keys is not rewritten — preserving the existing file rather than wiping it with an empty list.
Failure modes
| Symptom | Likely cause | Fix |
|---|---|---|
Permission denied (publickey) from a host that worked yesterday | Local key changed but BW still has the old pub key | For an intentional rotation, run dots-apply -e ssh_keys_force_push=true on the source host with BW unlocked, then run dots-apply on the failing host. If the mismatch was accidental, restore the Bitwarden key locally instead of forcing it. |
Some hosts missing from authorized_keys | Those hosts never round-tripped their key to BW | Run dots-apply on each missing host once, with BW unlocked. |
| New ed25519 key generated on a host that should have inherited an existing key | BW item missing/empty for this hostname | Verify the BW entry name matches james@<hostname> exactly. |
How to verify
# Check this host's key is in BW
bw get item "james@$(hostname)" | jq '.sshKey.publicKey' | grep -F "$(awk '{print $2}' ~/.ssh/id_ed25519.pub)"
# Check who can SSH in here
cat ~/.ssh/authorized_keys
# Test SSH to every fleet host
for h in $(yq '.all.hosts | keys | join(" ")' inventory.yml); do
ssh -o BatchMode=yes -o ConnectTimeout=3 "$h" "echo OK" 2>&1 | head -1
done
Notes
- Every machine SSHes to every other machine without password prompts after one round-trip apply.
- The same key is used for GitHub auth + commit signing (registered by the
githubrole). - The role tolerates a locked vault and never destructively rewrites
authorized_keyswhen BW returned no data.
ssh_mesh
Self-healing passwordless SSH mesh across the fleet.
What it does
- Ensures the machine has an ed25519 keypair (generates if missing).
- Merges all fleet public keys from
machine_pubkeys(ingroup_vars/all.yml) into~/.ssh/authorized_keys— non-destructivelineinfile, so Bitwarden-managed keys and extras survive. - Warns if this machine's own public key is missing or stale in git.
- Probes passwordless SSH to every fleet machine currently online on
the tailnet:
scripts/online_hosts.py --peers(the same matcher asjust apply-all) gives the online MagicDNS names, each inventory host is matched by itstailnet_name(or its own name), then ∩machines−ssh_mesh_exclude− thevpsgroup. Port and login user come frominventory.yml(ansible_port,ansible_user). - On failure: reaps stale ControlMaster sockets, refreshes the host key
in
known_hosts(rebuilt machines), and retries. - Fails the apply if the mesh is still broken, naming the unreachable hosts.
Why it's self-healing
- Public keys live in git, not only in Bitwarden — safe to publish,
and every apply replants them. A machine missing a peer's key heals on
its next
just applyor the dailydotfiles-update.timer(apply-nosecrets), no vault unlock needed. - The role is tagged
[ssh, mesh_check]but notsecrets, so the daily no-secrets timer runs it. - Probes bypass SSH multiplexing (
ControlMaster=no) so a hung mux socket can't fake an outage, and are capped withtimeout.
Usage
just mesh-check # verify + heal from this machine
just apply-remote-tags HOST mesh_check # verify + heal from HOST
Adding a machine to the mesh
Add its public key to machine_pubkeys in group_vars/all.yml and its
entry to the machines dict. If its MagicDNS name differs from its
inventory name, set tailnet_name on it in inventory.yml. Machines whose
sshd isn't always reachable (phones, port-forwarded test VMs) go in
ssh_mesh_exclude: a failed probe fails the apply.
Relationship to ssh_keys
ssh_keys (tag secrets) still round-trips private keys through
Bitwarden and rebuilds authorized_keys from the vault when unlocked.
ssh_mesh is the vault-free safety net + verification layer on top.
github
Tags: secrets, github
Secrets needed: Yes (only on first auth — once gh is authenticated, BW is not required)
Runs on: All machines with gh installed
Logs gh in via a BW-stored PAT (only if not already authenticated), then registers this host's SSH key with GitHub for both auth and signing.
What it does
- Ensures
github.com's ed25519 host key is in~/.ssh/known_hosts(prevents MITM warnings on first push). gh auth status— if already authenticated, skip the token fetch entirely (no BW dependency for steady-state).- If not authenticated and BW is unlocked →
bw get password github-token→gh auth login --with-token. If the BW item is missing, emits a one-line warning and continues. - Sets
gh config set git_protocol ssh. - Lists existing keys on the GitHub account. For each key type (auth + signing), runs
gh ssh-key addif this host's key isn't already there.
Failure modes
| Symptom | Cause | Fix |
|---|---|---|
gh auth status says not logged in after apply, vault locked | First run on this host with BW locked | Unlock BW and re-run just apply-tags github, or gh auth login manually |
Warning: No 'github-token' item found | BW vault doesn't have the PAT | Create a BW login item named github-token with your PAT as the password |
| Key registration silently fails | gh ssh-key add returned non-zero (failed_when:false) — possibly key was just deleted on github.com side | Inspect with gh ssh-key list and re-run the role |
| Stale keys clutter github.com | This host's previous key was rotated; the old gh entry stays forever | Delete the old key manually at https://github.com/settings/keys — no automated cleanup yet |
How to verify
gh auth status # Should say "Logged in" with this account
gh ssh-key list | grep "$(hostname)" # Should show <hostname>-auth and <hostname>-signing
git log --show-signature -1 # Latest commit should show "Good \"git\" signature for ..."
Notes
- Depends on the
ssh_keysrole running first (the key must exist on disk). - The signing key registration enables the "Verified" badge on commits — only if your local
~/.gitconfigis also set to sign with the SSH key (thegitrole handles this). - The PAT in BW needs
admin:public_key,write:gpg_key, andread:userscopes for the registrations to succeed.
tailscale
Tags: secrets, tailscale
Secrets needed: Yes (only to join — once connected, BW is not required)
Runs on: All machines
Installs Tailscale, fetches a reusable auth key from BW, joins the network, configures DNS, and keeps device hostname + advertised routes in sync.
What it does
- Configures
systemd-resolvedso MagicDNS works (link/etc/resolv.confto the stub resolver, drop inDNSStubListenerExtra=::1for container DNS). - Sets
net.ipv4.conf.{all,default}.rp_filter=2and IP forwarding on, so exit-node and subnet-router traffic actually routes. - If
tailscale statusreports connected → updates hostname, prefs, operator, systray, then exits the role. - If not connected and the vault is unlocked → fetches
tailscale-apikeyfrom BW, deletes any stale device with this hostname via the Tailscale API, thentailscale up --authkey …. - If not connected and the vault is locked → emits a one-line warning and ends the role. Joining without a key would either fail noisily or, worse, succeed with an empty key.
Failure modes
| Symptom | Cause | Fix |
|---|---|---|
| Role aborts with "vault locked" warning, host is not on tailnet | First-time apply on this host without BW unlocked | Unlock BW (`bw unlock --raw |
| Tailscale connected but MagicDNS names don't resolve | systemd-resolved link to /etc/resolv.conf was overwritten by NetworkManager / DHCP client | The role re-applies it; just re-run dots-apply. To make it sticky, check network-manager settings (Bluefin does this correctly out of the box) |
| Duplicate device entries in admin console after rebuild | API-key device cleanup didn't run (vault locked at the time) | Manually delete in tailscale admin console or unlock vault and re-apply |
| Joins but no routes advertised | tailscale_advertise_routes not set in host_vars | Add it: tailscale_advertise_routes: "<lan-cidr>" |
How to verify
tailscale status # should show all fleet machines as peers
tailscale ip -4 # this host's 100.x.x.x
host bihar.manatee-basking.ts.net # MagicDNS resolves to a 100.x.x.x
Notes
- All fleet machines are on the
manatee-basking.ts.nettailnet. - The auth key is reusable — new machines join without re-authentication.
- The Tailscale API key (
tailscale-apikeyin BW) is the same item used for joining and for device cleanup; rotate by editing the BW item and re-running this role on each host.
kube
Tags: secrets, kube
Secrets needed: Yes
Runs on: Desktop group only (unless skip_kube: true)
Pulls kube + Talos configs from BW for the home cluster (Bihar + Karnataka — powered down, expected back) and the historical AWS cluster, terminated 2026-10-25. Neither cluster is currently reachable.
What it does
When secrets_available is true:
- Ensures
~/.kube/and~/.talos/exist (mode 0700). bw get notes kubeconfig→ writes to~/.kube/config(0600). Skipped if BW returns empty.bw get notes talosconfig→ writes to~/.talos/config(0600). Skipped if BW returns empty.- Pins
tls-server-nameon every cluster in~/.kube/configtokube_tls_server_name(default: bihar's LAN address) — the home API-server cert lacks the tailnet address in its SANs. - Emits a one-line warning if either home-cluster note is missing in BW.
bw get notes kubeconfig-aws-migration→~/.kube/config-aws-migrationandbw get notes talosconfig-aws-migration→~/.talos/config-aws-migration(0600, each skipped if empty). These historical AWS files remain separate from the home configs.
When the vault is locked the role no-ops cleanly — existing local configs are preserved.
Failure modes
| Symptom | Cause | Fix |
|---|---|---|
kubectl returns Unable to connect to the server after onboarding | Vault was locked, configs never deployed | Unlock BW, re-run just apply-tags kube |
Warning: kubeconfig missing in BW | Vault was never seeded | Run just seed-kube from a host that already has a working config |
| Configs deployed but auth fails | Cluster PKI rotated — local configs are stale | Re-run just seed-kube from a healthy host, then re-apply on consumers |
| Role skipped silently on a server host that should have it | skip_kube: true in host_vars/<host>.yml, or host not in desktop group | Adjust inventory or host_vars |
Seeding BW
# From a machine that already has working configs:
just seed-kube
This packages ~/.kube/config and ~/.talos/config into BW notes named kubeconfig and talosconfig. It does not seed the historical *-aws-migration notes. After seeding, every desktop's next dots-apply (or just apply-tags kube) pulls the home configs down.
How to verify
Both commands below require the home cluster to be powered on; they fail while it remains in storage. The former AWS cluster cannot be queried.
kubectl get nodes # home cluster: bihar + karnataka (fails while it's powered down)
talosctl --talosconfig ~/.talos/config get members
Notes
- These files contain cluster PKI material; the repo's
.gitignoreexcludes the cluster source files, and these caches live in~/only — never commit. - The role is gated by
is_desktop | bool and not skip_kube. Servers don't get a kubeconfig — the cluster nodes themselves are Talos and not in the inventory.
forgejo_registry
Tags: secrets, forgejo, registry
Secrets needed: Yes (Login item named after the registry host)
Runs on: All machines except machine_profile: vm-test (unless skip_forgejo: true)
Logs podman in to the private Forgejo container registry.
What It Does
When secrets_available and podman is on PATH:
- Checks
podman login --get-login {{ forgejo_registry }}— skips if already logged in - Fetches
bw get username/bw get passwordfor the item named{{ forgejo_registry }} - Runs
podman login … --password-stdin - Warns if the Bitwarden item is missing
Configuration
# roles/forgejo_registry/defaults/main.yml
forgejo_registry: forgejo.manatee-basking.ts.net
The Bitwarden Login item's name must equal this value; its password is a registry/package token.
Notes
- Forgejo runs on the home Talos cluster, which is currently powered down — a fresh
podman logincan't succeed until it returns. Hosts already logged in skip the step.
gnome
Tags: desktop, gnome
Secrets needed: No
Runs on: Desktop group only
Configures GNOME desktop environment: keybindings, terminal, avatar, extensions, and remote desktop.
What It Does
Keybindings
Deploys custom keyboard shortcuts via dconf:
- Custom keybindings for terminal, browser, file manager
- Workspace navigation and window management shortcuts
Ptyxis Terminal
Sets the default Ptyxis profile to use fish from Homebrew with JetBrains Mono Nerd Font at 12pt.
User Avatar
- Downloads avatar from
https://reilly.asia/profile.png - Installs to both AccountsService and GDM cache
- Shows on login screen and user menu
GNOME Extensions
Deploys and enables extensions from group_vars/all.yml:
| Extension | Purpose |
|---|---|
| Caffeine | Disable screen blanking |
| AppIndicator | System tray icons |
| Blur My Shell | Blur effects |
| Dash to Dock | macOS-style dock |
| GSConnect | Phone integration |
| PaperWM | Tiling window manager |
| PaperShell | PaperWM companion |
| Search Light | Spotlight-style search |
Remote Desktop
Enables GNOME Remote Desktop (RDP) via grdctl and starts the systemd user service.
Extension Sync
Extensions are synced omnidirectionally — the gnome_enabled_extensions list in group_vars/all.yml is the canonical list. To update it from a running desktop:
gsettings get org.gnome.shell enabled-extensions
zen_browser
Tags: desktop, browser, zen_browser
Secrets needed: No
Runs on: Desktop group only
Configures Zen Browser (Firefox-based) as the default browser.
What It Does
- Pins Zen Browser to the GNOME dock
- Removes Firefox and Thunderbird from the dock if present
- Sets Zen as the default web browser
Notes
- Zen is installed via Flatpak (
app.zen_browser.zen) - Firefox and Thunderbird are intentionally removed from the dock — Zen replaces both
browser_fxa
Tags: none of its own — pulled in by zen_browser via include_role, so it runs under desktop, browser, zen_browser
Secrets needed: Yes (Firefox Account credentials from Bitwarden)
Runs on: Desktop group only
Automates Firefox Account sign-in for Zen Browser (and optionally Firefox).
What It Does
- Fetches
accounts.firefox.comcredentials + TOTP from Bitwarden - Writes
signedInUser.jsonto the browser profile directory - This pre-authenticates the browser with Firefox Sync
Configuration
# group_vars/all.yml
fxa_bw_item: "accounts.firefox.com"
fxa_bw_username: "<your-fxa-email>"
fxa_browsers:
- zen
- firefox
- firefox-flatpak
Notes
- This only enables FxA sign-in, not the full Sync key bundle
- Opening the browser once after deployment may still be needed to complete Sync
- If credentials are unavailable, the task fails silently — manual sign-in works fine
bluefin_common
Tags: desktop, bluefin
Secrets needed: No
Runs on: Desktop group only (skipped on Bluefin hosts unless force_bluefin: true)
Syncs Bluefin branding, fonts, wallpapers, and GNOME extensions from the Project Bluefin OCI image.
What It Does
- Pulls the
ghcr.io/projectbluefin/common:latestcontainer image - Extracts Bluefin gsettings/dconf overrides, backgrounds, icons, and logos
- Installs branding assets to
~/.local/share/ - Fixes XML background paths to point to home directory
- Applies dconf settings via a Python script
- Installs Bluefin-standard fonts (Inter, JetBrains Mono)
- Installs Bluefin-recommended GNOME extensions
- Deploys a wallpaper rotation service and timer
- Deploys the
corral(Go) VM utility
Fonts
- Inter — UI font (from GitHub releases)
- JetBrains Mono — monospace font for terminals and editors
Wallpaper Rotation
A systemd timer (bluefin-wallpaper-rotate.timer) cycles through Bluefin backgrounds daily. The Python script applies the wallpaper via D-Bus.
Corral
Deploys corral, the Go VM manager
(CLI + TUI + web). Corral is downloaded from the rolling binaries release of
hanthor/corral (checksum-verified, so it
updates whenever CI publishes a new build). It is no longer embedded in this
repo.
Notes
- Skips by default on actual Bluefin hosts (they already have this)
- Set
force_bluefin: trueto apply on Bluefin (e.g., after an image reset) - Set
skip_bluefin: trueinhost_varsto skip entirely
pipewire_audio
Tags: desktop, audio, pipewire
Secrets needed: No
Runs on: Desktop group only (unless skip_pipewire_audio: true)
Replaces Easy Effects with native PipeWire filter-chain modules using LSP LV2 and LADSPA plugins — no Flatpak, no GUI, no D-Bus service.
What It Does
- Deploys
speaker-dsp.conf— a PipeWire filter-chain that creates a virtual "Speakers (DSP)" sink - Deploys
mic-dsp.conf— a PipeWire filter-chain that creates a virtual "Microphone (DSP)" source - Configures WirePlumber to prefer the DSP nodes as default devices
- Cleans up any previous Easy Effects installation (Flatpak, autostart, presets)
Internal task boundaries
The role keeps one public pipewire tag and the skip_pipewire_audio gate, but
the entrypoint delegates to focused task files in this order:
tasks/platform.ymldetects the host platform, installs mutable-system prerequisites, and publishes plugin availability facts.tasks/dsp.ymldeploys the speaker and microphone filter-chain configs.tasks/systemd.ymlconfigures the LADSPA path and persistent microphone routing service.tasks/cleanup.ymlremoves obsolete WirePlumber and Easy Effects state.
The split is organizational only: conditions, handlers, paths, and the role's execution order remain unchanged.
Output Chain: Speaker DSP
Signal path: 16-band Parametric EQ → Bass Enhancer → Exciter → Limiter (all LSP LV2).
| Stage | LV2 Plugin | What It Does |
|---|---|---|
eq | LSP ParaEQ x16 LR | Corrective loudness curve with subsonic HPF @ 120 Hz |
bass | LSP Bass Enhancer | Harmonic bass synthesis for small speakers |
exciter | LSP Exciter | Crystalizer substitute — adds presence and air |
lim | LSP Limiter | Brickwall peak limiter @ -0.5 dBFS |
The EQ curve is a direct translation of the Easy Effects 15-band preset — ~2.5 dB cut through low-mids (220-600 Hz), ~3.5 dB peak at 3.2 kHz, tapering to ~1.5 dB boost above 8 kHz.
Simplification: To run just EQ + Limiter (the most critical, crash-proof chain), comment out the bass and exciter nodes in speaker-dsp.conf and remove their links.
Input Chain: Microphone DSP
Signal path: RNNoise → Compressor (LADSPA + LSP LV2).
| Stage | Plugin | What It Does |
|---|---|---|
rnnoise | librnnoise_ladspa.so | Real-time neural network noise suppression (VAD-enabled) |
comp | LSP Compressor Stereo | Downward compression, -24 dB threshold, 2:1 ratio |
Bootc Image Requirements
These packages must be in the bootc image (Containerfile):
lsp-plugins-lv2
noise-suppression-for-voice
The filter-chain configs reference:
- LSP LV2 bundles under
/usr/lib64/lv2/(Fedora default LV2 path) - RNNoise LADSPA at
/usr/lib64/ladspa/librnnoise_ladspa.so
Verifying
On a target machine after deploy:
# Check DSP nodes appeared
pw-cli ls Node | grep -E 'speaker_dsp|mic_dsp'
# Check defaults
wpctl status | head -20
# Watch for plugin loading errors
journalctl --user -u pipewire -f | grep -iE 'error|filter-chain|lv2'
# Verify LSP plugins are discoverable
lv2ls | grep lsp-plug
lv2info http://lsp-plug.in/plugins/lv2/para_equalizer_x16_lr | head -5
# Verify LADSPA plugin
ls /usr/lib64/ladspa/librnnoise_ladspa.so
Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| DSP node doesn't appear | LSP/LADSPA plugins missing from bootc image | Verify with lv2ls, ls /usr/lib64/ladspa/ |
| Port name mismatch | LSP plugin version changed port symbols | Run lv2info <uri> | grep 'port.*symbol' and update config |
| Audio routing loop | PipeWire/WirePlumber bug | Don't set the DSP sink as default manually; let WirePlumber handle it |
| No bass enhancer/exciter effect | Plugin UID mismatch | Comment them out in config for a leaner chain |
Notes
- PipeWire must be restarted after deploying configs (handled by Ansible handlers)
- Select "Speakers (DSP)" and "Microphone (DSP)" in GNOME Sound settings if WirePlumber doesn't auto-select them
- The virtual sink approach works because PipeWire prevents self-looping when capture/playback share the same
node.name - To skip: set
skip_pipewire_audio: trueinhost_vars, or exclude thepipewiretag
syncthing
Tags: services, syncthing
Secrets needed: No
Runs on: All except VPS
Deploys Syncthing as a systemd user service for continuous file synchronization.
What It Does
- Creates systemd user directory
- Deploys
syncthing.serviceuser unit - Enables and starts the service
Notes
- Runs as the user's systemd service (not system-wide)
- Web UI available at
http://localhost:8384 - Skipped on VPS hosts (no file sync needed)
systemd
Tags: services, systemd
Secrets needed: No
Runs on: All machines
Deploys systemd user services and timers for background tasks.
Services Deployed
dotfiles-update
Keeps the dotfiles repo up to date automatically.
- Desktops/servers: Daily timer (
dotfiles-update.timer) - Laptops: Runs on login via
graphical-session.target(no timer)
atuin-daemon
Keeps Atuin shell history synced in the background.
Podman socket (conditional)
If podman is installed:
- Creates the
podmangroup - Adds the user to the group
- Configures the podman socket for group access (mode 0660)
- Enables and starts
podman.socket
system-mitigator (himachal only)
A Python daemon that monitors for Lenovo EC firmware bugs:
- Detects false
BD_PROCHOTsignals that lock CPU cores to 400 MHz - Watches for
systemd-journaldwatchdog loops - Sends desktop notifications when issues are detected
Notes
- The
dotfiles-update.timeron laptops is removed during the laptop-specific path - All services run as user-level systemd units (no root needed)
proxy (Caddy)
Tags: services, proxy, caddy
Secrets needed: Yes (TLS certificates)
Runs on: All except VPS (skip_proxy: false)
Deploys Caddy as a reverse proxy with automatic TLS certificates.
What It Does
- Creates Caddy config directory
- Deploys
Caddyfilefrom template with per-host service routes - Proxies internal services to
*.manatee-basking.ts.netsubdomains - Uses Tailscale certificates for TLS
Proxied Services
The Caddyfile (listening on :9000, fronted by tailscale serve) carries:
/service/<name>— a redirect for each entry in the host'sweb_services/bst/— the BuildStream dashboard (on build machines)
There is no homepage at /.
Notes
- Skip with
skip_proxy: trueinhost_vars - Caddyfile is templated per-host — each machine proxies its own services
tailscale_cert
Tags: services, tailscale_cert
Secrets needed: No
Runs on: All except VPS
Fetches and deploys Tailscale TLS certificates for HTTPS on *.manatee-basking.ts.net.
What It Does
- Creates certificate storage directory
- Fetches TLS certificates from Tailscale for the machine's MagicDNS name
- Installs certificates where Caddy and other services can use them
Notes
- Tailscale provides free TLS certificates for nodes on your tailnet
- Certificates auto-renew — no Let's Encrypt needed for internal services
- The Caddy proxy uses these certificates for all
*.manatee-basking.ts.netservices
bst_dashboard
Tags: services, bst_dashboard
Secrets needed: No
Runs on: Machines with build infrastructure
Deploys a BuildStream build monitoring dashboard.
What It Does
- Creates dashboard directory based on hostname
- Deploys dashboard web UI files
- Configures dashboard to track local BuildStream build queues
Access
Proxied through Caddy at https://<host>.manatee-basking.ts.net/bst/.
Notes
- Used to monitor BuildStream build jobs on machines that run builds
- Dashboard shows build queue status, job history, and artifact cache
kirocrew
Keeps the Kiro Crew gateway (dashboard + Slack + cron) healthy on hosts that run it — currently punjab.
- Enable:
kirocrew_enabled: trueinhost_vars/<host>.yml - Tags:
services,kirocrew
The venv (~/.kiro/crew-venv) and the systemd unit are owned by Kiro's own
installer (kirocrew service install), so the role does not template them.
It:
- warns if the binary or unit is missing (install per upstream first),
- keeps
/etc/kirocrew/kirocrew.envroot:root 0600— tokens go there, - enables and starts
kirocrew.service, - fails the play if the dashboard listens on anything but localhost.
appflowy
Migrated to Kubernetes (2026-06-12).
This Ansible role (Podman Quadlets on bihar) has been removed. AppFlowy Cloud now runs in the Talos K8s cluster.
See talos-k8s/appflowy/ for the commercial Helm chart deployment manifest and operational instructions.
Access: https://appflowy.manatee-basking.ts.net
authentik
Migrated to Kubernetes (2026-06-12).
This Ansible role (Podman Quadlets on bihar) has been removed. Authentik now runs in the Talos K8s cluster.
See talos-k8s/authentik/ for the Helm-based deployment manifest and operational instructions.
Access: https://auth.manatee-basking.ts.net
n8n
Migrated to Kubernetes (2026-06-12).
This Ansible role (Podman Quadlet on bihar) has been removed. n8n now runs in the Talos K8s cluster.
See talos-k8s/n8n/ for the plain-K8s deployment manifest and operational instructions.
Access: https://n8n.manatee-basking.ts.net
Talos Kubernetes Cluster
ARCHIVED (2026-11-18). This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See Fleet status.
Two-node Kubernetes cluster running on Talos Linux, managed from the talos-k8s/ directory.
Powered down and in storage, expected back — commands below fail until it returns. Matrix and School, Reef and Hub run on the OCI Spindle VPS; the AWS Talos cluster was terminated on 2026-10-25.
Hardware
| Node | Role | Hardware |
|---|---|---|
| bihar | Control plane | Intel (bare metal) |
| karnataka | Worker | AMD Strix Halo APU (GPU-enabled) |
Software Stack
| Component | Version | Notes |
|---|---|---|
| Talos Linux | v1.13.2 | Immutable, API-driven OS |
| Kubernetes | v1.36.1 | Via Talos |
| CNI | flannel | Simple overlay network |
| Tailscale Operator | — | Ingress via *.manatee-basking.ts.net |
| KubeVirt | v1.8.2 | VM workloads on K8s |
Directory structure
talos-k8s/
├── ai/ lemonade
├── appflowy/ appflowy cloud (Helm)
├── authentik/ sso (Helm)
├── forgejo/ git + ci/cd
├── hive/ ai agent swarm
├── homepage/ cluster landing page
├── infrastructure/ kubevirt + corral
├── longhorn/ storage
├── monitoring/ prometheus + grafana + metrics-server
├── n8n/ workflow automation
├── networking/ tailscale configs
└── testing-lab/ ci/cd test infra
Running Workloads
Lemonade
Lemonade — AMD-optimized local AI runtime providing omni-modal endpoints (chat, vision, image gen, speech, transcription) on karnataka's integrated AMD GPU (amdgpu driver baked into the Talos image via Image Factory).
Manifest: talos-k8s/ai/lemonade.yaml
Additional services
See the cluster handbook for the full list — Forgejo, Hive, Authentik, n8n, AppFlowy Cloud, Grafana, Corral, and more.
Cluster Access
# kubectl (from kubeconfig in Bitwarden)
kubectl get nodes
# Talos CLI (LAN addresses; off-LAN they're reachable via bihar's subnet route)
talosctl -n <bihar-ip> version # bihar
talosctl -n <karnataka-ip> version # karnataka
Image Factory
Karnataka uses a custom Talos image from Image Factory that includes the siderolabs/amdgpu system extension for GPU passthrough to containers.
Handbook
Full cluster documentation at docs/src/servers/talos-k8s/cluster.md (hardware details, reinstall procedure, troubleshooting).
Cluster Handbook — Talos Linux on Bihar + Karnataka
ARCHIVED (2026-11-18). This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See Fleet status.
Status: powered down and in storage (as of 2026-08-27), expected back. Temporarily offline, not decommissioned — keep its configs, Bitwarden notes (
kubeconfig/talosconfig), andtalos-k8s/manifests.kubectlandjust doctorwill fail against it until it returns. Matrix and the School, Reef and Hub Hives now run on the OCI Spindle VPS. The AWS Talos cluster was terminated on 2026-10-25.When it returns, before applying anything under
talos-k8s/ai/: runtalos-k8s/ai/migrate-pv-path.shfirst.lemonade.yamlalready points its PVs at/var/lib/…, but the models are still under/var/tmp/…on karnataka. Applying the manifest first orphans them (#29). Then follow the prerequisites intalos-k8s/ai/README.mdfor the backup job and runlemonade-warmup.sh(#26, #27).Home Hummingbird migration: prepared, not deployed. The build/install and activation procedure is in section 12. The home machines remain offline/in storage; the Talos recovery instructions below and all existing home/AWS credentials remain valid recovery material.
Two-node Talos Linux Kubernetes cluster. AMD Strix Halo GPU acceleration · Lemonade · KubeVirt · Tailscale ingress. Everything you need to bring this cluster back from zero.
Table of contents
- Overview
- Hardware
- Network
- Talos installer image (with AMD GPU)
- Cluster bootstrap
- AMD GPU exposed to Kubernetes
- Production workloads
- Tailscale operator (ingress)
- Day-to-day operations
- Reinstall procedure
- Troubleshooting
- Hummingbird/bootc migration (home only)
1. Overview
┌──────────────────────────────────────────────────────────────┐
│ Kubernetes v1.36.1 │
│ (Talos v1.13.2) │
│ │
│ ┌────────────────────────┐ ┌────────────────────────┐ │
│ │ bihar (control plane)│ │ karnataka (worker) │ │
│ │ Intel CPU │ │ AMD Ryzen AI MAX+ 395 │ │
│ │ /dev/nvme0n1 │ │ AMD Radeon 8060S iGPU │ │
│ │ │ │ /dev/nvme0n1 │ │
│ │ kube-apiserver │ │ amdgpu-device-plugin │ │
│ │ etcd / scheduler / │ │ amdgpu-labeller │ │
│ │ controller-manager │ │ Lemonade │ │
│ │ KubeVirt control │ │ Forgejo + runner │ │
│ │ Forgejo + runner │ │ Hive agent swarm │ │
│ │ ArgoCD + Argo WF │ │ KubeVirt virt-handler │ │
│ │ Grafana / Prometheus │ │ Longhorn CSI │ │
│ │ Authentik + n8n │ │ AppFlowy Cloud │ │
│ │ Longhorn CSI / UI │ │ Corral VMs │ │
│ │ KubeVirt Manager │ │ │ │
│ └────────────────────────┘ └────────────────────────┘ │
│ │
│ CNI: flannel · kube-proxy: on · Storage: longhorn + local-path │
│ Tailscale Operator → ts.net ingress for cluster services │
└──────────────────────────────────────────────────────────────┘
The cluster is not diskless — Talos installs itself onto /dev/nvme0n1 on each node and persists. Reboots survive.
Source of truth for manifests: talos-k8s/ in this repo.
Source of truth for secrets (talosconfig, controlplane.yaml, worker.yaml, kubeconfig): Bitwarden, fetched onto your workstation by the kube role.
2. Hardware
Karnataka (worker)
| Component | Detail |
|---|---|
| CPU | AMD Ryzen AI MAX+ 395 (Strix Halo) — 16 cores / 32 threads, 5.18 GHz boost |
| RAM | 62 GB unified memory (CPU + GPU share via UMA) |
| GPU | AMD Radeon 8060S — RDNA 3.5 (gfx1151), integrated |
| NIC | Realtek RTL8126 5GbE (enp191s0) |
| WiFi | MediaTek MT7925 Wi-Fi 7 |
| Boot disk | Crucial P3 1TB (/dev/nvme0n1) |
| Secondary | WD Black SN850X 1TB (/dev/nvme1n1) — FAILING (I/O errors). Do not use. Replace with new drive. |
Bihar (control plane)
| Component | Detail |
|---|---|
| CPU | Intel Core i7-13700K |
| Integrated GPU | Intel UHD 770 |
| Boot disk | /dev/nvme0n1 |
3. Network
LAN — flat home /24
DHCP/DNS are served by the home router, which hands out fixed addresses from its address-reservation table — see Gateway. Older revisions of this page said dnsmasq on bihar; that predates Talos, which can't run it on the host.
| Host | Address | Role |
|---|---|---|
| bihar | reserved | K8s control plane, home services |
| karnataka | reserved | K8s worker, LLM host |
| goa | reserved | Raspberry Pi, fleet control |
| kvm | reserved | NanoKVM (plugged into karnataka) |
The commands below use <bihar-ip> / <karnataka-ip> for the nodes' LAN
addresses. Run just inventory to rescan with nmap and get the current values
(they are also the endpoints in the talosconfig).
Internet — data-capped
The home connection has a monthly data cap; exceeding it throttles to a crawl until the next billing cycle.
May 2026: The cap was hit (largely model downloads and image pulls) and the link was throttled for about a week. Lesson: budget large model downloads and bulk image pulls against the cap. LAN traffic is unaffected.
Tailscale — manatee-basking.ts.net
Cluster ingress goes through the Tailscale Operator. Services with an Ingress resource targeting tailscale ingressClassName get a <name>.manatee-basking.ts.net URL automatically.
| Service | URL |
|---|---|
| Cluster homepage | https://home.manatee-basking.ts.net |
| Lemonade (AI) | https://lemonade.manatee-basking.ts.net/v1 |
| Hive (agent swarm) | https://hive.manatee-basking.ts.net |
| Forgejo (git + CI) | https://forgejo.manatee-basking.ts.net |
| ArgoCD (GitOps) | https://argocd.manatee-basking.ts.net |
| Grafana (metrics) | https://grafana.manatee-basking.ts.net |
| Corral (VM dashboard) | https://corral.manatee-basking.ts.net |
| KubeVirt Manager | https://kubevirt-manager.manatee-basking.ts.net |
| Authentik (SSO) | https://auth.manatee-basking.ts.net |
| n8n (workflows) | https://n8n.manatee-basking.ts.net |
| AppFlowy (collab) | https://appflowy.manatee-basking.ts.net |
Kubernetes internal
| Network | CIDR |
|---|---|
| Pod | 10.244.0.0/16 (flannel) |
| Service | 10.96.0.0/12 |
4. Talos installer image (with AMD GPU)
Talos ships with an immutable root filesystem, so kernel modules cannot be added at runtime. The AMD GPU driver and the Longhorn iSCSI tooling are baked into the boot image via the Talos Image Factory.
Schematic ID (current, 2026-06-11): 3a33ec6dfc8cfd61d2a3db3caf97894f31e913952d71ce3c3fbbe565a3f08339
Extensions baked in:
siderolabs/amdgpu— AMD GPU kernel driversiderolabs/iscsi-tools— iscsid, for Longhorn block storagesiderolabs/util-linux-tools— fstrim/nsenter, for Longhorn
Both nodes run this schematic. Earlier schematics were e5912b95… (amdgpu
only) and, in older docs, b6ab12e…. Storage details: talos-k8s/longhorn/.
machine:
install:
image: factory.talos.dev/installer/3a33ec6dfc8cfd61d2a3db3caf97894f31e913952d71ce3c3fbbe565a3f08339:v1.13.2
disk: /dev/nvme0n1
After install, the node exposes /dev/dri/card0, /dev/dri/renderD128, and /dev/kfd on the host.
Talos version note: Pinned to
v1.13.2.v1.13.3had multi-arch registry pull errors on AMD64 at the time of writing.
User volume for /var/mnt/storage
nvme0n1 is the Crucial P3 (healthy, boot disk). The WD Black failed with I/O errors
and was replaced as the boot disk. Forgejo / local-path PVCs use a directory-type
user volume on the Crucial P3's EPHEMERAL partition:
# Apply the user volume config (idempotent):
talosctl -n <karnataka-ip> patch mc --patch @storage-volume.yaml
The patch is stored at talos-k8s/longhorn/storage-volume.yaml.
It creates /var/mnt/storage as a directory on the EPHEMERAL partition.
The mount is automatically propagated into the kubelet namespace.
WD Black failure (2026-05-30): The WD Black SN850X (
/dev/nvme1n1) developed I/O errors on its META partition during the 35B model download stress test. Error:error writing config to file: input/output error. The drive should be physically replaced. In the meantime, do not configure it as a Talos install disk or user volume.
5. Cluster bootstrap
Configs are generated once with talosctl gen config, edited, and applied to each node. The generated files contain cluster PKI secrets (machine token, bootstrap token, shared secret) and are not committed — they live in Bitwarden.
One-time generation
cd talos-k8s
talosctl gen config talos-k8s https://<bihar-ip>:6443
# Produces: controlplane.yaml, worker.yaml, talosconfig
Then patch machine.install.image and machine.install.disk on each (see talos-k8s/README.md for the schematic ID).
Apply configs
talosctl apply-config --insecure --nodes <bihar-ip> --file controlplane.yaml
talosctl apply-config --insecure --nodes <karnataka-ip> --file worker.yaml
talosctl config endpoint <bihar-ip>
talosctl config node <bihar-ip>
talosctl bootstrap --nodes <bihar-ip>
talosctl kubeconfig # writes ~/.kube/config
Seed Bitwarden with secrets
Once the cluster is up and you have working ~/.kube/config and ~/.talos/config:
just seed-kube # uploads kubeconfig + talosconfig to Bitwarden as secure notes
Other workstations then pull them automatically:
just apply-tags kube
6. AMD GPU exposed to Kubernetes
The driver is loaded on the host (from the Talos image). Kubernetes needs two more pieces from ROCm to actually schedule on the GPU:
kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml
kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml
Once the DaemonSets are running, karnataka exposes:
amd.com/gpu: 1as an allocatable resource- Labels such as
amd.com/gpu.product-name=AMD_Radeon_8060S_Graphics,amd.com/gpu.vram=64G
Workloads request the GPU with:
resources:
limits:
amd.com/gpu: "1"
7. Production workloads
All manifests live in talos-k8s/, organized by purpose:
talos-k8s/
├── ai/ Lemonade (LLM runtime)
├── appflowy/ AppFlowy Cloud (collaboration)
├── authentik/ Authentik (SSO)
├── forgejo/ Forgejo + runner + registry-mirror
├── hive/ Hive (AI agent swarm)
├── homepage/ Cluster landing page
├── infrastructure/ KubeVirt + Corral
├── kubestellar-hive/ KubeStellar multi-cluster hive
├── longhorn/ Storage (CSI, snapshots)
├── monitoring/ Prometheus + Grafana + metrics-server
├── n8n/ n8n (workflow automation)
├── networking/ Tailscale node configs
├── searxng/ SearXNG (self-hosted search)
├── testing-lab/ CI/CD test infrastructure
└── topolvm/ TopoLVM (dynamic LVM storage)
Lemonade — AMD-optimized local AI runtime
Single-replica Lemonade omni-modal server on the Strix Halo APU.
Manifest: talos-k8s/ai/lemonade.yaml.
Lemonade is an AMD-optimized, open-source local AI runtime that auto-detects hardware and provides standard OpenAI-compatible endpoints for chat, vision, image generation, image editing, speech generation, and transcription. Built on llama.cpp, ONNX Runtime, whisper.cpp, and stable-diffusion.cpp.
Key env overrides (RDNA 3.5 needs special handling):
| Env var | Value | Why |
|---|---|---|
HSA_OVERRIDE_GFX_VERSION | 11.5.1 | Forces ROCm to treat gfx1151 as supported |
PYTORCH_ROCM_ARCH | gfx1151 | Compile shaders for exact arch |
HSA_XNACK | 1 | Enables unified memory |
HSA_FORCE_FINE_GRAIN_PCIE | 1 | Fine-grained VM access |
Access:
- LAN:
http://<karnataka-ip>:31305/v1(NodePort) - Tailscale:
https://lemonade.manatee-basking.ts.net/v1
Model weights cached on karnataka:/var/tmp/lemonade-cache (HuggingFace) and /var/tmp/lemonade-models (llama models) via local-storage PersistentVolumes.
Web UI: Point a browser at http://<karnataka-ip>:31305 for the built-in control panel to download models, configure endpoints, and test chat/vision/image generation.
KubeVirt v1.8.2
Installed from upstream operator manifests:
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-operator.yaml
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-cr.yaml
The default namespace is labelled pod-security.kubernetes.io/enforce=privileged so privileged virtualization pods can run.
KubeVirt Manager (web UI)
Manifest: deployed from upstream bundle. Also see talos-k8s/infrastructure/ for KubeVirt CR and Corral VM dashboard.
kubectl apply -f https://raw.githubusercontent.com/kubevirt-manager/kubevirt-manager/main/kubernetes/bundled.yaml
Reachable at https://kubevirt-manager.manatee-basking.ts.net via the Tailscale ingress (manifest in cluster — not in this repo since it's a stock bundle).
Port-forward as fallback:
kubectl port-forward svc/kubevirt-manager 8080:8080 -n kubevirt-manager
# http://localhost:8080
Additional services
All other production services have their own directory in talos-k8s/:
| Service | Directory | Description |
|---|---|---|
| Forgejo | forgejo/ | Git hosting + Actions CI/CD |
| Hive | hive/ | AI coding agent swarm backed by Lemonade |
| Authentik | authentik/ | SSO (passkeys, OAuth2/OIDC) — Helm |
| n8n | n8n/ | Workflow automation — plain K8s manifest |
| AppFlowy Cloud | appflowy/ | Collaborative workspace — commercial Helm chart |
| Grafana + Prometheus | monitoring/ | kube-prometheus-stack + metrics-server |
| ArgoCD | (Helm) | GitOps deployment |
| Longhorn | longhorn/ | Distributed block storage (CSI, snapshots) |
| KubeStellar Hive | kubestellar-hive/ | KubeStellar multi-cluster hive |
| SearXNG | searxng/ | Self-hosted metasearch |
| TopoLVM | topolvm/ | Dynamic LVM-backed storage |
| Testing lab | testing-lab/ | Argo Workflows CI/CD test infra |
Each directory has a README.md with deploy and operational instructions.
8. Tailscale operator (ingress)
The Tailscale Operator creates a Tailscale-side proxy pod for any Ingress resource with ingressClassName: tailscale. The proxy joins the tailnet with a name derived from the Ingress, and routes incoming traffic to the backing Service.
Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lemonade
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: lemonade
port:
number: 13305
tls:
- hosts:
- lemonade
The operator namespace is tailscale. Auth keys are configured during operator install — see Tailscale docs for renewal.
9. Day-to-day operations
From your workstation
The kube role syncs ~/.kube/config and ~/.talos/config from Bitwarden, so on any desktop:
kubectl get pods -A
talosctl -n <bihar-ip> health
talosctl -n <karnataka-ip> dmesg | grep -i amdgpu
Common kubectl
kubectl get nodes -o wide
kubectl top nodes # requires metrics-server (monitoring/metrics-server.yaml)
kubectl logs -f deployment/lemonade
kubectl describe pod -l app=lemonade # check GPU scheduling / events
Common talosctl
talosctl -n <karnataka-ip> list /dev/dri/ # confirm GPU devices visible to host
talosctl -n <karnataka-ip> services # systemd-equivalent services
talosctl -n <bihar-ip> etcd status # control-plane etcd health
talosctl -n <bihar-ip> logs kubelet
talosctl -n <bihar-ip> reboot # graceful reboot
10. Reinstall procedure
If a node is wiped or replaced:
- Boot Talos ISO on the node (or PXE if you set that up separately). Default Talos boots into maintenance mode and accepts an apply-config.
- Apply config:
talosctl apply-config --insecure --nodes <node-ip> --file <controlplane|worker>.yaml - Bootstrap etcd (control plane only, first time):
talosctl bootstrap --nodes <bihar-ip> - Verify:
talosctl -n <bihar-ip> health kubectl get nodes - Create user volume (karnataka/worker only):
talosctl -n <karnataka-ip> patch mc --patch @talos-k8s/longhorn/storage-volume.yaml - Create host directories for local PVs (if re-creating from scratch):
kubectl exec -n kube-system $(kubectl get pods -n kube-system --field-selector spec.nodeName=karnataka -l app=kube-flannel -o name) -- \ mkdir -p /proc/1/root/var/tmp/lemonade-cache mkdir -p /proc/1/root/var/tmp/lemonade-models - Re-apply workloads (idempotent):
kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-operator.yaml kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-cr.yaml kubectl apply -f https://raw.githubusercontent.com/kubevirt-manager/kubevirt-manager/main/kubernetes/bundled.yaml kubectl apply -f talos-k8s/ai/lemonade.yaml
If only the worker was reset, the existing kubeconfig still works — only steps 1, 2, 5, and 6 are needed for that node, plus re-applying any DaemonSet pods.
WD Black failure: Do not use the WD Black (
/dev/nvme1n1) for installs or user volumes. Install only to the Crucial P3 (/dev/nvme0n1). Storage uses a directory-type volume on the Crucial P3's EPHEMERAL partition.
11. Troubleshooting
amd.com/gpu shows 0 on karnataka
Check the device plugin actually scheduled and started:
kubectl -n kube-system get pods -l name=amdgpu-device-plugin-ds
kubectl -n kube-system logs -l name=amdgpu-device-plugin-ds --tail=50
talosctl -n <karnataka-ip> dmesg | grep -i amdgpu # confirm driver loaded
talosctl -n <karnataka-ip> list /dev/dri/ # confirm devices exist
Lemonade pod stuck in ContainerCreating
Most often: pulling models from HuggingFace. Check:
kubectl describe pod -l app=lemonade
kubectl get pv,pvc # confirm PVCs bound
Tailscale Ingress not reachable
kubectl -n tailscale get pods # operator + per-Ingress ts-* proxies
kubectl -n tailscale logs deploy/operator
tailscale status | grep -i <ingress-name>
Talos node unresponsive
talosctl -n <ip> health --wait
talosctl -n <ip> dmesg --tail
talosctl -n <ip> reset # nuclear: wipes node, requires re-apply
Container runtime crash (karnataka)
Symptoms:
kubectl get nodesshows karnatakaNotReadykubectl describe node karnatakashowsReady: Falsewithcontainer runtime is down- All pods on karnataka stuck in
Terminating/Pending - Ping to karnataka still works (node is up, just CRI is dead)
Root cause: Memory pressure from simultaneous large container builds exhausts karnataka's 62GB unified memory. The node runs Lemonade (48-56Gi), Argo Workflow image builds, Forgejo Actions (podman-in-Docker), KubeVirt, and monitoring — when multiple 4GB image builds run concurrently, containerd OOMs and crashes.
Recovery:
# Option 1: Graceful reboot via Talos
talosctl -n <karnataka-ip> reboot
# Option 2: Hard reboot via the NanoKVM (see network/kvm)
After reboot (~2 min), all pods recover automatically. PVC-backed services (Forgejo, local-path PVs) survive.
Prevention:
- Don't run more than one large container build concurrently on karnataka
- Set
SKIP_CHUNKAH=1for Forgejo builds (avoids extra 2GB memory spike) - Consider moving non-GPU workloads to bihar or a third node
- The Forgejo runner has
timeout-minutes: 45— builds that exceed this will be killed
12. Hummingbird/bootc migration (home only)
Scope and acceptance status
This is a replacement OS and new K3s cluster for the two home nodes, not a Talos upgrade. Bihar is the Intel server/control plane; Karnataka is the AMD worker. No physical migration has been performed. The home cluster is still powered down/in storage. Nothing here changes the terminated AWS Talos cluster's recovery material; Corral acceptance uses separate disposable KubeVirt VMs, not Talos node changes. Sections 1–11 remain the Talos recovery handbook, not instructions for the new OS.
Image builds alone do not prove installation. The GitHub Actions acceptance
workflow builds all three images and offline ISOs, drives the real
bootc-installer-tui, boots the installed disks without their ISOs, and
switches the installed starter to Bihar before activating a separate disposable
two-node K3s cluster.
Publication run 39493512586
passed for release source 4ec3fa6470d6b02e7eaf1b0eaae03d1db05a31a7: all three offline TUI
installs, installed-disk UEFI boot, admin login/sudo, Enforcing SELinux,
unenrolled Tailscale, and a real starter-to-Bihar bootc switch with exact digest,
reboot, and administrator/hostname/identity//var persistence checks. Gated K3s
activation then passed two Ready nodes, cross-node HTTP, cluster DNS, and
local-path PVC data surviving writer-pod replacement. The reviewed PR also
passed normal repository CI.
On 2026-10-20, Playwright drove both installers through the actual Corral
noVNC canvases on disposable KubeVirt VMs. Both reached “Installation
Complete”; their ISO volumes were then detached before installed-disk boots.
Real admin-password login/sudo and Enforcing SELinux passed on both. The installed
cluster passed the same DNS, cross-node HTTP, and PVC pod-replacement scenario
used by CI. Bihar's Intel kvm_intel module loaded on this Intel Corral host;
that is not a test of a nested KubeVirt guest or the physical home machines.
The deployed Corral service reported its cluster unreachable, so acceptance used
a temporary localhost Corral dashboard against admin@aws-migration, without
changing the deployed service. All test VMs/disks were isolated in namespace
home-k8s-acceptance. Configure the disposable node LAN as 192.168.0.5/24
and 192.168.0.6/24 before activating K3s: Corral's provider pod addresses
overlap this new cluster's 10.244.0.0/16 pod CIDR. The initial overlapping
setup reached Ready nodes and working DNS but failed cross-node HTTP; the
non-overlapping guest LAN passed without changing the images' production
network configuration. Changing an already-initialized embedded-etcd server's
IP also requires its membership to be corrected; do not initialize it with the
provider pod IP in the first place.
Native Corral media predates publication-only OCI label changes; it is not
byte-identical to the subsequently rebuilt release. The publication dispatch
repeats complete KVM acceptance on the exact bytes it publishes. Corral evidence
is saved under ~/.local/share/home-k8s-build/ on the acceptance controller:
corral-{bihar,karnataka}-install-complete.webp,
corral-{bihar,karnataka}-installed-boot.webp,
corral-installed-kubernetes.webp, and corral-cluster-proof.txt.
The optional starter also passed a 2026-10-20 Playwright/noVNC installation
in namespace home-k8s-starter-acceptance, followed by ISO detachment and an
installed-disk UEFI boot. Actual admin-password login/sudo, hostname
starter-smoke, Enforcing SELinux, absent K3s role/configuration, driver/firmware
payload checks, and the canonical starter bootc reference passed. The real
installed tailscaled daemon was active and responsive in NeedsLogin, with
no tailnet and root-owned 0700 state; no VM was enrolled into the fleet.
The dedicated tailscale0 SSH firewall zone was present. All disposable VMs,
PVCs, namespaces, and temporary Corral/upload services were removed afterward.
That native starter ISO was built from source
2f8be86e3f98782f8dc7857e688e274318494f5b, is 2,057,453,568 bytes, and has SHA256
1dc796f217e2d4ce6140511bed59399d2e031092299e2fa09e0436bea305ee27.
Evidence is under ~/.local/share/home-k8s-build/unified-starter-native/:
corral-starter-install-summary.png, corral-starter-install-complete.png,
corral-starter-admin-sudo.png, corral-starter-installed-tailscale.png, and
corral-starter-installed-proof.txt. These are native acceptance artifacts,
not a claim that the later publication rebuild is byte-identical.
VM acceptance cannot establish GPU, physical NIC, disk health, firmware,
or nested-virtualization readiness on the real machines. The recorded runner
was AMD EPYC: Bihar's Intel-only kvm_intel preload fails in that VM, so its
module-load unit is not evidence of working Intel virtualization. Verify that
unit and a real KubeVirt guest on physical Bihar before cutover.
Image contents and state
The source is server-images/Containerfile, compose.sh, nodes/, and rootfs/.
All three targets use the digest-pinned Hummingbird Community os-minimal base,
without a desktop. The Bihar/Karnataka images remain the direct installation
path for those machines; starter is optional for future x86_64 AMD/Intel nodes.
Explicit AMD/Intel GPU and CPU, Intel Wi-Fi, Realtek, MediaTek, Atheros, and
Broadcom firmware subpackages come through the digest-pinned Utah package factory.
That local repository is authenticated by its OCI digest, not RPM signatures.
The lower-priority Fedora 44 repository checks Fedora RPM signatures and supplies
missing storage dependencies; it does not replace the Hummingbird kernel.
The separate live-media build uses Hummingbird's dracut-live and
systemd-boot-unsigned, with missing dependencies such as fuse, parted, and
ntfs-3g supplied by signed Fedora RPMs. Those live additions do not modify the
embedded installed-system OCI payload.
K3s and its SELinux policy RPM are separately SHA256-pinned.
The common hardware payload includes:
| Hardware | Shipped support |
|---|---|
| Bihar: Intel Core i7-13700K / UHD 770 | Intel microcode, i915 and its firmware, kvm_intel; the xe driver/firmware is also available |
Karnataka: Strix Halo / Radeon 8060S (gfx1151) | amdgpu, gc_11_5_1 GPU firmware, AMD microcode, kvm_amd |
| Karnataka: MediaTek MT7925 Wi-Fi | mt7925e plus its Wi-Fi code and patch firmware |
| Karnataka: Realtek RTL8126 Ethernet | r8169 plus Realtek NIC firmware |
| Other common Wi-Fi families | Intel iwlwifi, Realtek rtw89, Atheros/Qualcomm ath9k/ath10k/ath11k/ath12k, Broadcom brcmfmac, and their selected firmware packages |
NetworkManager-wifi supplies the actual Wi-Fi backend, with wpa_supplicant
for authentication. After installed-system login, connect interactively with
sudo nmcli --ask device wifi connect "your-ssid"; never put Wi-Fi credentials
in image source or command-line password arguments. Live-session network
configuration is not copied by the pinned installer. Driver/module/firmware
presence is checked during builds and starter acceptance, but physical RF,
GPU compute, device passthrough, and board-specific compatibility remain untested.
This is not an ARM image or a proprietary NVIDIA/CUDA stack.
K3s is pinned to v1.36.1+k3s1, with embedded containerd, Flannel networking,
and embedded etcd on Bihar (cluster-init: true). Pod/service CIDRs remain
10.244.0.0/16 and 10.96.0.0/12; cluster DNS is 10.96.0.10.
Bihar disables bundled Traefik, ServiceLB, and metrics-server: restore any needed
equivalents deliberately. This is one control plane, not HA.
NetworkManager, sshd, firewalld, Tailscale, and iSCSI services are enabled on all
three images. Bihar enables k3s@server; Karnataka enables k3s@agent, but
both are gated by /etc/rancher/k3s/activate. The starter selects neither role
and supplies no hostname, K3s configuration, or cluster endpoint.
No activation marker, cluster token, Tailscale auth key, or enrolled identity
is baked into an image. Automatic bootc fetch/apply services are masked so a
reboot of the only control plane is an explicit maintenance decision.
NFS GSSAPI state directories are initialized privately (0700) through
tmpfiles before the daemon's sandbox is assembled; CI requires gssproxy to
start successfully with SELinux enforcing.
Mutable state includes /etc/rancher/k3s, /var/lib/rancher/k3s (including etcd
and containerd), user homes under /var/home, and these retained workload paths:
| Path | Purpose |
|---|---|
/var/mnt/storage | Bihar's K3s local-path default and existing host-path data |
/var/lib/longhorn | Longhorn persistent storage |
/var/lib/lemonade-cache | Lemonade cache, matching the current home PV manifest |
/var/lib/lemonade-models | Lemonade models, matching the current home PV manifest |
The current talos-k8s/ai/lemonade.yaml uses /var/lib/lemonade-*. The offline
Talos node may still hold the data under /var/tmp/lemonade-*: back up those
source directories before wiping and restore into /var/lib/lemonade-* on the
new node. Creating an empty PV directory does not migrate the models.
Back up before building a physical migration plan
Before any disk wipe, recover the old nodes if necessary and copy application backups, PV data, models, manifests, and required secrets off both machines. Inventory PV-to-node/path mappings, ownership, application restore procedures, and Longhorn volumes; verify the backups can be read/restored independently. An etcd snapshot is useful for Talos recovery, but Talos etcd cannot be directly restored into K3s. A matching directory name is not a data migration.
Keep the old Bitwarden secure notes kubeconfig / talosconfig and their
~/.kube/config / ~/.talos/config files. Also keep the separate AWS
kubeconfig-aws-migration / talosconfig-aws-migration notes and files unchanged.
Use a new home kubeconfig below; do not overwrite any of these.
Karnataka's /dev/nvme1n1 (WD Black) is failing: never select it.
Only the healthy /dev/nvme0n1 is a candidate, after off-node backups and an
explicit physical operator choice. Identify disks by model, serial, size,
and current contents at the console (lsblk -o NAME,SIZE,MODEL,SERIAL,FSTYPE,MOUNTPOINTS);
do not assume device enumeration is sufficient. Apply the same backup and
operator-choice gate to Bihar's intended /dev/nvme0n1.
Build node images and optional starter media
Run from this repository on a Linux builder with Podman, Go compatible with
Fisherman's tui/go.mod, Python 3, squashfs-tools (mksquashfs), dosfstools
(mkfs.fat), mtools (mmd, mcopy), and xorriso available. Local source
defaults are ~/dev/tuna-os/fisherman and ~/dev/utah; override with
FISHERMAN_SRC and UTAH_SRC. For the CI source selection, use Fisherman commit
014e704be5f7365c4a1ccdef37fcadf97ade792a and Utah commit
f2736b6b774551cc8971af85cedaa65a272354ee; the Fisherman Go requirement is
1.26.2. The media builder compiles the real Fisherman backend and TUI from
these trees, not a substitute installer. Use a clean Fisherman checkout at that
revision: it includes the target-disk OCI staging fix from
the merged Fisherman PR 345, required for
offline installation without exhausting the live filesystem.
just server-images
# Produces localhost/home-k8s-{starter,bihar,karnataka}:testing.
mkdir -p artifacts/home-k8s
export FISHERMAN_SRC="$HOME/dev/tuna-os/fisherman"
export UTAH_SRC="$HOME/dev/utah"
# Choose canonical refs in a registry namespace you control.
export BIHAR_TARGET_REF=ghcr.io/hanthor/home-k8s-bihar:testing
export KARNATAKA_TARGET_REF=ghcr.io/hanthor/home-k8s-karnataka:testing
just server-installer bihar "$BIHAR_TARGET_REF" artifacts/home-k8s/bihar-installer.iso
just server-installer karnataka "$KARNATAKA_TARGET_REF" artifacts/home-k8s/karnataka-installer.iso
# Optional: a hardware-generic install with no selected cluster role.
just server-installer starter ghcr.io/hanthor/home-k8s-starter:testing artifacts/home-k8s/starter-installer.iso
The target references above are day-2 tracking metadata, not downloads needed
by the offline installation. Each ISO embeds its matching unchanged OCI payload.
These local commands and PR/push acceptance do not publish registry images.
Before bootc upgrade, publish the intended image to its canonical ref and verify
the digest; building an ISO alone does not make that ref available.
To build just one profile, use just server-image bihar, just server-image karnataka, or just server-image starter. The canonical recipe disables
Podman's injected /etc/hostname so image composition can clear the base hostname
before applying a specialization. The UEFI media keeps its boot files inside
the FAT ESP rather than duplicating them as loose ISO files; the embedded
installed-system payload is unchanged by that size reduction.
Publish accepted images and installation media
Published acceptance release:
home-k8s-20261021.
These are public prerelease artifacts, not proof of physical hardware
readiness. All three profiles' date tags and testing channels passed anonymous
GHCR access and match the exact tested payloads and source revision.
| Profile | Published image | Offline installer ISO |
|---|---|---|
| Bihar | ghcr.io/hanthor/home-k8s-bihar:20261021 | bihar-installer.iso |
| Karnataka | ghcr.io/hanthor/home-k8s-karnataka:20261021 | karnataka-installer.iso |
| Optional starter | ghcr.io/hanthor/home-k8s-starter:20261021 | starter-installer.iso |
Download SHA256SUMS
and provenance.json
from the same release; the latter records image digests, exact source/upstream
revisions, and the acceptance run. OCI archives, media contracts, and the
starter-switch proof
are also release assets. The publisher treats date tags as write-once; use the
recorded @sha256: digest when a registry reference must be intrinsically immutable.
The older 20261020 release remains available but predates the optional starter
and completed GPU/Wi-Fi firmware and host Tailscale payload.
Publication is an explicit master-only dispatch, after interactive Corral acceptance. It repeats all three real offline TUI installations, the installed starter's bootc switch, and the two-node Kubernetes scenario before transferring any release artifacts:
gh workflow run server-images.yml --repo hanthor/dotfiles --ref master \
-f publish=true -f version=20261022
Choose an unused, valid YYYYMMDD date; 20261020 and 20261021 are already published.
Ordinary PR/push runs and manual runs
with publish=false remain read-only. Only the separate publishing job receives
registry/release write permissions; it pushes the exact accepted OCI archives,
not rebuilt images. It publishes ghcr.io/hanthor/home-k8s-{starter,bihar,karnataka}
with immutable date tags and the mutable testing channel, plus the public
GitHub prerelease home-k8s-YYYYMMDD (not the repository's latest stable release).
Assets include all three tested ISOs, OCI archives, media contracts, checksums,
source/upstream revisions, registry digests, explicit acceptance markers, and
the sanitized starter-switch proof.
The canonical just server-images build records this repository's Git revision
in the OCI source/revision labels, rather than inheriting Hummingbird's revision.
Installed VM disks and credentials are never release assets.
New GitHub container packages default to private, even when linked to a public repository. The packages above are already public. For a new package, change visibility in its GitHub package settings if necessary. The publisher refuses to create the release until anonymous access to both image tags succeeds. If this gate fails, change visibility and rerun only the failed publishing job within one day: the accepted artifact transfer has one-day retention. Identical existing date tags are reused; different content or an existing release/tag name is rejected. Never rebuild over an immutable date tag.
Install through the actual TUI
For each node, first meet the backup/disk-selection gate above. Boot its matching ISO in UEFI mode with Secure Boot disabled. At the live root console run:
bootc-installer-tui
- Skip network configuration to use DHCP. The pinned TUI does not serialize selected network configuration; do not depend on it applying static addressing.
- Choose Custom URL and enter
oci:/usr/share/home-k8s/payload. - Choose the verified intended installation disk. Installation destroys its contents; do not choose the failing Karnataka secondary disk.
- Choose ext4, and set the hostname to
biharorkarnatakaas appropriate, or a unique hostname for a new starter installation. - Leave disk encryption disabled for the tested installation path. Encrypted installation is not covered by this acceptance run.
- Create an actual administrative user with a strong password. Record it securely and check login/sudo after installation. Do not rely on root live access as an installed-system account.
- Skip SSH-key import: this pinned TUI does not serialize imported keys into the installation. Add authorized keys securely after first login instead.
- Review the destructive disk choice, install, shut down/reboot, and remove the ISO so the next boot is from the installed disk.
The live ISO is SELinux permissive only (enforcing=0, not selinux=0).
That exception belongs to live assembly, not the target. On the installed node,
verify getenforce prints Enforcing, sudo -v succeeds for the new user,
bootc status shows the intended target reference, and the corresponding K3s
unit is inactive before activation. Stop and resolve any mismatch before joining
the cluster. Install SSH keys and verify remote access using authenticated,
trusted connections; never paste private keys or cluster tokens into CI logs.
Optional starter and a deliberate image switch
The starter is not a third cluster node and does not join anything on first boot. It installs the same hardware, storage, Tailscale, and K3s tooling, but neither K3s instance is enabled and there is no default K3s configuration. Install Bihar and Karnataka directly when replacing those machines; the starter is an optional base for additional machines or a later switch to another bootc image.
For example, to adopt Bihar's server profile from an installed starter:
sudo bootc switch ghcr.io/hanthor/home-k8s-bihar:testing
sudo systemctl reboot
After reboot, inspect sudo bootc status and verify the booted image, not
merely a staged deployment. The administrator, chosen hostname, and /var state
survive; a switch does not rename the host or create the K3s activation marker.
/etc uses a three-way merge, so review existing configuration rather than
assuming the destination image replaces it.
The named profiles are home-machine specializations, not generic role
templates: they contain fixed K3s node names, home addresses, and CPU-specific
module preloads. On additional machines, use an appropriate derived image or
review those defaults before activation, especially node-name, server address,
TLS SANs, and virtualization modules. The starter also retains the common
home-LAN firewall policy; new sites need an explicit network-policy review.
Enroll host Tailscale after installation
The pinned Utah factory supplies Tailscale 1.98.8. tailscaled starts
automatically but remains NeedsLogin until the operator enrolls that node.
On the installed system—not the live ISO—run:
sudo tailscale up --hostname="$(uname -n)" --advertise-tags=tag:fleet \
--accept-dns=false --accept-routes=false
tailscale status
Complete the displayed login URL using the fleet tailnet account; tag:fleet
requires the corresponding tag-owner permission. This keeps the existing fleet
tag convention without importing tailnet routes or replacing the node's DNS,
which could conflict with the home/Flannel networks. No subnet routes, exit-node
service, or Tailscale SSH are enabled automatically.
The dedicated tailscale0 firewalld zone allows ordinary SSH only; it does not
open the Kubernetes API, kubelet, VXLAN, or NodePorts to every tailnet peer.
Tailnet ACLs and installed SSH authorized keys still govern access. Host Tailscale
does not replace Flannel or the separate in-cluster Tailscale Operator.
Node keys and enrollment state live under private, root-owned
/var/lib/tailscale (0700) and survive bootc updates/switches. Never bake that
directory into an image or copy it between machines. A clean reinstall needs its
own enrollment; a live-session identity is not migrated by this installer.
CI checks the real installed daemon's unauthenticated state and private storage
with SELinux enforcing; it does not enroll disposable VMs into the fleet.
Explicit activation and separate kubeconfig
Verify router DHCP reservations and actual addresses before starting K3s:
Bihar 192.168.0.5, Karnataka 192.168.0.6. The worker configuration and
server TLS SAN depend on Bihar's address. Make sure the isolated/new home
cluster cannot conflict with a running old cluster at those addresses.
On Bihar, as root:
touch /etc/rancher/k3s/activate
systemctl start k3s@server
k3s kubectl get nodes
Wait for the API/server to be healthy. Securely transfer
/var/lib/rancher/k3s/server/node-token from Bihar to
/etc/rancher/k3s/token on Karnataka. Treat this server token as
cluster-admin-sensitive, not as a harmless worker password. Use a verified SSH
connection or other authenticated encrypted channel; do not print it, commit it,
store it in artifacts, or copy it through chat. On Karnataka ensure the file is
owned by root and mode 0600 before activation:
chown root:root /etc/rancher/k3s/token
chmod 0600 /etc/rancher/k3s/token
touch /etc/rancher/k3s/activate
systemctl start k3s@agent
Securely copy Bihar's /etc/rancher/k3s/k3s.yaml to the workstation as
~/.kube/config-home-hummingbird (mode 0600); change only its loopback API
endpoint to https://192.168.0.5:6443. Keep this admin credential out of Git and
logs. Use it explicitly instead of changing the old home or AWS kubeconfigs:
export KUBECONFIG="$HOME/.kube/config-home-hummingbird"
kubectl get nodes -o wide
kubectl wait --for=condition=Ready nodes/bihar nodes/karnataka --timeout=300s
Restore home workloads and validate before cutover
Review and apply only selected home workload manifests from talos-k8s/.
Do not recursively apply that whole directory: it also contains AWS manifests.
Do not apply Talos machine patches, extension/schematic settings, or Talos-only
bootstrap configuration to K3s. Reconcile storage classes, node labels,
selectors, host paths, privileged-device access, and security requirements
against the new cluster before restoring applications.
Restore data from off-node backups to the intended node/path with correct
ownership and SELinux labels (use restorecon on restored host paths). Follow
application-consistent restore procedures rather than blindly copying running
databases or treating Longhorn's data directory as a complete backup. Validate
PVC binding, persistence after pod/node restarts, DNS, cross-node traffic,
ingress, application health, and data integrity before declaring a cutover.
On real Karnataka hardware, verify amdgpu, /dev/kfd, /dev/dri, the AMD
device plugin's allocatable GPU, and an actual Lemonade inference workload.
Verify /dev/kvm, firmware virtualization settings, the correct Intel/AMD KVM
modules, KubeVirt scheduling, and a real guest boot on the relevant nodes.
Build-time module checks and virtual acceptance do not establish these facts.
Use planned maintenance and fresh data/etcd backups for day-2 bootc updates.
bootc rollback rolls back the OS deployment, not application data, etcd,
mutable /etc configuration, or PV contents. A data rollback requires its
own tested restore plan; retain the Talos recovery material independently.
Disposable VM acceptance (GitHub Actions and local)
GitHub Actions workflow Home server images
(server-images.yml) runs for matching pull requests and master pushes.
Manual dispatch defaults to acceptance only (publish=false); review branches
cannot publish. Acceptance builds all three images/ISOs, installs each through
the real TUI, and checks installed-disk boot, admin/sudo, Enforcing SELinux,
unenrolled Tailscale, and gated K3s. It switches the installed starter to the exact
new Bihar payload through a loopback-only registry, verifies the staged digest,
reboots, and checks the booted digest plus persistent admin/hostname/identity
and /var data. The fixture registry, starter disk, and NVRAM are removed before
the two-node cluster scenario. The VMs use an isolated virtual LAN plus QEMU NAT, not the physical
home LAN. CI requires KVM and fails if it is unavailable. No fleet deployment
or physical-disk access occurs, including on a publication dispatch.
It runs on ubuntu-24.04 and uploads *.log, *.json, and *.txt diagnostics
as home-server-images-diagnostics with seven-day retention, never installed
disks. Only an explicit master publication retains the exact tested ISOs and
exports an allowlisted, one-day release-artifact transfer to the publishing job
described above. Retain any needed diagnostics before they expire.
For local acceptance, install uv, QEMU x86_64, Skopeo, and OVMF, and use the ISO names
from the build commands above. These commands create disposable installed
qcow2 disks and per-node UEFI variable stores in the artifact directory:
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm install starter
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm install bihar
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm install karnataka
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm cluster
To exercise a published-image switch on that installed starter, resolve the immutable date tag's digest first (the guest needs outbound registry access):
image=ghcr.io/hanthor/home-k8s-bihar:20261021
digest=$(skopeo inspect --format '{{.Digest}}' "docker://$image")
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm \
switch-starter --image "$image" --expected-digest "$digest"
starter-switch-proof.txt records only explicit successful checks and public
image references/digests, never the administrator password or private state.
The workflow's fixture instead uses the exact newly built Bihar archive before
publication, binds its HTTP registry only to 127.0.0.1:5000, and scopes its
temporary insecure-registry setting to 10.0.2.2:5000 inside the disposable guest.
On a machine without usable KVM, explicitly replace --accel kvm with
--accel tcg in every command; this is slower emulation, not hardware
acceleration proof. The harness pins pexpect==4.9.0 through its inline uv
dependency metadata. It drives the actual TUI, powers off, removes the ISO,
boots the installed UEFI disk, checks login/sudo, Enforcing SELinux, target
reference, and inactive K3s, then activates both nodes and checks Ready status,
cross-node traffic, DNS, and PVC behavior. Preserve the generated *.log,
*.json, and *.txt proof outputs with the CI run. The passing run and physical
hardware limitations are recorded under “Scope and acceptance status” above.
Bihar
ARCHIVED (2026-11-18). This page describes bihar on Talos, which was replaced by Bluefin + k3s. See Fleet status.
Kubernetes control plane. Talos Linux node — not in inventory.yml, managed with talosctl/kubectl only.
Powered down and in storage, expected back. See the cluster handbook.
Hardware
- Arch: x86_64 (Intel)
- Motherboard: ASRock
- LAN: reserved address on the home /24
- Role: Control plane (Talos K8s)
- Tailscale:
bihar(MagicDNS)
OS
Talos Linux v1.13.2 (Kubernetes v1.36.1)
Services
Bihar no longer serves anything directly — the old per-host Caddy paths
(bihar.manatee-basking.ts.net/grafana, /auth, /n8n, …) predate Talos.
Cluster services are reached through the Tailscale Operator; see the
ingress list in the cluster handbook.
Networking
Tailscale runs as a Talos system extension, configured by
talos-k8s/networking/tailscale-bihar.yaml:
- Advertises the home LAN /24 as a subnet route
- Tailscale DNS: disabled (
TS_ACCEPT_DNS=false)
See also
Karnataka
ARCHIVED (2026-11-18). This page describes karnataka on Talos, which was replaced by Bluefin + k3s. See Fleet status.
Kubernetes worker node with AMD GPU. Talos Linux node — not in inventory.yml, managed with talosctl/kubectl only.
Powered down and in storage, expected back. See the cluster handbook.
Hardware
- Arch: x86_64
- System: Framework Computer
- CPU: AMD Strix Halo APU
- GPU: AMD integrated (exposed via Talos Image Factory schematic with
siderolabs/amdgpu) - LAN: reserved address on the home /24
- Role: Worker (Talos K8s)
- Tailscale IP: via MagicDNS
OS
Talos Linux v1.13.2 (Kubernetes v1.36.1)
K8s Workloads
- Lemonade — AMD-optimized local AI runtime (omni-modal: chat, vision, image gen, speech, transcription) on iGPU
- KubeVirt v1.8.2 — VM workloads
- Tailscale Operator — Ingress to
*.manatee-basking.ts.net
See also
AWS Talos Cluster (archive)
Terminated 2026-10-25. This is a historical record, not an operating runbook. The fleet-backups S3 bucket and a mirror on himachal-1 (
~/backups/tunaos-aws-2026-10-25) were retained; the current status of individual volumes, snapshots, IPs and other AWS resources is not established here. Matrix and standalone LiveKit are deployed on the OCI spindle VPS.OpenTofu still describes removed cluster resources.
just aws-planmay propose recreating them, andjust aws-applycan recreate infrastructure and incur costs. Review live state and every proposed action before applying; see AWS account and IaC.
The following describes the former TunaOS two-node Talos Kubernetes cluster in
AWS eu-north-1, built 2026-08-27 to consolidate the two Hetzner VPSes
(matrix and telengana). Those retired hosts must not be restarted. The home
Talos cluster is separately powered down in storage and expected back.
Historical descriptions and commands below refer to the cluster as it existed
before termination; they cannot be used to access it now. The nodes were not
in inventory.yml and no Ansible role targeted them.
Nodes
| Role | Instance | Labels |
|---|---|---|
| control-plane | m6i.2xlarge (8 vCPU / 32 GiB) | workload-role=hive |
| worker | m6i.xlarge | workload-role=matrix |
Each node had a private VPC address and an Elastic IP; addresses were recorded in OpenTofu state and talosconfig.
Talos v1.13.9, Kubernetes v1.36.2, flannel CNI, local-path storage.
The control-plane taint was removed so it could schedule workloads too. m6a
(AMD) was not offered in eu-north-1; m6i was the equivalent.
Network
Dedicated VPC 10.20.0.0/16, single AZ eu-north-1a, public subnet + IGW, no
NAT gateway.
Three security groups:
| Security group | Purpose |
|---|---|
| intra-cluster | self-referencing, node-to-node traffic |
| public | web ingress, Matrix federation, and the MatrixRTC media ports |
| admin | Talos and Kubernetes APIs, admin allowlist only |
Both nodes carried the public group, allowing either to serve web traffic. The
RTC ports matched what the ESS chart's MatrixRTC SFU exposed as NodePorts.
The admin allowlist was in aws/terraform.tfvars (Bitwarden aws-tofu-tfvars).
Ingress — the important gotcha
The Traefik Service was type: LoadBalancer and remained <pending>:
Talos on EC2 had no cloud-controller-manager to provision an ELB. Public traffic
worked because the Traefik pod bound hostPorts 80/443.
Driven from talos-k8s/traefik/values.yaml.
Three non-obvious settings, all load-bearing:
deployment.kind: DaemonSet. A single-replica Deployment is a SPOF whose rescheduling silently kills all ingress.maxSurge: 0 / maxUnavailable: 1. hostPorts make the chart default (surge-then-delete) impossible — the new pod can never bind ports the old one still holds, so rollouts hang with the replacementPendingforever. Costs a few seconds of downtime per Traefik upgrade.nodeSelector: workload-role=matrix. Running Traefik on every node was tried and reverted — see the incident note below.
DNS pointed at the worker's Elastic IP at the time.
Incident 2026-08-27: don't put workloads on an undersized control-plane
Running Traefik on both nodes tipped the control-plane over. Root cause was not
Traefik: the node was an m6i.large (7.7Gi) running etcd, the API server and
hive, whose agy/codex agent processes had grown into a 4Gi limit. Free memory
hit ~650Mi and kube-scheduler and kube-controller-manager were OOM-killed
(exit 137).
That deadlocks the cluster in a way that resists the obvious fix: with no
scheduler nothing can be placed, and kubectl scale cannot relieve the pressure
because actioning a scale-down is the controller-manager's job — and it was dead
too. The scale-to-zero appeared to succeed and did nothing. Recovery required
force-deleting the hive pods so the kubelet released memory.
Fixed properly by resizing the control-plane to m6i.xlarge (15.7Gi) — free
memory went from ~650Mi to ~12Gi — and keeping an explicit memory limit on hive
so the kubelet evicts hive, never the control plane.
2026-09-24: resized again to m6i.2xlarge. Three Hives on the control
plane pushed m6i.xlarge to a load of about 85 on 4 vCPU. Every kubectl exec took 10–35 s, the ops CronJobs hung until their deadlines, and agents
were slow to reach a ready prompt. The in-place resize through OpenTofu took
about 3 minutes of Hive downtime; Matrix on the worker stayed up.
The node-bound local-path PVC prevented hive from moving to the worker, so
the control-plane needed enough headroom for its workloads.
Former workloads
| Namespace | What | Exposure |
|---|---|---|
ess | Matrix (Synapse + workers, MAS, MatrixRTC, haproxy, redis, element-admin) | matrix/auth/call/matrixadmin.reilly.asia |
hive | tuna-os Hive (SCHOOL) + Discord bots | school.tunaos.org (legacy alias hive.tunaos.org), Cloudflare-proxied |
hive-reef | tuna-os Hive (REEF) | reef.tunaos.org |
hive-hub | Hive hub — every Hive's status in one place | hub.tunaos.org |
postgres | Postgres 16 — former production DB for Synapse + MAS | in-cluster only |
default | CFP review dashboard, searxng | Tailscale ingress |
cert-manager | Let's Encrypt via Cloudflare DNS-01 | — |
tailscale | Tailscale operator (ingress proxies) | — |
Former Postgres storage
Talos made the original plan to install Postgres on the host impossible. The
cluster's production Postgres ran in the postgres namespace, with durability
depending on local-path on the worker's root volume. A separate 50GB gp3
pgdata volume was attached at the EC2 level but not mounted by Talos.
Historical access
The old cluster is no longer reachable. Its kubectl/talosctl setup required
endpoints = public EIP, nodes = private IP —
not the same address for both. Talos does a two-hop call (client → endpoint →
node); pointing both at the public EIP makes the node try to reach itself via
its own Elastic IP from inside the VPC, which AWS's hairpin NAT does not
support. The symptom is a maximally confusing dial tcp …: i/o timeout that
looks like a raw network fault.
export KUBECONFIG=~/.kube/config-aws-migration
kubectl get nodes -o wide
# talosctl: endpoint = public EIP, node = private IP
talosctl --talosconfig ~/.talos/config-aws-migration \
-e <node-eip> -n <node-private-ip> version
The configs were stored in Bitwarden as kubeconfig-aws-migration /
talosconfig-aws-migration; the kube role still
fetches them separately from the home cluster's files. They no longer provide
access to this terminated cluster.
Historical cost context
The initial deployment used AWS credits, which have since been exhausted. See AWS account costs for the current caution.
Historical gaps
- The 50GB Postgres volume was unmounted.
- Nightly Postgres dumps were copied to S3 and size-verified; DLM snapshot schedules covered the root volumes. Retention of individual snapshots is not established by this historical record.
- API-server OIDC and finer-grained RBAC were deferred; access used an admin client certificate.
See also
AWS cluster teardown, 2026-10-25
The AWS Talos cluster, its KubeVirt node and all their disks were deleted on 2026-10-25 because the account's credits ran out. This page records what is gone, what was saved and where, and how to rebuild.
Why
The account ran on one credit: AWS Open Source Promotional Credits, issued $1,435.00, valid 2026-08-01 to 2028-07-31. It is not monthly. It was consumed in under three months (Aug $108, Sep $550, Oct $932 applied). October was the first month with a net bill ($66.15 by the 25th, AWS forecast $307). At the footprint before the shutdown the cost was about $41/day gross.
Earlier text in this repo calls the credits "permanent" or "$1400/mo". That was wrong. $1,400 is the migration budget's ceiling, not an allowance.
State now
| Thing | State |
|---|---|
aws-migration-worker-matrix, aws-migration-controlplane-hive, aws-migration-kubevirt | terminated |
Their EBS volumes (200, 400, 150 GB roots, 50 GB migration-pgdata) | deleted |
| 5 blank detached volumes created 2026-10-24 for an unstarted Hive-to-EBS migration | deleted |
| EBS snapshots in eu-north-1 (35) | deleted |
| punjab (t3.large, us-east-1), its 250 GB volume, one snapshot | kept, running |
Elastic IPs 13.62.161.5 (the matrix.reilly.asia target) and 13.63.243.56 | kept, unassociated |
| VPC, subnet, security groups, IAM, budgets, S3 bucket | kept |
Matrix (reilly.asia), the Hives (school., reef., hub.tunaos.org,
hive.reilly.asia), the CFP dashboard and Postgres are offline.
aws/ (OpenTofu) was not changed. The instances were removed outside it,
so just aws-plan will plan to create them again. That plan is the rebuild.
Where the data is
Bucket hanthor-fleet-backups-181185361136 (eu-north-1).
Cluster state: cluster-state-2026-10-25/cluster-state.tar.gz.age
Contains a Talos etcd snapshot (2,507 keys, includes every Kubernetes
Secret: the MAS and Matrix secrets generated by ESS, tokens, etc.) and a YAML
dump of 58 resource types. It is encrypted with age to five SSH keys: punjab,
and the james@kanpur, himachal, james@Kerala and
james@Jamess-MacBook-Air.local keys. Any one private key opens it:
aws s3 cp s3://hanthor-fleet-backups-181185361136/cluster-state-2026-10-25/cluster-state.tar.gz.age .
age -d -i ~/.ssh/id_ed25519 cluster-state.tar.gz.age | tar xz # etcd.snapshot, resources/*.yaml
Round trip checked: the decrypted archive hashes identically to the original. The private keys' own backup is not verified from here (Bitwarden was locked).
Volumes: postgres/shutdown-export-2026-10-25/
| Object | Content |
|---|---|
ess_ess-spindle-data.tar.gz | spindle store (the live homeserver) |
ess_synapse-media-preseed.tar.gz, ess_spindle-v0-1-0-rollback.tar.gz | media, rollback store |
hive-reef_hive-data.tar.gz, hive_hive-data.tar.gz, hive-hanthor_hive-data.tar.gz | the three Hives' data (about 43 GiB stored) |
hive_discord-reports-data, default_review-data, hive-hub_hive-hub-data, hive_hive-ops-state | small state |
tar | gzip -1, storage class GLACIER_IR (retrieval about $0.03/GB). Each
archive's tar size, recomputed on the node, equals the uncompressed size in its
gzip trailer. They were not restore-tested.
Postgres: postgres/2026-10-25/
Final dump, 4.1 GB: Synapse and MAS.
Permanently gone (not exported)
Contributor homes (claude-home 46 GB, pi-codex-home 43 GB, agy-home
15 GB), the spindle-rehearsal volumes, raw pgdata (the dump covers it),
hive-shared-auth (provider logins, sign in again), and everything that lived
on the KubeVirt node (monitoring data, VM disks, wootc). Workload replica
counts and cronjob suspend flags were saved as annotations and are inside the
etcd snapshot.
Rebuilding (outline, untested)
just aws-applyrecreates the three instances fromaws/. The Elastic IPs and private addresses are unchanged, so DNS and the Talos config still match.- Restore etcd onto the new control plane so the Secrets and workload
definitions come back, using the snapshot from the archive:
talosctl --talosconfig ~/.talos/config-aws-migration -e <eip> -n 10.20.1.10 bootstrap --recover-from=./etcd.snapshot(the machine secrets come from the OpenTofu state, which is what the nodes were built with). - Recreate each volume's directory under
/opt/local-path-provisioner/on the node that holds it (the PV objects name the directories) and untar the matching archive into it. A read-writehostPathhelper pod inkube-systemdoes this; other namespaces rejecthostPath. - Restore Postgres from the dump (see
talos-k8s/backup/), then scale the workloads back up from the saved annotationstunaos.org/pre-shutdown-replicasandtunaos.org/pre-shutdown-suspend.
Cold start of the control plane from its stopped disk was exercised once during the teardown (the API server and etcd came back intact). Nothing else here has been run end to end.
Matrix
Retired 2026-08-27. The Matrix homeserver moved first to the AWS Talos cluster, terminated 2026-10-25, and now runs on the OCI VPS. Restart nothing on this retired host; the page below is historical. See the Matrix cutover runbook.
Former VPS node in the fleet (vps group in inventory.yml) that hosted the
reilly.asia Matrix homeserver.
What it ran
- Ubuntu 24.04 LTS, x86_64
- Synapse + MAS on a single-node Kubernetes install, with PostgreSQL as the database
- Managed by the fleet playbook (daily cron apply, no secrets/homebrew)
Retirement
Synapse and MAS were scaled to zero at cutover and must stay that way: two live homeservers sharing one federation identity is unrecoverable. Automatic upgrades on the box were disabled so nothing resurrects them.
Lessons carried forward
- A single VPS with no swap and unmanaged journald growth was fragile for K8s; the former AWS cluster needed more headroom (see the OOM incident in its archived handbook).
reilly.asiaDNS is on Cloudflare, which made the cutover a DNS change rather than a server move.
Telengana
Retired 2026-08-27. The tuna-os Hive moved from this host to the former AWS Talos cluster, terminated 2026-10-25. School, Reef and Hub now run on the OCI VPS. Restart nothing on this retired host; the page below is historical.
Former VPS node in the fleet (vps group in inventory.yml).
What it ran
- Ubuntu 24.04 LTS, x86_64
- A single-node k3s hosting the tuna-os Hive
- Managed by the fleet playbook (daily cron apply, no secrets/homebrew)
Lessons carried forward
The box drifted from the fleet's hardening baseline (host firewall and
fail2ban were never enabled, and Tailscale had been logged out, leaving admin
access on the public interface). The former AWS cluster kept its admin APIs
behind an allowlisted security group, and fleet servers use the
server_hardening role.