Dotfiles Fleet Handbook

Ansible-driven dotfiles and infrastructure for a personal fleet. Three distinct concerns live here:

  1. Workstation config — shells, packages, browser, SSH, Tailscale, kubeconfig, GNOME, and more
  2. Home servers — bihar and karnataka (Bluefin + k3s, linked by KubeStellar); the hives run on bihar with karnataka as standby. The Talos manifests are archived
  3. Hosted infrastructure — OCI spindle (Matrix, public front door, hive failover coordinator) and the AWS account; the AWS Talos cluster is terminated

Current state of every host: Fleet status.

This handbook is public. It covers architecture, decisions and runbooks. Never put secrets in it. Fleet addresses live in inventory or come from just inventory; service runbooks may document public endpoints.

Quick Start

# Apply to local machine (most common)
just apply

# Apply specific tags only
just apply-tags kube,shell

# Apply to a remote machine
just apply-remote himachal

# Dry-run to see what would change
just check

# Lint all YAML and Ansible
just lint

Machine Fleet

Ansible-managed hosts, from inventory.yml:

MachineGroupTypeRole
kanpurdesktopLaptopPortable workstation
himachaldesktopLaptopPortable workstation; legacy hive_ops timers retired
dillidesktopDesktopSecondary workstation
keraladesktopMobilepostmarketOS ARM device
mumbaidesktopVMDebian VM on the phone, machine_profile: cli-only
punjabserverVMAgent box, KubeVirt VM on karnataka
bihardesktopServerHome server: active hive host, KubeStellar hub
goaserverServerRaspberry Pi 5 ARM server
vmserverVMLocal dev VM
termuxtermux_hostsPhoneRaw Termux/Android layer
test-fleet-fedora, test-fleet-node2test_fleetVMKubeVirt test VMs (vm-test)
matrixvpsVPSRetired — see the cutover runbook
telenganavpsVPSRetired — see the cutover runbook

Outside this inventory:

  • karnataka — home k3s/KubeVirt server, configured from the native-units branch. See Karnataka.
  • Spindle — OCI Frankfurt VPS: Matrix, LiveKit, the Caddy front door for the hives, and the hive failover coordinator.
  • The AWS Talos cluster was terminated on 2026-10-25, and the home Talos cluster was replaced by k3s (archive).

The llm group is dormant (commented out in inventory.yml).

Playbook Phases

site.yml groups roles into four commented phases. Tags cut across them — use just apply-tags <tag> to target a subset.

PhaseRolesMain tags
1 — System + packages + dotfilessudo, sshd, apk_packages, homebrew, termux_packages, bitwarden, shell_fonts, shell_dotfiles, omp, hive_ops, git, neovimsystem, packages, dotfiles
2 — Secrets + authshell_atuin, shell_ai, ssh_keys, ssh_mesh, github, tailscale, kube, forgejo_registrysecrets
3 — Desktop appsflatpak, bluefin_common, gnome, zen_browser (+ browser_fxa), pipewire_audiodesktop
4 — Servicessyncthing, systemd, bst_dashboard, kirocrew, proxy, tailscale_cert, server_hardeningservices

Only the secrets-tagged work needs Bitwarden. With the vault locked (or --skip-tags secrets), phases 1, 3 and 4 run normally and BW-touching tasks skip cleanly.

Architecture

Repository Layout

dotfiles/
├── site.yml              # Main playbook — entrypoint for all machines
├── inventory.yml         # Host inventory + group definitions
├── Justfile              # Task runner (just apply, just lint, etc.)
├── aws/                  # OpenTofu for the TunaOS AWS account and cluster
├── group_vars/
│   ├── all.yml           # Shared vars: packages, flatpaks, fleet config
│   └── vps.yml           # VPS-specific overrides
├── host_vars/
│   └── <machine>.yml     # Per-machine settings (is_laptop, skip_*, etc.)
├── roles/
│   └── <role>/           # Each role: tasks/, files/, templates/, vars/
├── docs/                 # This handbook
├── scripts/              # Utility scripts (inventory scan, BW seed)
├── talos-k8s/            # Talos Linux + Kubernetes cluster manifests
└── tools/                # Standalone tools, including the secret-broker

How It Works

Each machine manages itself via ansible-playbook --connection=local. There's no central control node. This is an Ansible pull model — the playbook lives on every machine and each host applies it locally.

Local Apply

just apply

Runs the playbook locally with --connection=local. The playbook auto-detects which host it's running on by matching /etc/dotfiles-machine or falling back to localhost.

Remote Apply

just apply-remote himachal

SSHes to the target, forwards the Bitwarden session token, pulls the latest dotfiles, then runs just apply locally on the remote machine.

Tagged Apply

just apply-tags kube,shell

Runs only roles tagged with the given tags. Useful for deploying specific configs without running the full playbook. --skip-tags works the same way in reverse — just apply-nosecrets is --skip-tags secrets under the hood (see Bitwarden).

Available Tags

Every role in site.yml carries one or more tags. Tags fall into two kinds: phase tags shared by roles in that phase, and role tags naming (or further scoping) one specific role. Phases follow the # Phase N comments in site.yml.

PhaseRoleTags
(pre/post tasks)fact setup, platform detection, secrets summaryalways — runs on every tagged apply unless --skip-tags always
(before Phase 1)sudosystem, sudo
System + packagessshdsystem, sshd
System + packagesapk_packagessystem, packages, apk
System + packageshomebrewpackages, homebrew
System + packagestermux_packagessystem, packages, termux
System + packagesbitwardensecrets, bitwarden
System + packagesshell_fontsdotfiles, shell, fonts
System + packagesshell_dotfilesdotfiles, shell, dotfiles
System + packagesompdotfiles, omp
System + packageshive_opshive, hive_ops
System + packagesgitdotfiles, git
System + packagesneovimdotfiles, neovim, editor
Secrets + authshell_atuindotfiles, shell, atuin, secrets
Secrets + authshell_aidotfiles, shell, ai, secrets
Secrets + authssh_keyssecrets, ssh, ssh_keys
Secrets + authssh_meshssh, mesh_check
Secrets + authgithubsecrets, github
Secrets + authtailscalesecrets, tailscale
Secrets + authkubesecrets, kube
Secrets + authforgejo_registrysecrets, forgejo, registry
Desktop appsflatpakpackages, desktop, flatpak
Desktop appsbluefin_commondesktop, bluefin
Desktop appsgnomedesktop, gnome
Desktop appszen_browserdesktop, browser, zen_browser
Desktop appspipewire_audiodesktop, audio, pipewire
Servicessyncthingservices, syncthing
Servicessystemdservices, systemd
Servicesbst_dashboardservices, bst_dashboard
Serviceskirocrewservices, kirocrew
Servicesproxyservices, proxy, caddy
Servicestailscale_certservices, tailscale_cert
Servicesretired_servicessystem, retired_services
Servicesserver_hardeningsystem, hardening, security
Servicesfleet_factsfacts, fleet_facts

Common phase-wide selections:

  • --tags secrets — every BW-backed role only (bitwarden, shell_atuin, shell_ai, ssh_keys, github, tailscale, kube, forgejo_registry)
  • --tags desktop — every desktop-app role (flatpak, bluefin_common, gnome, zen_browser, pipewire_audio)
  • --tags services — every long-running service role (syncthing, systemd, bst_dashboard, kirocrew, proxy, tailscale_cert)
  • --skip-tags secrets — everything except BW-backed roles. This is just apply-nosecrets, which the daily dotfiles-update.timer and the dots alias run; bootstrap.sh and phone-apply.service pass --skip-tags secrets directly

Tags do not replace the skip_* host_vars flags or inventory-group gating described below — a role can be tag-selected and still no-op if its when: condition (group membership, skip_*, machine_profile) evaluates false.

Role Structure

Each role follows the standard Ansible layout:

roles/<name>/
├── tasks/
│   └── main.yml         # Role entrypoint
├── files/               # Static files to deploy
├── templates/           # Jinja2 templates (with .j2 extension)
├── vars/                # Role-specific variables
│   └── main.yml
├── meta/
│   └── main.yml         # Explicit prerequisites for this role
└── handlers/            # Notify targets
    └── main.yml

Role Dependencies

Prerequisites that provide required facts, files, or services are declared in roles/<name>/meta/main.yml, not only by their position in site.yml. The playbook still owns fleet-wide phase ordering, host conditions, and tags; the metadata makes the prerequisite relationship part of the role interface.

RoleDependencyContract
shell_atuinbitwardenReads the Atuin login item and bw_unlocked
shell_aibitwardenReads API credentials for generated shell files
ssh_keysbitwardenRecovers and mirrors the machine key
githubssh_keysRegisters the public key maintained by ssh_keys
kubebitwardenReads the cluster configuration notes
forgejo_registrybitwardenReads the registry login item
tailscalebitwardenReads the join key when the host is not connected

Optional integrations keep their existing standalone behavior. For example, browser_fxa can receive credentials directly and therefore does not declare a Bitwarden dependency.

Conditional Execution

Roles are gated by:

  • Inventory groups — when: is_desktop | bool runs only on desktop group members
  • Host vars — skip_*: true flags in host_vars/<machine>.yml skip specific roles
  • Tags — --tags kube,shell limits which roles execute

Skip Flags

Every flag site.yml reads, all defaulting to false (| default(false)). Set skip_<name>: true in host_vars/<machine>.yml to skip that role on that machine. No role reads a skip_* variable itself — the gating is all in site.yml.

FlagRole skippedOther when: gates on the roleSet in host_vars/
skip_sudosudo—termux
skip_ssh_meshssh_meshnot vm-test—
skip_kubekubedesktop, not vm-testmumbai, termux
skip_forgejoforgejo_registrynot vm-test—
skip_flatpakflatpakdesktop, not vm-testmumbai, termux
skip_bluefinbluefin_commondesktop, not vm-testmumbai, termux
skip_gnomegnomedesktop, not vm-testmumbai, termux
skip_zen_browserzen_browserdesktop, not vm-testmumbai, termux
skip_pipewire_audiopipewire_audiodesktop, not vm-testmumbai
skip_syncthingsyncthingnot vps, not vm-testmumbai, termux
skip_systemdsystemd—termux
skip_proxyproxynot vps, not vm-testmumbai, punjab, termux
skip_tailscale_certtailscale_certnot vps, not vm-testmumbai, termux

"desktop" is is_desktop (membership of the desktop group unless overridden); "not vm-test" is machine_profile != 'vm-test'. Roles without a flag either always run or are gated another way: termux_packages (termux_hosts group), omp (not Termux), kirocrew (kirocrew_enabled: true), server_hardening (server/vps groups), retired_services (not vps, not Termux, systemd hosts).

Secrets Flow

  1. Ansible requests a Bitwarden session (env var, cached file, or interactive unlock)
  2. The bitwarden role resolves BW_SESSION and caches it
  3. Secrets-tagged roles consume the session token for API calls
  4. For remote applies, scripts/bw-resolve.sh remote <host> unlocks the vault on the target and the recipe exports that session inline over ssh -t (no SendEnv) — see Remote Apply

No secrets are stored in git — the repo is fully public.

Inventory & Groups

Host Groups

inventory.yml defines these groups:

desktop

Workstations and laptops with a full desktop environment (GNOME, browsers, etc.).

desktop:
  hosts:
    mumbai:     # Debian VM on the phone (cli-only, skips desktop roles)
    kerala:     # postmarketOS ARM device
    kanpur:     # Laptop
    himachal:   # Laptop (runs hive_ops)
    dilli:      # Secondary desktop

Desktop roles: flatpak, gnome, zen_browser, bluefin_common, pipewire_audio, kube

server

Headless hosts.

server:
  hosts:
    punjab:   # agent box, KubeVirt VM on karnataka (100.103.187.46)
    vm:       # Local dev VM
    goa:      # Raspberry Pi 5 (ARM)

Server roles: server_hardening, plus the service roles.

vps

The inventory lists two retired Hetzner VPSes, formerly migrated to the AWS Talos cluster, which was terminated on 2026-10-25. Restart nothing on these hosts; follow the Matrix cutover runbook before touching either. Active spindle is an OCI Frankfurt VPS running Matrix and standalone LiveKit; it is not in this inventory.

vps:
  hosts:
    matrix:     # Retired — former Matrix homeserver
    telengana:  # Retired — hosted the Hive on k3s

vps members skip syncthing, proxy and tailscale_cert, and get server_hardening. group_vars/vps.yml is currently empty.

test_fleet

KubeVirt VMs reached via virtctl port-forward (see the host's host_vars). Both set machine_profile: vm-test, which skips desktop and service roles.

test_fleet:
  hosts:
    test-fleet-fedora:
    test-fleet-node2:

termux_hosts

The raw Termux/Android layer on the phone. Gates the termux_packages role.

termux_hosts:
  hosts:
    termux:

llm (dormant)

Commented out in inventory.yml — karnataka, its former member, now runs k3s workloads.

Not in the inventory

bihar is in the desktop group (Bluefin, bootstrapped 2026-11-17). karnataka is configured from the native-units branch (#348) and is not listed here. Neither is a Talos node any more; see Fleet status. spindle is not in the inventory. The terminated AWS cluster's nodes were not in the inventory either.

Per-Machine Configuration

Each machine has a host_vars/<name>.yml file with common knobs:

is_laptop: true        # Laptop (runs on login, not timer)
skip_flatpak: true     # Skip Flatpak installs
skip_gnome: true       # Skip GNOME config
skip_kube: true        # Skip kubeconfig deploy
skip_proxy: true       # Skip Caddy proxy
machine_profile: cli-only  # Free-form profile; `vm-test` skips desktop + service roles
web_services: []       # Service links for the local dashboard

The full set of skip_* flags site.yml honours: skip_sudo, skip_ssh_mesh, skip_kube, skip_forgejo, skip_flatpak, skip_bluefin, skip_gnome, skip_zen_browser, skip_pipewire_audio, skip_syncthing, skip_systemd, skip_proxy, skip_tailscale_cert.

Group Variables

  • group_vars/all.yml — Homebrew package lists, Flatpak remotes, Firefox Accounts config, GNOME extensions, authorized SSH keys
  • roles/flatpak/vars/main.yml — the system_flatpaks list
  • group_vars/vps.yml — currently empty

Secrets with Bitwarden

All secrets live in Bitwarden and are resolved at runtime.

How It Works

  1. The bitwarden role runs first in the secrets phase.
  2. It resolves a session token in order: BW_SESSION env var → /tmp/bw_session → interactive bw unlock.
  3. It runs bw sync (with a 15 s hard timeout, so a flaky vault server can't hang the play) to refresh the local cached vault.
  4. It calls bw status and publishes a secrets_available host fact: true only if the CLI is installed, a session was resolved, and the vault actually unlocks.
  5. Every BW-using role gates on when: secrets_available | default(false). The end-of-play post_tasks summary prints one of:
    • vault unlocked — all secret roles executed normally
    • Bitwarden vault was LOCKED — … skipped. To enable: bw unlock …
    • SKIPPED (--skip-tags secrets). Run dots-apply to include …

Remote Apply

just apply-remote <host> (and just apply-remote-tags <host> <tags>):

  1. Runs scripts/bw-resolve.sh remote <host>, which unlocks Bitwarden on the remote and prints that host's session token. If the remote CLI is logged out it tries bw login --apikey with the bw-api-key item from your local vault; to unlock it pipes the master password (item James Bitwarden) over stdin, falling back to an interactive bw unlock over ssh -t.
  2. Runs ssh -t <host> "export BW_SESSION='…'; …; just apply" — the session is exported inline in the remote command, not via SendEnv. apply-remote also caches it in the remote's /tmp/bw_session.
  3. The bitwarden role on the remote runs bw status: if it reports unlocked, secrets_available becomes true and secrets work proceeds.

Caveat — session portability is not guaranteed. A BW_SESSION token is only valid on the machine that unlocked it — that's why the recipes unlock on the remote instead of forwarding yours. If the remote unlock fails, the remote bw status returns locked, the role prints a one-line warning, and every BW-touching task is skipped cleanly — nothing fails, no files get clobbered with empty values. Unlock the vault directly on that host (bw unlock, then write the session to /tmp/bw_session) to enable secrets there.

Vault Items

Every item a role or script reads (grep bw get / bw list under roles/ and scripts/):

Bitwarden ItemTypeUsed By
james@<machine>SSH Keyssh_keys role — per-machine ed25519 keys
tailscale-apikeyLogin (password)tailscale role — key for tailscale up --authkey and stale-device cleanup via the API
kubeconfigSecure Notekube role — home cluster kubeconfig
talosconfigSecure Notekube role — home cluster Talos config
kubeconfig-aws-migrationSecure Notekube role — retained kubeconfig for the terminated AWS cluster
talosconfig-aws-migrationSecure Notekube role — retained Talos config for the terminated AWS cluster
accounts.firefox.comLogin + TOTPbrowser_fxa role — Firefox Account credentials
github-tokenLogin (password)github role — PAT for gh auth login
atuin.shLogin + custom field keyshell_atuin role — atuin sync login + mnemonic
deepseek-api-keyLogin (password)shell_ai role — pi auth.json
forgejoSecure Note (API Token: line)shell_ai role — FORGEJO_TOKEN
tavily-api-keyLogin (password)shell_ai role — TAVILY_API_KEY
forgejo.manatee-basking.ts.netLoginforgejo_registry role — podman registry login
bw-api-keyLoginscripts/bw-resolve.sh — bw login --apikey on a remote
James BitwardenLoginscripts/bw-resolve.sh — non-interactive unlock on a remote

Recovery items (not read by any role)

These items are kept so that you can rebuild a lost host by hand. No role reads them.

Bitwarden ItemTypeHolds
hive-github-app-hanthor-hive-agentSecure NoteSchool's GitHub App: App ID, installation ID, PEM private key (the hub mounts the same key). See Hive
hive-github-app-tuna-reef-hiveSecure NoteReef's GitHub App: App ID, installation ID, PEM private key
hive-spindle-envSecure NoteThe hives' .env (hub secret, dashboard token, omp broker token, provider keys). It was /srv/hives/.env on spindle and is now /var/srv/hives/.env on bihar and karnataka. The item name has not changed

Running Without Secrets

For fast iteration that doesn't need the vault — packages, dotfiles, desktop config — use the explicit no-secrets path:

just apply-nosecrets   # git pull + apply with --skip-tags secrets
dots                   # alias: cd to the repo + just apply-nosecrets

The daily systemd timer (dotfiles-update.service) calls apply-nosecrets for the same reason: it can't unlock the vault non-interactively, so it doesn't try.

No Session?

If Bitwarden can't unlock (first run, no BW CLI installed, vault genuinely locked on this host), every secret-touching task is skipped automatically. Non-secret roles (system, packages, dotfiles, desktop, most services) run normally and the play exits successfully — the end-of-play summary tells you exactly which roles didn't get to do their BW work.

Seeding New Secrets

# Push kubeconfig + talosconfig (home cluster) to Bitwarden from a machine that has them:
just seed-kube

seed-kube only handles the two home-cluster notes; the *-aws-migration notes are maintained by hand.

Onboarding a New Machine

Before secrets or SSH can flow, the machine needs to be on the tailnet. Instead of copying an auth key, join it by scanning a QR with your phone — run this on the new machine:

just tailscale-qr              # uses default machine name
just tailscale-qr name=custom  # or provide a custom name

This runs interactive tailscale up --qr: it renders the login URL as a QR in the terminal. Scan it with your phone and approve the device in the Tailscale admin (passwordless via your IdP). No auth key, no Bitwarden, no master password.

Once the machine is on the tailnet, its tailnet identity becomes its first credential — every other onboarding step (secret delivery, apply, SSH) flows from there. This is Phase 0 of the QR secret onboarding effort; the later phases replace the auth-key/master-password handshake below entirely.

Requires you to be a tagOwner of tag:fleet in the tailnet ACL (every fleet node carries this tag). Drop --advertise-tags in the recipe to join as a personal device instead.

Then continue with Path A or B below.


Two paths depending on whether the machine is already reachable.

Path A: Machine Already Online (Tailscale + SSH)

# From any machine with the dotfiles repo:
just add-machine <name> <type>

This:

  1. If the machine isn't in inventory.yml yet, adds it (ansible_host: localhost, ansible_connection: local) and puts it in the <type> group
  2. Creates an empty host_vars/<name>.yml — add per-machine knobs there afterwards
  3. Commits and pushes
  4. SSHes in (ssh -t <name>) and runs bootstrap.sh --name <name> --type <type> there — the same bootstrap as Path B

Path B: Fresh Install

# From any machine with the dotfiles repo:
just onboard <name> <type>

This prints a bootstrap command to run on the new machine:

# On the new machine:
curl -fsSL https://raw.githubusercontent.com/hanthor/dotfiles/master/bootstrap.sh \
  | bash -s -- --name <name> --type <type>

The bootstrap script:

  1. Ensures python, git, uv and Ansible, then clones the dotfiles repo from GitHub
  2. Registers the machine in inventory.yml if needed (prompts for the type when --type isn't given)
  3. Writes /etc/dotfiles-machine so the playbook can self-identify, sets the hostname, sets up sudo and Tailscale
  4. Runs site.yml with --skip-tags secrets (installs Homebrew and core packages among the rest)
  5. If the Bitwarden CLI is now available and unlocks, runs site.yml --tags secrets and pushes any inventory changes

After Bootstrap Completes

Bootstrap ends with shell environment setup instructions. The critical next step is to start a new shell and run the first full apply:

# Start a fresh shell to load updated environment (e.g., Homebrew paths)
exec zsh

# Run the first full apply (with secrets)
just apply

Why this step matters:

  • Bootstrap runs site.yml --skip-tags secrets, which skips Bitwarden integration
  • just apply runs the full playbook with secrets, unlocking Bitwarden and completing setup
  • This pull-in step distributes SSH keys across the fleet, registers with GitHub and Tailscale, and sets up daily/on-login automation

Do not edit host_vars/<name>.yml before the first just apply — bootstrap sets sensible defaults, and just apply will pull in critical secrets (SSH keys, GitHub tokens, Tailscale auth). Any per-machine customizations can follow once the first apply succeeds; the skip_* flags that turn individual roles off are listed in Architecture → Skip Flags.

The details of what happens during this first apply are covered in the Post-Onboarding section below.

Machine Types

<type> is the inventory group to join. bootstrap.sh offers desktop, server and vps:

TypeGroupHas Desktop?Notes
desktopdesktopYesLaptops are desktops too — set is_laptop: true in host_vars
serverserverNoGets server_hardening; desktop roles skip automatically
vpsvpsNoAlso skips syncthing, proxy, tailscale_cert

There is no laptop type — a type that doesn't match a group leaves the host in no group. test_fleet and termux_hosts hosts are added to inventory.yml by hand.

Post-Onboarding

  1. The ssh_keys role generates a fresh ed25519 key on the new host, then pushes the pub key up to Bitwarden as james@<hostname>.
  2. The new host now needs to trust the rest of the fleet, and the rest of the fleet needs to trust it. The first apply on the new host downloads every other host's pub key into ~/.ssh/authorized_keys. The other hosts will pick up the new host's pub key on their next full apply (dots-apply on each — the timer runs without secrets, so it skips ssh_keys).
  3. The github role registers the SSH key with GitHub for both auth and commit signing.
  4. The tailscale role joins the machine to the Tailscale network using the auth key in BW.
  5. The daily dotfiles-update.timer (servers) or on-login service (laptops) runs just apply-nosecrets — secrets stay on the manual dots-apply cadence because BW can't be unlocked non-interactively.

Run just doctor on the new host once it's bootstrapped — it'll tell you whether BW is unlocked, whether the timer is wired up, and the timestamp of the last successful apply.

See Architecture for how the playbook runs on each machine and Secrets with Bitwarden for the details of the BW handshake.

QR Secret Onboarding

Tracking issue: hanthor/dotfiles#49

Onboard (or re-align) a machine and deliver its secrets seamlessly and securely by scanning a QR from your phone — no master password typed on the new box, no hand-copied BW_SESSION, no chicken-and-egg SSH setup.

The core problem: bootstrap-of-trust

The hard part is delivering the first credential to a machine that has nothing. Once the box holds any one of {tailnet membership, a single Bitwarden credential}, everything else — the full BW_SESSION, kubeconfig, SSH keys, Tailscale auth key — already flows through existing automation (bw-resolve.sh, the tailscale/kube/ssh_keys roles). So the whole question is: how does that first secret arrive?

The spine: Tailscale identity as the bootstrap credential

We already run Tailscale everywhere, so the passwordless + QR experience is mostly already in the stack:

  1. Approve the new device on your phone. tailscale up --qr renders the login URL as a QR; scan it and approve in the Tailscale admin (passwordless via the IdP). The box joins the tailnet, tagged tag:fleet.
  2. The machine's tailnet identity is then its first credential. A small tailnet-only secrets broker sees the authenticated node identity and hands it exactly the secrets it needs, read from Bitwarden server-side.
  3. No secret ever rides in the QR — the QR carries a login URL; the phone approves; the identity does the rest.

End-to-end flow

sequenceDiagram
    participant N as New machine
    participant P as Your phone
    participant TS as Tailscale admin
    participant B as secret-broker (tailnet)
    participant BW as Bitwarden Secrets Manager

    Note over N,TS: Phase 0 — join the tailnet
    N->>N: just tailscale-qr (renders login QR)
    P->>TS: scan QR + approve device
    TS-->>N: authenticated; tagged tag:fleet, gets a StableNodeID

    Note over N,BW: Phase 1 — first secrets, over the tailnet
    N->>B: GET /v1/secrets (identity = WhoIs, unforgeable)
    B-->>N: 202 pending + fingerprint
    P->>B: just broker-pending (see stableID + fingerprint)
    P->>B: just broker-approve <stableID> (match fingerprint)
    N->>B: GET /v1/secrets (retry)
    B->>BW: bws get (only this node's policy refs)
    BW-->>B: secret values
    B-->>N: 200 + least-privilege secrets

Bitwarden stays the source of truth — we're replacing the bootstrap, not the vault. (The bw CLI cannot do passwordless "log in with device"; it only supports password / --apikey / --sso, so we don't build on that.)

Bonus: adopting Tailscale identity here also opens the door to Tailscale SSH (auth by tailnet identity, ACL-gated), which would have prevented the "kerala couldn't SSH anywhere" incident — there'd be no authorized_keys to fall out of.

Phases

PhaseWhatStatus
0QR tailnet join — just tailscale-qr renders the tailscale up login URL as a console QR; scan + approve on phone.Done
1Identity broker — a tailnet-only service (tools/secret-broker) that issues per-node least-privilege secrets from Bitwarden Secrets Manager, gated on unforgeable StableNodeID + operator approval (mirrors Tailscale device approve).Code-complete — deny-path unit-tested; live run pending (needs TS authkey + BWS token per the runbook)
2Hardening — ACL restricting tag:fleet→broker port, wiring the client into onboarding once run live, richer per-node policy config.Planned

Phase 0 (implemented)

# On the new machine:
just tailscale-qr <name>

Runs interactive tailscale up --qr --hostname=<name> --advertise-tags=tag:fleet --accept-routes. Uses the Tailscale CLI's native --qr flag (confirmed on the fleet's 1.98.x) — no qrencode dependency. See the tailscale-qr recipe in the Justfile; if tailscale isn't installed yet, the recipe installs it first.

Phase 1 (code-complete, live run pending)

Once the node is on the tailnet, it fetches its first secrets from the broker, gated on its unforgeable tailnet identity plus a one-time operator approval:

just broker-request              # on the NEW node — prints its fingerprint + "pending"
just broker-pending              # on your ADMIN node — see the queued request
just broker-approve <stableID>   # approve that specific node (verify the fingerprint)
just broker-request              # on the NEW node — now returns its secrets

Authorization is scoped to the StableNodeID (unforgeable), never hostname; approval is per-node (no time-window race); secrets come per-node least-privilege from Bitwarden Secrets Manager (read-scoped machine-account token, no vault master password). Full design, security model, blast-radius note, and the operator runbook for the live setup (mint TS authkey, provision the BWS project/token, systemd unit, ACL) live in tools/secret-broker/README.md. The client recipes are manual/opt-in — deliberately not wired into just apply/onboarding until the flow has been run live once.

Security requirements (all phases)

  • No long-lived secret encoded in a QR that persists on screen/scrollback.
  • Tokens are short-TTL and single-use; pairing nonces expire fast.
  • Out-of-band verification fingerprint on both phone and machine (anti-MITM).
  • Master password stays on the phone; never keyed into the onboarding box.
  • Transport is the tailnet (WireGuard) or otherwise E2E-encrypted.
  • Repo stays public — code and docs only, never secrets.

References

Operational runbook

Recipes for the questions that recur. If you find yourself answering one of these from memory or commit logs more than twice, add it here.


I rotated my SSH key on host X — how do I propagate it?

An intentional local rotation is explicit: when the local public key differs from the Bitwarden item, the ssh_keys role leaves Bitwarden unchanged and warns. This protects the stored key if a new key was generated while the vault was locked. To publish an intentional rotation, on host X with BW unlocked:

# `dots-apply` forwards these arguments to the Ansible playbook.
dots-apply -e ssh_keys_force_push=true

That updates the james@<hostname> item in Bitwarden and refreshes this host's authorized_keys while it is at it. Then on every other host:

dots-apply           # picks up the fresh pubkey from BW into authorized_keys

If the mismatch was not intentional, do not use the force flag: move the local key aside and re-apply so the host fetches the Bitwarden copy.

If you can't reach an "other" host because of the stale key situation, you can also push the new key directly first:

# From a host that can still reach the broken one
ssh-copy-id <broken-host>     # adds the rotating host's pub key by hand
# then `dots-apply` on broken-host as above

BW is locked on a remote — how do I unlock it there?

just apply-remote / apply-remote-tags try to unlock the vault on the remote themselves (scripts/bw-resolve.sh remote), but that can fail — e.g. no bw-api-key item, or the remote CLI was never logged in. To unlock directly on that host:

# From your laptop, let bw-resolve.sh handle unlock internally:
just apply-remote-tags <host> bitwarden,ssh_keys

The scripts/bw-resolve.sh remote <host> script (invoked internally) will SSH to the remote, ensure Bitwarden is logged in and unlocked (either via stored API key, master password from your local vault, or interactive prompt), and return the session token. The bitwarden role will then confirm the vault is unlocked, and downstream secrets roles will run.


The timer apply failed last night — what do I do?

The user-level dotfiles-update.service runs just apply-nosecrets daily, which now writes ~/.cache/dotfiles/last-apply.json. Quickest signal:

just doctor                 # `→ Last apply` line shows status + age

If rc != 0, see what blew up:

journalctl --user -u dotfiles-update.service -e --no-pager | tail -80

The most common causes:

  • git pull failure (DNS / network blip): the recipe tolerates it with git pull --ff-only || true; the ansible run continues with whatever's on disk. If the working tree drifted from main, fix manually.
  • a role started failing after a config change: re-run interactively with just apply-nosecrets and read the failure.
  • mkswap: /swapfile is mounted (resolved): server_hardening now gates mkswap/swapon on swapon --show, so an already-active swapfile is left alone. If you still see this, the host is running an old checkout.

BW master password just returned "decryption failed" — vault corrupt?

Two BW CLI processes can corrupt the local encrypted vault if they sync at the same time. Symptom: bw unlock --raw returns Cryptography error, The decryption operation failed.

Fix is to re-login (downloads a clean vault):

bw logout
bw login              # email + master password + 2FA
export BW_SESSION=$(scripts/bw-resolve.sh local)

The bitwarden role now has a 15 s timeout on bw sync, which prevents the race in most cases going forward.


I want to onboard a new machine — full sequence

# 1. From an existing fleet machine (with BW unlocked):
just onboard <newhost> desktop      # or server / vps

# 2. Copy the printed bootstrap command, then on the new machine:
curl -fsSL https://raw.githubusercontent.com/hanthor/dotfiles/master/bootstrap.sh \
  | bash -s -- --name <newhost> --type desktop

# 3. After bootstrap finishes, on the new machine, get BW into a usable state:
bw login                            # if not logged in
export BW_SESSION=$(scripts/bw-resolve.sh local)  # unlocks vault

# 4. Run the full apply (this pushes the new machine's pubkey to BW):
dots-apply

# 5. Back on each existing host (the no-secrets timer skips ssh_keys):
dots-apply                          # picks up the new host's pubkey

Verify with just doctor on the new host — should be all green and report "last apply ago".


I added a flatpak / brew package — how do I roll it out?

Edit the package list — system_flatpaks in roles/flatpak/vars/main.yml, or the brew lists (core_brews, desktop_brews, …) in group_vars/all.yml, which render the Brewfile — then commit, push. Every host picks it up on the next dots-apply or the next timer fire.

To force-apply immediately on the local machine:

just apply-tags packages

I want to disable a role on one host

In that host's host_vars/<name>.yml, add a skip_<role>: true line. The flags site.yml actually checks:

skip_sudo, skip_ssh_mesh, skip_kube, skip_forgejo, skip_flatpak, skip_bluefin, skip_gnome, skip_zen_browser, skip_pipewire_audio, skip_syncthing, skip_systemd, skip_proxy, skip_tailscale_cert.

Roles without a flag are gated some other way — server_hardening on server/vps membership, termux_packages on termux_hosts, many roles on machine_profile != 'vm-test'. Grep site.yml.


How do I verify a change made it to every host?

just doctor-fleet

Runs just doctor on every online host in parallel. The → Last apply line tells you which hosts have an old or failed convergence — those are the ones that need attention.

To roll a single change out manually across online hosts instead of waiting for the timer:

# Check online tailnet hosts in fleet inventory
just online

# Apply specific tags to all online fleet machines in parallel
just apply-online-tags <tag-or-tags>

# Or perform a full apply across all online fleet machines
just apply-all

Self-improving repo: bots, agents and guardrails

This repo is edited by bots, and every fleet machine pulls master daily and applies it with sudo. A bad merge therefore reaches every machine within 24h. The automation below is set up so that routine improvements land without you, and anything that executes on the fleet still waits for you.

Who changes what

ActorWhat it doesMerges by itself?
Renovateminor/patch/digest bumps (pi npm deps, Go, GitHub Actions, OpenTofu providers), weekly lockfile maintenanceyes, once CI is green
Renovate (cluster)image/chart bumps in talos-k8s/, grouped weekly, label deploy-neededno — manifests are applied by hand, so a merge would only make git lie about the cluster
Hive (school and reef spokes on bihar, with karnataka as standby; Apps hanthor-hive-agent and tuna-reef-hive)its agents open PRs labelled hold; the Hive itself only merges PRs labelled lgtmonly if the PR stays inside the carve-outs below. Neither hive's roster includes this repo today (both cover only tuna-os/*), so no Hive PRs are expected here unless a hanthor spoke is rebuilt
Youdirect pushes, just onboard, etc.admin bypass — unaffected

The gate

Three pieces, and none of them trusts the PR itself:

  1. Branch protection on master: required checks lint, python, tests, tofu, justfile, broker, deps-guard, plus code-owner review. Admins can bypass, so your own pushes still work.
  2. .github/CODEOWNERS is deny-by-default (* @hanthor). The paths carved out are the ones that never run on a fleet machine: docs/, *.md, tests/, talos-k8s/, pi skill text, and dependency manifests. .github/ is always owned, so nobody can loosen the gate from inside a PR.
  3. agent-automerge.yml queues GitHub auto-merge on every Hive PR. It runs from master's copy (pull_request_target), never checks out PR code, and labels PRs that touch owned paths needs-human, with a comment listing the paths.

deps-guard lets Renovate change versions in roles/pi/files/package.json but fails if a PR adds or removes a dependency. The manifest is tracked by repository automation; the omp role copies roles/pi/files/skills/ and installs OMP plugins separately. It does not run npm install from this manifest on fleet machines.

Day to day

  • Nothing to do for docs, tests or dependency bumps. They merge once green.
  • Review anything labelled needs-human (Hive) or deploy-needed (Renovate, cluster). The Discord daily digest lists them under Needs you.
  • To let the Hive merge more, carve the path out in CODEOWNERS (and mirror it in agent-automerge.yml's comment logic). Only do that for paths that never execute on the fleet.

Hive config lives on its volume

Each live spoke persists its configuration to data/hive.yaml.runtime next to the seed config/hive.yaml, and the runtime file wins on restart, so roster or level changes made in the dashboard stick. hive.reilly.asia is the hanthor spoke on bihar (rebuilt 2026-11-19); its seed is hives/hanthor/hive.yaml. The AWS-era ConfigMap in talos-k8s/hive/spokes/hanthor/manifest.yaml is history. For the live files, read-only inspection and the safe edit procedure, see the Hive operations guide:

ssh james@bihar 'sudo podman ps --filter name=hives-; sudo cat /var/srv/hives/hanthor/data/hive.yaml.runtime'

Utility Scripts

The scripts/ directory contains helper utilities for inventory management, Bitwarden integration, and operational tasks. They are not deployed to machines — they run on your local development machine or during CI.

Inventory Management

register-machine.py

Register a new machine in inventory.yml and create its host_vars file.

Usage:

python3 scripts/register-machine.py <name> [type] [path-to-inventory.yml]

Arguments:

  • <name> — machine name (e.g., kanpur, goa)
  • [type] — group/machine type: desktop, server, vps, termux_hosts (default: desktop)
  • [path-to-inventory.yml] — inventory file path (default: inventory.yml)

What it does:

  • Adds the machine to the all.hosts section
  • Adds it to the specified group under all.children
  • Sets ansible_host: localhost and ansible_connection: local
  • Creates host_vars/<name>.yml with a template

Example:

python3 scripts/register-machine.py goa server

purge-machine.py

Remove a machine from inventory.yml and its host_vars file. Mirror of register-machine.py.

Usage:

python3 scripts/purge-machine.py <name> [path-to-inventory.yml]

Important: Does NOT delete the corresponding Bitwarden SSH key item — you can re-onboard the machine later using the same name.

Example:

python3 scripts/purge-machine.py matrix

nmap-inventory.sh

Scan the LAN using nmap and cross-reference with Tailscale to show network status.

Usage:

./scripts/nmap-inventory.sh [CIDR]

Arguments:

  • [CIDR] — network range to scan (default: 192.168.0.0/24). Override with a different subnet.

Output columns:

  • IP address
  • MAC address
  • Hostname (reverse DNS or nmap service detection)
  • Tailscale status (online/offline)

Prerequisites:

  • nmap installed
  • Tailscale running and authenticated
  • Network access to the scanned range

Example:

./scripts/nmap-inventory.sh 10.0.0.0/24

Bitwarden Integration

bw-resolve.sh

Resolve a usable BW_SESSION for local and remote operations. Tries multiple fallback tiers.

Usage:

./scripts/bw-resolve.sh local
./scripts/bw-resolve.sh remote <host>

Modes:

  • local — Resolve a session on your local machine. Returns export BW_SESSION=... on stdout or exits 1 if unlock fails.

  • remote <host> — Unlock Bitwarden on a remote machine, export that session inline over ssh -t, and return it. Used by just apply-remote to drive secret-dependent tasks on the target.

Session resolution order:

  1. Existing BW_SESSION environment variable (if still valid)
  2. Cached session in /tmp/bw_session (if not expired)
  3. Interactive bw unlock prompt (asks for master password)

Example — local:

eval "$(./scripts/bw-resolve.sh local)"
bw list items  # Now BW_SESSION is set

Example — remote:

./scripts/bw-resolve.sh remote himachal
# Executes commands on himachal with Bitwarden access

bw-unlock.sh

Simpler variant of bw-resolve.sh — resolves a session on the local machine only.

Usage:

./scripts/bw-unlock.sh

Output: Prints <session-token> on stdout, or exits 1 if all tiers fail.

bw-seed-ssh-keys.sh

Pull SSH public keys from all online machines and seed them into a Bitwarden collection.

Usage:

./scripts/bw-seed-ssh-keys.sh

What it does:

  • Connects to each machine in the online fleet
  • Retrieves their ~/.ssh/id_ed25519.pub
  • Creates Bitwarden login items named james@<hostname> with the public key in the password field
  • Updates existing items in place (safe to re-run)

Prerequisites:

  • SSH access to all machines (keys already in ~/.ssh/authorized_keys)
  • Bitwarden CLI (bw) authenticated
  • Tailscale or direct network access to all machines

Run once: Before onboarding a new machine, so their SSH keys are available in Bitwarden during setup.

bw-seed-kube.sh

Seed kubeconfig and talosconfig from this machine into Bitwarden.

Usage:

./scripts/bw-seed-kube.sh

What it does:

  • Exports your local ~/.kube/config to Bitwarden as a secure note
  • Exports your local ~/.talos/config to Bitwarden
  • Updates existing items in place

Prerequisites:

  • kubectl configured with cluster access
  • talosctl configured with cluster access
  • Bitwarden CLI (bw) authenticated

Use when: You've set up cluster access locally and want to share it to other machines.

Apply and Status

record-apply.py

Record the outcome of a just apply run for tracking and status checks.

Usage:

python3 scripts/record-apply.py <exit-code> [label] [skip-tags]

Arguments:

  • <exit-code> — exit code from the Ansible apply (0 = success, non-zero = failure)
  • [label] — optional descriptive label (default: inferred from context)
  • [skip-tags] — optional comma-separated skip tags (for partial applies)

Output: Writes to ~/.cache/dotfiles/last-apply.json with:

  • Timestamp
  • Exit code (0 = success, 1 = failure)
  • Label and skip tags (if provided)

Used by: just doctor to check health status and flag old or failed applies.

last-apply-status.py

Pretty-print the last recorded apply status. Used by just doctor.

Usage:

python3 scripts/last-apply-status.py

Exit codes:

  • 0 — apply succeeded and timestamp is recent (< 48 hours)
  • 1 — apply failed OR timestamp is stale (≥ 48 hours)

Output: Human-readable status with timestamp, result, and any tags.

online_hosts.py

Print the online fleet hosts: inventory hosts whose tailnet name is online, excluding the vps group and this machine. A host's tailnet name is its tailnet_name in inventory.yml (e.g. termux → pixel-9a), otherwise its inventory name. A peer's name is its MagicDNS short name; the OS HostName is used only when a peer has no DNSName, because HostNames aren't unique (several phones report localhost).

Usage:

python3 scripts/online_hosts.py          # inventory names, space-separated
python3 scripts/online_hosts.py --peers  # raw online tailnet names, one per line

Used by: just _online_hosts (bulk applies, doctor-fleet, push-terminfo), and roles/ssh_mesh, which runs --peers on the target via the script module.

Silently fails: If Tailscale is not running, returns empty output (so scripts gracefully degrade).

Cluster and Infrastructure

kubevirt-node

Operations on KubeVirt nodes for the test cluster.

Usage:

./scripts/kubevirt-node [command] [node-name]

Supported commands:

  • list — List all KubeVirt test nodes
  • start <node> — Power on a node
  • stop <node> — Power off a node
  • status <node> — Show node status

Prerequisites:

  • kubectl configured with cluster access
  • Permission to manage VirtualMachine resources

sync-brews.sh

Sync the Homebrew package state: fetch the current installed packages from Homebrew and update group_vars/all.yml with the new list.

Usage:

./scripts/sync-brews.sh

What it does:

  1. Runs brew list --formulae and brew list --casks
  2. Extracts the current package list
  3. Updates the homebrew_packages and homebrew_casks sections in group_vars/all.yml
  4. Prints a diff so you can review changes

Use when: You've manually installed packages with brew install and want to record them in the inventory for automated management.

ess-render-values.py

Historical helper for rendering ESS Helm values for the terminated AWS Talos cluster from Bitwarden. It does not configure the standalone OCI deployment.

Usage:

python3 scripts/ess-render-values.py < ess-helm-values-template.yml > values.yml

What it does:

  1. Reads the ess-helm-values secure note from Bitwarden
  2. Applies environment and variable substitutions (e.g., ${DOMAIN})
  3. Outputs the rendered Helm values to stdout

Previously used by: AWS cluster deployment pipelines to generate Helm configurations.

Prerequisites:

  • bw CLI authenticated with Bitwarden vault access
  • The ess-helm-values secure note exists in Bitwarden

Prerequisites and Setup

All scripts assume:

  • Python 3 (for .py scripts)
  • Bash (for .sh scripts)
  • Bitwarden CLI (bw) installed and configured when BW scripts are used
  • SSH access to target machines (for remote operations)
  • Tailscale installed for machine discovery (for inventory scripts)

Set up Bitwarden CLI once:

brew install bitwarden-cli  # macOS
# or: apt-get install bw   # Linux

bw login <email>

Session tokens are cached locally in /tmp/ for convenience; they expire and are renewed automatically.

Fleet status

Snapshot of 2026-11-18, made from tailscale status and the live hosts. Update this page when a host changes role. To refresh it, run tailscale status --json | jq -r '.Peer[] | [.HostName,.Online,.LastSeen] | @tsv'.

How it fits together

Internet ─▶ Cloudflare ─▶ spindle (Oracle, Frankfurt): Caddy, Matrix, LiveKit, OMP auth broker, hive failover coordinator
                               │ tailnet
           ┌───────────────────┴────────────────────┐
           ▼                                        ▼
   bihar (home, WiFi)                        karnataka (home)
   k3s + KubeStellar hub                     k3s + KubeVirt + ARC runners + LMS
   hives: ACTIVE ───── hive-sync 15 min ───▶ hives: cold standby
                                             └─ VM: punjab (agent box, Ansible host)

Live hosts

HostKindOSRoleDocs
biharHome server (Intel, 62 GB)Bluefin utah:testingActive hive host; k3s + KubeStellar hubBihar
karnatakaHome server (AMD Strix Halo, 125 GB)Bluefin utah:testingk3s: KubeVirt VMs, CI runners, LMS, Corral; hive standbyKarnataka
spindleOracle A1 VM, eu-frankfurt-1 (2 ARM cores, 12 GB)Ubuntu 24.04Public front door (Caddy, 92.5.82.232), Matrix (reilly.asia), LiveKit, OMP auth broker, hive failover coordinatorSpindle VPS
punjabKubeVirt VM on karnataka (12 vCPU, 23 GB, 500 GB)Ubuntu 24.04Agent box and fleet Ansible host. Disk 96% full.Punjab
corralTailscale sidecar pod on karnatakan/aTailnet name for corral-webKarnataka
pixel-9a (termux)PhoneAndroid + TermuxAnsible termux hostInventory

Cross-host services:

Intermittent (desktops and laptops)

HostLast seenNotesDocs
kerala (kerala-1)2026-11-17Desktop. Earlier tailnet node kerala last seen 2026-08-29Kerala
himachal / himachal-12026-11-09 / 2026-11-10Desktop. Two tailnet nodes; remove the earlier oneHimachal
dilli2026-10-21DesktopDilli
kanpur2026-07-21Desktop or Windows test machineKanpur
mumbai2026-07-20Debian VM on the phoneInventory
goa2026-07-08ServerGoa
MacBook Air (2 nodes), CMF Phone 12026-10Personal devices, not managednone

Retired

HostRetiredNotesDocs
Home Talos cluster (bihar and karnataka on Talos)2026-10/11Replaced by Bluefin + k3s on the same hardwareArchive
AWS Talos cluster (eu-north-1)2026-10-25Terminated. Configurations kept as recovery materialArchive
punjab EC2 (punjab-aws-retired, 100.78.73.8)2026-10-28Replaced by the punjab VM on karnatakaPunjab
matrix (reilly-asia-matrix), telengana2026-09Hetzner VPSesArchive
Hives on spindle (/srv/hives)2026-11-18Moved to bihar. The copy is stopped and kept for rollbackHives hosting

Tailnet nodes to remove

The tagged devices from the retired clusters are offline and can be deleted from the Tailscale admin console. On 2026-11-18 these were:

  • since 2026-07: appflowy*, argocd, auth, forgejo, grafana, loki, prometheus, n8n, lemonade, home, kubevirt-manager*, default-deepseek-14b, repro-ctr-vm-vm, gate-*-vm
  • since 2026-10-25: hive, hive-auth-egress-0/1, review, search, spindle-grafana, aws-migration-operator, tuna-v1-preflight-*
  • since 2026-09: tailscale-operator*, secret-broker
  • since 2026-04: uisp, chuppvm
  • since 2026-07 and 2026-08: vm, WIN-G9NAVN79GN0

hanthor/dotfiles#333 (scripts/tailnet.py) automates this cleanup.

Network

The home LAN is a single flat private /24 behind a consumer router (DNS, DHCP). Most fleet traffic goes over Tailscale instead. Per-host addresses are not published here — run just inventory for the live LAN + Tailscale crosswalk.

Devices

NameRole
gatewayRouter (DNS, DHCP) — see Gateway
karnatakaHome server: k3s, KubeVirt, AMD GPU — reserved address
goaFleet control, Debian ARM (Raspberry Pi) — reserved address
phoneAndroid phone (reserved address)
printerNetwork printer (DHCP)
macbookMacBook Air (DHCP)
kvmNanoKVM (reserved address) — see KVM
esp32Unidentified IoT device (static address)
dilliLaptop (WiFi, when online)

Fleet nodes (Tailscale MagicDNS)

Hosts are reached by MagicDNS name; ansible_host in inventory.yml holds the tailnet address where one is needed.

HostnameLANGroup
goareservedserver (applies locally)
punjab— (VM on karnataka)server
himachalDHCPdesktop
dilliDHCPdesktop
kanpurDHCPdesktop
keralacellulardesktop
mumbai (ssh :2222)phonedesktop (phone VM)
termux (ssh :8022)phonetermux_hosts
matrixremotevps — retired
telenganaremotevps — retired
bihar— (separate site, WiFi)desktop — home server, hive host
karnatakareservednot in inventory (native units) — k3s host

Gateway

The home router: a consumer WiFi 6 router doing DNS and DHCP for the flat LAN.

Network

  • One private /24, DHCP pool covering most of the range, short lease time
  • Fixed addresses for infrastructure (bihar, karnataka, goa, kvm) come from the router's address-reservation table, not from static config on the hosts
  • ISP fiber, ONT in bridge mode, router holds the WAN address directly
  • Nothing is port-forwarded; remote access to the fleet is via Tailscale

Third-party firmware

The router's chipset (MediaTek Filogic 820) has only snapshot-level OpenWrt support so far, so it runs stock firmware. Revisit once a stable OpenWrt release supports it.

IPv6

The ISP delegates a single /64 via SLAAC only. The stock firmware has no LAN DHCPv6 server and no configurable IPv6 firewall. Hosts use privacy extensions, so their addresses rotate — impractical for hosting or stable SSH. Options if stable v6 addresses are ever wanted:

  1. Stable SLAAC token per machine:
    # /etc/systemd/network/eth0.network
    [IPv6AcceptRA]
    Token=::1  # becomes <prefix>::1
    
  2. Disable privacy extensions (EUI-64, based on MAC):
    net.ipv6.conf.eth0.use_tempaddr = 0
    
  3. DHCPv6 — not available on the stock firmware.

Tailscale remains the default path for fleet SSH.

NanoKVM

IP KVM for the fleet: a Sipeed NanoKVM running OpenWrt.

Notes

  • Plugged into Karnataka; provides remote keyboard/video/mouse for it
  • Web UI on its reserved LAN address (find it with just inventory)

Dilli

Secondary workstation in the hanthor fleet.

Hardware

  • Arch: x86_64
  • WiFi on the home LAN (DHCP)
  • Tailscale: dilli (MagicDNS)

OS

Bluefin (Fedora Atomic)

Services

  • BuildStream Dashboard — build monitor

Notes

  • PaperWM, Papershell, Copyous extensions synced from fleet
  • Brew packages managed via dotfiles playbook

Himachal

Laptop workstation in the hanthor fleet.

Hardware and OS

  • Architecture: x86_64
  • OS: Bluefin
  • The reachable Tailscale SSH name used for the 2026-10-27 inspection was james@himachal-1 (hostname himachal). The inventory entry for himachal still specifies 100.73.3.51; it should not be assumed to identify the reachable node without checking.

Services

  • Dotfiles updates: host_vars/himachal.yml sets is_laptop: true and dotfiles_auto_update: false. On 2026-10-27, dotfiles-update.service was not found on himachal-1. The configured 90-second delay applies only if login-time automatic updates are enabled; it is not evidence that updates currently run. The host-vars comment suggests just apply-nosecrets for a manual update.
  • Hive operations: hive_ops_enabled: false in host vars. Read-only inspection on 2026-10-27 found the seven former Hive and pi peak timers not-found and inactive on himachal-1. They previously targeted AWS Kubernetes, which was terminated on 2026-10-25; the host-vars comment saying those jobs now run in-cluster is stale. See hive_ops and the current Hive deployment.

The fleet's GNOME role includes PaperWM, PaperShell and Copyous; their enabled state on this host was not checked.

Kanpur

Laptop workstation in the hanthor fleet.

Hardware

  • Arch: x86_64
  • Laptop (runs on login, not periodic timer)
  • Tailscale: kanpur (MagicDNS)

OS

Bluefin (Fedora Atomic)

Notes

  • dotfiles_update_delay_seconds: 90 — applies dotfiles 90s after login

Kerala

PostMarketOS mobile device in the hanthor fleet.

Hardware

  • Arch: aarch64 (ARM)
  • Mobile device (laptop power profile)
  • Tailscale: kerala (MagicDNS)

OS

PostMarketOS (Alpine/musl-based)

Notes

  • Uses apk for package management (no Homebrew)
  • Packages installed via apk_packages role
  • Musl libc — Homebrew incompatible

Bihar

Home server, the primary host for the hives, and the KubeStellar hub. Since 2026-11-17 it runs Bluefin, not Talos. The Talos page is archived.

HardwareASRock B660M-STX, Intel, 24 threads, 62 GB of RAM
Disk1.9 TB NVMe (/var)
OSBluefin ghcr.io/projectbluefin/utah:testing (bootc: read-only /usr, writable /etc and /var)
NetworkWiFi (wlp3s0). Not on the same LAN as karnataka.
Tailnetbihar / 100.103.44.48 (bihar.manatee-basking.ts.net)
Accessssh james@bihar (passwordless sudo through /etc/sudoers.d/zz-james)
AnsibleYes: host_vars/bihar.yml, skip_pipewire_audio: true

Roles

  1. Active hive host. The hive containers (hub, hub-front, school and reef) run under rootful podman compose, and spindle's Caddy sends public traffic to them. See Hives hosting and failover.
  2. k3s and KubeStellar hub. A single-node k3s runs KubeStellar 0.30.2, with ITS its1 and WDS wds1. bihar and karnataka are both workload clusters. See Home k3s and KubeStellar.

Host settings (applied 2026-11-17)

  • No sleep: the sleep, suspend and hibernate targets are masked, the 90-nosleep.conf drop-ins are in /etc/systemd/{sleep,logind}.conf.d/, and the GDM dconf sleep-inactive-ac-type='nothing' is set. Before this change, GNOME suspended the host after 15 min idle, and the host went offline.
  • Default target is multi-user.target: no graphical login. Use systemctl isolate graphical.target to start one.
  • Wake-on-LAN is set to magic on enp0s31f6. The host uses WiFi, so wake-on-LAN works only if you connect a cable.
  • To do: in the firmware, set "Restore on AC/Power Loss" to Power On. This setting is necessary because the hives depend on this host.
  • Dotfiles bootstrap fixes for a new bootc host are in hanthor/dotfiles#385.
  • Bitwarden phase 2 is not done yet. Run just apply -e skip_pipewire_audio=true from an interactive terminal.

Timers

UnitPurpose
hive-reconcile.timerFollows the hive lease on spindle (every 1 min)
hive-sync.timerPushes the hive data to karnataka while bihar is active (every 15 min)
fleet-osquery-report.timerFleet facts
tailscale-cert-renew.timerTailscale TLS certificate

Gotchas

  • Do not use pkill -f <pattern> over SSH if the pattern also occurs in the SSH command line. The command stops your own session.
  • gsettings over SSH needs DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus.
  • semanage cannot write on bootc. Use chcon for SELinux labels.

Karnataka

The main home workload server: k3s with KubeVirt (punjab and the test VMs run here), GitHub Actions runners, Frappe LMS, Corral, and the cold standby for the hives. Since late 2026-10 it runs Bluefin, not Talos. The Talos page is archived.

HardwareAMD Strix Halo APU (32 threads, integrated GPU), 125 GB of RAM
Disk912 GB NVMe (/var), 88% used on 2026-11-18
OSBluefin ghcr.io/projectbluefin/utah:testing
Tailnetkarnataka / 100.66.158.127
Accessssh james@karnataka (passwordless sudo). kubectl from punjab uses ~/.kube/config
AnsibleYes: host_vars/karnataka.yml

k3s

k3s v1.36.5+k3s1 runs as a binary in /var/usrlocal/bin. It binds to the tailnet IP only, uses flannel on tailscale0, and has traefik and servicelb disabled. The kit is in home-k3s/karnataka/:

  • install-host.sh installs k3s.
  • apply-stack.sh installs KubeVirt v1.9.1, CDI v1.66.1, corral-web, the AMD GPU device plugin and Lemonade.

See Home k3s and KubeStellar for the shared design and the firewalld rules.

Workloads (2026-11-18)

NamespaceWhatExposed as
corral-vmsKubeVirt VMs: punjab (agent box, 500 Gi root), messages and the macos-beeper* disks (macOS for Beeper/iMessage), tuna-q test VMs, mandelbrot-visual (stopped)punjab is on the tailnet
corralcorral-web (VM manager UI) and its Tailscale sidecarhttps://karnataka.manatee-basking.ts.net/, tailnet node corral
front-doorCaddy link page and redirects (/course, /learn, /frappe, /books, /corral)http://karnataka/
lmsFrappe Learning v16.1.0 (frappe, db, redis), local image from frappe-lms-image/https://karnataka.manatee-basking.ts.net:8443
bealebeale-course (course site)beale.karnataka.ts.net
beeper-bridgesrustpush (iMessage bridge)none
arc-systems, tuna-ci, tuna-ci-storageGitHub Actions runner scale sets tuna-ci-24.04 (1-6), tuna-ci-26.04 (0-2), tuna-iso-gpu (0-1), with scratch storagenone
lemonadeLemonade AMD LLM server, scaled to 0:13305 (tailscale serve)
open-cluster-management-agent*KubeStellar WEC agent for the hub on biharnone

tailscale serve (tailnet only):

  • :80 goes to front-door.
  • :443 goes to corral-web.
  • :8443 goes to LMS.
  • :8444 goes to 127.0.0.1:8080 (RaveBSD books).
  • :13305 goes to Lemonade.

Hive standby

/var/srv/hives is a copy of the hive data. When bihar is active, bihar pushes it here every 15 min. The containers stay stopped until the lease on spindle names karnataka. See Hives hosting and failover.

Notes

  • Disk: /var is 88% full. The ARC runner PVCs and the macOS VM disks use most of the space. Check this before you add VMs.
  • punjab is a VM on this host. If karnataka restarts, punjab also restarts.

Home k3s and KubeStellar

bihar and karnataka each run a separate single-node k3s on Bluefin. They replace the two-node Talos cluster (archive). KubeStellar on bihar connects the two clusters. The hosts are not on the same LAN, so all cluster traffic uses the tailnet.

The kits are in home-k3s/: bihar/ and karnataka/ each contain install-host.sh, config.yaml, k3s.service, wait-tailnet.sh and resolv.conf.

Shared design

  • The k3s binary is in /var/usrlocal/bin/k3s. /usr is read-only on bootc. k3s.service waits for the tailnet (k3s-wait-tailnet).
  • The API server, kubelet and node IP bind to the tailnet IP only. Flannel uses tailscale0.
  • traefik and servicelb are disabled. Services are exposed with tailscale serve (karnataka) or ingress-nginx (bihar, KubeStellar only).
  • Pod DNS uses resolv.conf (1.1.1.1 and 1.0.0.1).
  • SELinux on the host stays enforcing. The k3s SELinux module cannot be installed on bootc.

firewalld (required: without these rules metrics-server fails and pods cannot resolve DNS)

# cni0 goes in the internal zone; pods can reach the kubelet
firewall-cmd --permanent --zone=internal --change-interface=cni0
firewall-cmd --permanent --zone=internal --add-rich-rule='rule family=ipv4 source address=10.42.0.0/16 destination address=<tailnet-ip>/32 port port=10250 protocol=tcp accept'
# pods can reach the internet
firewall-cmd --permanent --new-policy k3s-pod-egress
firewall-cmd --permanent --policy k3s-pod-egress --set-target CONTINUE
firewall-cmd --permanent --policy k3s-pod-egress --add-ingress-zone internal
firewall-cmd --permanent --policy k3s-pod-egress --add-egress-zone public
firewall-cmd --permanent --policy k3s-pod-egress --add-rich-rule='rule family=ipv4 source address=10.42.0.0/16 accept'
# API (and KubeStellar on bihar) from the tailnet only
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=100.64.0.0/10 destination address=<tailnet-ip>/32 port port=6443 protocol=tcp accept'

bihar's install-host.sh applies these rules. On karnataka the rules were applied by hand.

Differences

biharkarnataka
service CIDR10.96.0.0/12 (KubeFlex's vcluster hard-codes the kind range)k3s default 10.43.0.0/16
Extrasingress-nginx (SSL passthrough), KubeFlex, KubeStellarKubeVirt, CDI, ARC, workloads (see Karnataka)
Resetreset-k3s.shnone

KubeStellar (hub on bihar)

KubeStellar 0.30.2 core chart (oci://ghcr.io/kubestellar/kubestellar/core-chart):

  • ITS its1: vcluster with the OCM cluster-manager.
  • WDS wds1: where you apply workloads.
  • Both are served at https://<name>.100.103.44.48.sslip.io:9443. Port 9443 is open to the tailnet only.
  • kubectl contexts on bihar: bihar, its1, wds1 (in ~/.kube/config).
  • The join command is in bihar:~/.kube/its1-join.cmd. It is a secret.

Workload execution clusters (joined with clusteradm 1.0.0 --singleton):

ClusterLabels
biharlocation-group=home name=bihar cpu=intel role=hub
karnatakalocation-group=home name=karnataka cpu=amd gpu=amd kubevirt=true

Use it like this: apply manifests with kubectl --context wds1, and add a BindingPolicy with a clusterSelectors label match. Set wantSingletonReportedState: true to get status back from one cluster. Notes are in home-k3s/bihar/KUBESTELLAR.md.

Problems found during installation (2026-11-17)

  • If metrics-server is not ready, the KubeFlex hooks fail with "stale GroupVersion discovery". Correct the firewall, then restart kubeflex-controller-manager.
  • A failed ks-core release stopped on the uninstall-postgresql hook. To recover, run helm uninstall ks-core --no-hooks, delete the kubeflex-system, its1-system and wds1-system namespaces, then install again.

Hives hosting and failover

TunaOS infrastructure. This page describes where and how the hives run. For the hives themselves (agents, configuration, health), see the Hive operations guide.

Last updated: 2026-11-19 (hanthor added). The source for the scripts, compose layers and systemd units is hives/ in this repository. Each host also has a copy of this runbook at /var/srv/hives-host/RUNBOOK.md (on spindle: /var/lib/hive-failover/RUNBOOK.md).

Overview

The hives run as one compose project named hives, with five containers:

ServicePurposePublic URL
hubHub API that all spokes report tohttps://hub.tunaos.org/api/*, /login/*
hub-frontStatic hub page (nginx)https://hub.tunaos.org/
schoolSpoke for the tuna-os org (GitHub App 3942065)https://school.tunaos.org
reefSpoke (GitHub App 5546574)https://reef.tunaos.org
hanthorSpoke for the hanthor/* personal repositories (App 3942065, installation 158987150). Added 2026-11-19https://hive.reilly.asia

Until 2026-11-18 the hives ran on spindle (Oracle ARM VM, Frankfurt, 2 cores and 11 GB of RAM). They now run on bihar (x86, 24 cores, 62 GB of RAM). karnataka is a cold standby. spindle keeps the public IP and Caddy, and it coordinates failover.

Internet ─▶ Cloudflare ─▶ spindle Caddy (92.5.82.232)
                              │  tailnet, ports 3100-3104
               ┌──────────────┴──────────────┐
               ▼ (active, holds the lease)    ▼ (standby, stopped)
        bihar 100.103.44.48          karnataka 100.66.158.127
               └── hive-sync every 15 min ──▶

Outbound traffic from the hives (to GitHub and the model APIs) leaves through the active host's own internet connection. It does not go through spindle.

The OMP auth broker still runs on spindle at http://100.113.75.88:8765, reached over the tailnet.

Host layout (bihar and karnataka are identical)

PathContents
/var/srv/hives/Hive data and configuration, copied from spindle's /srv/hives: compose.yaml, compose.override.yaml, .env, {hub,school,reef,hanthor}/{config,data,secrets}, hub-front/, proxy/, bin/dashboard-*.html
/var/srv/hives-host/compose.<host>.yamlHost layer: ports, memory limits, broker URL, image digests, and the amd64 binary mount
/var/srv/hives-host/bin/hive-2ef7dc7c0-amd64tuna-os/hive at 2ef7dc7c0 (branch deploy/upstream-v6-20261113) built with CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath ./cmd/hive
/var/srv/hives-host/hivesWrapper that runs docker-compose against rootful podman with the three compose files
/var/srv/hives-host/hive-reconcileMakes the host follow the lease (timer, every 1 min)
/var/srv/hives-host/hive-syncPushes the data to the peer host (timer, every 15 min)
/var/srv/hives-host/keys/lease (read-only access to the lease on spindle) and sync (rsync to the peer host)
/var/srv/hives-host/maintenanceIf this file exists, reconcile does not start the hives

Each host publishes its ports on its own tailnet IP only:

PortService
3100hub-front
3101hub
3102school
3103reef
3104hanthor

A firewalld rich rule in the public zone accepts these ports only from spindle (100.113.75.88). Inside the compose network the services reach each other by name, for example http://hub:3001. The podman network is named spindle_default so that compose.yaml works unchanged.

Images are pinned by digest, and the pin must match the binary:

  • ghcr.io/hivecommons/hive@sha256:d56425fb… (school, reef and hanthor)
  • ghcr.io/hivecommons/hive-hub@sha256:3c0162eb… (hub)

Do not change to :v6-latest. On 2026-11-18 a newer image made the hub log "failed to write token cache". Upgrade the binary and the images together.

Daily operations (on the active host)

sudo /var/srv/hives-host/hives ps
sudo /var/srv/hives-host/hives logs -f --tail 50 school
sudo /var/srv/hives-host/hives restart reef
sudo podman stats --no-stream

To find the active host, run this on spindle:

sudo hive-failover status

The command shows the lease, the health and container count of each host, and the targets in Caddy.

Change the hive configuration

  1. Edit the files under /var/srv/hives/ on the active host only.
  2. Run sudo /var/srv/hives-host/hives up -d.

The next hive-sync copies the change to the standby.

Change a host-specific setting

  1. Edit /var/srv/hives-host/compose.<host>.yaml on both hosts. This covers ports, memory and image pins.
  2. Run sudo chcon -t container_file_t on the file.

Upgrade the hive binary

  1. Build the amd64 binary from the fork commit.
  2. Copy it to /var/srv/hives-host/bin/ on both hosts.
  3. Change the binary mount and the image digests in both host files.
  4. Run hives up -d on the active host.

Failover

The lease

The file spindle:/var/lib/hive-failover/active contains bihar, karnataka or none. Only the host named in the lease may run the hives. This rule prevents two hosts from running the same GitHub Apps at once.

ComponentWhereScheduleWhat it does
hive-failover checkspindle, hive-failover.timerevery 1 minChecks the active host on ports 3101-3103. After 5 min of failures it sets the lease to none, stops the old host if it can, and waits. When the old host has 0 containers, or after 10 min, it starts the standby, sets the lease, changes Caddy and sends a ntfy notification on $NTFY_TOPIC.
hive-reconcileboth hosts, hive-reconcile.timerevery 1 minIf the lease names this host and fewer than 4 containers run, it starts them (unless maintenance exists). If the lease names a different host, it stops them. If it cannot read the lease for 8 min, it stops them (self-fence).
hive-syncboth hosts, hive-sync.timerevery 15 minIf the lease names this host, it uses rsync to push /var/srv/hives to the peer, without caches and toolchains.
hive-cleanupboth hosts, hive-cleanup.timerevery 1 hOn running spokes only: removes idle Claude session scratch and stale work dirs in /tmp inside the containers, removes dangling images and old backups, and warns when /var has less than 10 GB free. Ported from spindle, where the writable layers of the hive containers had reached 16.8 GB.

Logs:

journalctl -t hive-failover                # on spindle
journalctl -u hive-reconcile -u hive-sync  # on the hosts

The podman containers have restart policy unless-stopped and podman-restart.service is disabled. As a result, a host that boots does not start the hives itself. Only reconcile starts them, and only when the lease names that host.

Commands (on spindle)

sudo hive-failover status
sudo hive-failover move bihar       # planned move or failback: stop, sync, start, Caddy
sudo hive-failover move karnataka
sudo hive-failover hold on          # pause automatic failover (for example during maintenance)
sudo hive-failover hold off

move sets hold on while it runs, and clears it when it succeeds. If move fails, the lease stays at none and hold stays on. Correct the problem and run move again.

After an automatic failover

The hives run on karnataka. Failback is manual, so that an unstable WiFi connection cannot move the hives back and forth.

  1. Make sure bihar is up: ssh james@bihar uptime.
  2. Run sudo hive-failover move bihar on spindle. It takes about 1 min.

Planned maintenance on bihar

Use one of these methods:

  • Run sudo hive-failover move karnataka first.
  • Or run sudo hive-failover hold on on spindle, then sudo touch /var/srv/hives-host/maintenance and hives stop on bihar. When you finish, remove the file and run hold off.

Limits

  • The data on the standby can be up to 15 min old. Rsync copies files while they are in use, so a file that changes during the copy can arrive in an inconsistent state.
  • The self-fence after 8 min without spindle was tested by hand only. A real network partition has not been tested.
  • If spindle is down, there is no public access and no failover. The hosts stop the hives after 8 min, because they cannot read the lease.

Drill on 2026-11-18

  1. Hives stopped on bihar at 13:55:44Z.
  2. Spindle revoked the lease at 14:01:29Z.
  3. Karnataka was healthy and Caddy changed at 14:01:56Z.
  4. move bihar completed in 50 s, including the sync back.

Gotchas

  • SELinux (bihar and karnataka are bootc and enforcing):
    • Files mounted into containers must have the type container_file_t. semanage cannot write on bootc, so hive-sync runs chcon -R after each copy. Do not copy SELinux attributes with rsync (-X).
    • systemd cannot run scripts that have the label var_t (the result is 203/EXEC Permission denied). For this reason the units start the scripts as /usr/bin/bash /var/srv/hives-host/<script>. Do not use chcon on the scripts.
    • Under systemd, rsync changes to the confined rsync_t domain, and its ssh cannot read the key. hive-sync.service therefore runs as SELinuxContext=system_u:system_r:unconfined_t:s0. setsebool -P has no effect on bootc.
    • Until 2026-11-18 18:50Z, the reconcile and sync timers did not run because of these two problems. The failover drill passed only because spindle sent the commands directly over SSH. After you change a unit, use systemctl show <unit> -p Result to make sure that it ran.
  • The Caddyfile on spindle is a single-file bind mount: sed -i makes a new inode, and Caddy continues to see the old file. Write in place (cat new > Caddyfile), or run docker cp Caddyfile spindle-caddy-1:/tmp/Caddyfile then docker exec spindle-caddy-1 caddy reload --adapter caddyfile --config /tmp/Caddyfile. hive-failover does both.
  • The spindle-to-host path uses a relay: it goes through Tailscale DERP (Frankfurt), with approximately 170 ms of latency and 1 MB/s. This is sufficient for dashboard and API traffic.
  • Disk: the caches (Go build, rustup and others) are not synced. After a move, the first builds on the new host are slower.
  • Use separate SSH keys:
    • spindle:/root/.ssh/hive_failover gives access to james@ on both hosts.
    • The lease key of each host gives ubuntu@spindle with a forced command: cat /var/lib/hive-failover/active.
    • The sync key of each host gives james@ on the peer host.

Install or rebuild a host

The files in hives/ go to these locations:

Repository fileDestination
hives, hive-reconcile, hive-sync/var/srv/hives-host/ on bihar and karnataka (mode 0755)
compose.<host>.yaml/var/srv/hives-host/ on that host
hive-cleanup/var/srv/hives-host/ on bihar and karnataka (mode 0755)
systemd/hive-{reconcile,sync,cleanup}.*/etc/systemd/system/ on bihar and karnataka
hive-failover/usr/local/sbin/ on spindle
systemd/hive-failover.*/etc/systemd/system/ on spindle

On a new host, also do these steps:

  1. Install docker-compose with brew.
  2. Run systemctl enable --now podman.socket.
  3. Create the podman network spindle_default.
  4. Pull the pinned images.
  5. Copy bin/hive-2ef7dc7c0-amd64 from the other host. If neither host has it, build it from tuna-os/hive at 2ef7dc7c0.
  6. Create the SSH keys in /var/srv/hives-host/keys/, and add the firewalld rule for ports 3100-3104 from spindle. Also install /etc/sysctl.d/90-hives-inotify.conf (fs.inotify.max_user_instances = 1024, fs.inotify.max_user_watches = 1048576): with five containers the per-agent permission watchers exhaust the default of 128 instances.
  7. Label the data and compose files (not the scripts): chcon -R -t container_file_t /var/srv/hives /var/srv/hives-host/compose.*.yaml /var/srv/hives-host/bin.
  8. Enable hive-reconcile.timer, hive-sync.timer and hive-cleanup.timer. Start each service one time and make sure that its Result is success.

Rollback to spindle (emergency only)

On 2026-11-18 the cleanup removed the stale data copies, the arm64 binaries, and the hive containers and images on spindle. /srv/hives contains only the compose files, .env, <svc>/config, <svc>/secrets and backups/. karnataka is the fallback. Use spindle only if both hosts are lost:

  1. On spindle, run sudo hive-failover hold on.
  2. Build an arm64 binary from tuna-os/hive at 2ef7dc7c0 (GOARCH=arm64) and copy it to /srv/hives/bin/.
  3. Copy the newest /var/srv/hives from bihar or karnataka to spindle (<svc>/data, without .cache).
  4. Change the hive blocks in Caddy back to school:3001, reef:3001, hub:3001 and hub-front:80.
  5. Run docker compose up -d in /srv/hives. Spindle has 2 ARM cores, so reduce HIVE_SHARED_AGENT_POOL_MAX_SESSIONS to 2.

Spindle VPS (Matrix and the public front door)

The Oracle Cloud VPS in Frankfurt runs the Matrix homeserver that was on the AWS Talos cluster, terminated 2026-10-25. Spindle, Postgres, standalone LiveKit and lk-jwt-service (MatrixRTC; no ESS chart/runtime), synctap and Caddy run as Docker Compose services under /srv/spindle (state on 2026-11-18). On 2026-11-16, Spindle's built-in OIDC replaced MAS (auth.reilly.asia now goes to Spindle). The MAS sections below are history.

The hives moved off this host on 2026-11-18. School, Reef, Hub and the hub frontend now run on bihar, with karnataka as a cold standby. The hanthor spoke (hive.reilly.asia) joined them on 2026-11-19. Spindle keeps three hive roles:

  • Caddy front door: school, reef, hub.tunaos.org and hive.reilly.asia go to the active host's tailnet IP on ports 3100-3104 (hive.reilly.asia on 3104). hive-failover rewrites these upstreams.
  • Failover coordinator: the lease in /var/lib/hive-failover/active, /usr/local/sbin/hive-failover, and hive-failover.timer.
  • OMP auth broker: the omp-auth-broker user service on 100.113.75.88:8765. The hives connect to it over the tailnet. The omp-auth-broker-docker-proxy socket (172.18.0.1:18765) was used only by the hives when they ran on this host.

See Hives hosting and failover. The stopped copy in /srv/hives is kept for rollback only (MOVED-TO-BIHAR.txt). This removes the 2560 MB per-spoke memory budget from this host.

PieceSource
HostOracle Cloud Always Free, VM.Standard.A1.Flex, Ubuntu 24.04 aarch64, eu-frankfurt-1 AD-1; public IPv4 92.5.82.232, Tailscale 100.113.75.88
Spindleghcr.io/tuna-os/spindle:v0.1.6 running on 2026-11-18 (v0.1.1 deployed 2026-11-07; arm64 release images). The upgrade from the outbox-hotfix-arm64 build and its rehearsal are in upgrade-to-main.md
MatrixRTClivekit-server:v1.13.5 + lk-jwt-service:0.6.0 as standalone containers
Dataess_ess-spindle-data.tar.gz (the store, taken 15:33 UTC) and postgres/2026-10-25/mas.dump (15:22 UTC), from ~/backups/tunaos-aws-2026-10-25 on himachal-1
Configproduction runtime + delegated-auth block, MAS under/overrides and LiveKit template; deploy.sh renders credentials from cluster-state

MatrixRTC is enabled by the standalone services above. element-admin (matrixadmin.reilly.asia) remains unmigrated.

1. Create the instance (Oracle console)

  • Shape VM.Standard.A1.Flex. spindle is in eu-frankfurt-1, AD-1, with 2 OCPU / 12 GB; stay within that Always Free ceiling. Spindle is capped at 7 GiB, LiveKit at 2 GiB / 1 OCPU.
  • Image: Canonical Ubuntu 24.04 (aarch64); 100-GB boot volume.
  • Assign and reserve a public IPv4. Paste cloud-init.yaml; it installs Docker, opens direct host 80/443, and authorizes the punjab and himachal-1 SSH keys.
  • In the OCI security list allow TCP 80, 443, 30001 and UDP 32700–32767 from 0.0.0.0/0. Remove the default public SSH rule after tailnet access works.
  • Oracle may reclaim idle Always Free instances. We accept that risk and will not upgrade this tenancy to a paid account. Cloud-init package installation failed on the existing instance and required manual installation; check cloud-init status --long and installed packages before deploy.

A1 capacity is often "out of host capacity"; retry another availability domain or region later.

2. Deploy (on himachal-1)

cd ~/.local/share/dotfiles/docs/src/servers/spindle-vps   # a checkout of this branch
./deploy.sh <vps-tailnet-ip-or-host>  # SSH destination; use the tailnet address after lockdown
./deploy.sh --render-only /tmp/r   # just render the configs (contains secrets; delete after)

It renders the secrets from cluster-state.himachal.tar.gz.age, copies the kit to /srv/spindle, streams the store (excluding store.pre723), restores mas.dump, runs mas database migrate and starts the stack. Restore markers prevent re-importing the original snapshots on a normal rerun; do not remove those markers or restore old data over live data. The script restarts Spindle and Caddy, causing an interruption. Nothing secret is printed or committed.

The supplied Caddyfile includes the Hive routes. Do not deploy a Matrix-only Caddyfile over the shared live configuration: doing so removes Hive TLS sites and causes Cloudflare 525 errors.

2b. Join the tailnet and lock down

ssh ubuntu@<ip> 'sudo tailscale up --hostname=spindle --ssh=false'   # prints a login URL; approve it
ssh ubuntu@spindle true                                              # confirm SSH over the tailnet
scp lockdown.sh ubuntu@spindle: && ssh ubuntu@spindle 'sudo ./lockdown.sh'

Then delete the public SSH rule from the OCI security list. Docker publishes the listed LiveKit ports; keep the OCI ingress list in sync with all public ports in compose.yaml.

3. DNS cutover

Cloudflare DNS-only A records for matrix.reilly.asia, auth.reilly.asia, and call.reilly.asia point to the VPS at 92.5.82.232 with a 60-second TTL. call must remain DNS-only: its UDP media ports do not pass through Cloudflare's HTTP proxy.

Verify the public endpoints:

curl https://matrix.reilly.asia/_matrix/client/versions
curl https://auth.reilly.asia/.well-known/openid-configuration
curl https://reilly.asia/.well-known/matrix/client
curl -i https://call.reilly.asia/sfu/get  # GET should return 405
curl https://school.tunaos.org/api/health
curl https://reef.tunaos.org/api/health
curl https://hub.tunaos.org/api/fleet-stats

The apex reilly.asia is a Cloudflare Worker custom domain (reilly-asia) serving the personal portfolio and active Matrix client .well-known document. Keep it on the Worker; pointing the apex at this VPS or removing the Worker domain would take the homepage offline. The Worker advertises the same https://call.reilly.asia MatrixRTC service as this deployment.

call.reilly.asia routes JWT requests to the standalone lk-jwt-service and LiveKit signaling to the standalone SFU. Since v0.1.1 Spindle's built-in minter also serves federated participants; moving to it is the consolidation step, not yet applied here.

This is manual DNS cutover, not automatic failover. The zone is on Cloudflare Free; health-checked Load Balancing is an add-on, and no second Matrix origin is available. A 60-second TTL only limits resolver caching; it does not detect failures or switch records automatically.

The authorizer resolves matrix.reilly.asia and call.reilly.asia through Docker's host gateway to avoid public-IP hairpin routing; reilly.asia remains on the portfolio Worker.

Incidents

Caveats

  • The MAS dump is from 15:22 UTC. The Spindle store's last write was 15:25 UTC and its archive was taken at 15:33 UTC; the old cluster continued until 17:16 UTC. Writes after the store archive are only in the encrypted cluster-state backup and are not in this VPS store. A real login and joined room read passed against the VPS, but that does not prove the later writes were restored.
  • The AWS cluster is terminated. The S3 and himachal-1 backups are untouched, but stopping the VPS stack is an outage, not a rollback; recovery requires provisioning and restoring from backup.
  • v0.1.1 contains #723, #736 and #794, so the ops/arm64-perf-723, ops/legacy-media-compat and ops/outbox-hotfix branches can be deleted once production has run it for a week.
  • LiveKit is capped at 2 GiB and 1 OCPU to leave room for Spindle and the Hives on this 2-OCPU host; suitable for light personal calls, not high-concurrency conferencing. No TURN relay is configured, so clients on networks that block UDP may fail to connect.

Upgrading the VPS Spindle from the hotfix image to main

Status: done 2026-11-07. Production runs ghcr.io/tuna-os/spindle:v0.1.1 (main 1d8dfd99, arm64 release image). The rehearsal below was run first, against a copy of production; both blockers it found were cleared before the upgrade (see Blockers).

Before the upgrade production ran ghcr.io/tuna-os/spindle:outbox-hotfix-arm64, built from branch ops/outbox-hotfix. That branch is the Oct 22 main (79f00253) plus #723, the legacy-media patch and the concurrent outbox (#794).

The production upgrade (2026-11-07)

The procedure under Production procedure was followed as written, with the release tag v0.1.1:

StepResult
docker pullv0.1.1, arm64 linux
Stop (05:22:53 UTC) and cp -a of the storedata/store.pre-v0.1.1, 3.4 GB
migrate --list / --dry-runstore and binary at schema 1/1/3; intern-state-keys "not needed"; "already at this binary's schema"
Startlistening at 05:24:58 (outage about 2 minutes); room warm-up finished, 118 rooms in 21.2 s
/_matrix/client/versions, .well-known/matrix/client, MAS discovery, call.reilly.asia/sfu/get200, 200, 200, 405
First 8 minutes324 inbound /send answered 200, 156 PDUs accepted, 0 ERROR log lines

The compose file before the upgrade is /srv/spindle/compose.yaml.bak-20261107T052253.

What the rehearsal showed

The rehearsal compared main 1e8f6973 with hotfix fe76489a. Both were x86_64 release builds, run on punjab, pinned to two cores (taskset -c 0,1, TOKIO_WORKER_THREADS=2 as in compose.yaml). The config was production's spindle.toml with these changes: scratch store path, localhost binds, federation and push disabled, and no [auth.delegated] block. A scratch password replaced MAS login for one account, set on the copy only.

The store was the live production store (2.7 GB including 2,371 media blobs). It was copied read-only over Tailscale: sudo nice -n 19 ionice -c3 tar, then rsync, then a final rsync pass at 01:58 UTC. That final pass changed only the active journal (89.jnl). Nothing was written on the VPS.

CheckHotfix (baseline)Main
Opens the copied store; verify-media2,371 blobs, all present (0.5 s)2,371 blobs, all present (0.4 s)
migrate --list / --dry-runn/astore at 1/1/3, only step intern-state-keys is "not needed"; dry-run and migrate: "already at this binary's schema" (0.5–0.6 s, writes nothing)
Rooms (admin API)118; 70,919 joined members; 105,020 state eventsidentical, same room-ID hash
State of the 15 largest rooms (up to 51,884 events)event-ID hashesidentical hashes
Last 100 log entries of those rooms; /messages in 5 joined roomsevent-ID hashesidentical hashes
Accounts (admin API)1,3601,360, same hash
Initial /sync for a local user (115 joined rooms, 2,103 timeline events)200 in 9.7–10.1 s200 in 9.1–11.0 s; same counts
Incremental /sync0.03 s0.03–0.06 s
Cold start (page cache dropped): listening / warm-up done (118 rooms)1.0 s / 14.2 s0.4 s / 12.0 s
Largest room cold load (≥1M bucket)1.04 s cold, 5.37 s on first run1.06 s cold, 1.70 s on first run
RSS after probe2.9 GiB2.8 GiB
WARN/ERROR log lines00
Write path: send message, upload media, restart, read backn/amessage found, media 200, verify-media 2,372/2,372
Rollback: hotfix binary on the store after main ran and wroten/aopens; same room/state/timeline/account hashes; 0 WARN/ERROR
Legacy GET /_matrix/media/v3/download and /thumbnail200404 (see Blockers)

Other measurements on the copy (main binary, offline):

  • spindle backup took 15.7 s and wrote 16.7 M rows into a 4.0 GB file. The backup holds rows, not media.
  • spindle gc --dry-run took 64.5 s and found 364,697 unreachable state nodes (290 MB) that could be swept.

Production warm-up is slower than on punjab. The 2026-11-06 restarts logged 19.2 s and 59.9 s for the same 118 rooms on the hotfix image. Budget about a minute of degraded service after the container starts.

No data migration runs. The store's schema (key 1, record 1, content digest 3, fjall 3) is the same in both builds, so migrate is a no-op. The rollback is a plain image swap, and the rehearsal exercised it.

Behaviour that changes with main and is not a data question:

  • Per-user rate limits now default to Synapse's numbers: messages 0.2/s with burst 10, local joins 0.1/s with burst 10, room creation 0.016/s with burst 10, media uploads 10/s with burst 50. Production's config sets none of these. There are no appservices, so no bridge is affected.
  • The MAS introspection verdict is cached for 120 s ([auth.delegated] introspection_cache_ttl_secs). A revoked MAS session can work for up to two minutes.
  • Production's spindle.toml parses unchanged under main. The rehearsal checked this by starting main with that file, delegated block included.

Blockers

Both were cleared before the upgrade:

  1. Legacy media. Production depends on the frozen unauthenticated media endpoints (/_matrix/media/v3/download|thumbnail), which pre-MSC3916 clients use for avatars and thumbnails. The rehearsal found main returned 404 there. tuna-os/spindle#736 merged before v0.1.1; it serves local media and remote media already cached, and never fetches uncached remote media without a token.
  2. No arm64 image of main. release.yml builds amd64 and arm64 images for every v* tag; v0.1.1 was cut from main 1d8dfd99 after #736, #800, #804 and #805 merged.

Re-running the rehearsal (punjab)

The rehearsal needs about 3 GB for the copy, 3 GB for a working copy and 8 GB more if a backup is taken. Everything stays in one mode-700 directory, which is deleted at the end because it holds user data.

S=~/rehearsal-scratch; mkdir -p "$S/prod-copy/store" && chmod 700 "$S"
# Read-only copy of the live store. Run it twice; the second pass should only touch the journal.
rsync -a --delete --partial -z --rsync-path='sudo -n nice -n 19 ionice -c3 rsync' \
  ubuntu@100.113.75.88:/srv/spindle/data/store/ "$S/prod-copy/store/"
cp -a "$S/prod-copy/store" "$S/store"
# Binaries: cargo build --release -p spindle-server --bin spindle --locked at each commit.
OLD=spindle-hotfix; NEW=spindle-<tag>
CFG=$S/cfg/prodlike.toml   # production spindle.toml: storage path -> $S/store, binds on
                           # 127.0.0.1, federation/push disabled, [auth.delegated] removed
$OLD verify-media "$CFG"
$NEW promote-admin "$CFG" james
$NEW set-password-hash "$CFG" james < pw.hash   # argon2id of a throwaway password
$NEW migrate --list "$CFG"; $NEW migrate "$CFG" --dry-run
$NEW migrate "$CFG"; $NEW verify-media "$CFG"
TOKIO_WORKER_THREADS=2 taskset -c 0,1 $NEW "$CFG"
# Probe the server: admin API /_spindle/admin/v1/rooms, rooms/{id}/state and
# rooms/{id}/timeline, users; client /joined_rooms, /messages and /sync;
# metrics spindle_room_cold_* and spindle_room_warmup_*. Compare the results
# with a run of $OLD on a fresh copy.
rm -rf "$S"

The rehearsal compared runs by hashing event and room IDs, so the comparison files never held message content. An incoming message on production during the copy window would show up as a difference between the copy and production. It would not show up as a difference between the two binaries.

The run on 2026-11-07 took about 40 minutes. Most of that was the copy: 2.7 GB over a DERP-relayed Tailscale path at about 0.7 MB/s. All the checks together took under 5 minutes.

Production procedure (once the blockers are cleared)

Run on the VPS in /srv/spindle. <tag> is the release tag, for example v0.1.1. Expected outage: the time to stop and copy the store (seconds; 2.7 GB on local disk), plus about a minute of warm-up.

cd /srv/spindle
# 1. Pull ahead of time; no outage.
docker pull ghcr.io/tuna-os/spindle:<tag>
cp compose.yaml compose.yaml.bak-$(date -u +%Y%m%dT%H%M%S)
sed -i 's#ghcr.io/tuna-os/spindle:outbox-hotfix-arm64#ghcr.io/tuna-os/spindle:<tag>#' compose.yaml

# 2. Stop Spindle and keep a byte-for-byte copy of the stopped store (includes media).
docker compose stop spindle
sudo cp -a data/store data/store.pre-<tag>

# 3. Migration plan; must say "already at this binary's schema" (rehearsed: no steps).
docker compose run --rm --no-deps spindle migrate --list /etc/spindle/spindle.toml
docker compose run --rm --no-deps spindle migrate /etc/spindle/spindle.toml --dry-run
docker compose run --rm --no-deps spindle verify-media /etc/spindle/spindle.toml

# 4. Start and watch warm-up.
docker compose up -d spindle
docker compose logs -f spindle | grep -m1 'room warm-up finished'

If the dry run lists any step other than "not needed", stop. Swap the image back and investigate: the rehearsal did not cover that step.

Checks after the start:

curl -s https://matrix.reilly.asia/_matrix/client/versions | jq '.versions[-1]'
curl -s -o /dev/null -w '%{http_code}\n' \
  https://matrix.reilly.asia/_matrix/media/v3/download/reilly.asia/<known-media-id>   # 200 (#736)
curl -i https://call.reilly.asia/sfu/get   # 405

Then:

  • Log in through MAS in Element and confirm that sync completes.
  • Send a message in a federated room and confirm a remote server receives it.
  • Start a call.

Rollback

The schema did not change, so rolling back is an image swap:

docker compose stop spindle
cp compose.yaml.bak-<stamp> compose.yaml
docker compose up -d spindle

The rehearsal opened a store that main had already written with the hotfix binary: same rooms, state and timelines, no errors. Data written by features only main has (new keyspaces) is ignored by the old binary.

Restore data/store.pre-<tag> only if the store itself is damaged:

sudo mv data/store data/store.bad && sudo mv data/store.pre-<tag> data/store

That restore loses everything written since the stop.

Afterwards

  • Delete data/store.pre-<tag> after a week without problems, and remove the old images (outbox-hotfix-arm64, legacy-media-compat-arm64, perf-723-de17805-arm64).
  • Optional, offline, with a fresh copy taken first: spindle gc reclaims about 290 MB of unreachable state nodes. It took 64 s on the copy.
  • Update the image row in README.md and delete the ops/outbox-hotfix branch.

2026-11-16/17: slow sync, missing avatars, Element X image errors

Symptoms on matrix.reilly.asia: sync felt slow, profile and room pictures were missing, and Element X showed "Sorry, an error occurred" for every image. Four separate faults, plus host load.

The general fault guide (checks and fixes for any Spindle operator) is in the Spindle repo: docs/troubleshooting.md.

What was wrong and where it was fixed

FaultSymptomFix
Element X 26.11 requests media at /_matrix/client/unstable/org.matrix.msc3916/media/…; Spindle served only /v1/mediaEvery image and avatar 404s (route="unmatched" in metrics); Element X error dialogCaddy rewrite (live now, also in this kit's Caddyfile); Spindle #940 serves the prefix itself
Federation media answered with a Location header was cached as 0 bytesBeeper avatars blank forever#933, in v0.1.5
CDN labels images application/octet-streamM_UNSUPPORTED: no thumbnails for application/octet-stream#937, in v0.1.6
Partial-state (faster-join) rooms never finished resync: one bad member signature, or one auth event the resident refuses with 403, failed the whole room; keys were fetched one server at a timeNewly joined big rooms have no member list or member avatars; /joined_members hangs#935 (v0.1.5), #938 (v0.1.6)

Not Spindle bugs, still open:

  • A browser tab looping on a dead token. Firefox at https://hanthor.github.io sent sync/whoami/keys ~20 req/s, all 401 M_UNKNOWN_TOKEN, after the MAS → built-in OIDC switch on 2026-11-16. Sign out and back in (or close it); the client should have logged itself out.
  • Host contention. 2 OCPU / 12 GB, no swap; the School and Reef Hives use ~2.6 GB and heavy CPU/disk. Spindle's first sync after a restart takes up to a minute while caches warm. Moving the Hives off this host is the biggest remaining speed-up.
  • synctap still proxies sliding sync for @james (0.5 CPU cap). Remove the service and the @ssync Caddy handle when that diagnosis is done.

Live state left behind (as of 2026-11-17)

  • compose.yaml runs ghcr.io/tuna-os/spindle:v0.1.5 (backup compose.yaml.bak-pre-v0.1.5-*). Bump to the release that contains #940, then delete the @msc3916 rewrite.
  • Caddyfile has two temporary additions, each with a comment:
    • the @msc3916 rewrite (keep until #940 is deployed);
    • an access log to /data/access-diag.log inside the Caddy container (./caddy/access-diag.log on the host). Headers and access_token are stripped. Remove the log block when done; the file grows.
  • @claude-test:reilly.asia is a non-admin test account, joined to #fedora:fedoraproject.org. Its password is in ~/.spindle-testuser-pass (mode 600) on the VPS. Deactivate it when it is no longer needed.

Diagnosing on this host

  • Metrics: docker exec spindle-caddy-1 wget -qO- http://spindle:9100/metrics. spindle_http_requests_total{route="unmatched"} counts paths Spindle does not serve.
  • docker logs spindle-caddy-1 shows nothing new: the container's json log file was deleted from disk. Log to a file in /data instead (above).
  • Editing Caddyfile or spindle.toml: both are single-file bind mounts. sed -i and most editors replace the inode, and the container keeps reading the old file. Write in place (cat new > Caddyfile, or Python open(p, "r+") + truncate), or restart the container. Check with docker exec spindle-caddy-1 cat /etc/caddy/Caddyfile.
  • Seeing what a client requests: sudo tcpdump -i any -A "tcp port 8008 and host <spindle container IP>" shows plaintext proxy→Spindle traffic (it includes bearer tokens; do not save it).
  • Partial-state rooms: docker logs spindle-spindle-1 | grep -E "resync|partial-state".
  • Test account while registration is closed: add shared_secret under [registration], restart, register via /_synapse/admin/v1/register (Synapse HMAC), remove the secret, restart. Details in the Spindle fault guide.

Releasing a Spindle fix to this host

release.yml (dispatch with bump=patch) tags main only after every check on the commit has finished, including the Matrix compliance suite (~1 h after merge). Dispatching earlier logs "CI is still running" or "The Rust quality gate has not passed" and builds nothing. Then:

cd /srv/spindle
cp compose.yaml compose.yaml.bak-pre-vX.Y.Z-$(date -u +%Y%m%dT%H%M%S)
sed -i 's#tuna-os/spindle:vOLD#tuna-os/spindle:vNEW#' compose.yaml   # compose.yaml is not bind-mounted
docker compose pull spindle && docker compose up -d spindle

Punjab

Headless agent and development box, and the fleet's Ansible host. Since 2026-10-27 it is a KubeVirt VM on karnataka. The EC2 instance it replaced (tailnet node punjab-aws-retired, 100.78.73.8) was last seen on 2026-10-28. See EC2 history.

Runs onkarnataka k3s, corral-vms/punjab (VM created 2026-10-27)
Resources12 cores, 24 Gi of RAM, 500 Gi root PVC (punjab-root, local-path on karnataka)
OSUbuntu 24.04 LTS
Tailnetpunjab / 100.103.187.46
Accessssh ubuntu@punjab (there is no james user). For the VM console, run virtctl console -n corral-vms punjab from a host that has karnataka's kubeconfig

Role

  • Agent host: Claude Code, OMP and Codex sessions in tmux. Most git worktrees are in ~/dev.
  • kubectl against karnataka's k3s: ~/.kube/config.
  • Hive fork builds: ~/dev/hive-v6-port (tuna-os/hive, deploy branches). The amd64 binary that bihar and karnataka run was built here. See Hives hosting.
  • sccache.service (user) is a shared compiler cache for cargo builds.
  • The Kiro Crew gateway is retired: kirocrew.service is masked.

Disk

On 2026-11-18 / was 96% full (19 GB free). ~/dev used 156 GB. Delete stale worktrees under ~/dev and ~/tuna-wt before you make the PVC larger.

Dependencies

punjab is a VM on karnataka. When karnataka restarts or k3s restarts its VMs, punjab also restarts, and the agent tmux sessions stop.

Ansible notes (host_vars/punjab.yml)

  • skip_proxy: true: there is no podman on stock Ubuntu, so there is no Caddy quadlet.
  • Not a desktop: no GNOME, flatpak or browser roles.
  • syncthing runs only if its brew binary exists.

EC2 history (retired)

From 2026-08 to 2026-10, punjab was an EC2 t3.large in us-east-1, the Ansible host of the TunaOS AWS account (AWS account). It had SSM break-glass access, daily DLM snapshots and Kiro Crew. It is described in aws/punjab.tf. Its tailnet node is now punjab-aws-retired. Before you delete the instance and its snapshots, confirm that the VM has all the data.

Goa

Home server / Ansible control node for the hanthor fleet.

Hardware

  • Arch: aarch64 (ARM Cortex-A76)
  • Board: Raspberry Pi 5
  • Tailscale: goa (MagicDNS)

OS

Debian GNU/Linux 13 (trixie), kernel 6.18.29

Role

  • Fleet control plane — runs Ansible to manage all nodes
  • Primary pi-coding-agent development host
  • Inventory + health check hub (just inventory, just doctor)

Notes

  • Homebrew at /home/linuxbrew/.linuxbrew (ARM — some bottles unavailable)
  • System packages via apt, userland tools via brew
  • Hosts the dotfiles git repo at ~/.local/share/dotfiles
  • Pi agent config at ~/.pi/agent/

Home Kubernetes Server Images

Not deployed (2026-11-18). bihar and karnataka run stock Bluefin (ghcr.io/projectbluefin/utah:testing), not these images; k3s is installed on the host from home-k3s/. The server-images CI still builds them. Treat this page as a design for a future rebuild.

Bootable container images for the home K8s cluster (bihar, karnataka, starter). Built from Fedora Hummingbird + GNOME desktop + K3s, with graphical target disabled by default.

Image Variants

VariantRoleCPU/GPUImage URIInstaller ISO
biharK3s control planeIntelghcr.io/hanthor/home-k8s-bihar:latestRelease
karnatakaK3s worker (AMD APU + GPU)AMD Strix Haloghcr.io/hanthor/home-k8s-karnataka:latestRelease
starterDisposable test/staging nodeAnyghcr.io/hanthor/home-k8s-starter:latestRelease

Image Composition

Each image includes:

  • Base: projectbluefin/utah (Hummingbird + GNOME 51 desktop + Bluefin theming)
  • K3s: Kubernetes v1.36.1 + SELinux policy
  • Storage: iSCSI utils, NFS utils, Longhorn tools
  • Networking: NetworkManager, Tailscale, iptables-nft
  • GPU: AMD firmware + amdgpu kernel driver (built into boot image via Talos Image Factory)
  • Filesystems: ext4, xfs, btrfs (all tools included)
  • Default state: Multi-user target (text console), GDM masked

Obtaining Images

Download dated ISO releases from GitHub Releases:

  • {bihar,karnataka,starter}-installer.iso — offline bootc-installer-tui ISOs
  • {bihar,karnataka,starter}.oci.tar — OCI archives for bootc switch
  • SHA256SUMS — checksums for all artifacts
  • provenance.json — builder details (source commits, acceptance proofs)

Development/Testing

Build locally:

just server-images          # Builds all three variants to localhost/home-k8s-*:testing
just server-installer bihar ghcr.io/hanthor/home-k8s-bihar:testing artifacts/bihar.iso

Container Registry

  • Tracking tags (mutable): ghcr.io/hanthor/home-k8s-{bihar,karnataka,starter}:testing
  • Immutable tags (dated): ghcr.io/hanthor/home-k8s-{bihar,karnataka,starter}:YYYYMMDD

Both are public (no login required).

Installation

From Installer ISO

  1. Boot the ISO from USB stick (UEFI, Secure Boot must be disabled)
  2. At root console:
    bootc-installer-tui
    
  3. Select image: Custom URL → oci:/usr/share/home-k8s/payload
  4. Choose filesystem: ext4 (default), xfs, or btrfs
  5. Set hostname: bihar, karnataka, or starter (enables K3s role)
  6. Create admin user: nodeadmin (or any name)
  7. Confirm and install

The TUI automatically configures:

  • Disk partitioning (EFI + /boot + root)
  • K3s service enablement (based on hostname)
  • Tailscale daemon (unenrolled at install time)
  • SELinux enforcing mode

After reboot, the node boots to text console (multi-user.target). K3s is gated (disabled) until you manually enable it.

From OCI Archive (bootc switch)

To upgrade an existing machine from a previous image:

# Download the OCI archive
wget https://github.com/hanthor/dotfiles/releases/download/home-k8s-20261021/home-k8s-bihar.oci.tar

# Extract locally
podman load -i home-k8s-bihar.oci.tar
# → Loads as ghcr.io/hanthor/home-k8s-bihar:testing

# Switch to new image
sudo bootc switch ghcr.io/hanthor/home-k8s-bihar:testing

# Reboot to apply
sudo systemctl reboot

Filesystem Options

All three mkfs tools are included. Choose at install time or format a custom layout:

FilesystemUse casePerformanceNotes
ext4Default, safeStable, widely supportedVerity-enabled for composefs
xfsHigh-throughput, serversFast journalRequires mkfs.xfs (included)
btrfsSnapshots, COWModern, complexRequires mkfs.btrfs + btrfs CLI

Deployed images support all three without additional tools. If a filesystem option fails to appear in the TUI, verify the mkfs tool exists: which mkfs.xfs etc.

Graphical Target (Desktop)

Utah ships with GNOME desktop, but the images disable it by default (multi-user.target) for server use.

Enable Graphical Target

# Simple toggle script (on the machine)
sudo toggle-graphical --reboot

# Or manually
sudo systemctl set-default graphical.target
sudo systemctl reboot

What happens:

  • GDM (GNOME Display Manager) starts
  • Full GNOME 51 desktop available
  • Tailscale MagicDNS in system tray
  • K3s still running (if enabled)

Disable Graphical Target

sudo toggle-graphical --reboot

K3s Enablement

K3s is installed but gated (disabled) at boot. Roles are determined by hostname:

HostnameK3s RoleServiceEnabled at boot
biharserver (control plane)k3s@server.serviceNo (gated)
karnatakaagent (worker)k3s@agent.serviceNo (gated)
starter(none)——

To enable K3s on bihar (control plane):

sudo systemctl enable k3s@server.service
sudo systemctl start k3s@server.service

# Wait for API server
sudo systemctl status k3s@server.service

To enable K3s on karnataka (worker):

sudo systemctl enable k3s@agent.service
sudo systemctl start k3s@agent.service

Check that nodes joined the cluster:

kubectl get nodes -o wide

Networking & Access

  • SSH: nodeadmin@{hostname}.manatee-basking.ts.net (via Tailscale)
  • Kubeconfig: At /etc/rancher/k3s/k3s.yaml (after K3s start)
  • Tailscale: Daemon running, identity unenrolled (register via web UI or sudo tailscale up)

Storage & Persistence

iSCSI (Longhorn)

Both nodes are configured for Longhorn iSCSI workloads:

  • iscsid.socket running
  • iscsi-initiator-utils installed
  • Unique initiator name generated at first boot
  • /var/lib/longhorn/ mounted to second NVMe (karnataka only)

Temporary Disks

# View all mounts
findmnt /

# Root filesystem type
lsblk -f /dev/sda3    # or appropriate root partition

Image History & Migration

Changes in Latest Release

home-k8s-20261024:

  • Base switched from os-minimal to utah (added GNOME desktop)
  • Filesystem detection in TUI (only shows available mkfs tools)
  • toggle-graphical script for easy desktop on/off
  • xfsprogs + btrfs-progs included (all filesystem tools)

Previous release (home-k8s-20261021):

  • Hummingbird base (minimal, no desktop)
  • ext4 only (xfs/btrfs not available)
  • No toggle script

Upgrading from Old Images

If you're running home-k8s-20261021 or earlier, you can upgrade:

# Get the new OCI archive
wget https://github.com/hanthor/dotfiles/releases/download/home-k8s-20261024/home-k8s-bihar.oci.tar

podman load -i home-k8s-bihar.oci.tar
sudo bootc switch ghcr.io/hanthor/home-k8s-bihar:testing
sudo systemctl reboot

Note: Old images do not have toggle-graphical. Update to get the script.

Troubleshooting

TUI Filesystem Option Missing

If xfs or btrfs don't appear in the installer TUI filesystem picker:

# On the live ISO (at root console)
which mkfs.xfs    # Check if tool exists
which mkfs.btrfs

If missing, the image was built without those packages. Rebuild or wait for next release.

K3s Won't Start

# Check service status
sudo systemctl status k3s@server.service
sudo journalctl -u k3s@server.service -n 50

# Verify kubeconfig exists
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml get nodes

Can't SSH After Graphical Enable

If you SSH while the system is switching targets, the SSH session may hang:

# On local machine, from another terminal
ssh nodeadmin@karnataka.manatee-basking.ts.net "sudo systemctl is-active graphical.target"

# Or just reboot the graphical machine and reconnect

Tailscale Not Enrolled

Machines boot with Tailscale daemon running but unenrolled:

sudo tailscale up
# or web UI at: https://login.tailscale.com/admin/machines

See Also

TunaOS AWS Account

Cluster terminated 2026-10-25; IaC is not reconciled. The AWS Talos cluster is historical. Matrix and standalone LiveKit are deployed on the OCI spindle VPS. S3 backups and a mirror on himachal-1 were retained; do not assume other AWS resources have been removed or retained without checking the account.

Do not blindly run just aws-plan or just aws-apply. aws/ still describes removed cluster resources. A plan can propose recreating them; applying it can recreate infrastructure and incur costs. Review live state, configuration and every proposed action before any apply.

This TunaOS account's OpenTofu configuration is in aws/; runs-on was managed separately by CloudFormation. The table describes the configuration and prior deployment, not a verified live resource inventory.

RegionConfigured or historical resourceConfiguration
eu-north-1Terminated Talos cluster — VPC, nodes, Elastic IPs, security groups, pgdata volumeaws/cluster.tf
eu-north-1Backup configuration — DLM policy and retained fleet-backups bucketaws/backups.tf
us-east-1punjab EC2 agent box; running as of 2026-10-27aws/punjab.tf
us-east-2runs-on GitHub Actions runners; current stack status unverifiedCloudFormation stack runs-on
globalIAM, backup, account and budget configurationaws/iam.tf, aws/backups.tf, aws/account.tf, aws/budgets.tf

OpenTofu layout

aws/
├── versions.tf     # provider (eu-north-1 default, us_east_1 alias) + S3 backend
├── cluster.tf      # VPC/subnet/IGW/routes, 3 SGs, 2 Talos nodes, EIPs, pgdata
├── backups.tf      # DLM role + policy, fleet-backups bucket (+ versioning/SSE/lifecycle)
├── punjab.tf       # key pair, SG, instance, EIP, SSM role, DLM
├── account.tf      # EBS encryption by default (both regions)
├── iam.tf          # scoped admin user + its policies
├── budgets.tf      # 5 budgets, one for_each
├── imports.tf      # one-time adoption record (no-op once imported)
├── variables.tf
└── terraform.tfvars.example
  • State: in the fleet-backups bucket under tofu/aws/ (versioned, SSE-S3, lockfile). It contains the Talos nodes' user_data — i.e. cluster PKI — so it never goes in git.
  • Variables: aws/terraform.tfvars holds the alert email and the admin ingress allowlists. The repo is public, so it is gitignored and stored in the Bitwarden secure note aws-tofu-tfvars.
  • Talos user_data is not in the config. The node resources ignore user_data and ami; prevent_destroy is configured on the nodes, punjab, worker EIP, pgdata volume and backup bucket. It does not prevent creation of resources missing from the account.

Usage and drift

The former expectation of a no-change plan no longer applies after termination. Do not treat drift as a reason to restore resources. Before using just aws-plan, inspect the account and configuration; review the resulting plan carefully and reconcile the intended resource lifecycle before any just aws-apply. Planning does not itself create resources, but applying proposed creates can restore infrastructure and incur charges. aws/terraform.tfvars remains sensitive; just aws-seed-tfvars stores it in Bitwarden after an intentional edit.

IAM & access

  • Root historically had MFA and no access keys. Verify current access configuration before relying on it.
  • A scoped admin user was the day-to-day identity, with EC2, S3, budgets, DLM, SSM and read-only IAM access. The historical key item is aws-james-admin; verify its current permissions and status before use.
  • A backup user was configured for put/get under postgres/ in the backup bucket, with no delete. Its key was stored in Bitwarden as postgres-backup-s3 and in the former cluster's postgres/postgres-backup-s3 Secret, not tofu state.
  • punjab's configured instance role has AmazonSSMManagedInstanceCore.
  • DNS was managed on Cloudflare. Verify current AWS security and DNS settings before relying on this historical inventory.

Backups

The S3 fleet-backups bucket was retained, as was a mirror on himachal-1 at ~/backups/tunaos-aws-2026-10-25. Protect these backups and the OpenTofu state; do not assume their presence means every former cluster volume or snapshot survived. The former cluster uploaded verified Postgres dumps to postgres/<date>/; configured S3 lifecycle rules moved them to IA after 30 days, Glacier IR after 90, and expired them after a year. Check live bucket settings before relying on that schedule.

The configuration also describes DLM schedules for the former Talos root volumes and punjab. Those schedules and the availability of individual snapshots are not a current backup inventory.

Cost

The AWS credits were exhausted. Remaining resources can incur charges, and recreating cluster infrastructure would add costs. aws/budgets.tf describes budget alerts; verify current budgets and actual charges in the AWS account.

Hive — AI Agent Supervisor

Current deployment: bihar (karnataka standby)

Three Hive spokes, school, reef and hanthor, and their hub run as compose services under rootful podman on bihar, in /var/srv/hives (ssh james@bihar, commands need sudo). Until 2026-11-18 they ran on the Spindle VPS under /srv/hives. Spindle's Caddy still routes school.tunaos.org, reef.tunaos.org, hive.reilly.asia and hub.tunaos.org, over the tailnet to the active host. karnataka is a cold standby. A lease on spindle selects the active host. Run every command below on the host the lease names (ssh spindle sudo hive-failover status). Hosting, failover and the host-specific compose layer are described in Hives hosting and failover.

hanthor (hive.reilly.asia)

The personal-account spoke from the old AWS cluster was rebuilt on bihar on 2026-11-19. It is the hanthor service in compose.override.yaml (port 3104), with its data in /var/srv/hives/hanthor. It uses the same GitHub App as school (hanthor-hive-agent, App ID 3942065) through the App's installation on the hanthor account (158987150), and the same key file. Its seed is hives/hanthor/hive.yaml: ACMM level 5, four agents (supervisor, scanner, quality, guide) on the claude backend, and the repositories dotfiles, reilly.asia, homebrew-tap, shrimply and indiafoss-chat-android. Compared with the AWS seed it drops indiafoss-companion (archived), hummingbird-github (deleted) and rust-wayland-desktop (now tuna-os/tuna-desktop, which school covers). The AWS data volume was not kept, so the hive started with empty state.

The layout and roster below were read on spindle on 2026-11-13. The move changed the paths and the commands, but not the hive configuration. Container state alone does not prove that agents are healthy. See Checking health.

This section is the current operations guide. The archived cluster guide after it describes the retired Kubernetes deployment and must not be applied here.

Layout

/var/srv/hives/          # on the active host; the host layer is /var/srv/hives-host/compose.<host>.yaml
├── compose.yaml            # services: hub, hub-front, school, reef (network spindle_default)
├── compose.override.yaml   # pins the running binary/dashboard, memory limits, session pool;
│                           # also defines the hanthor spoke (added 2026-11-19)
├── .env                    # secrets and NTFY topic (mode 600; never print or commit)
├── bin/hive-<sha>          # fork builds; compose.yaml mounts bin/hive, the override re-pins one
├── <spoke>/config/hive.yaml        # seed, mounted read-only at /etc/hive
├── <spoke>/data/hive.yaml.runtime  # live config, in the spoke's /data volume
├── hub/  hub-front/        # hub config/data/secrets; static front page
└── backups/                # timestamped copies of every config and data snapshot
  • Seed vs runtime. The spoke starts from the seed and persists its live configuration to data/hive.yaml.runtime, which then wins. Hive can rewrite the runtime file (dashboard changes, ACMM level), so the two files drift: reef's runtime is ACMM 6 while its seed says 5. Edit the seed and the runtime for anything you want to stick, then verify the parsed result.
  • Binary. compose.yaml bind-mounts bin/hive over /usr/local/bin/hive; on 2026-11-13 compose.override.yaml re-pins hub, school and reef to bin/hive-2ef7dc7c0 (a tuna-os/hive fork build on upstream v6) and mounts a matching dashboard HTML on the spokes. The image tag (ghcr.io/hivecommons/hive:v6-latest, hub hive-hub:v6-latest) does not identify the running code; the header of the override file is the change log and rollback note.
  • Agents. All agents use the claude backend (sonnet, opus, haiku and fable tiers) on both spokes; the seed's backends.allow also lists omp (and muse on school). Runtime model assignments differ from the seed's.
  • Dashboard. dashboard.authorized_users is hanthor:owner. The dashboard.auth_token in the runtime file and the GitHub App key are secrets: never print, paste or commit them.

Roster

Each non-empty tuna-os repo is in exactly one hive, except the deliberate exclusions below (state of 2026-11-13; xfce-linux was renamed brussels).

HiveRepos
school (21)tunaos, tromso, tacklebox, brussels, tunaos-packages, wootc, fisherman, bootc-installer, bootc-migrate, bootsahi, remora, bst-ci, iso-builder, branding, homebrew-tap, scoop-bucket, athens, docs, .github, bluefin-cli, tuna-desktop
reef (22)Tavern, corral, flatpak-index, gtk-office-suite, blueshell, dualcut, gnome-hive-monitor, hive, mandelbrot, mariner, protota, spindle, PaperWM, compass, ccleft, hive-operator, quickcast, postcard, lifafa, tuna-mail, proton-unofficial-sdk, tuta-unofficial-sdk
excludediced, libcosmic, rust-atomicwrites, rdocx, changelog-action (patch-carrier forks, no issues); flatpark (excluded from automation); publish-proof (empty)

The roster is two lists, project.repos and dashboard.matrix.repos, and both must match in both files of the spoke (four edits per change). The seed uses either bare names or tuna-os/<name> in project.repos (reef bare, school full); the runtime file uses bare names there and tuna-os/<name> in the matrix. Keep whatever style the file already has. school also has a separate revert_sheriff.repos list (shadow mode); it is not part of the roster.

As of 13:01Z on 2026-11-13 the reef seed lacked postcard (21 repos) while the runtime listed it (22); the intent of that seed edit is unverified.

Identity

HiveGitHub AppApp id
schoolhanthor-hive-agent3942065
reeftuna-reef-hive5546574

Both set github.app_authored_prs: true and project.ai_author: "", so PRs and comments come from the App's [bot] account, not from a human or AI-author trailer. Each hive lists the other App in github.additional_bot_logins. Both Apps are installed on every org repo (per the maintainer; not checkable from the host), so adding a repo needs only a roster edit, no App change.

Both hives set sentinel.exempt_logins: [renovate[bot]]. Sentinel's sensitive_path rule flagged every dependency bump and re-flagged after each rebase. The alternative, auto_merge.trusted_bot_authors, would also let Hive auto-merge Renovate major bumps, which the org forbids.

Adding or moving a repo

  1. Pick the hive: school for the OS, packaging, installer and org-level repos; reef for apps, libraries and the Matrix/Hive tooling (see the table). Check the repo is in no other roster.
  2. Apply the safe edit procedure with add <repo> on the target hive's two files. To move a repo, also run remove <repo> on the old hive's two files, in the same maintenance window, so it is never in both or neither.
  3. Re-parse both files and check the list lengths and that the new repo shows up on the hive's dashboard; no App installation step is needed.

GitHub Apps

Since 2026-10-30 School and Reef use separate GitHub Apps, so each has its own installation rate limit (6,150 requests/hour across the 43 org repositories). Before that change they shared one installation and used all of its limit every hour.

HiveApp (bot login)App IDInstallationKey in Bitwarden
school (the hub mounts the same key)hanthor-hive-agent (hanthor-hive-agent[bot])3942065137498420hive-github-app-hanthor-hive-agent
reeftuna-reef-hive (tuna-reef-hive[bot])5546574177007234hive-github-app-tuna-reef-hive

The IDs occur in three places for each hive:

  • GH_APP_ID and GH_APP_INSTALLATION_ID in compose.yaml;
  • github.app_id and installation_id in the seed;
  • the same keys in hive.yaml.runtime.

Both hives manage .github, Tavern, bluefin-cli, corral, docs and flatpak-index. Each hive lists the bot of the other hive in github.additional_bot_logins, so that the issues and PRs of the other hive count as hive-authored.

To restore a key:

  1. Write the key block of the Bitwarden note to /var/srv/hives/<svc>/secrets/gh-app-key.pem on the active host (mode 600).
  2. Run chcon -t container_file_t on the file.
  3. Recreate the service with sudo /var/srv/hives-host/hives up -d --no-deps <svc>.

The next hive-sync copies the key to the standby host.

To make sure that each hive has its own rate limit, run:

ssh james@bihar 'for c in hives-school-1 hives-reef-1; do sudo podman exec $c sh -c \
  "curl -s -H \"Authorization: Bearer \$(cat /var/run/hive-metrics/gh-app-token.cache)\" https://api.github.com/rate_limit" \
  | jq -c .resources.core; done'

Different reset times show separate installations.

Safe edit procedure

More than one operator edits /var/srv/hives (including agents). Re-read the files immediately before editing and never overwrite a file you read earlier.

Do not hives up a spoke that is running to apply config: it triggered a container-rename bug. Remove the container first:

sudo -i                                   # the files are root-owned on bihar/karnataka
cd /var/srv/hives
H=/var/srv/hives-host/hives               # docker-compose wrapper: podman + host layer
$H rm -f -s reef                          # stop and remove the spoke container only
python3 hive-roster.py add NAME reef/config/hive.yaml reef/data/hive.yaml.runtime
$H up -d --no-deps reef                   # recreate only this spoke
podman ps --filter name=hives-            # verify: Up, and the right name

The edit step must (a) write a timestamped copy of each file into /var/srv/hives/backups/ first, (b) parse the YAML before and after, and (c) abort, writing nothing, if anything other than the intended change would differ. This helper does that for roster edits (save it as /var/srv/hives/hive-roster.py and run it from /var/srv/hives; it needs PyYAML):

#!/usr/bin/env python3
"""hive-roster.py add|remove REPO FILE...  (run with the spoke container removed)"""
import datetime, shutil, sys, yaml

ORG = "tuna-os"
BACKUPS = "/var/srv/hives/backups"
PATHS = (("project", "repos"), ("dashboard", "matrix", "repos"))

def seq_node(root, path):
    node = root
    for key in path:
        node = next(v for k, v in node.value if k.value == key)
    return node

def edit(text, op, repo):
    lines = text.split("\n")
    root = yaml.compose(text)
    # bottom-most list first so earlier line numbers stay valid
    seqs = sorted((seq_node(root, p) for p in PATHS), key=lambda n: -n.start_mark.line)
    for seq in seqs:
        full = seq.value[0].value.startswith(ORG + "/")
        item = f"{ORG}/{repo}" if full else repo
        hit = [n for n in seq.value if n.value in (repo, f"{ORG}/{repo}")]
        col, last = seq.value[0].start_mark.column, seq.value[-1].start_mark.line
        if op == "add" and not hit:
            lines.insert(last + 1, " " * (col - 2) + "- " + item)
        elif op == "remove":
            for n in sorted(hit, key=lambda n: -n.start_mark.line):
                del lines[n.start_mark.line]
    return "\n".join(lines)

def main(op, repo, *files):
    assert op in ("add", "remove") and files
    stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    new = {}
    for f in files:
        old = open(f).read()
        want = yaml.safe_load(old)
        for path in PATHS:
            d = want
            for k in path[:-1]:
                d = d[k]
            style = d[path[-1]][0].startswith(ORG + "/")
            item = f"{ORG}/{repo}" if style else repo
            d[path[-1]] = [x for x in d[path[-1]] if x not in (repo, f"{ORG}/{repo}")]
            if op == "add":
                d[path[-1]].append(item)
        text = edit(old, op, repo)
        if yaml.safe_load(text) != want:
            sys.exit(f"{f}: unrelated change detected, aborting (nothing written)")
        new[f] = text
    for f, text in new.items():
        shutil.copy2(f, f"{BACKUPS}/{f.replace('/', '-')}.{stamp}")
        open(f, "w").write(text)

main(*sys.argv[1:])

The script was exercised only on synthetic copies of the two file styles, not on the live files. Backups from the 2026-11-13 edits are backups/*.20261113T040153Z (sentinel exemption) and backups/*.roster.20261113T125525Z (roster); to roll back, remove the container, restore both files of the spoke from the same stamp, and up it. Rolling back the binary is described in the header of compose.override.yaml.

Operating from GitHub

You rarely need the host. On a hive-managed issue or PR:

  • /hive decision <instruction> records a maintainer decision for the agents; /hive approve lifts Hive's own needs-human hold on a PR; /fixed tells Hive an issue is resolved.
  • On PRs that Hive did not author, a GitHub review (approve or request changes) is honoured.
  • When Hive un-parks an issue it removes the blocked label itself. Decision comments written by the maintainer's agent start with Decision (for @hanthor):.

(The command names are present in the running binary and the usage comes from the maintainer; their exact semantics were not exercised for this page.)

Checking health

curl -s -o /dev/null -w '%{http_code}\n' https://school.tunaos.org/api/health   # 200
curl -s -o /dev/null -w '%{http_code}\n' https://reef.tunaos.org/api/health     # 200
curl -s https://hub.tunaos.org/api/fleet-stats                                  # hub: /api/health is 404
ssh james@bihar 'sudo podman ps --filter name=hives-'   # or karnataka after a failover

For authenticated status from the host, call the API inside the container with the container's own token, so it is never printed:

ssh james@bihar 'sudo podman exec hives-school-1 sh -c \
  "curl -sS -H \"X-Hive-Internal: \$HIVE_DASHBOARD_TOKEN\" http://127.0.0.1:3002/api/status"' \
  | jq '.agents[] | {name, paused, state}'

Agents that show stopped are normal between governor kicks; check authored PRs and issues (gh search prs --author "hanthor-hive-agent[bot]" and "tuna-reef-hive[bot]") to see whether a hive is producing.

Known limits

  • Memory. On bihar and karnataka, each spoke can use 8 GB and the hub allows 4 concurrent shared-pool sessions (compose.<host>.yaml). On spindle the limits were 2560 MB per spoke and 2 sessions, and spokes were OOM-killed during rustc/govulncheck builds.
  • Pinned images. The hive and hive-hub images are pinned by digest with the amd64 fork binary hive-2ef7dc7c0-amd64. Do not change to :v6-latest.
  • Hive's own CI for tuna-os/hive is blocked by GitHub ("Actions has been disabled"; a support ticket is pending), so fork builds are not CI-verified.
  • Reply spam: upstream hivecommons/hive#17512 (draft) fixes Hive repeating its un-park replies because it does not recognise its own earlier reply when posting as an App login.
  • Concurrent edits and drift: see the layout notes above; verify after every restart that the parsed seed and runtime agree on the roster.
  • The spokes call the omp-auth-broker on spindle (OMP_AUTH_BROKER_URL, http://100.113.75.88:8765). Caddy routing and failover are described in Hives hosting and failover. This page does not describe the hub's own configuration. The hive_ops role targets the retired cluster, not this deployment.

Archived cluster guide

Historical reference only — do not apply or run the Kubernetes, image-build, secret, restart, or state-reset instructions below. The AWS Talos cluster was terminated on 2026-10-25; its Kubernetes resources and CronJobs are not a current deployment. The home Talos cluster (bihar and karnataka) is powered down temporarily, not decommissioned. Preserve its configurations. The sections below describe older home-cluster and AWS configurations at different points in time, not the current Spindle service. See the AWS cluster history and historical cluster manifests. Examples are retained for context, not as instructions for either cluster.

The earlier home-cluster Hive used pi with DeepSeek, tmux agents, a governor, and a GitHub App. The architecture and examples below are snapshots of that period; later AWS configurations and the current Compose deployment differ.

1. Architecture (historical)

┌──────────────────────────────────────────────────────────────┐
│ Namespace: hive (node: bihar)                                │
│                                                              │
│  ┌─────────┐  ┌──────────┐  ┌────────────────────────────┐  │
│  │ hive    │  │ proxy    │  │ 9x pi agents (tmux sessions)│  │
│  │ Go bin  │  │ node.js  │  │                            │  │
│  │ :3002   │  │ :3001    │  │ supervisor  ADVISORY       │  │
│  │ (API)   │  │ (web UI) │  │ scanner     ISSUES_AND_PRS │  │
│  └────┬────┘  └────┬─────┘  │ ci-maintain ISSUES_AND_PRS │  │
│       │            │        │ quality     ISSUES_AND_PRS │  │
│       └────────────┘        │ sec-check   ISSUES_AND_PRS │  │
│                             │ guide       ISSUES_AND_PRS │  │
│  ┌──────────────────────────│ architect   ISSUES_AND_PRS │  │
│  │  pi → DeepSeek API       │ strategist  ISSUES_AND_PRS │  │
│  │  (native, no proxy)      │ brainstorm  on-demand     │  │
│  │  GitHub App → gh CLI     └────────────┬───────────────┘  │
│  └───────────────────────────────────────┘                  │
│                                                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │
│  │ ConfigMap│  │ Secret   │  │ PVC      │                  │
│  │ hive.yaml│  │ DEEPSEEK │  │ hive-data│                  │
│  │ backends │  │ GH App   │  │ (10Gi)   │                  │
│  └──────────┘  └──────────┘  └──────────┘                  │
└──────────────────────────────────────────────────────────────┘
         │
    ┌────▼────┐     ┌──────────────────┐
    │ GitHub  │     │ Tailscale Ingress │
    │ tuna-os │     │ hive.manatee-     │
    │ repos   │     │ basking.ts.net    │
    └─────────┘     └──────────────────┘

Components

ComponentLanguagePurpose
hive (Go binary)GoAgent manager, governor, scheduler, dashboard API
proxy (Node.js)JSDashboard web UI + SSE, proxies to Go API
pi (coding agent)Node.jsAI agent CLI running in tmux sessions
pi-wrapper.shShellTmux interface adapter — restart loop, ready markers

How agents work

Each agent runs as a long-lived pi process inside a tmux session. The Go binary:

  1. Creates a tmux session per agent (hive-supervisor, hive-scanner, etc.)
  2. Writes a boot prompt (agent role, authorized repos, instructions) via send-keys
  3. Detects agent readiness from CLI markers in the pane output
  4. Governor evaluates the issue queue every 5 minutes and sends kick prompts to due agents
  5. clear_on_kick: true agents receive ^C before each kick to reset their session

Agents share a persistent /data volume (10Gi PVC) for beads, session state, and agent configs.


2. What Hive did (historical)

Advisory digest

The dashboard shows agent findings — CI failures, test coverage gaps, security issues, documentation problems. Each agent produces beads (structured findings) that appear in the advisory digest.

Issue creation

At ACMM L3+, agents create GitHub issues. At L4+, they can create hold-gated PRs. The sec-check agent scans dependencies for CVEs. The ci-maintainer agent monitors CI failures.

Strategy Lab

Inception feature: describe what you want to build, and Hive scaffolds a new project with knowledge base facts and spec-kit specifications. Powered by the brainstorm agent (on-demand).

Knowledge Base

Agents can contribute facts to a shared knowledge base. Facts are tagged by type (pattern, gotcha, regression, decision, etc.) and searchable from the dashboard.


3. Deployment (historical)

Namespace and resources

The home-cluster deployment used the hive namespace on bihar:

hive (namespace)
├── deploy/hive           — main controller pod
├── svc/hive              — ClusterIP :3001 (dashboard), :3002 (API), :7681 (ttyd)
├── ingress/hive          — Tailscale ingress → hive.manatee-basking.ts.net
├── configmap/hive-config — hive.yaml (project config, agents, governor)
├── secret/hive-secrets   — DEEPSEEK_API_KEY, GH App credentials
├── pvc/hive-data         — 10Gi persistent state
└── secret/ghcr-auth      — GHCR pull credentials

Historical files (not the current repository layout)

talos-k8s/hive/
├── hive.yaml              — All K8s resources (namespace, deploy, svc, ingress, configmap)
├── Dockerfile             — Custom image (hive v2 + pi + fd/ripgrep)
├── pi-wrapper.sh          — Wrapper: Hive tmux interface → pi interactive TUI
├── pi-marker.patch        — Adds "pi" to hive's cliPaneMarkers (source patch)
├── build-job.yaml         — Kaniko build job (deprecated, use CI)
├── deepseek-chat.py       — Python fallback (unused with pi)
├── goose-wrapper.sh       — Legacy goose wrapper (replaced by pi-wrapper.sh)
├── goose-marker.patch     — Legacy goose patch (replaced by pi-marker.patch)
├── SKILL.md               — Skill file for pi coding agent
├── README.md              — Quick reference
└── HANDOFF.md             — Debugging session notes

CI workflow

The former .github/workflows/hive-build.yml workflow did the following:

  1. Clones kubestellar/hive v2 source
  2. Applies pi-marker.patch and three sed patches to the Go source:
    • Adds "pi" to cliPaneMarkers (so agents aren't detected as crashed)
    • Adds "pi": true to validBackends (config validation)
    • Adds "pi": "pi" to backendBinary() (binary resolution)
  3. Cross-compiles Go binary for amd64
  4. Builds Docker image with pi + pi-wrapper.sh + fd/ripgrep
  5. Pushes to ghcr.io/hanthor/hive:latest

Secrets and deployment (historical)

The home-cluster pod used a Kubernetes Secret for DeepSeek and GitHub App credentials. Later, the AWS deployment exported spoke manifests under talos-k8s/hive/spokes/ and used the hive-operator to manage images. These manifests are historical records, not deployment instructions. The home cluster's former Tailscale ingress was hive.manatee-basking.ts.net.


4. AI backend: pi + DeepSeek (historical)

Why pi (not goose)

Hive originally used goose (v1.36 from AAIF) as the AI agent CLI. Goose does not natively support DeepSeek, requiring a Python proxy (deepseek-proxy.py) to:

  • Strip reasoning_content from message history
  • Inject thinking:{type:disabled} to prevent thinking block generation
  • Route through litellm pointing at a local proxy

This was fragile. pi natively supports DeepSeek via its deepseek provider — no proxy, no litellm, no workaround.

Why pi works for unattended agents

Pi intentionally has no permission popups — tools (read, write, edit, bash) auto-execute without user confirmation. This makes it ideal for Hive's unattended agent workflow where no human is available to approve tool calls.

pi-wrapper.sh

The wrapper translates Hive's backend CLI interface to pi:

# Hive calls: pi --no-confirm --model deepseek-v4-pro
# Wrapper strips --no-confirm (pi doesn't need it), passes --model to pi
# Sets up ~/.pi/agent/settings.json with DeepSeek config
# Runs pi in interactive TUI mode inside the tmux pane
# Restart loop: ^C from clear_on_kick kills pi-real, wrapper re-emits ready markers

pi configuration

Written at runtime by the wrapper:

{
  "defaultProvider": "deepseek",
  "defaultModel": "deepseek-v4-pro",
  "quietStartup": true,
  "hideThinkingBlock": true,
  "enableInstallTelemetry": false
}

Helper tools

fd (fd-find) and ripgrep are pre-installed via apt in the Docker image so pi doesn't need to download them at runtime (the Hive MITM proxy would corrupt the downloads).


5. Agents (historical)

Historical agents (ACMM L5)

AgentModeCadenceRole
supervisorADVISORY5mAgent health monitoring, sweep analysis, stall detection
scannerISSUES_AND_PRS4hIssue triage, PR dispatch, merge management
ci-maintainerISSUES_AND_PRS4hCI/CD health, workflow fixes, build monitoring
qualityISSUES_AND_PRS2hTest coverage, integration tests, quality gates
sec-checkISSUES_AND_PRS4hSecurity scanning, dependency audit, CVE monitoring
guideISSUES_AND_PRS4hDocumentation gaps, onboarding analysis
architectISSUES_AND_PRS4hCross-cutting RFCs, refactors, new features
strategistISSUES_AND_PRS4hExperiment design, A/B testing, strategy lab
brainstormon-demand—Ideation via Inception; proposes features and architecture

Agent modes

ModeCan create issuesCan create PRsCan merge
ADVISORY✗✗✗
ISSUES_ONLY✓✗✗
ISSUES_AND_PRS✓✓ (hold-gated)✗

Mode is determined by ACMM level + agent config. At L5, most agents run in ISSUES_AND_PRS.

Agent lifecycle

  1. Boot: Agent starts in tmux session, receives role prompt + authorized repos
  2. Kick: Governor sends a task prompt (e.g., "scan for open issues", "check CI health")
  3. Work: pi processes the prompt, runs tools, produces beads or GitHub issues/PRs
  4. Idle: Agent waits for next kick
  5. Clear on kick (clear_on_kick: true): ^C kills the pi process before the next kick, wrapper restarts it fresh

Adding a new agent

# In hive.yaml ConfigMap, under agents:
my-agent:
  enabled: true
  backend: pi
  model: deepseek-v4-pro
  mode: ISSUES_AND_PRS
  beads_dir: /data/beads/my-agent
  clear_on_kick: true

Also add the agent to governor cadences for each mode:

governor:
  modes:
    idle:
      my-agent: 4h
    quiet:
      my-agent: 4h
    busy:
      my-agent: 1h
    surge:
      my-agent: pause

6. ACMM levels (historical)

The ACMM (Autonomous Code Maintenance Maturity) framework defines 6 levels:

LevelNameBehavior
L1AdvisoryAdvisory-only reports, no writes
L2IssuesCan create issues, no PRs
L3CI/CDIssues + hold-gated PRs, merge requires human LGTM
L4GuardedBroader write access, more agents active
L5Guarded Auto-MergeAuto-merge on green CI with guardrails
L6Full AutonomyUnrestricted agent actions

That historical configuration used L5; this is not a statement about the current spokes.

Changing ACMM level

# In hive.yaml ConfigMap
acmm_level: 5

The old procedure reset persistent state and restarted the pod; do not use it on the current deployment or on retained backups.


7. Governor (historical)

The governor evaluates the issue queue every 5 minutes (eval_interval_s: 300) and switches between four modes:

ModeThresholdBehavior
idle0 issuesSlow cadences, minimal activity
quiet≤2 issuesNormal cadences
busy≤50 issuesFaster scanner, ci-maintainer still active
surge>50 issuesScanner every 15m, ci-maintainer paused

Historical cadences

governor:
  eval_interval_s: 300
  modes:
    surge:
      threshold: 50
      scanner: 15m
      ci-maintainer: pause
      quality: pause
      sec-check: pause
      guide: pause
      architect: pause
      strategist: pause
    busy:
      threshold: 10
      scanner: 15m
      ci-maintainer: 1h
    quiet:
      threshold: 2
      scanner: 15m
      ci-maintainer: 45m
    idle:
      threshold: 0
      scanner: 4h
      ci-maintainer: 4h

The supervisor always runs at 5m regardless of mode.


8. Configuration (historical)

Backends (backends.conf)

The backends.conf in the ConfigMap defines supported AI backends. Adding a new backend requires entries in three places:

  1. KNOWN_BACKENDS list
  2. backend_binary() — maps backend name to binary in PATH
  3. backend_perm_flag() — permission flag (pi returns empty — no permission popups)

The old supported list was claude copilot bob gemini codex amazonq goose aider pi.

Agent configuration (hive.yaml)

agents:
  supervisor:
    enabled: true
    backend: pi
    model: deepseek-v4-pro
    beads_dir: /data/beads/supervisor
    clear_on_kick: true

Persistent state

Two locations store state that survives pod restarts:

  • /data/hive-state.json — Governor mode, agent overrides, kick history, cadence overrides
  • /data/agent-configs/*.yaml — Per-agent persistent config (backend, model, mode, enabled, display name)

Persistent overrides could cause stale values in that deployment. Do not copy its old state-reset and backend-rewrite commands to the current deployment.


9. Building the image (historical)

The former CI workflow built ghcr.io/hanthor/hive:latest; an older Kaniko job was also used. Neither is a current deployment procedure.

What's in the image

  • Base: node:24-slim
  • Runtime deps: bash, curl, git, gosu, iptables, jq, procps, python3, tmux, fd-find, ripgrep
  • Go binaries: hive (agent manager), bd (bead CLI) — cross-compiled from kubestellar/hive v2
  • Node.js proxy: Dashboard UI + SSE, serves on :3001
  • pi: @earendil-works/pi-coding-agent installed via npm
  • pi-wrapper.sh: Replaces /usr/local/bin/pi with the tmux interface wrapper
  • gh CLI: GitHub CLI with wrapper for App token auth
  • Entrypoint: UID isolation, tmux setup, config backup, App token generation

10. Historical operations

Operators inspected the old pod's health, tmux panes, pi processes and logs with kubectl. Those commands targeted the retired Kubernetes deployment; the read-only Compose inventory at the top of this page is the current starting point instead.

Historical restarts and state resets

The former procedure restarted the pod and sometimes reset persistent state. It is not a recovery procedure for Spindle; do not reset retained data.


11. Historical troubleshooting

SymptomCheck
ImagePullBackOffThe home cluster once encountered IPv6 timeouts to ghcr.io
Agent CLI crashedCheck ps aux | grep pi — should see 9 pi processes
Agent stuck in ADVISORY modeHistorical state-file ACMM overrides
Dashboard shows old backend/modelHistorical /data/agent-configs/*.yaml overrides
No kicks sentGovernor eval cycle: 5min. Agent cadences vary per mode
Issues not createdGitHub App installation and authentication were relevant
"CLI did not reach input prompt"pi is mid-response when kick arrives. Normal — agent will pick up next kick
fd/ripgrep download errorspi can't download tools through MITM proxy. Pre-installed in Docker image
"backend binary not found"Missing binary in PATH. Check backends.conf and verify wrapper is at /usr/local/bin/pi
"failed to persist config"Read-only ConfigMap mount. Cosmetic — config is cached in memory

Historical agent issues

Agent shows bare bash shell:

  • pi process died or never started
  • Check: tmux capture-pane -t hive-scanner -p | tail -5
  • Should show pi's footer (deepseek-v4-pro • high), not $ bash prompt

Agent stuck at /clear:

  • Hive sent /clear to reset the agent before a kick
  • Agent processed it but new kick hasn't arrived yet
  • Wait for next governor cycle

Agent mode mismatch:

  • The state file held persistent mode overrides in that deployment.

12. GitHub App (historical)

Hive authenticates to GitHub via a GitHub App installed on tuna-os (App and installation IDs are in the hive-secrets Secret).

Token generation

The entrypoint generates an installation token on startup and caches it at /var/run/hive-metrics/gh-app-token.cache. Agents access it via $GH_TOKEN.

Required permissions

PermissionScopeWhy
contents: readRepoRead code, files
contents: writeRepoCreate branches, push commits for PRs
issues: writeRepoCreate and update issues
pull_requests: writeRepoCreate PRs
metadata: readRepoBasic repo access
security_events: readRepoDependabot alerts (sec-check agent)

Rate limits

  • GitHub App: 15,000 req/hr (shared across all installations)
  • Personal Access Token: 5,000 req/hr

The App gives Hive its own rate limit pool, separate from personal API usage.

Permission changes (historical)

GitHub App permission changes required re-acceptance by the installation. Do not use the old Kubernetes restart procedure to apply them.

Images and upgrades on the former AWS cluster

On 2026-09-24 the AWS Hives used stock upstream images (ghcr.io/hivecommons/hive, hive-hub), not the tuna-os/hive fork. The configuration described below was subsequently superseded by the HiveRelease operator controller before the cluster was terminated.

HiveNamespaceBranding
hive.reilly.asia (canary)hive-hanthornone
reef.tunaos.orghive-reefREEF
school.tunaos.org (legacy alias hive.tunaos.org)hiveSCHOOL
hub.tunaos.orghive-hub—

At that time the hive-upgrade CronJob checked upstream v5 releases daily, upgrading one Hive at a time with a soak and rollback. It was later deleted in favor of the operator's HiveRelease controller. The retained upgrade history is not an active upgrade runbook.

The old hive-fork-{switch,verify,drift,ai-check} CronJobs had been suspended; they do not run on the terminated AWS cluster.

sshd

Tags: system, sshd
Secrets needed: No
Runs on: All machines

Configures the SSH daemon to accept Bitwarden environment variables from clients that explicitly use SSH environment forwarding. The current just apply-remote recipe does not depend on this drop-in: it resolves the session on the target and exports it inside the remote command.

What It Does

Drops a config file at /etc/ssh/sshd_config.d/99-bw-env.conf:

AcceptEnv BW_SESSION BW_CLIENTID BW_CLIENTSECRET

This drop-in is compatibility for explicit SendEnv/AcceptEnv workflows. It is not required by just apply-remote, which uses target-side resolution instead of SSH environment forwarding.

Handlers

  • Reload sshd — restarts the SSH service after config changes

Notes

  • Only runs if /etc/ssh/sshd_config.d/ exists (Debian/Ubuntu/Fedora convention)
  • On Alpine/musl systems this directory may not exist — the role skips gracefully

sudo

Tags: system, sudo
Secrets needed: No
Runs on: All machines

Grants passwordless sudo to the current user and cleans up old sudoers entries.

What It Does

  1. Writes a sudoers drop-in at /etc/sudoers.d/zz-<user> granting NOPASSWD:ALL
  2. Validates the file with visudo -cf before deploying
  3. Removes old/conflicting sudoers files from previous naming conventions

Notes

  • Required for unattended playbook runs — many tasks use become: true
  • The zz- prefix ensures this file loads last, overriding any earlier restrictions

apk_packages

Tags: system, packages, apk
Secrets needed: No
Runs on: All machines (skips if not musl/Alpine)

Installs core packages on Alpine/musl-based systems where Homebrew is unavailable.

What It Does

  1. Detects musl libc (Alpine, postmarketOS) — skips entirely on glibc systems
  2. Installs core CLI tools via apk: bash, fish, zsh, git, neovim, tmux, ripgrep, fzf, jq, podman, etc.
  3. Installs Bitwarden CLI via npm (not available in apk repos)
  4. Enables podman and podman.socket system services
  5. Adds the user to the podman group

When It's Skipped

On glibc systems (Fedora, Debian, Ubuntu, Bluefin), this role does nothing — Homebrew handles all package management.

server_hardening

Tags: system, hardening, security
Secrets needed: No
Runs on: server and vps groups (site.yml gates on group membership)

Applies reliability and light security hardening for long-running server hosts.

What It Does

  • journald: sets SystemMaxUse=500M and vacuums journals to that size
  • swap: creates a 2G /swapfile if missing, sets 0600, runs mkswap + swapon only when it isn't already active, adds it to /etc/fstab
  • NTP: enables and starts systemd-timesyncd
  • SSH rate limits: MaxStartups 3:50:10, MaxSessions 10, ClientAliveCountMax 3 in sshd_config (reloads sshd)
  • UFW: turns logging on if UFW is already active (doesn't enable UFW or add rules)
  • Livepatch: runs pro enable livepatch if Ubuntu Pro is attached (best-effort)
  • Reboot check: prints a warning if /var/run/reboot-required exists

Notes

  • Does not install fail2ban, change SSH auth/cipher settings, set sysctls, or configure automatic updates
  • No skip_ flag — to keep a host out, take it out of server/vps
  • The retired VPSes (matrix, telengana) are still in vps; don't apply to them (see the cutover runbook) — this role reloads sshd and restarts journald

fleet_facts

Publishes each machine's public device facts for the live fleet view at reilly.asia/infra.

  • Runs on: every host (opt out with fleet_facts_publish: false), last in the play, including the no-secrets daily timer
  • Tags: facts, fleet_facts
  • Needs: a logged-in gh CLI (skipped quietly otherwise)

Each run writes facts/<host>.json to the fleet-facts branch of this repo through the GitHub contents API. There's no clone and no merge, and master isn't touched. The Worker behind reilly.asia/infra reads those files.

FieldSource
os.{name,version,codename,kernel}Ansible distribution facts
hardware.modelAndroid getprop, else device-tree model (Pi), else DMI product name (placeholders like "System Product Name" dropped)
hardware.{vendor,form_factor,arch,cpu,cpu_threads,memory_gb}Ansible hardware facts
hardware.gpuslspci -mm display devices (unnamed virtual adapters dropped)
virtualization, uptime_days, last_applyAnsible facts, ~/.cache/dotfiles/last-apply.json

Never published: IP/MAC addresses, serial numbers, usernames, anything from Bitwarden.

Private drift monitoring (osquery)

The separate osquery role watches changes that must not go to the public fleet-facts branch. It installs checksum-pinned upstream osquery 5.23.1 under /opt, including on immutable desktops: no package layering, writable /usr, reboot, Fleet server, enrollment token, or new network listener.

Tags: osquery, services, security. Requires root, glibc Linux, systemd, and x86_64 or aarch64. Nura/Alpine musl (Kerala) and unrooted Android/Termux are unsupported; osquery_supported: false records that exclusion. Offline hosts install on their next successful normal apply. skip_osquery: true skips the role; it does not uninstall an already-running daemon. The retired VPS group is never modified, and Talos nodes are not Ansible-managed.

What's watched

  • TCP listeners and bound UDP ports: address, port, protocol, executable path. PIDs are excluded so a process restart alone does not create drift. UDP client sockets and temporary development listeners can legitimately change.
  • Enabled system startup triggers: service, timer, socket and path symlinks in /etc/systemd/system/*.wants, *.requires and their /run equivalents. This includes enabled-but-not-started units, without mistaking ordinary active/inactive transitions for startup changes. User units and vendor dependencies compiled into unit files are outside this query.
  • Kernel, OS and osquery versions.

osqueryd.service samples every five minutes, writing differential JSON logs only under /var/log/osquery (10 MiB rotation, three retained archives). fleet-osquery-report.timer refreshes a private baseline comparison every 15 minutes. Both have bounded CPU/memory use; audit/event capture and distributed queries are disabled. Sampling is not real-time intrusion detection or unit-file content integrity checking.

Review the fleet

From an updated checkout:

just apply-tags osquery
just osquery-fleet
just osquery-fleet --host punjab --host dilli
just osquery-fleet --json

The controller uses effective Ansible inventory, Tailscale online status and existing authenticated SSH plus noninteractive sudo. Host-key verification stays enabled. It shows ok, drift, offline, unsupported, excluded, stale or error; missing/bad reports and failed reads never become ok. Reports older than 30 minutes are stale. Stale/errors return nonzero; ordinary drift and offline coverage do not. Each result shows its collection time, not a claim of live state. --inventory PATH (or DOTFILES_INVENTORY) selects another inventory.

Always-on poller: Punjab

Punjab has osquery_poller_enabled: true. Its system-level fleet-osquery-poll.timer runs every 15 minutes at :02/:17/:32/:47, after the hosts' report-refresh window, and two minutes after boot. It runs as ubuntu, using that account's existing SSH keys, known hosts and noninteractive sudo. It does not depend on an interactive SSH agent or copy credentials into Kubernetes.

The role installs a root-managed collector and a minimal inventory under /opt/fleet-osquery and /etc/fleet-osquery/inventory.yml. Only connection and exclusion metadata is copied—not the full effective inventory or Bitwarden facts. Apply the osquery role on Punjab after changing fleet membership or connection details to refresh this inventory snapshot.

Each poll atomically replaces one aggregate: /home/ubuntu/.local/state/fleet-osquery/latest.json (0600, inside a 0700 directory, owned by ubuntu). No unbounded history is kept. Journald gets coverage counts, not raw addresses or executable paths. Polling never accepts host baselines.

ssh ubuntu@punjab 'cat ~/.local/state/fleet-osquery/latest.json'
ssh ubuntu@punjab 'systemctl status fleet-osquery-poll.timer --no-pager'
ssh ubuntu@punjab 'sudo -n systemctl start fleet-osquery-poll.service'

Offline machines remain explicitly offline. Failed reads/stale reports are saved as degraded coverage and fail the service; an unusable or empty inventory saves a timestamped error instead of retaining a supposedly healthy snapshot. If the process itself cannot run or write, check the service journal and the snapshot's collection time—an old file is not current health.

osquery_poller_enabled defaults to false on other hosts. Setting it false on a previous poller and applying the role stops its timer and in-flight service; the last private snapshot remains for inspection. The manual fleet CLI still works from other machines and supports --output PATH for explicit private snapshots.

Baseline and local investigation

First apply records the machine's existing state, not an audited known-safe state. Later applies never accept new drift automatically. Changes remain pending against that baseline until reviewed and explicitly accepted:

sudo systemctl start fleet-osquery-report.service
sudo /opt/fleet-osquery/fleet-osquery report
sudo journalctl -u fleet-osquery-report.service
# Only after reviewing the changes:
sudo /opt/fleet-osquery/fleet-osquery baseline --accept
sudo systemctl start fleet-osquery-report.service

Baseline/report/lock files are root-owned mode 0600 in /var/lib/fleet-osquery (0700). Failed collection does not replace the last successful report or baseline; inspect the failed service and its journal, and treat an aging cached sample accordingly. Manual CLI runs do not save fetched reports unless --output is supplied; Punjab's scheduled aggregate is described above. Keep all report output private: addresses and executable paths are sensitive; do not attach raw JSON to public issues or publish it through fleet_facts. No command lines, process environments, credentials or serial numbers are queried.

Check mode reports platform support only; actual installation and baseline collection require a normal apply. CI exercises the actual role and sudo python3 tests/osquery_smoke.py on x86_64 and aarch64 Ubuntu runners. That smoke creates a temporary loopback listener and an enabled, never-started unit, checks pending/accepted/removed drift, and compares a synthetic prior kernel against the real current kernel. It uses isolated state and removes its fixtures; the fleet baseline is untouched. The native CI jobs also enable the persistent poller and run tests/osquery_poller_smoke.py: a real temporary listener must appear in the aggregate, disappear after removal, and leave the host baseline unchanged.

retired_services

Tags: system, retired_services
Secrets needed: No
Runs on: every systemd host except the vps group and Termux (site.yml gates on vps membership, is_termux and service_mgr)

Deleting a role from the repo stops the fleet from deploying its services, but nothing ever removed what it had already installed — so retired podman quadlets (Prometheus, Grafana, Cockpit, …) kept running on laptops long after their roles were gone (#170). This role removes those leftover unit files.

What It Does

For every unit file in its lists that exists on the machine:

  1. stops the service generated from it — quadlet service names are the file stem (foo.container → foo.service, foo.volume → foo-volume.service, foo.network → foo-network.service); plain .service/.timer files are also disabled
  2. deletes the unit file
  3. runs one daemon-reload (system and/or user, whichever scope had leftovers)

On a machine with none of the files, everything after one find over the unit directories is skipped — the role reports no changes.

Only unit files are removed. Podman volumes (the data of Prometheus, Grafana, NetBox's Postgres, …), images, and config directories such as /etc/cockpit, /etc/homepage, /etc/netbox and ~/.config/monitoring are left alone; delete them by hand once you're sure nothing is wanted (podman volume ls, sudo podman volume ls).

Units

RoleRemoved inScopeUnit files
cockpit2af694asystem (/etc/containers/systemd/)cockpit.container
homepagef0143a6systemhomepage.container
netbox7f3c122systemnetbox.container, netbox-worker.container, netbox-redis.container, netbox-postgres.container, netbox-postgres.volume, netbox-media.volume, netbox.network
netbox_agent7f3c122system (/etc/systemd/system/)netbox-agent.timer, netbox-agent.service
limaf0143a6user (~/.config/containers/systemd/)lima-bootc.container
monitoring8c9b171usernode-exporter, podman-exporter, alloy, promtail, prometheus (also the hub variant), perses, grafana, loki, alertmanager, pve-exporter, matrix-notifier .container; alloy-data, prometheus-data, perses-data, grafana-data, loki-data, alertmanager-data .volume; monitoring.network
appflowyf7f2a2auserappflowy-cloud, appflowy-gotrue, appflowy-s3, appflowy-minio, appflowy-redis, appflowy-db .container; appflowy-s3-data, appflowy-minio-data, appflowy-db-data .volume; appflowy.network
authentikf7f2a2auserauthentik-server, authentik-worker, authentik-redis, authentik-db .container; authentik-db-data, authentik-media, authentik-certs, authentik-custom-templates .volume; authentik.network
n8nf7f2a2ausern8n-server, n8n-db .container; n8n-data, n8n-db-data .volume; n8n.network

caddy.container is not on the list — the proxy role still deploys it.

Variables

VariablePurpose
retired_services_system_unitsAbsolute paths of rootful quadlets / system units to remove
retired_services_user_unitsFile names under ~/.config/containers/systemd/ to remove

Notes

  • When you delete a role that installed units, add them here in the same PR.
  • User-scope systemctl gets XDG_RUNTIME_DIR=/run/user/<uid> pinned so it works from shells that don't export it.
  • No skip_ flag — the role is a no-op wherever the files are absent.

homebrew

Tags: packages, homebrew
Secrets needed: No
Runs on: All glibc machines (skipped on musl/Alpine)

Installs and manages Homebrew packages on Linux.

What It Does

  1. Installs Homebrew to /home/linuxbrew/.linuxbrew if missing
  2. Generates a Brewfile from group_vars/all.yml package lists
  3. Runs brew bundle to install/update all packages
  4. Ensures Python packaging module is available (required by Ansible's systemd module)

Package Categories

VariableWhere DefinedScope
core_brewsgroup_vars/all.ymlAll machines
core_tap_brewsgroup_vars/all.ymlAll machines (tapped)
desktop_brewsgroup_vars/all.ymlDesktop group only

Key Packages

  • Shells: fish, zsh, bash-preexec
  • CLI tools: neovim, tmux, direnv, just, make, ripgrep, fzf, eza, bat, fd, zoxide
  • Bluefin extras: atuin, starship, carapace, tealdeer (via bluefin-cli)
  • Dev tools: go, kubernetes-cli, helm, k9s, podman-compose, ansible, uv
  • AI/ML: ollama, ramalama, openai-whisper, pi-coding-agent

Notes

  • Homebrew on Linux installs to /home/linuxbrew/.linuxbrew — no root needed
  • The playbook adds brew_bin to PATH for all subsequent tasks

flatpak

Tags: packages, desktop, flatpak
Secrets needed: No
Runs on: Desktop group only (skip_flatpak: false)

Installs and manages Flatpak applications from Flathub.

What It Does

  1. Fixes broken Flatpak symlinks if present
  2. Adds the Flathub remote
  3. Removes unwanted system flatpaks (Firefox, Thunderbird — replaced by Zen)
  4. Installs missing flatpaks in batch (fast path), falling back to individual installs
  5. Updates all flatpaks when upgrade: true is set
  6. Creates Chromium wrapper scripts for CLI compatibility

Package List

Defined in roles/flatpak/vars/main.yml as system_flatpaks (the Flathub remote is in flatpak_remotes, also set in group_vars/all.yml). Key apps:

AppFlatpak ID
Zen Browserapp.zen_browser.zen
VS Codecom.visualstudio.code
Ungoogled Chromiumcom.github.Eloston.UngoogledChromium
Flatsealcom.github.tchx84.Flatseal
GNOME appsCalculator, Calendar, Clocks, Contacts, Loupe, etc.
Vesktopdev.vencord.Vesktop
Obsidianmd.obsidian.Obsidian

Wrapper Scripts

Creates ~/.local/bin/chromium, ~/.local/bin/google-chrome, and ~/.local/bin/chromium-browser that exec into the Ungoogled Chromium flatpak — tools that hardcode google-chrome or chromium-browser still work.

Notes

  • Installs system-wide (--system) with become: true
  • Batch install tolerates partial failures — individual retries catch stragglers

termux_packages

Tags: system, packages, termux
Secrets needed: No
Runs on: termux_hosts group only (the raw Termux/Android layer on the phone)

Package and sshd management for Termux, which has no root, no systemd, no Homebrew and bionic libc.

What It Does

  1. pkg update -y (retried 3×), then pkg install -y the termux_pkgs list
  2. npm install -g the termux_npm_global list (the pi coding agent)
  3. Deploys ~/.termux/boot/start-sshd (termux-wake-lock + sshd) for Termux:Boot
  4. Starts sshd if it isn't running
  5. Schedules the same script as a persisted periodic job via termux-job-scheduler (job id 1), so sshd comes back after Android kills Termux

Configuration

roles/termux_packages/defaults/main.yml:

  • termux_pkgs — openssh, git, python, nodejs, fish, vim, nano, termux-api, curl, unzip, plus file and e2fsprogs (Ansible's stat/copy modules need them)
  • termux_npm_global — @earendil-works/pi-coding-agent
  • termux_keepalive_period_ms — 1800000 (30 min; Android's floor is 15)

Notes

  • Requires the Termux:Boot and Termux:API companion apps, installed separately (F-Droid/GitHub)
  • Claude Code, Codex etc. ship no android-arm64 binary — they live on the mumbai phone VM instead
  • Connection quirks (remote tmp, scp, bash as shell executable) are pinned in host_vars/termux.yml

shell_fonts

Tags: dotfiles, shell, fonts
Secrets needed: No
Runs on: All machines (font install only when is_desktop)

Installs the JetBrains Mono Nerd Font for terminals and the prompt.

What It Does

  1. Ensures ~/.local/share/fonts/ exists (every host)
  2. On desktops, if JetBrainsMono/JetBrainsMonoNerdFont-Regular.ttf is missing: downloads the latest JetBrainsMono.zip release, unpacks it into ~/.local/share/fonts/JetBrainsMono/, and runs fc-cache -f

Notes

  • Idempotent on the presence of the Regular .ttf — delete it to force a re-download
  • The gnome role sets Ptyxis to JetBrainsMono Nerd Font 12, which depends on this

shell_dotfiles

Tags: dotfiles, shell
Secrets needed: No
Runs on: All machines

Deploys the shell, prompt, multiplexer and terminal configs.

What It Does

  1. ~/.config/shell/aliases.sh (bash/zsh) and ~/.config/fish/conf.d/aliases.fish — shared aliases, templated
  2. ~/.bashrc, ~/.zshrc, ~/.zprofile — templated
  3. ~/.config/fish/config.fish, ~/.tmux.conf, ~/.inputrc, ~/.config/starship.toml — copied as-is
  4. Ghostty terminfo: ships xterm-ghostty.terminfo and compiles it into ~/.terminfo with tic, so SSH sessions from Ghostty don't break ncurses apps. Warns and skips if tic isn't installed.
  5. ~/.config/ghostty/config — auto light/dark following GNOME (inert on headless hosts)

Fleet aliases

AliasExpands to
dotscd ~/.local/share/dotfiles && just apply-nosecrets (git pull + --skip-tags secrets)
dots-applycd ~/.local/share/dotfiles && git pull --ff-only && just apply (full, with BW unlock)
dots-editopen the repo in $EDITOR

In fish these are abbreviations (abbr), not aliases.

Notes

  • Shell secrets (~/.config/shell/secrets.sh, ~/.config/fish/conf.d/secrets.fish) are written by shell_ai, not here
  • Source files: roles/shell_dotfiles/files/ and roles/shell_dotfiles/templates/

shell_atuin

Tags: dotfiles, shell, atuin, secrets
Secrets needed: Yes, for login (atuin.sh from Bitwarden)
Runs on: All machines

Configures Atuin shell history and logs it in to sync.

What It Does

  1. Writes ~/.config/atuin/config.toml — sync to api.atuin.sh every 5m, fuzzy search, filter_mode = "global" (including the Up-arrow binding)
  2. Checks atuin status; if already logged in, skips the Bitwarden round-trip
  3. Otherwise, when secrets_available, reads the atuin.sh item (username, password, and a custom field key holding the mnemonic) and runs atuin login -u … -p … -k …
  4. Runs a best-effort atuin sync
  5. Imports existing shell history once (atuin import auto, sentinel ~/.local/share/atuin/.history_imported)

Notes

  • Skips login with a warning if the atuin binary is missing (it's in core_brews)
  • The mnemonic is passed to atuin login -k, never written to ~/.local/share/atuin/key directly — atuin expects that file to be a binary key
  • Don't set the Up-arrow filter to session: it looks broken in fresh terminals

shell_ai

Tags: dotfiles, shell, ai, secrets
Secrets needed: Yes (deepseek-api-key, forgejo, tavily-api-key)
Runs on: All machines

Deploys API credentials for the AI coding tools and shell.

What It Does

  1. When secrets_available:
    • bw get password deepseek-api-key → DEEPSEEK_API_KEY in shell secret files
    • bw get notes forgejo (the API Token: line) → FORGEJO_TOKEN
    • bw get password tavily-api-key → TAVILY_API_KEY
    • Writes these exports to ~/.config/shell/secrets.sh (bash/zsh) and ~/.config/fish/conf.d/secrets.fish (fish), both 0600
  2. Warns if deepseek-api-key is missing; Kiro OAuth remains a per-user OMP sign-in

Notes

  • A locked vault leaves any previously written secret files untouched — nothing is overwritten with empty values
  • TAVILY_API_KEY is read by the tavily-search skill deployed by the omp role

omp

Tags: dotfiles, omp
Secrets needed: No for OMP/Kiro installation; shell_ai optionally provides the DeepSeek API key
Runs on: All supported hosts except raw Termux

Installs Oh My Pi and the Kiro provider on each supported computer. Homebrew hosts install OMP and Bun from the can1357/tap tap. musl hosts use OMP's checksum-verified binary installer and Bun's installer. The Kiro provider is installed from github:elkibalion/omp-kiro-provider.

What It Does

  1. Installs and verifies omp.
  2. Installs the Kiro provider plugin for the current user; it is available as kiro/* models.
  3. Deploys personal skills to ~/.omp/agent/skills/james/ and symlinks them into ~/.claude/skills/.
  4. Exposes the Bitwarden DeepSeek key as DEEPSEEK_API_KEY through the shell secret files, not an agent-specific credential file.
  5. Removes the legacy Pi CLI from raw Termux. OMP has no native Android/Bionic build; the Debian VM (mumbai) is supported normally.

Skills and the package manifest

The playbook copies roles/pi/files/skills/ to ~/.omp/agent/skills/james/ and links those skill directories into ~/.claude/skills/. The roles/pi path is skill source data, not a separate Ansible role.

roles/pi/files/package.json is tracked by Renovate and protected by deps-guard; it is not installed by this role. OMP plugin installation is handled separately by the role's explicit plugin tasks.

Kiro authentication

Plugin installation is unattended. OAuth is per user and must be completed on each computer:

omp
/login kiro

Complete the browser sign-in. Then select kiro/auto with /model kiro/auto. The playbook does not set Kiro as the default model because the OAuth session is not present until a user signs in.

Package source

The OMP and Bun Homebrew formulas are in can1357/tap; both are core packages for Homebrew-supported hosts. On musl, the official OMP binary installer verifies its SHA-256 and requires libgcc and libstdc++. Raw Termux receives neither OMP nor the obsolete Pi npm package.

hive_ops

Tags: hive, hive_ops
Default: hive_ops_enabled: false
Credentials: no Bitwarden lookup in this role; its Kubernetes kubeconfig contains credentials.

Retired AWS-cluster integration, not the current Hive operations guide. The AWS Talos cluster was terminated on 2026-10-25. The current school and reef spokes and hub run as Docker Compose services on the bihar (karnataka standby). Do not enable this role or run its old Kubernetes helpers against that deployment.

The role still exists in the repository. When enabled it installs host user units and scripts that use kubectl against ~/.kube/config-aws-migration by default; they do not manage the Compose services. Historically these units ran on telengana against its local k3s, then on himachal against AWS. The cluster CronJobs and operator controllers that later replaced parts of this work are also historical: the AWS cluster no longer exists.

Observed and configured state

host_vars/himachal.yml sets hive_ops_enabled: false. On 2026-10-27, read-only SSH to james@himachal-1 found all seven role timer units not-found and inactive. This observation is not a fleet-wide timer audit; a host-vars value alone does not establish a host's live state.

When disabled, the role's cleanup runs only if ~/.config/systemd/user/hive-watchdog.timer exists. In that case it attempts to stop/disable the seven timers and remove their units, drop-ins and helper scripts. If the sentinel is absent, the cleanup block does not run; do not assume that it removed every possible leftover. The role was not applied as part of this documentation update.

Historical unit inventory

The checked-in unit files describe the old DeepSeek-era setup, not the current claude-backend configuration on Spindle. Their schedules are:

TimerChecked-in scheduleHistorical purpose
hive-rotate10 min after boot, then 20 min after activationRotate backends based on provider headroom
hive-watchdog3 min after boot, then every 5 minProbe and heal agents
hive-tiersMonday 04:30 UTC, up to 30 min random delayRefresh model tiers
hive-peak-pause / hive-peak-resumeWeekdays 01:00 and 06:00 / 04:00 and 10:00 UTCPause/resume around DeepSeek peak windows
pi-peak-stop / pi-peak-startWeekdays 01:00 and 06:00 / 04:00 and 10:00 UTCStop/start local pi.service around those windows

When enabled, the role installs hive-peak.sh, hive-rotate.sh, and hive-tiers.sh and writes service drop-ins for KUBECONFIG and PATH. The drop-ins made Kubernetes and Homebrew tools available to the systemd user manager; they do not make the legacy scripts suitable for the VPS.

Variables retained in the role

VariableDefaultHistorical purpose
hive_ops_enabledfalseOpt a host in; multiple operators could conflict
hive_ops_kubeconfig~/.kube/config-aws-migrationAWS cluster credentials
hive_ops_pathHomebrew plus system pathsFind kubectl and jq in user services
hive_ops_timersSeven timers aboveUnits deployed or cleaned up

git

Tags: dotfiles, git
Secrets needed: No
Runs on: All machines

Deploys Git configuration with SSH commit signing.

What It Does

  1. Deploys ~/.gitconfig with name, email, and signing preferences
  2. Configures SSH commit signing using ~/.ssh/id_ed25519
  3. Deploys ~/.ssh/allowed_signers for signature verification
  4. Deploys GitHub CLI config at ~/.config/gh/config.yml

Key Settings

  • Signing: SSH (ssh-ed25519 key) for all commits
  • Allowed signers: All fleet machine keys from Bitwarden
  • Delta: Git diff viewer (git-delta) for side-by-side diffs

neovim

Tags: dotfiles, neovim, editor
Secrets needed: No
Runs on: All machines

Deploys Neovim configuration.

What It Does

  1. Creates ~/.config/nvim/ directory structure
  2. Deploys init.lua and plugin configuration
  3. Installs Lazy.nvim plugin manager and configured plugins

Key Plugins

  • LSP: nvim-lspconfig, mason.nvim (language server management)
  • Completion: nvim-cmp with LuaSnip snippets
  • UI: Catppuccin theme, lualine statusline, nvim-tree file explorer
  • Editing: treesitter, comment.nvim, autopairs
  • Git: gitsigns (inline git blame/diff)
  • Navigation: telescope.nvim (fuzzy finder)

bitwarden

Tags: secrets, bitwarden Secrets needed: Yes (provides secrets to other roles) Runs on: All machines

Resolves a Bitwarden session, refreshes the local vault cache, and publishes a single secrets_available host fact that every other BW-using role consumes. Failure to unlock is never fatal to the play; it's reported once at the end and downstream BW operations are skipped cleanly.

What it does

  1. Resolves BW_SESSION in this order: environment → /tmp/bw_session cache → interactive bw unlock.
  2. Runs bw sync with a hard 15 s timeout (so a flaky vault server can't hang the whole play).
  3. Calls bw status and sets secrets_available (boolean host fact) — true only if bw is installed, a session was resolved, and the vault actually reports unlocked.
  4. If a session is set but the vault is still locked, emits a one-line debug warning so the operator knows secrets work is being skipped here. On a remote apply, this means target-side session resolution did not unlock the target vault.

Every BW-using role downstream gates on when: secrets_available | default(false). The end-of-play post_tasks summary then prints one of three messages:

StateMessage
unlockedvault unlocked — all secret roles executed normally
lockedBitwarden vault was LOCKED — … skipped. To enable: bw unlock …
--skip-tags secretsSKIPPED (--skip-tags secrets). Run dots-apply to include …

Remote applies

just apply-remote <host> (and just apply-remote-tags <host> <tags>) resolve the Bitwarden session on the target host; they do not forward the local host's BW_SESSION:

  1. scripts/bw-resolve.sh remote <host> unlocks Bitwarden on the target and prints that host's session token. If the remote CLI is logged out, it tries the configured API-key and master-password paths, then falls back to an interactive unlock over ssh -t.
  2. The recipe exports the returned BW_SESSION inline in the remote command and caches it in the target's /tmp/bw_session.
  3. The remote bitwarden role runs bw status; when it reports unlocked, secrets_available becomes true and secrets work proceeds.

A session token is valid only on the host that unlocked it. If target-side resolution fails, the remote vault remains locked and BW-using tasks skip cleanly. See the full remote apply flow.

How to verify it ran correctly

  • just doctor will show vault unlocked if BW is good on this host.
  • Check the end-of-play summary line printed by every just apply*.
  • Look for any task in ansible-playbook output with a [bitwarden : …] prefix that ended in failed: — should never happen; only "skipping" is acceptable.

Notes

  • Must run before any other secrets role (it does, by position in site.yml).
  • The cached /tmp/bw_session persists for the user across reboots only as long as /tmp survives (most modern systemd boots clear it). The role re-resolves the session on every play.
  • bw sync failure is non-fatal; downstream operations use the locally cached vault, which may be stale.

ssh_keys

Tags: secrets, ssh, ssh_keys Secrets needed: Yes Runs on: All machines

Manages per-machine ed25519 SSH keys, distributes the fleet's public keys via authorized_keys, and maintains the allowed_signers list for git commit verification.

Source of truth is the local disk. Bitwarden is the durable mirror that lets new/rebuilt machines recover and lets the rest of the fleet stay in sync.

What it does

For the local machine's key (james@<hostname> in Bitwarden):

  1. No key on disk → fetch from BW; if absent there too, ssh-keygen a fresh ed25519 pair.
  2. Local pub key matches BW → no-op.
  3. Local pub key differs from BW → leave the Bitwarden item unchanged and warn by default. An intentional rotation must opt in with dots-apply -e ssh_keys_force_push=true; only then is the BW item replaced so every other host sees the current key on its next apply. This gate prevents a key generated while the vault was locked from clobbering the stored key.

Then, regardless of whether the local item changed:

  • Slurps every other james@<machine> SSH item from BW in a single bw list call.
  • Rewrites ~/.ssh/authorized_keys with the union of those + extra_authorized_keys.
  • Refreshes ~/.ssh/known_hosts for every fleet host via ssh-keyscan, removing stale entries first.
  • Updates ~/.ssh/allowed_signers with this host's pub key for git commit signing.

If secrets_available is false (vault locked, BW not installed, etc.), every BW-touching step is skipped and authorized_keys is not rewritten — preserving the existing file rather than wiping it with an empty list.

Failure modes

SymptomLikely causeFix
Permission denied (publickey) from a host that worked yesterdayLocal key changed but BW still has the old pub keyFor an intentional rotation, run dots-apply -e ssh_keys_force_push=true on the source host with BW unlocked, then run dots-apply on the failing host. If the mismatch was accidental, restore the Bitwarden key locally instead of forcing it.
Some hosts missing from authorized_keysThose hosts never round-tripped their key to BWRun dots-apply on each missing host once, with BW unlocked.
New ed25519 key generated on a host that should have inherited an existing keyBW item missing/empty for this hostnameVerify the BW entry name matches james@<hostname> exactly.

How to verify

# Check this host's key is in BW
bw get item "james@$(hostname)" | jq '.sshKey.publicKey' | grep -F "$(awk '{print $2}' ~/.ssh/id_ed25519.pub)"

# Check who can SSH in here
cat ~/.ssh/authorized_keys

# Test SSH to every fleet host
for h in $(yq '.all.hosts | keys | join(" ")' inventory.yml); do
  ssh -o BatchMode=yes -o ConnectTimeout=3 "$h" "echo OK" 2>&1 | head -1
done

Notes

  • Every machine SSHes to every other machine without password prompts after one round-trip apply.
  • The same key is used for GitHub auth + commit signing (registered by the github role).
  • The role tolerates a locked vault and never destructively rewrites authorized_keys when BW returned no data.

ssh_mesh

Self-healing passwordless SSH mesh across the fleet.

What it does

  1. Ensures the machine has an ed25519 keypair (generates if missing).
  2. Merges all fleet public keys from machine_pubkeys (in group_vars/all.yml) into ~/.ssh/authorized_keys — non-destructive lineinfile, so Bitwarden-managed keys and extras survive.
  3. Warns if this machine's own public key is missing or stale in git.
  4. Probes passwordless SSH to every fleet machine currently online on the tailnet: scripts/online_hosts.py --peers (the same matcher as just apply-all) gives the online MagicDNS names, each inventory host is matched by its tailnet_name (or its own name), then ∩ machines − ssh_mesh_exclude − the vps group. Port and login user come from inventory.yml (ansible_port, ansible_user).
  5. On failure: reaps stale ControlMaster sockets, refreshes the host key in known_hosts (rebuilt machines), and retries.
  6. Fails the apply if the mesh is still broken, naming the unreachable hosts.

Why it's self-healing

  • Public keys live in git, not only in Bitwarden — safe to publish, and every apply replants them. A machine missing a peer's key heals on its next just apply or the daily dotfiles-update.timer (apply-nosecrets), no vault unlock needed.
  • The role is tagged [ssh, mesh_check] but not secrets, so the daily no-secrets timer runs it.
  • Probes bypass SSH multiplexing (ControlMaster=no) so a hung mux socket can't fake an outage, and are capped with timeout.

Usage

just mesh-check                        # verify + heal from this machine
just apply-remote-tags HOST mesh_check # verify + heal from HOST

Adding a machine to the mesh

Add its public key to machine_pubkeys in group_vars/all.yml and its entry to the machines dict. If its MagicDNS name differs from its inventory name, set tailnet_name on it in inventory.yml. Machines whose sshd isn't always reachable (phones, port-forwarded test VMs) go in ssh_mesh_exclude: a failed probe fails the apply.

Relationship to ssh_keys

ssh_keys (tag secrets) still round-trips private keys through Bitwarden and rebuilds authorized_keys from the vault when unlocked. ssh_mesh is the vault-free safety net + verification layer on top.

github

Tags: secrets, github Secrets needed: Yes (only on first auth — once gh is authenticated, BW is not required) Runs on: All machines with gh installed

Logs gh in via a BW-stored PAT (only if not already authenticated), then registers this host's SSH key with GitHub for both auth and signing.

What it does

  1. Ensures github.com's ed25519 host key is in ~/.ssh/known_hosts (prevents MITM warnings on first push).
  2. gh auth status — if already authenticated, skip the token fetch entirely (no BW dependency for steady-state).
  3. If not authenticated and BW is unlocked → bw get password github-token → gh auth login --with-token. If the BW item is missing, emits a one-line warning and continues.
  4. Sets gh config set git_protocol ssh.
  5. Lists existing keys on the GitHub account. For each key type (auth + signing), runs gh ssh-key add if this host's key isn't already there.

Failure modes

SymptomCauseFix
gh auth status says not logged in after apply, vault lockedFirst run on this host with BW lockedUnlock BW and re-run just apply-tags github, or gh auth login manually
Warning: No 'github-token' item foundBW vault doesn't have the PATCreate a BW login item named github-token with your PAT as the password
Key registration silently failsgh ssh-key add returned non-zero (failed_when:false) — possibly key was just deleted on github.com sideInspect with gh ssh-key list and re-run the role
Stale keys clutter github.comThis host's previous key was rotated; the old gh entry stays foreverDelete the old key manually at https://github.com/settings/keys — no automated cleanup yet

How to verify

gh auth status                  # Should say "Logged in" with this account
gh ssh-key list | grep "$(hostname)"   # Should show <hostname>-auth and <hostname>-signing
git log --show-signature -1     # Latest commit should show "Good \"git\" signature for ..."

Notes

  • Depends on the ssh_keys role running first (the key must exist on disk).
  • The signing key registration enables the "Verified" badge on commits — only if your local ~/.gitconfig is also set to sign with the SSH key (the git role handles this).
  • The PAT in BW needs admin:public_key, write:gpg_key, and read:user scopes for the registrations to succeed.

tailscale

Tags: secrets, tailscale Secrets needed: Yes (only to join — once connected, BW is not required) Runs on: All machines

Installs Tailscale, fetches a reusable auth key from BW, joins the network, configures DNS, and keeps device hostname + advertised routes in sync.

What it does

  1. Configures systemd-resolved so MagicDNS works (link /etc/resolv.conf to the stub resolver, drop in DNSStubListenerExtra=::1 for container DNS).
  2. Sets net.ipv4.conf.{all,default}.rp_filter=2 and IP forwarding on, so exit-node and subnet-router traffic actually routes.
  3. If tailscale status reports connected → updates hostname, prefs, operator, systray, then exits the role.
  4. If not connected and the vault is unlocked → fetches tailscale-apikey from BW, deletes any stale device with this hostname via the Tailscale API, then tailscale up --authkey ….
  5. If not connected and the vault is locked → emits a one-line warning and ends the role. Joining without a key would either fail noisily or, worse, succeed with an empty key.

Failure modes

SymptomCauseFix
Role aborts with "vault locked" warning, host is not on tailnetFirst-time apply on this host without BW unlockedUnlock BW (`bw unlock --raw
Tailscale connected but MagicDNS names don't resolvesystemd-resolved link to /etc/resolv.conf was overwritten by NetworkManager / DHCP clientThe role re-applies it; just re-run dots-apply. To make it sticky, check network-manager settings (Bluefin does this correctly out of the box)
Duplicate device entries in admin console after rebuildAPI-key device cleanup didn't run (vault locked at the time)Manually delete in tailscale admin console or unlock vault and re-apply
Joins but no routes advertisedtailscale_advertise_routes not set in host_varsAdd it: tailscale_advertise_routes: "<lan-cidr>"

How to verify

tailscale status                # should show all fleet machines as peers
tailscale ip -4                 # this host's 100.x.x.x
host bihar.manatee-basking.ts.net   # MagicDNS resolves to a 100.x.x.x

Notes

  • All fleet machines are on the manatee-basking.ts.net tailnet.
  • The auth key is reusable — new machines join without re-authentication.
  • The Tailscale API key (tailscale-apikey in BW) is the same item used for joining and for device cleanup; rotate by editing the BW item and re-running this role on each host.

kube

Tags: secrets, kube Secrets needed: Yes Runs on: Desktop group only (unless skip_kube: true)

Pulls kube + Talos configs from BW for the home cluster (Bihar + Karnataka — powered down, expected back) and the historical AWS cluster, terminated 2026-10-25. Neither cluster is currently reachable.

What it does

When secrets_available is true:

  1. Ensures ~/.kube/ and ~/.talos/ exist (mode 0700).
  2. bw get notes kubeconfig → writes to ~/.kube/config (0600). Skipped if BW returns empty.
  3. bw get notes talosconfig → writes to ~/.talos/config (0600). Skipped if BW returns empty.
  4. Pins tls-server-name on every cluster in ~/.kube/config to kube_tls_server_name (default: bihar's LAN address) — the home API-server cert lacks the tailnet address in its SANs.
  5. Emits a one-line warning if either home-cluster note is missing in BW.
  6. bw get notes kubeconfig-aws-migration → ~/.kube/config-aws-migration and bw get notes talosconfig-aws-migration → ~/.talos/config-aws-migration (0600, each skipped if empty). These historical AWS files remain separate from the home configs.

When the vault is locked the role no-ops cleanly — existing local configs are preserved.

Failure modes

SymptomCauseFix
kubectl returns Unable to connect to the server after onboardingVault was locked, configs never deployedUnlock BW, re-run just apply-tags kube
Warning: kubeconfig missing in BWVault was never seededRun just seed-kube from a host that already has a working config
Configs deployed but auth failsCluster PKI rotated — local configs are staleRe-run just seed-kube from a healthy host, then re-apply on consumers
Role skipped silently on a server host that should have itskip_kube: true in host_vars/<host>.yml, or host not in desktop groupAdjust inventory or host_vars

Seeding BW

# From a machine that already has working configs:
just seed-kube

This packages ~/.kube/config and ~/.talos/config into BW notes named kubeconfig and talosconfig. It does not seed the historical *-aws-migration notes. After seeding, every desktop's next dots-apply (or just apply-tags kube) pulls the home configs down.

How to verify

Both commands below require the home cluster to be powered on; they fail while it remains in storage. The former AWS cluster cannot be queried.

kubectl get nodes               # home cluster: bihar + karnataka (fails while it's powered down)
talosctl --talosconfig ~/.talos/config get members

Notes

  • These files contain cluster PKI material; the repo's .gitignore excludes the cluster source files, and these caches live in ~/ only — never commit.
  • The role is gated by is_desktop | bool and not skip_kube. Servers don't get a kubeconfig — the cluster nodes themselves are Talos and not in the inventory.

forgejo_registry

Tags: secrets, forgejo, registry
Secrets needed: Yes (Login item named after the registry host)
Runs on: All machines except machine_profile: vm-test (unless skip_forgejo: true)

Logs podman in to the private Forgejo container registry.

What It Does

When secrets_available and podman is on PATH:

  1. Checks podman login --get-login {{ forgejo_registry }} — skips if already logged in
  2. Fetches bw get username / bw get password for the item named {{ forgejo_registry }}
  3. Runs podman login … --password-stdin
  4. Warns if the Bitwarden item is missing

Configuration

# roles/forgejo_registry/defaults/main.yml
forgejo_registry: forgejo.manatee-basking.ts.net

The Bitwarden Login item's name must equal this value; its password is a registry/package token.

Notes

  • Forgejo runs on the home Talos cluster, which is currently powered down — a fresh podman login can't succeed until it returns. Hosts already logged in skip the step.

gnome

Tags: desktop, gnome
Secrets needed: No
Runs on: Desktop group only

Configures GNOME desktop environment: keybindings, terminal, avatar, extensions, and remote desktop.

What It Does

Keybindings

Deploys custom keyboard shortcuts via dconf:

  • Custom keybindings for terminal, browser, file manager
  • Workspace navigation and window management shortcuts

Ptyxis Terminal

Sets the default Ptyxis profile to use fish from Homebrew with JetBrains Mono Nerd Font at 12pt.

User Avatar

  • Downloads avatar from https://reilly.asia/profile.png
  • Installs to both AccountsService and GDM cache
  • Shows on login screen and user menu

GNOME Extensions

Deploys and enables extensions from group_vars/all.yml:

ExtensionPurpose
CaffeineDisable screen blanking
AppIndicatorSystem tray icons
Blur My ShellBlur effects
Dash to DockmacOS-style dock
GSConnectPhone integration
PaperWMTiling window manager
PaperShellPaperWM companion
Search LightSpotlight-style search

Remote Desktop

Enables GNOME Remote Desktop (RDP) via grdctl and starts the systemd user service.

Extension Sync

Extensions are synced omnidirectionally — the gnome_enabled_extensions list in group_vars/all.yml is the canonical list. To update it from a running desktop:

gsettings get org.gnome.shell enabled-extensions

zen_browser

Tags: desktop, browser, zen_browser
Secrets needed: No
Runs on: Desktop group only

Configures Zen Browser (Firefox-based) as the default browser.

What It Does

  1. Pins Zen Browser to the GNOME dock
  2. Removes Firefox and Thunderbird from the dock if present
  3. Sets Zen as the default web browser

Notes

  • Zen is installed via Flatpak (app.zen_browser.zen)
  • Firefox and Thunderbird are intentionally removed from the dock — Zen replaces both

browser_fxa

Tags: none of its own — pulled in by zen_browser via include_role, so it runs under desktop, browser, zen_browser
Secrets needed: Yes (Firefox Account credentials from Bitwarden)
Runs on: Desktop group only

Automates Firefox Account sign-in for Zen Browser (and optionally Firefox).

What It Does

  1. Fetches accounts.firefox.com credentials + TOTP from Bitwarden
  2. Writes signedInUser.json to the browser profile directory
  3. This pre-authenticates the browser with Firefox Sync

Configuration

# group_vars/all.yml
fxa_bw_item: "accounts.firefox.com"
fxa_bw_username: "<your-fxa-email>"
fxa_browsers:
  - zen
  - firefox
  - firefox-flatpak

Notes

  • This only enables FxA sign-in, not the full Sync key bundle
  • Opening the browser once after deployment may still be needed to complete Sync
  • If credentials are unavailable, the task fails silently — manual sign-in works fine

bluefin_common

Tags: desktop, bluefin
Secrets needed: No
Runs on: Desktop group only (skipped on Bluefin hosts unless force_bluefin: true)

Syncs Bluefin branding, fonts, wallpapers, and GNOME extensions from the Project Bluefin OCI image.

What It Does

  1. Pulls the ghcr.io/projectbluefin/common:latest container image
  2. Extracts Bluefin gsettings/dconf overrides, backgrounds, icons, and logos
  3. Installs branding assets to ~/.local/share/
  4. Fixes XML background paths to point to home directory
  5. Applies dconf settings via a Python script
  6. Installs Bluefin-standard fonts (Inter, JetBrains Mono)
  7. Installs Bluefin-recommended GNOME extensions
  8. Deploys a wallpaper rotation service and timer
  9. Deploys the corral (Go) VM utility

Fonts

Wallpaper Rotation

A systemd timer (bluefin-wallpaper-rotate.timer) cycles through Bluefin backgrounds daily. The Python script applies the wallpaper via D-Bus.

Corral

Deploys corral, the Go VM manager (CLI + TUI + web). Corral is downloaded from the rolling binaries release of hanthor/corral (checksum-verified, so it updates whenever CI publishes a new build). It is no longer embedded in this repo.

Notes

  • Skips by default on actual Bluefin hosts (they already have this)
  • Set force_bluefin: true to apply on Bluefin (e.g., after an image reset)
  • Set skip_bluefin: true in host_vars to skip entirely

pipewire_audio

Tags: desktop, audio, pipewire
Secrets needed: No
Runs on: Desktop group only (unless skip_pipewire_audio: true)

Replaces Easy Effects with native PipeWire filter-chain modules using LSP LV2 and LADSPA plugins — no Flatpak, no GUI, no D-Bus service.

What It Does

  1. Deploys speaker-dsp.conf — a PipeWire filter-chain that creates a virtual "Speakers (DSP)" sink
  2. Deploys mic-dsp.conf — a PipeWire filter-chain that creates a virtual "Microphone (DSP)" source
  3. Configures WirePlumber to prefer the DSP nodes as default devices
  4. Cleans up any previous Easy Effects installation (Flatpak, autostart, presets)

Internal task boundaries

The role keeps one public pipewire tag and the skip_pipewire_audio gate, but the entrypoint delegates to focused task files in this order:

  1. tasks/platform.yml detects the host platform, installs mutable-system prerequisites, and publishes plugin availability facts.
  2. tasks/dsp.yml deploys the speaker and microphone filter-chain configs.
  3. tasks/systemd.yml configures the LADSPA path and persistent microphone routing service.
  4. tasks/cleanup.yml removes obsolete WirePlumber and Easy Effects state.

The split is organizational only: conditions, handlers, paths, and the role's execution order remain unchanged.

Output Chain: Speaker DSP

Signal path: 16-band Parametric EQ → Bass Enhancer → Exciter → Limiter (all LSP LV2).

StageLV2 PluginWhat It Does
eqLSP ParaEQ x16 LRCorrective loudness curve with subsonic HPF @ 120 Hz
bassLSP Bass EnhancerHarmonic bass synthesis for small speakers
exciterLSP ExciterCrystalizer substitute — adds presence and air
limLSP LimiterBrickwall peak limiter @ -0.5 dBFS

The EQ curve is a direct translation of the Easy Effects 15-band preset — ~2.5 dB cut through low-mids (220-600 Hz), ~3.5 dB peak at 3.2 kHz, tapering to ~1.5 dB boost above 8 kHz.

Simplification: To run just EQ + Limiter (the most critical, crash-proof chain), comment out the bass and exciter nodes in speaker-dsp.conf and remove their links.

Input Chain: Microphone DSP

Signal path: RNNoise → Compressor (LADSPA + LSP LV2).

StagePluginWhat It Does
rnnoiselibrnnoise_ladspa.soReal-time neural network noise suppression (VAD-enabled)
compLSP Compressor StereoDownward compression, -24 dB threshold, 2:1 ratio

Bootc Image Requirements

These packages must be in the bootc image (Containerfile):

lsp-plugins-lv2
noise-suppression-for-voice

The filter-chain configs reference:

  • LSP LV2 bundles under /usr/lib64/lv2/ (Fedora default LV2 path)
  • RNNoise LADSPA at /usr/lib64/ladspa/librnnoise_ladspa.so

Verifying

On a target machine after deploy:

# Check DSP nodes appeared
pw-cli ls Node | grep -E 'speaker_dsp|mic_dsp'

# Check defaults
wpctl status | head -20

# Watch for plugin loading errors
journalctl --user -u pipewire -f | grep -iE 'error|filter-chain|lv2'

# Verify LSP plugins are discoverable
lv2ls | grep lsp-plug
lv2info http://lsp-plug.in/plugins/lv2/para_equalizer_x16_lr | head -5

# Verify LADSPA plugin
ls /usr/lib64/ladspa/librnnoise_ladspa.so

Troubleshooting

SymptomLikely CauseFix
DSP node doesn't appearLSP/LADSPA plugins missing from bootc imageVerify with lv2ls, ls /usr/lib64/ladspa/
Port name mismatchLSP plugin version changed port symbolsRun lv2info <uri> | grep 'port.*symbol' and update config
Audio routing loopPipeWire/WirePlumber bugDon't set the DSP sink as default manually; let WirePlumber handle it
No bass enhancer/exciter effectPlugin UID mismatchComment them out in config for a leaner chain

Notes

  • PipeWire must be restarted after deploying configs (handled by Ansible handlers)
  • Select "Speakers (DSP)" and "Microphone (DSP)" in GNOME Sound settings if WirePlumber doesn't auto-select them
  • The virtual sink approach works because PipeWire prevents self-looping when capture/playback share the same node.name
  • To skip: set skip_pipewire_audio: true in host_vars, or exclude the pipewire tag

syncthing

Tags: services, syncthing
Secrets needed: No
Runs on: All except VPS

Deploys Syncthing as a systemd user service for continuous file synchronization.

What It Does

  1. Creates systemd user directory
  2. Deploys syncthing.service user unit
  3. Enables and starts the service

Notes

  • Runs as the user's systemd service (not system-wide)
  • Web UI available at http://localhost:8384
  • Skipped on VPS hosts (no file sync needed)

systemd

Tags: services, systemd
Secrets needed: No
Runs on: All machines

Deploys systemd user services and timers for background tasks.

Services Deployed

dotfiles-update

Keeps the dotfiles repo up to date automatically.

  • Desktops/servers: Daily timer (dotfiles-update.timer)
  • Laptops: Runs on login via graphical-session.target (no timer)

atuin-daemon

Keeps Atuin shell history synced in the background.

Podman socket (conditional)

If podman is installed:

  • Creates the podman group
  • Adds the user to the group
  • Configures the podman socket for group access (mode 0660)
  • Enables and starts podman.socket

system-mitigator (himachal only)

A Python daemon that monitors for Lenovo EC firmware bugs:

  • Detects false BD_PROCHOT signals that lock CPU cores to 400 MHz
  • Watches for systemd-journald watchdog loops
  • Sends desktop notifications when issues are detected

Notes

  • The dotfiles-update.timer on laptops is removed during the laptop-specific path
  • All services run as user-level systemd units (no root needed)

proxy (Caddy)

Tags: services, proxy, caddy
Secrets needed: Yes (TLS certificates)
Runs on: All except VPS (skip_proxy: false)

Deploys Caddy as a reverse proxy with automatic TLS certificates.

What It Does

  1. Creates Caddy config directory
  2. Deploys Caddyfile from template with per-host service routes
  3. Proxies internal services to *.manatee-basking.ts.net subdomains
  4. Uses Tailscale certificates for TLS

Proxied Services

The Caddyfile (listening on :9000, fronted by tailscale serve) carries:

  • /service/<name> — a redirect for each entry in the host's web_services
  • /bst/ — the BuildStream dashboard (on build machines)

There is no homepage at /.

Notes

  • Skip with skip_proxy: true in host_vars
  • Caddyfile is templated per-host — each machine proxies its own services

tailscale_cert

Tags: services, tailscale_cert
Secrets needed: No
Runs on: All except VPS

Fetches and deploys Tailscale TLS certificates for HTTPS on *.manatee-basking.ts.net.

What It Does

  1. Creates certificate storage directory
  2. Fetches TLS certificates from Tailscale for the machine's MagicDNS name
  3. Installs certificates where Caddy and other services can use them

Notes

  • Tailscale provides free TLS certificates for nodes on your tailnet
  • Certificates auto-renew — no Let's Encrypt needed for internal services
  • The Caddy proxy uses these certificates for all *.manatee-basking.ts.net services

bst_dashboard

Tags: services, bst_dashboard
Secrets needed: No
Runs on: Machines with build infrastructure

Deploys a BuildStream build monitoring dashboard.

What It Does

  1. Creates dashboard directory based on hostname
  2. Deploys dashboard web UI files
  3. Configures dashboard to track local BuildStream build queues

Access

Proxied through Caddy at https://<host>.manatee-basking.ts.net/bst/.

Notes

  • Used to monitor BuildStream build jobs on machines that run builds
  • Dashboard shows build queue status, job history, and artifact cache

kirocrew

Keeps the Kiro Crew gateway (dashboard + Slack + cron) healthy on hosts that run it — currently punjab.

  • Enable: kirocrew_enabled: true in host_vars/<host>.yml
  • Tags: services, kirocrew

The venv (~/.kiro/crew-venv) and the systemd unit are owned by Kiro's own installer (kirocrew service install), so the role does not template them. It:

  1. warns if the binary or unit is missing (install per upstream first),
  2. keeps /etc/kirocrew/kirocrew.env root:root 0600 — tokens go there,
  3. enables and starts kirocrew.service,
  4. fails the play if the dashboard listens on anything but localhost.

appflowy

Migrated to Kubernetes (2026-06-12).

This Ansible role (Podman Quadlets on bihar) has been removed. AppFlowy Cloud now runs in the Talos K8s cluster.

See talos-k8s/appflowy/ for the commercial Helm chart deployment manifest and operational instructions.

Access: https://appflowy.manatee-basking.ts.net

authentik

Migrated to Kubernetes (2026-06-12).

This Ansible role (Podman Quadlets on bihar) has been removed. Authentik now runs in the Talos K8s cluster.

See talos-k8s/authentik/ for the Helm-based deployment manifest and operational instructions.

Access: https://auth.manatee-basking.ts.net

n8n

Migrated to Kubernetes (2026-06-12).

This Ansible role (Podman Quadlet on bihar) has been removed. n8n now runs in the Talos K8s cluster.

See talos-k8s/n8n/ for the plain-K8s deployment manifest and operational instructions.

Access: https://n8n.manatee-basking.ts.net

Talos Kubernetes Cluster

ARCHIVED (2026-11-18). This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See Fleet status.

Two-node Kubernetes cluster running on Talos Linux, managed from the talos-k8s/ directory.

Powered down and in storage, expected back — commands below fail until it returns. Matrix and School, Reef and Hub run on the OCI Spindle VPS; the AWS Talos cluster was terminated on 2026-10-25.

Hardware

NodeRoleHardware
biharControl planeIntel (bare metal)
karnatakaWorkerAMD Strix Halo APU (GPU-enabled)

Software Stack

ComponentVersionNotes
Talos Linuxv1.13.2Immutable, API-driven OS
Kubernetesv1.36.1Via Talos
CNIflannelSimple overlay network
Tailscale Operator—Ingress via *.manatee-basking.ts.net
KubeVirtv1.8.2VM workloads on K8s

Directory structure

talos-k8s/
├── ai/              lemonade
├── appflowy/        appflowy cloud (Helm)
├── authentik/       sso (Helm)
├── forgejo/         git + ci/cd
├── hive/            ai agent swarm
├── homepage/        cluster landing page
├── infrastructure/  kubevirt + corral
├── longhorn/        storage
├── monitoring/      prometheus + grafana + metrics-server
├── n8n/             workflow automation
├── networking/      tailscale configs
└── testing-lab/     ci/cd test infra

Running Workloads

Lemonade

Lemonade — AMD-optimized local AI runtime providing omni-modal endpoints (chat, vision, image gen, speech, transcription) on karnataka's integrated AMD GPU (amdgpu driver baked into the Talos image via Image Factory).

Manifest: talos-k8s/ai/lemonade.yaml

Additional services

See the cluster handbook for the full list — Forgejo, Hive, Authentik, n8n, AppFlowy Cloud, Grafana, Corral, and more.

Cluster Access

# kubectl (from kubeconfig in Bitwarden)
kubectl get nodes

# Talos CLI (LAN addresses; off-LAN they're reachable via bihar's subnet route)
talosctl -n <bihar-ip> version       # bihar
talosctl -n <karnataka-ip> version   # karnataka

Image Factory

Karnataka uses a custom Talos image from Image Factory that includes the siderolabs/amdgpu system extension for GPU passthrough to containers.

Handbook

Full cluster documentation at docs/src/servers/talos-k8s/cluster.md (hardware details, reinstall procedure, troubleshooting).

Cluster Handbook — Talos Linux on Bihar + Karnataka

ARCHIVED (2026-11-18). This page describes the home Talos cluster, which was replaced by Bluefin + k3s. See Fleet status.

Status: powered down and in storage (as of 2026-08-27), expected back. Temporarily offline, not decommissioned — keep its configs, Bitwarden notes (kubeconfig / talosconfig), and talos-k8s/ manifests. kubectl and just doctor will fail against it until it returns. Matrix and the School, Reef and Hub Hives now run on the OCI Spindle VPS. The AWS Talos cluster was terminated on 2026-10-25.

When it returns, before applying anything under talos-k8s/ai/: run talos-k8s/ai/migrate-pv-path.sh first. lemonade.yaml already points its PVs at /var/lib/…, but the models are still under /var/tmp/… on karnataka. Applying the manifest first orphans them (#29). Then follow the prerequisites in talos-k8s/ai/README.md for the backup job and run lemonade-warmup.sh (#26, #27).

Home Hummingbird migration: prepared, not deployed. The build/install and activation procedure is in section 12. The home machines remain offline/in storage; the Talos recovery instructions below and all existing home/AWS credentials remain valid recovery material.

Two-node Talos Linux Kubernetes cluster. AMD Strix Halo GPU acceleration · Lemonade · KubeVirt · Tailscale ingress. Everything you need to bring this cluster back from zero.

Table of contents

  1. Overview
  2. Hardware
  3. Network
  4. Talos installer image (with AMD GPU)
  5. Cluster bootstrap
  6. AMD GPU exposed to Kubernetes
  7. Production workloads
  8. Tailscale operator (ingress)
  9. Day-to-day operations
  10. Reinstall procedure
  11. Troubleshooting
  12. Hummingbird/bootc migration (home only)

1. Overview

┌──────────────────────────────────────────────────────────────┐
│                       Kubernetes v1.36.1                      │
│                        (Talos v1.13.2)                        │
│                                                               │
│   ┌────────────────────────┐   ┌────────────────────────┐    │
│   │  bihar  (control plane)│   │ karnataka  (worker)    │    │
│   │  Intel CPU             │   │ AMD Ryzen AI MAX+ 395  │    │
│   │  /dev/nvme0n1          │   │ AMD Radeon 8060S iGPU  │    │
│   │                        │   │ /dev/nvme0n1           │    │
│   │  kube-apiserver        │   │ amdgpu-device-plugin   │    │
│   │  etcd / scheduler /    │   │ amdgpu-labeller        │    │
│   │  controller-manager    │   │ Lemonade               │    │
│   │  KubeVirt control      │   │ Forgejo + runner       │    │
│   │  Forgejo + runner      │   │ Hive agent swarm       │    │
│   │  ArgoCD + Argo WF      │   │ KubeVirt virt-handler  │    │
│   │  Grafana / Prometheus  │   │ Longhorn CSI           │    │
│   │  Authentik + n8n       │   │ AppFlowy Cloud         │    │
│   │  Longhorn CSI / UI     │   │ Corral VMs             │    │
│   │  KubeVirt Manager      │   │                        │    │
│   └────────────────────────┘   └────────────────────────┘    │
│                                                               │
│   CNI: flannel · kube-proxy: on · Storage: longhorn + local-path      │
│   Tailscale Operator → ts.net ingress for cluster services    │
└──────────────────────────────────────────────────────────────┘

The cluster is not diskless — Talos installs itself onto /dev/nvme0n1 on each node and persists. Reboots survive.

Source of truth for manifests: talos-k8s/ in this repo. Source of truth for secrets (talosconfig, controlplane.yaml, worker.yaml, kubeconfig): Bitwarden, fetched onto your workstation by the kube role.


2. Hardware

Karnataka (worker)

ComponentDetail
CPUAMD Ryzen AI MAX+ 395 (Strix Halo) — 16 cores / 32 threads, 5.18 GHz boost
RAM62 GB unified memory (CPU + GPU share via UMA)
GPUAMD Radeon 8060S — RDNA 3.5 (gfx1151), integrated
NICRealtek RTL8126 5GbE (enp191s0)
WiFiMediaTek MT7925 Wi-Fi 7
Boot diskCrucial P3 1TB (/dev/nvme0n1)
SecondaryWD Black SN850X 1TB (/dev/nvme1n1) — FAILING (I/O errors). Do not use. Replace with new drive.

Bihar (control plane)

ComponentDetail
CPUIntel Core i7-13700K
Integrated GPUIntel UHD 770
Boot disk/dev/nvme0n1

3. Network

LAN — flat home /24

DHCP/DNS are served by the home router, which hands out fixed addresses from its address-reservation table — see Gateway. Older revisions of this page said dnsmasq on bihar; that predates Talos, which can't run it on the host.

HostAddressRole
biharreservedK8s control plane, home services
karnatakareservedK8s worker, LLM host
goareservedRaspberry Pi, fleet control
kvmreservedNanoKVM (plugged into karnataka)

The commands below use <bihar-ip> / <karnataka-ip> for the nodes' LAN addresses. Run just inventory to rescan with nmap and get the current values (they are also the endpoints in the talosconfig).

Internet — data-capped

The home connection has a monthly data cap; exceeding it throttles to a crawl until the next billing cycle.

May 2026: The cap was hit (largely model downloads and image pulls) and the link was throttled for about a week. Lesson: budget large model downloads and bulk image pulls against the cap. LAN traffic is unaffected.

Tailscale — manatee-basking.ts.net

Cluster ingress goes through the Tailscale Operator. Services with an Ingress resource targeting tailscale ingressClassName get a <name>.manatee-basking.ts.net URL automatically.

ServiceURL
Cluster homepagehttps://home.manatee-basking.ts.net
Lemonade (AI)https://lemonade.manatee-basking.ts.net/v1
Hive (agent swarm)https://hive.manatee-basking.ts.net
Forgejo (git + CI)https://forgejo.manatee-basking.ts.net
ArgoCD (GitOps)https://argocd.manatee-basking.ts.net
Grafana (metrics)https://grafana.manatee-basking.ts.net
Corral (VM dashboard)https://corral.manatee-basking.ts.net
KubeVirt Managerhttps://kubevirt-manager.manatee-basking.ts.net
Authentik (SSO)https://auth.manatee-basking.ts.net
n8n (workflows)https://n8n.manatee-basking.ts.net
AppFlowy (collab)https://appflowy.manatee-basking.ts.net

Kubernetes internal

NetworkCIDR
Pod10.244.0.0/16 (flannel)
Service10.96.0.0/12

4. Talos installer image (with AMD GPU)

Talos ships with an immutable root filesystem, so kernel modules cannot be added at runtime. The AMD GPU driver and the Longhorn iSCSI tooling are baked into the boot image via the Talos Image Factory.

Schematic ID (current, 2026-06-11): 3a33ec6dfc8cfd61d2a3db3caf97894f31e913952d71ce3c3fbbe565a3f08339

Extensions baked in:

  • siderolabs/amdgpu — AMD GPU kernel driver
  • siderolabs/iscsi-tools — iscsid, for Longhorn block storage
  • siderolabs/util-linux-tools — fstrim/nsenter, for Longhorn

Both nodes run this schematic. Earlier schematics were e5912b95… (amdgpu only) and, in older docs, b6ab12e…. Storage details: talos-k8s/longhorn/.

machine:
  install:
    image: factory.talos.dev/installer/3a33ec6dfc8cfd61d2a3db3caf97894f31e913952d71ce3c3fbbe565a3f08339:v1.13.2
    disk: /dev/nvme0n1

After install, the node exposes /dev/dri/card0, /dev/dri/renderD128, and /dev/kfd on the host.

Talos version note: Pinned to v1.13.2. v1.13.3 had multi-arch registry pull errors on AMD64 at the time of writing.

User volume for /var/mnt/storage

nvme0n1 is the Crucial P3 (healthy, boot disk). The WD Black failed with I/O errors and was replaced as the boot disk. Forgejo / local-path PVCs use a directory-type user volume on the Crucial P3's EPHEMERAL partition:

# Apply the user volume config (idempotent):
talosctl -n <karnataka-ip> patch mc --patch @storage-volume.yaml

The patch is stored at talos-k8s/longhorn/storage-volume.yaml. It creates /var/mnt/storage as a directory on the EPHEMERAL partition. The mount is automatically propagated into the kubelet namespace.

WD Black failure (2026-05-30): The WD Black SN850X (/dev/nvme1n1) developed I/O errors on its META partition during the 35B model download stress test. Error: error writing config to file: input/output error. The drive should be physically replaced. In the meantime, do not configure it as a Talos install disk or user volume.


5. Cluster bootstrap

Configs are generated once with talosctl gen config, edited, and applied to each node. The generated files contain cluster PKI secrets (machine token, bootstrap token, shared secret) and are not committed — they live in Bitwarden.

One-time generation

cd talos-k8s
talosctl gen config talos-k8s https://<bihar-ip>:6443
# Produces: controlplane.yaml, worker.yaml, talosconfig

Then patch machine.install.image and machine.install.disk on each (see talos-k8s/README.md for the schematic ID).

Apply configs

talosctl apply-config --insecure --nodes <bihar-ip> --file controlplane.yaml
talosctl apply-config --insecure --nodes <karnataka-ip> --file worker.yaml

talosctl config endpoint <bihar-ip>
talosctl config node <bihar-ip>

talosctl bootstrap --nodes <bihar-ip>
talosctl kubeconfig                  # writes ~/.kube/config

Seed Bitwarden with secrets

Once the cluster is up and you have working ~/.kube/config and ~/.talos/config:

just seed-kube     # uploads kubeconfig + talosconfig to Bitwarden as secure notes

Other workstations then pull them automatically:

just apply-tags kube

6. AMD GPU exposed to Kubernetes

The driver is loaded on the host (from the Talos image). Kubernetes needs two more pieces from ROCm to actually schedule on the GPU:

kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml
kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml

Once the DaemonSets are running, karnataka exposes:

  • amd.com/gpu: 1 as an allocatable resource
  • Labels such as amd.com/gpu.product-name=AMD_Radeon_8060S_Graphics, amd.com/gpu.vram=64G

Workloads request the GPU with:

resources:
  limits:
    amd.com/gpu: "1"

7. Production workloads

All manifests live in talos-k8s/, organized by purpose:

talos-k8s/
├── ai/              Lemonade (LLM runtime)
├── appflowy/        AppFlowy Cloud (collaboration)
├── authentik/       Authentik (SSO)
├── forgejo/         Forgejo + runner + registry-mirror
├── hive/            Hive (AI agent swarm)
├── homepage/        Cluster landing page
├── infrastructure/  KubeVirt + Corral
├── kubestellar-hive/ KubeStellar multi-cluster hive
├── longhorn/        Storage (CSI, snapshots)
├── monitoring/      Prometheus + Grafana + metrics-server
├── n8n/             n8n (workflow automation)
├── networking/      Tailscale node configs
├── searxng/         SearXNG (self-hosted search)
├── testing-lab/     CI/CD test infrastructure
└── topolvm/         TopoLVM (dynamic LVM storage)

Lemonade — AMD-optimized local AI runtime

Single-replica Lemonade omni-modal server on the Strix Halo APU. Manifest: talos-k8s/ai/lemonade.yaml.

Lemonade is an AMD-optimized, open-source local AI runtime that auto-detects hardware and provides standard OpenAI-compatible endpoints for chat, vision, image generation, image editing, speech generation, and transcription. Built on llama.cpp, ONNX Runtime, whisper.cpp, and stable-diffusion.cpp.

Key env overrides (RDNA 3.5 needs special handling):

Env varValueWhy
HSA_OVERRIDE_GFX_VERSION11.5.1Forces ROCm to treat gfx1151 as supported
PYTORCH_ROCM_ARCHgfx1151Compile shaders for exact arch
HSA_XNACK1Enables unified memory
HSA_FORCE_FINE_GRAIN_PCIE1Fine-grained VM access

Access:

  • LAN: http://<karnataka-ip>:31305/v1 (NodePort)
  • Tailscale: https://lemonade.manatee-basking.ts.net/v1

Model weights cached on karnataka:/var/tmp/lemonade-cache (HuggingFace) and /var/tmp/lemonade-models (llama models) via local-storage PersistentVolumes.

Web UI: Point a browser at http://<karnataka-ip>:31305 for the built-in control panel to download models, configure endpoints, and test chat/vision/image generation.

KubeVirt v1.8.2

Installed from upstream operator manifests:

kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-operator.yaml
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-cr.yaml

The default namespace is labelled pod-security.kubernetes.io/enforce=privileged so privileged virtualization pods can run.

KubeVirt Manager (web UI)

Manifest: deployed from upstream bundle. Also see talos-k8s/infrastructure/ for KubeVirt CR and Corral VM dashboard.

kubectl apply -f https://raw.githubusercontent.com/kubevirt-manager/kubevirt-manager/main/kubernetes/bundled.yaml

Reachable at https://kubevirt-manager.manatee-basking.ts.net via the Tailscale ingress (manifest in cluster — not in this repo since it's a stock bundle).

Port-forward as fallback:

kubectl port-forward svc/kubevirt-manager 8080:8080 -n kubevirt-manager
# http://localhost:8080

Additional services

All other production services have their own directory in talos-k8s/:

ServiceDirectoryDescription
Forgejoforgejo/Git hosting + Actions CI/CD
Hivehive/AI coding agent swarm backed by Lemonade
Authentikauthentik/SSO (passkeys, OAuth2/OIDC) — Helm
n8nn8n/Workflow automation — plain K8s manifest
AppFlowy Cloudappflowy/Collaborative workspace — commercial Helm chart
Grafana + Prometheusmonitoring/kube-prometheus-stack + metrics-server
ArgoCD(Helm)GitOps deployment
Longhornlonghorn/Distributed block storage (CSI, snapshots)
KubeStellar Hivekubestellar-hive/KubeStellar multi-cluster hive
SearXNGsearxng/Self-hosted metasearch
TopoLVMtopolvm/Dynamic LVM-backed storage
Testing labtesting-lab/Argo Workflows CI/CD test infra

Each directory has a README.md with deploy and operational instructions.


8. Tailscale operator (ingress)

The Tailscale Operator creates a Tailscale-side proxy pod for any Ingress resource with ingressClassName: tailscale. The proxy joins the tailnet with a name derived from the Ingress, and routes incoming traffic to the backing Service.

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lemonade
spec:
  ingressClassName: tailscale
  defaultBackend:
    service:
      name: lemonade
      port:
        number: 13305
  tls:
    - hosts:
        - lemonade

The operator namespace is tailscale. Auth keys are configured during operator install — see Tailscale docs for renewal.


9. Day-to-day operations

From your workstation

The kube role syncs ~/.kube/config and ~/.talos/config from Bitwarden, so on any desktop:

kubectl get pods -A
talosctl -n <bihar-ip> health
talosctl -n <karnataka-ip> dmesg | grep -i amdgpu

Common kubectl

kubectl get nodes -o wide
kubectl top nodes                        # requires metrics-server (monitoring/metrics-server.yaml)
kubectl logs -f deployment/lemonade
kubectl describe pod -l app=lemonade    # check GPU scheduling / events

Common talosctl

talosctl -n <karnataka-ip> list /dev/dri/   # confirm GPU devices visible to host
talosctl -n <karnataka-ip> services         # systemd-equivalent services
talosctl -n <bihar-ip> etcd status      # control-plane etcd health
talosctl -n <bihar-ip> logs kubelet
talosctl -n <bihar-ip> reboot           # graceful reboot

10. Reinstall procedure

If a node is wiped or replaced:

  1. Boot Talos ISO on the node (or PXE if you set that up separately). Default Talos boots into maintenance mode and accepts an apply-config.
  2. Apply config:
    talosctl apply-config --insecure --nodes <node-ip> --file <controlplane|worker>.yaml
    
  3. Bootstrap etcd (control plane only, first time):
    talosctl bootstrap --nodes <bihar-ip>
    
  4. Verify:
    talosctl -n <bihar-ip> health
    kubectl get nodes
    
  5. Create user volume (karnataka/worker only):
    talosctl -n <karnataka-ip> patch mc --patch @talos-k8s/longhorn/storage-volume.yaml
    
  6. Create host directories for local PVs (if re-creating from scratch):
    kubectl exec -n kube-system $(kubectl get pods -n kube-system --field-selector spec.nodeName=karnataka -l app=kube-flannel -o name) -- \
      mkdir -p /proc/1/root/var/tmp/lemonade-cache
      mkdir -p /proc/1/root/var/tmp/lemonade-models
    
  7. Re-apply workloads (idempotent):
    kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml
    kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml
    kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-operator.yaml
    kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.8.2/kubevirt-cr.yaml
    kubectl apply -f https://raw.githubusercontent.com/kubevirt-manager/kubevirt-manager/main/kubernetes/bundled.yaml
    kubectl apply -f talos-k8s/ai/lemonade.yaml
    

If only the worker was reset, the existing kubeconfig still works — only steps 1, 2, 5, and 6 are needed for that node, plus re-applying any DaemonSet pods.

WD Black failure: Do not use the WD Black (/dev/nvme1n1) for installs or user volumes. Install only to the Crucial P3 (/dev/nvme0n1). Storage uses a directory-type volume on the Crucial P3's EPHEMERAL partition.


11. Troubleshooting

amd.com/gpu shows 0 on karnataka

Check the device plugin actually scheduled and started:

kubectl -n kube-system get pods -l name=amdgpu-device-plugin-ds
kubectl -n kube-system logs -l name=amdgpu-device-plugin-ds --tail=50
talosctl -n <karnataka-ip> dmesg | grep -i amdgpu   # confirm driver loaded
talosctl -n <karnataka-ip> list /dev/dri/           # confirm devices exist

Lemonade pod stuck in ContainerCreating

Most often: pulling models from HuggingFace. Check:

kubectl describe pod -l app=lemonade
kubectl get pv,pvc                       # confirm PVCs bound

Tailscale Ingress not reachable

kubectl -n tailscale get pods            # operator + per-Ingress ts-* proxies
kubectl -n tailscale logs deploy/operator
tailscale status | grep -i <ingress-name>

Talos node unresponsive

talosctl -n <ip> health --wait
talosctl -n <ip> dmesg --tail
talosctl -n <ip> reset                   # nuclear: wipes node, requires re-apply

Container runtime crash (karnataka)

Symptoms:

  • kubectl get nodes shows karnataka NotReady
  • kubectl describe node karnataka shows Ready: False with container runtime is down
  • All pods on karnataka stuck in Terminating/Pending
  • Ping to karnataka still works (node is up, just CRI is dead)

Root cause: Memory pressure from simultaneous large container builds exhausts karnataka's 62GB unified memory. The node runs Lemonade (48-56Gi), Argo Workflow image builds, Forgejo Actions (podman-in-Docker), KubeVirt, and monitoring — when multiple 4GB image builds run concurrently, containerd OOMs and crashes.

Recovery:

# Option 1: Graceful reboot via Talos
talosctl -n <karnataka-ip> reboot

# Option 2: Hard reboot via the NanoKVM (see network/kvm)

After reboot (~2 min), all pods recover automatically. PVC-backed services (Forgejo, local-path PVs) survive.

Prevention:

  • Don't run more than one large container build concurrently on karnataka
  • Set SKIP_CHUNKAH=1 for Forgejo builds (avoids extra 2GB memory spike)
  • Consider moving non-GPU workloads to bihar or a third node
  • The Forgejo runner has timeout-minutes: 45 — builds that exceed this will be killed

12. Hummingbird/bootc migration (home only)

Scope and acceptance status

This is a replacement OS and new K3s cluster for the two home nodes, not a Talos upgrade. Bihar is the Intel server/control plane; Karnataka is the AMD worker. No physical migration has been performed. The home cluster is still powered down/in storage. Nothing here changes the terminated AWS Talos cluster's recovery material; Corral acceptance uses separate disposable KubeVirt VMs, not Talos node changes. Sections 1–11 remain the Talos recovery handbook, not instructions for the new OS.

Image builds alone do not prove installation. The GitHub Actions acceptance workflow builds all three images and offline ISOs, drives the real bootc-installer-tui, boots the installed disks without their ISOs, and switches the installed starter to Bihar before activating a separate disposable two-node K3s cluster. Publication run 39493512586 passed for release source 4ec3fa6470d6b02e7eaf1b0eaae03d1db05a31a7: all three offline TUI installs, installed-disk UEFI boot, admin login/sudo, Enforcing SELinux, unenrolled Tailscale, and a real starter-to-Bihar bootc switch with exact digest, reboot, and administrator/hostname/identity//var persistence checks. Gated K3s activation then passed two Ready nodes, cross-node HTTP, cluster DNS, and local-path PVC data surviving writer-pod replacement. The reviewed PR also passed normal repository CI.

On 2026-10-20, Playwright drove both installers through the actual Corral noVNC canvases on disposable KubeVirt VMs. Both reached “Installation Complete”; their ISO volumes were then detached before installed-disk boots. Real admin-password login/sudo and Enforcing SELinux passed on both. The installed cluster passed the same DNS, cross-node HTTP, and PVC pod-replacement scenario used by CI. Bihar's Intel kvm_intel module loaded on this Intel Corral host; that is not a test of a nested KubeVirt guest or the physical home machines.

The deployed Corral service reported its cluster unreachable, so acceptance used a temporary localhost Corral dashboard against admin@aws-migration, without changing the deployed service. All test VMs/disks were isolated in namespace home-k8s-acceptance. Configure the disposable node LAN as 192.168.0.5/24 and 192.168.0.6/24 before activating K3s: Corral's provider pod addresses overlap this new cluster's 10.244.0.0/16 pod CIDR. The initial overlapping setup reached Ready nodes and working DNS but failed cross-node HTTP; the non-overlapping guest LAN passed without changing the images' production network configuration. Changing an already-initialized embedded-etcd server's IP also requires its membership to be corrected; do not initialize it with the provider pod IP in the first place.

Native Corral media predates publication-only OCI label changes; it is not byte-identical to the subsequently rebuilt release. The publication dispatch repeats complete KVM acceptance on the exact bytes it publishes. Corral evidence is saved under ~/.local/share/home-k8s-build/ on the acceptance controller: corral-{bihar,karnataka}-install-complete.webp, corral-{bihar,karnataka}-installed-boot.webp, corral-installed-kubernetes.webp, and corral-cluster-proof.txt.

The optional starter also passed a 2026-10-20 Playwright/noVNC installation in namespace home-k8s-starter-acceptance, followed by ISO detachment and an installed-disk UEFI boot. Actual admin-password login/sudo, hostname starter-smoke, Enforcing SELinux, absent K3s role/configuration, driver/firmware payload checks, and the canonical starter bootc reference passed. The real installed tailscaled daemon was active and responsive in NeedsLogin, with no tailnet and root-owned 0700 state; no VM was enrolled into the fleet. The dedicated tailscale0 SSH firewall zone was present. All disposable VMs, PVCs, namespaces, and temporary Corral/upload services were removed afterward.

That native starter ISO was built from source 2f8be86e3f98782f8dc7857e688e274318494f5b, is 2,057,453,568 bytes, and has SHA256 1dc796f217e2d4ce6140511bed59399d2e031092299e2fa09e0436bea305ee27. Evidence is under ~/.local/share/home-k8s-build/unified-starter-native/: corral-starter-install-summary.png, corral-starter-install-complete.png, corral-starter-admin-sudo.png, corral-starter-installed-tailscale.png, and corral-starter-installed-proof.txt. These are native acceptance artifacts, not a claim that the later publication rebuild is byte-identical.

VM acceptance cannot establish GPU, physical NIC, disk health, firmware, or nested-virtualization readiness on the real machines. The recorded runner was AMD EPYC: Bihar's Intel-only kvm_intel preload fails in that VM, so its module-load unit is not evidence of working Intel virtualization. Verify that unit and a real KubeVirt guest on physical Bihar before cutover.

Image contents and state

The source is server-images/Containerfile, compose.sh, nodes/, and rootfs/. All three targets use the digest-pinned Hummingbird Community os-minimal base, without a desktop. The Bihar/Karnataka images remain the direct installation path for those machines; starter is optional for future x86_64 AMD/Intel nodes. Explicit AMD/Intel GPU and CPU, Intel Wi-Fi, Realtek, MediaTek, Atheros, and Broadcom firmware subpackages come through the digest-pinned Utah package factory. That local repository is authenticated by its OCI digest, not RPM signatures. The lower-priority Fedora 44 repository checks Fedora RPM signatures and supplies missing storage dependencies; it does not replace the Hummingbird kernel. The separate live-media build uses Hummingbird's dracut-live and systemd-boot-unsigned, with missing dependencies such as fuse, parted, and ntfs-3g supplied by signed Fedora RPMs. Those live additions do not modify the embedded installed-system OCI payload. K3s and its SELinux policy RPM are separately SHA256-pinned.

The common hardware payload includes:

HardwareShipped support
Bihar: Intel Core i7-13700K / UHD 770Intel microcode, i915 and its firmware, kvm_intel; the xe driver/firmware is also available
Karnataka: Strix Halo / Radeon 8060S (gfx1151)amdgpu, gc_11_5_1 GPU firmware, AMD microcode, kvm_amd
Karnataka: MediaTek MT7925 Wi-Fimt7925e plus its Wi-Fi code and patch firmware
Karnataka: Realtek RTL8126 Ethernetr8169 plus Realtek NIC firmware
Other common Wi-Fi familiesIntel iwlwifi, Realtek rtw89, Atheros/Qualcomm ath9k/ath10k/ath11k/ath12k, Broadcom brcmfmac, and their selected firmware packages

NetworkManager-wifi supplies the actual Wi-Fi backend, with wpa_supplicant for authentication. After installed-system login, connect interactively with sudo nmcli --ask device wifi connect "your-ssid"; never put Wi-Fi credentials in image source or command-line password arguments. Live-session network configuration is not copied by the pinned installer. Driver/module/firmware presence is checked during builds and starter acceptance, but physical RF, GPU compute, device passthrough, and board-specific compatibility remain untested. This is not an ARM image or a proprietary NVIDIA/CUDA stack.

K3s is pinned to v1.36.1+k3s1, with embedded containerd, Flannel networking, and embedded etcd on Bihar (cluster-init: true). Pod/service CIDRs remain 10.244.0.0/16 and 10.96.0.0/12; cluster DNS is 10.96.0.10. Bihar disables bundled Traefik, ServiceLB, and metrics-server: restore any needed equivalents deliberately. This is one control plane, not HA.

NetworkManager, sshd, firewalld, Tailscale, and iSCSI services are enabled on all three images. Bihar enables k3s@server; Karnataka enables k3s@agent, but both are gated by /etc/rancher/k3s/activate. The starter selects neither role and supplies no hostname, K3s configuration, or cluster endpoint. No activation marker, cluster token, Tailscale auth key, or enrolled identity is baked into an image. Automatic bootc fetch/apply services are masked so a reboot of the only control plane is an explicit maintenance decision. NFS GSSAPI state directories are initialized privately (0700) through tmpfiles before the daemon's sandbox is assembled; CI requires gssproxy to start successfully with SELinux enforcing.

Mutable state includes /etc/rancher/k3s, /var/lib/rancher/k3s (including etcd and containerd), user homes under /var/home, and these retained workload paths:

PathPurpose
/var/mnt/storageBihar's K3s local-path default and existing host-path data
/var/lib/longhornLonghorn persistent storage
/var/lib/lemonade-cacheLemonade cache, matching the current home PV manifest
/var/lib/lemonade-modelsLemonade models, matching the current home PV manifest

The current talos-k8s/ai/lemonade.yaml uses /var/lib/lemonade-*. The offline Talos node may still hold the data under /var/tmp/lemonade-*: back up those source directories before wiping and restore into /var/lib/lemonade-* on the new node. Creating an empty PV directory does not migrate the models.

Back up before building a physical migration plan

Before any disk wipe, recover the old nodes if necessary and copy application backups, PV data, models, manifests, and required secrets off both machines. Inventory PV-to-node/path mappings, ownership, application restore procedures, and Longhorn volumes; verify the backups can be read/restored independently. An etcd snapshot is useful for Talos recovery, but Talos etcd cannot be directly restored into K3s. A matching directory name is not a data migration.

Keep the old Bitwarden secure notes kubeconfig / talosconfig and their ~/.kube/config / ~/.talos/config files. Also keep the separate AWS kubeconfig-aws-migration / talosconfig-aws-migration notes and files unchanged. Use a new home kubeconfig below; do not overwrite any of these.

Karnataka's /dev/nvme1n1 (WD Black) is failing: never select it. Only the healthy /dev/nvme0n1 is a candidate, after off-node backups and an explicit physical operator choice. Identify disks by model, serial, size, and current contents at the console (lsblk -o NAME,SIZE,MODEL,SERIAL,FSTYPE,MOUNTPOINTS); do not assume device enumeration is sufficient. Apply the same backup and operator-choice gate to Bihar's intended /dev/nvme0n1.

Build node images and optional starter media

Run from this repository on a Linux builder with Podman, Go compatible with Fisherman's tui/go.mod, Python 3, squashfs-tools (mksquashfs), dosfstools (mkfs.fat), mtools (mmd, mcopy), and xorriso available. Local source defaults are ~/dev/tuna-os/fisherman and ~/dev/utah; override with FISHERMAN_SRC and UTAH_SRC. For the CI source selection, use Fisherman commit 014e704be5f7365c4a1ccdef37fcadf97ade792a and Utah commit f2736b6b774551cc8971af85cedaa65a272354ee; the Fisherman Go requirement is 1.26.2. The media builder compiles the real Fisherman backend and TUI from these trees, not a substitute installer. Use a clean Fisherman checkout at that revision: it includes the target-disk OCI staging fix from the merged Fisherman PR 345, required for offline installation without exhausting the live filesystem.

just server-images
# Produces localhost/home-k8s-{starter,bihar,karnataka}:testing.
mkdir -p artifacts/home-k8s
export FISHERMAN_SRC="$HOME/dev/tuna-os/fisherman"
export UTAH_SRC="$HOME/dev/utah"
# Choose canonical refs in a registry namespace you control.
export BIHAR_TARGET_REF=ghcr.io/hanthor/home-k8s-bihar:testing
export KARNATAKA_TARGET_REF=ghcr.io/hanthor/home-k8s-karnataka:testing
just server-installer bihar "$BIHAR_TARGET_REF" artifacts/home-k8s/bihar-installer.iso
just server-installer karnataka "$KARNATAKA_TARGET_REF" artifacts/home-k8s/karnataka-installer.iso
# Optional: a hardware-generic install with no selected cluster role.
just server-installer starter ghcr.io/hanthor/home-k8s-starter:testing artifacts/home-k8s/starter-installer.iso

The target references above are day-2 tracking metadata, not downloads needed by the offline installation. Each ISO embeds its matching unchanged OCI payload. These local commands and PR/push acceptance do not publish registry images. Before bootc upgrade, publish the intended image to its canonical ref and verify the digest; building an ISO alone does not make that ref available.

To build just one profile, use just server-image bihar, just server-image karnataka, or just server-image starter. The canonical recipe disables Podman's injected /etc/hostname so image composition can clear the base hostname before applying a specialization. The UEFI media keeps its boot files inside the FAT ESP rather than duplicating them as loose ISO files; the embedded installed-system payload is unchanged by that size reduction.

Publish accepted images and installation media

Published acceptance release: home-k8s-20261021. These are public prerelease artifacts, not proof of physical hardware readiness. All three profiles' date tags and testing channels passed anonymous GHCR access and match the exact tested payloads and source revision.

ProfilePublished imageOffline installer ISO
Biharghcr.io/hanthor/home-k8s-bihar:20261021bihar-installer.iso
Karnatakaghcr.io/hanthor/home-k8s-karnataka:20261021karnataka-installer.iso
Optional starterghcr.io/hanthor/home-k8s-starter:20261021starter-installer.iso

Download SHA256SUMS and provenance.json from the same release; the latter records image digests, exact source/upstream revisions, and the acceptance run. OCI archives, media contracts, and the starter-switch proof are also release assets. The publisher treats date tags as write-once; use the recorded @sha256: digest when a registry reference must be intrinsically immutable. The older 20261020 release remains available but predates the optional starter and completed GPU/Wi-Fi firmware and host Tailscale payload.

Publication is an explicit master-only dispatch, after interactive Corral acceptance. It repeats all three real offline TUI installations, the installed starter's bootc switch, and the two-node Kubernetes scenario before transferring any release artifacts:

gh workflow run server-images.yml --repo hanthor/dotfiles --ref master \
  -f publish=true -f version=20261022

Choose an unused, valid YYYYMMDD date; 20261020 and 20261021 are already published. Ordinary PR/push runs and manual runs with publish=false remain read-only. Only the separate publishing job receives registry/release write permissions; it pushes the exact accepted OCI archives, not rebuilt images. It publishes ghcr.io/hanthor/home-k8s-{starter,bihar,karnataka} with immutable date tags and the mutable testing channel, plus the public GitHub prerelease home-k8s-YYYYMMDD (not the repository's latest stable release). Assets include all three tested ISOs, OCI archives, media contracts, checksums, source/upstream revisions, registry digests, explicit acceptance markers, and the sanitized starter-switch proof. The canonical just server-images build records this repository's Git revision in the OCI source/revision labels, rather than inheriting Hummingbird's revision. Installed VM disks and credentials are never release assets.

New GitHub container packages default to private, even when linked to a public repository. The packages above are already public. For a new package, change visibility in its GitHub package settings if necessary. The publisher refuses to create the release until anonymous access to both image tags succeeds. If this gate fails, change visibility and rerun only the failed publishing job within one day: the accepted artifact transfer has one-day retention. Identical existing date tags are reused; different content or an existing release/tag name is rejected. Never rebuild over an immutable date tag.

Install through the actual TUI

For each node, first meet the backup/disk-selection gate above. Boot its matching ISO in UEFI mode with Secure Boot disabled. At the live root console run:

bootc-installer-tui
  1. Skip network configuration to use DHCP. The pinned TUI does not serialize selected network configuration; do not depend on it applying static addressing.
  2. Choose Custom URL and enter oci:/usr/share/home-k8s/payload.
  3. Choose the verified intended installation disk. Installation destroys its contents; do not choose the failing Karnataka secondary disk.
  4. Choose ext4, and set the hostname to bihar or karnataka as appropriate, or a unique hostname for a new starter installation.
  5. Leave disk encryption disabled for the tested installation path. Encrypted installation is not covered by this acceptance run.
  6. Create an actual administrative user with a strong password. Record it securely and check login/sudo after installation. Do not rely on root live access as an installed-system account.
  7. Skip SSH-key import: this pinned TUI does not serialize imported keys into the installation. Add authorized keys securely after first login instead.
  8. Review the destructive disk choice, install, shut down/reboot, and remove the ISO so the next boot is from the installed disk.

The live ISO is SELinux permissive only (enforcing=0, not selinux=0). That exception belongs to live assembly, not the target. On the installed node, verify getenforce prints Enforcing, sudo -v succeeds for the new user, bootc status shows the intended target reference, and the corresponding K3s unit is inactive before activation. Stop and resolve any mismatch before joining the cluster. Install SSH keys and verify remote access using authenticated, trusted connections; never paste private keys or cluster tokens into CI logs.

Optional starter and a deliberate image switch

The starter is not a third cluster node and does not join anything on first boot. It installs the same hardware, storage, Tailscale, and K3s tooling, but neither K3s instance is enabled and there is no default K3s configuration. Install Bihar and Karnataka directly when replacing those machines; the starter is an optional base for additional machines or a later switch to another bootc image.

For example, to adopt Bihar's server profile from an installed starter:

sudo bootc switch ghcr.io/hanthor/home-k8s-bihar:testing
sudo systemctl reboot

After reboot, inspect sudo bootc status and verify the booted image, not merely a staged deployment. The administrator, chosen hostname, and /var state survive; a switch does not rename the host or create the K3s activation marker. /etc uses a three-way merge, so review existing configuration rather than assuming the destination image replaces it.

The named profiles are home-machine specializations, not generic role templates: they contain fixed K3s node names, home addresses, and CPU-specific module preloads. On additional machines, use an appropriate derived image or review those defaults before activation, especially node-name, server address, TLS SANs, and virtualization modules. The starter also retains the common home-LAN firewall policy; new sites need an explicit network-policy review.

Enroll host Tailscale after installation

The pinned Utah factory supplies Tailscale 1.98.8. tailscaled starts automatically but remains NeedsLogin until the operator enrolls that node. On the installed system—not the live ISO—run:

sudo tailscale up --hostname="$(uname -n)" --advertise-tags=tag:fleet \
  --accept-dns=false --accept-routes=false
tailscale status

Complete the displayed login URL using the fleet tailnet account; tag:fleet requires the corresponding tag-owner permission. This keeps the existing fleet tag convention without importing tailnet routes or replacing the node's DNS, which could conflict with the home/Flannel networks. No subnet routes, exit-node service, or Tailscale SSH are enabled automatically.

The dedicated tailscale0 firewalld zone allows ordinary SSH only; it does not open the Kubernetes API, kubelet, VXLAN, or NodePorts to every tailnet peer. Tailnet ACLs and installed SSH authorized keys still govern access. Host Tailscale does not replace Flannel or the separate in-cluster Tailscale Operator.

Node keys and enrollment state live under private, root-owned /var/lib/tailscale (0700) and survive bootc updates/switches. Never bake that directory into an image or copy it between machines. A clean reinstall needs its own enrollment; a live-session identity is not migrated by this installer. CI checks the real installed daemon's unauthenticated state and private storage with SELinux enforcing; it does not enroll disposable VMs into the fleet.

Explicit activation and separate kubeconfig

Verify router DHCP reservations and actual addresses before starting K3s: Bihar 192.168.0.5, Karnataka 192.168.0.6. The worker configuration and server TLS SAN depend on Bihar's address. Make sure the isolated/new home cluster cannot conflict with a running old cluster at those addresses.

On Bihar, as root:

touch /etc/rancher/k3s/activate
systemctl start k3s@server
k3s kubectl get nodes

Wait for the API/server to be healthy. Securely transfer /var/lib/rancher/k3s/server/node-token from Bihar to /etc/rancher/k3s/token on Karnataka. Treat this server token as cluster-admin-sensitive, not as a harmless worker password. Use a verified SSH connection or other authenticated encrypted channel; do not print it, commit it, store it in artifacts, or copy it through chat. On Karnataka ensure the file is owned by root and mode 0600 before activation:

chown root:root /etc/rancher/k3s/token
chmod 0600 /etc/rancher/k3s/token
touch /etc/rancher/k3s/activate
systemctl start k3s@agent

Securely copy Bihar's /etc/rancher/k3s/k3s.yaml to the workstation as ~/.kube/config-home-hummingbird (mode 0600); change only its loopback API endpoint to https://192.168.0.5:6443. Keep this admin credential out of Git and logs. Use it explicitly instead of changing the old home or AWS kubeconfigs:

export KUBECONFIG="$HOME/.kube/config-home-hummingbird"
kubectl get nodes -o wide
kubectl wait --for=condition=Ready nodes/bihar nodes/karnataka --timeout=300s

Restore home workloads and validate before cutover

Review and apply only selected home workload manifests from talos-k8s/. Do not recursively apply that whole directory: it also contains AWS manifests. Do not apply Talos machine patches, extension/schematic settings, or Talos-only bootstrap configuration to K3s. Reconcile storage classes, node labels, selectors, host paths, privileged-device access, and security requirements against the new cluster before restoring applications.

Restore data from off-node backups to the intended node/path with correct ownership and SELinux labels (use restorecon on restored host paths). Follow application-consistent restore procedures rather than blindly copying running databases or treating Longhorn's data directory as a complete backup. Validate PVC binding, persistence after pod/node restarts, DNS, cross-node traffic, ingress, application health, and data integrity before declaring a cutover.

On real Karnataka hardware, verify amdgpu, /dev/kfd, /dev/dri, the AMD device plugin's allocatable GPU, and an actual Lemonade inference workload. Verify /dev/kvm, firmware virtualization settings, the correct Intel/AMD KVM modules, KubeVirt scheduling, and a real guest boot on the relevant nodes. Build-time module checks and virtual acceptance do not establish these facts.

Use planned maintenance and fresh data/etcd backups for day-2 bootc updates. bootc rollback rolls back the OS deployment, not application data, etcd, mutable /etc configuration, or PV contents. A data rollback requires its own tested restore plan; retain the Talos recovery material independently.

Disposable VM acceptance (GitHub Actions and local)

GitHub Actions workflow Home server images (server-images.yml) runs for matching pull requests and master pushes. Manual dispatch defaults to acceptance only (publish=false); review branches cannot publish. Acceptance builds all three images/ISOs, installs each through the real TUI, and checks installed-disk boot, admin/sudo, Enforcing SELinux, unenrolled Tailscale, and gated K3s. It switches the installed starter to the exact new Bihar payload through a loopback-only registry, verifies the staged digest, reboots, and checks the booted digest plus persistent admin/hostname/identity and /var data. The fixture registry, starter disk, and NVRAM are removed before the two-node cluster scenario. The VMs use an isolated virtual LAN plus QEMU NAT, not the physical home LAN. CI requires KVM and fails if it is unavailable. No fleet deployment or physical-disk access occurs, including on a publication dispatch. It runs on ubuntu-24.04 and uploads *.log, *.json, and *.txt diagnostics as home-server-images-diagnostics with seven-day retention, never installed disks. Only an explicit master publication retains the exact tested ISOs and exports an allowlisted, one-day release-artifact transfer to the publishing job described above. Retain any needed diagnostics before they expire.

For local acceptance, install uv, QEMU x86_64, Skopeo, and OVMF, and use the ISO names from the build commands above. These commands create disposable installed qcow2 disks and per-node UEFI variable stores in the artifact directory:

uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm install starter
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm install bihar
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm install karnataka
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm cluster

To exercise a published-image switch on that installed starter, resolve the immutable date tag's digest first (the guest needs outbound registry access):

image=ghcr.io/hanthor/home-k8s-bihar:20261021
digest=$(skopeo inspect --format '{{.Digest}}' "docker://$image")
uv run server-images/test-vm.py --artifacts artifacts/home-k8s --accel kvm \
  switch-starter --image "$image" --expected-digest "$digest"

starter-switch-proof.txt records only explicit successful checks and public image references/digests, never the administrator password or private state. The workflow's fixture instead uses the exact newly built Bihar archive before publication, binds its HTTP registry only to 127.0.0.1:5000, and scopes its temporary insecure-registry setting to 10.0.2.2:5000 inside the disposable guest.

On a machine without usable KVM, explicitly replace --accel kvm with --accel tcg in every command; this is slower emulation, not hardware acceleration proof. The harness pins pexpect==4.9.0 through its inline uv dependency metadata. It drives the actual TUI, powers off, removes the ISO, boots the installed UEFI disk, checks login/sudo, Enforcing SELinux, target reference, and inactive K3s, then activates both nodes and checks Ready status, cross-node traffic, DNS, and PVC behavior. Preserve the generated *.log, *.json, and *.txt proof outputs with the CI run. The passing run and physical hardware limitations are recorded under “Scope and acceptance status” above.

Bihar

ARCHIVED (2026-11-18). This page describes bihar on Talos, which was replaced by Bluefin + k3s. See Fleet status.

Kubernetes control plane. Talos Linux node — not in inventory.yml, managed with talosctl/kubectl only.

Powered down and in storage, expected back. See the cluster handbook.

Hardware

  • Arch: x86_64 (Intel)
  • Motherboard: ASRock
  • LAN: reserved address on the home /24
  • Role: Control plane (Talos K8s)
  • Tailscale: bihar (MagicDNS)

OS

Talos Linux v1.13.2 (Kubernetes v1.36.1)

Services

Bihar no longer serves anything directly — the old per-host Caddy paths (bihar.manatee-basking.ts.net/grafana, /auth, /n8n, …) predate Talos. Cluster services are reached through the Tailscale Operator; see the ingress list in the cluster handbook.

Networking

Tailscale runs as a Talos system extension, configured by talos-k8s/networking/tailscale-bihar.yaml:

  • Advertises the home LAN /24 as a subnet route
  • Tailscale DNS: disabled (TS_ACCEPT_DNS=false)

See also

Karnataka

ARCHIVED (2026-11-18). This page describes karnataka on Talos, which was replaced by Bluefin + k3s. See Fleet status.

Kubernetes worker node with AMD GPU. Talos Linux node — not in inventory.yml, managed with talosctl/kubectl only.

Powered down and in storage, expected back. See the cluster handbook.

Hardware

OS

Talos Linux v1.13.2 (Kubernetes v1.36.1)

K8s Workloads

See also

AWS Talos Cluster (archive)

Terminated 2026-10-25. This is a historical record, not an operating runbook. The fleet-backups S3 bucket and a mirror on himachal-1 (~/backups/tunaos-aws-2026-10-25) were retained; the current status of individual volumes, snapshots, IPs and other AWS resources is not established here. Matrix and standalone LiveKit are deployed on the OCI spindle VPS.

OpenTofu still describes removed cluster resources. just aws-plan may propose recreating them, and just aws-apply can recreate infrastructure and incur costs. Review live state and every proposed action before applying; see AWS account and IaC.

The following describes the former TunaOS two-node Talos Kubernetes cluster in AWS eu-north-1, built 2026-08-27 to consolidate the two Hetzner VPSes (matrix and telengana). Those retired hosts must not be restarted. The home Talos cluster is separately powered down in storage and expected back.

Historical descriptions and commands below refer to the cluster as it existed before termination; they cannot be used to access it now. The nodes were not in inventory.yml and no Ansible role targeted them.

Nodes

RoleInstanceLabels
control-planem6i.2xlarge (8 vCPU / 32 GiB)workload-role=hive
workerm6i.xlargeworkload-role=matrix

Each node had a private VPC address and an Elastic IP; addresses were recorded in OpenTofu state and talosconfig.

Talos v1.13.9, Kubernetes v1.36.2, flannel CNI, local-path storage.

The control-plane taint was removed so it could schedule workloads too. m6a (AMD) was not offered in eu-north-1; m6i was the equivalent.

Network

Dedicated VPC 10.20.0.0/16, single AZ eu-north-1a, public subnet + IGW, no NAT gateway.

Three security groups:

Security groupPurpose
intra-clusterself-referencing, node-to-node traffic
publicweb ingress, Matrix federation, and the MatrixRTC media ports
adminTalos and Kubernetes APIs, admin allowlist only

Both nodes carried the public group, allowing either to serve web traffic. The RTC ports matched what the ESS chart's MatrixRTC SFU exposed as NodePorts. The admin allowlist was in aws/terraform.tfvars (Bitwarden aws-tofu-tfvars).

Ingress — the important gotcha

The Traefik Service was type: LoadBalancer and remained <pending>: Talos on EC2 had no cloud-controller-manager to provision an ELB. Public traffic worked because the Traefik pod bound hostPorts 80/443.

Driven from talos-k8s/traefik/values.yaml. Three non-obvious settings, all load-bearing:

  • deployment.kind: DaemonSet. A single-replica Deployment is a SPOF whose rescheduling silently kills all ingress.
  • maxSurge: 0 / maxUnavailable: 1. hostPorts make the chart default (surge-then-delete) impossible — the new pod can never bind ports the old one still holds, so rollouts hang with the replacement Pending forever. Costs a few seconds of downtime per Traefik upgrade.
  • nodeSelector: workload-role=matrix. Running Traefik on every node was tried and reverted — see the incident note below.

DNS pointed at the worker's Elastic IP at the time.

Incident 2026-08-27: don't put workloads on an undersized control-plane

Running Traefik on both nodes tipped the control-plane over. Root cause was not Traefik: the node was an m6i.large (7.7Gi) running etcd, the API server and hive, whose agy/codex agent processes had grown into a 4Gi limit. Free memory hit ~650Mi and kube-scheduler and kube-controller-manager were OOM-killed (exit 137).

That deadlocks the cluster in a way that resists the obvious fix: with no scheduler nothing can be placed, and kubectl scale cannot relieve the pressure because actioning a scale-down is the controller-manager's job — and it was dead too. The scale-to-zero appeared to succeed and did nothing. Recovery required force-deleting the hive pods so the kubelet released memory.

Fixed properly by resizing the control-plane to m6i.xlarge (15.7Gi) — free memory went from ~650Mi to ~12Gi — and keeping an explicit memory limit on hive so the kubelet evicts hive, never the control plane.

2026-09-24: resized again to m6i.2xlarge. Three Hives on the control plane pushed m6i.xlarge to a load of about 85 on 4 vCPU. Every kubectl exec took 10–35 s, the ops CronJobs hung until their deadlines, and agents were slow to reach a ready prompt. The in-place resize through OpenTofu took about 3 minutes of Hive downtime; Matrix on the worker stayed up.

The node-bound local-path PVC prevented hive from moving to the worker, so the control-plane needed enough headroom for its workloads.

Former workloads

NamespaceWhatExposure
essMatrix (Synapse + workers, MAS, MatrixRTC, haproxy, redis, element-admin)matrix/auth/call/matrixadmin.reilly.asia
hivetuna-os Hive (SCHOOL) + Discord botsschool.tunaos.org (legacy alias hive.tunaos.org), Cloudflare-proxied
hive-reeftuna-os Hive (REEF)reef.tunaos.org
hive-hubHive hub — every Hive's status in one placehub.tunaos.org
postgresPostgres 16 — former production DB for Synapse + MASin-cluster only
defaultCFP review dashboard, searxngTailscale ingress
cert-managerLet's Encrypt via Cloudflare DNS-01—
tailscaleTailscale operator (ingress proxies)—

Former Postgres storage

Talos made the original plan to install Postgres on the host impossible. The cluster's production Postgres ran in the postgres namespace, with durability depending on local-path on the worker's root volume. A separate 50GB gp3 pgdata volume was attached at the EC2 level but not mounted by Talos.

Historical access

The old cluster is no longer reachable. Its kubectl/talosctl setup required endpoints = public EIP, nodes = private IP — not the same address for both. Talos does a two-hop call (client → endpoint → node); pointing both at the public EIP makes the node try to reach itself via its own Elastic IP from inside the VPC, which AWS's hairpin NAT does not support. The symptom is a maximally confusing dial tcp …: i/o timeout that looks like a raw network fault.

export KUBECONFIG=~/.kube/config-aws-migration
kubectl get nodes -o wide

# talosctl: endpoint = public EIP, node = private IP
talosctl --talosconfig ~/.talos/config-aws-migration \
  -e <node-eip> -n <node-private-ip> version

The configs were stored in Bitwarden as kubeconfig-aws-migration / talosconfig-aws-migration; the kube role still fetches them separately from the home cluster's files. They no longer provide access to this terminated cluster.

Historical cost context

The initial deployment used AWS credits, which have since been exhausted. See AWS account costs for the current caution.

Historical gaps

  • The 50GB Postgres volume was unmounted.
  • Nightly Postgres dumps were copied to S3 and size-verified; DLM snapshot schedules covered the root volumes. Retention of individual snapshots is not established by this historical record.
  • API-server OIDC and finer-grained RBAC were deferred; access used an admin client certificate.

See also

AWS cluster teardown, 2026-10-25

The AWS Talos cluster, its KubeVirt node and all their disks were deleted on 2026-10-25 because the account's credits ran out. This page records what is gone, what was saved and where, and how to rebuild.

Why

The account ran on one credit: AWS Open Source Promotional Credits, issued $1,435.00, valid 2026-08-01 to 2028-07-31. It is not monthly. It was consumed in under three months (Aug $108, Sep $550, Oct $932 applied). October was the first month with a net bill ($66.15 by the 25th, AWS forecast $307). At the footprint before the shutdown the cost was about $41/day gross.

Earlier text in this repo calls the credits "permanent" or "$1400/mo". That was wrong. $1,400 is the migration budget's ceiling, not an allowance.

State now

ThingState
aws-migration-worker-matrix, aws-migration-controlplane-hive, aws-migration-kubevirtterminated
Their EBS volumes (200, 400, 150 GB roots, 50 GB migration-pgdata)deleted
5 blank detached volumes created 2026-10-24 for an unstarted Hive-to-EBS migrationdeleted
EBS snapshots in eu-north-1 (35)deleted
punjab (t3.large, us-east-1), its 250 GB volume, one snapshotkept, running
Elastic IPs 13.62.161.5 (the matrix.reilly.asia target) and 13.63.243.56kept, unassociated
VPC, subnet, security groups, IAM, budgets, S3 bucketkept

Matrix (reilly.asia), the Hives (school., reef., hub.tunaos.org, hive.reilly.asia), the CFP dashboard and Postgres are offline.

aws/ (OpenTofu) was not changed. The instances were removed outside it, so just aws-plan will plan to create them again. That plan is the rebuild.

Where the data is

Bucket hanthor-fleet-backups-181185361136 (eu-north-1).

Cluster state: cluster-state-2026-10-25/cluster-state.tar.gz.age

Contains a Talos etcd snapshot (2,507 keys, includes every Kubernetes Secret: the MAS and Matrix secrets generated by ESS, tokens, etc.) and a YAML dump of 58 resource types. It is encrypted with age to five SSH keys: punjab, and the james@kanpur, himachal, james@Kerala and james@Jamess-MacBook-Air.local keys. Any one private key opens it:

aws s3 cp s3://hanthor-fleet-backups-181185361136/cluster-state-2026-10-25/cluster-state.tar.gz.age .
age -d -i ~/.ssh/id_ed25519 cluster-state.tar.gz.age | tar xz   # etcd.snapshot, resources/*.yaml

Round trip checked: the decrypted archive hashes identically to the original. The private keys' own backup is not verified from here (Bitwarden was locked).

Volumes: postgres/shutdown-export-2026-10-25/

ObjectContent
ess_ess-spindle-data.tar.gzspindle store (the live homeserver)
ess_synapse-media-preseed.tar.gz, ess_spindle-v0-1-0-rollback.tar.gzmedia, rollback store
hive-reef_hive-data.tar.gz, hive_hive-data.tar.gz, hive-hanthor_hive-data.tar.gzthe three Hives' data (about 43 GiB stored)
hive_discord-reports-data, default_review-data, hive-hub_hive-hub-data, hive_hive-ops-statesmall state

tar | gzip -1, storage class GLACIER_IR (retrieval about $0.03/GB). Each archive's tar size, recomputed on the node, equals the uncompressed size in its gzip trailer. They were not restore-tested.

Postgres: postgres/2026-10-25/

Final dump, 4.1 GB: Synapse and MAS.

Permanently gone (not exported)

Contributor homes (claude-home 46 GB, pi-codex-home 43 GB, agy-home 15 GB), the spindle-rehearsal volumes, raw pgdata (the dump covers it), hive-shared-auth (provider logins, sign in again), and everything that lived on the KubeVirt node (monitoring data, VM disks, wootc). Workload replica counts and cronjob suspend flags were saved as annotations and are inside the etcd snapshot.

Rebuilding (outline, untested)

  1. just aws-apply recreates the three instances from aws/. The Elastic IPs and private addresses are unchanged, so DNS and the Talos config still match.
  2. Restore etcd onto the new control plane so the Secrets and workload definitions come back, using the snapshot from the archive: talosctl --talosconfig ~/.talos/config-aws-migration -e <eip> -n 10.20.1.10 bootstrap --recover-from=./etcd.snapshot (the machine secrets come from the OpenTofu state, which is what the nodes were built with).
  3. Recreate each volume's directory under /opt/local-path-provisioner/ on the node that holds it (the PV objects name the directories) and untar the matching archive into it. A read-write hostPath helper pod in kube-system does this; other namespaces reject hostPath.
  4. Restore Postgres from the dump (see talos-k8s/backup/), then scale the workloads back up from the saved annotations tunaos.org/pre-shutdown-replicas and tunaos.org/pre-shutdown-suspend.

Cold start of the control plane from its stopped disk was exercised once during the teardown (the API server and etcd came back intact). Nothing else here has been run end to end.

Matrix

Retired 2026-08-27. The Matrix homeserver moved first to the AWS Talos cluster, terminated 2026-10-25, and now runs on the OCI VPS. Restart nothing on this retired host; the page below is historical. See the Matrix cutover runbook.

Former VPS node in the fleet (vps group in inventory.yml) that hosted the reilly.asia Matrix homeserver.

What it ran

  • Ubuntu 24.04 LTS, x86_64
  • Synapse + MAS on a single-node Kubernetes install, with PostgreSQL as the database
  • Managed by the fleet playbook (daily cron apply, no secrets/homebrew)

Retirement

Synapse and MAS were scaled to zero at cutover and must stay that way: two live homeservers sharing one federation identity is unrecoverable. Automatic upgrades on the box were disabled so nothing resurrects them.

Lessons carried forward

  • A single VPS with no swap and unmanaged journald growth was fragile for K8s; the former AWS cluster needed more headroom (see the OOM incident in its archived handbook).
  • reilly.asia DNS is on Cloudflare, which made the cutover a DNS change rather than a server move.

Telengana

Retired 2026-08-27. The tuna-os Hive moved from this host to the former AWS Talos cluster, terminated 2026-10-25. School, Reef and Hub now run on the OCI VPS. Restart nothing on this retired host; the page below is historical.

Former VPS node in the fleet (vps group in inventory.yml).

What it ran

  • Ubuntu 24.04 LTS, x86_64
  • A single-node k3s hosting the tuna-os Hive
  • Managed by the fleet playbook (daily cron apply, no secrets/homebrew)

Lessons carried forward

The box drifted from the fleet's hardening baseline (host firewall and fail2ban were never enabled, and Tailscale had been logged out, leaving admin access on the public interface). The former AWS cluster kept its admin APIs behind an allowlisted security group, and fleet servers use the server_hardening role.