2026-11-16/17: slow sync, missing avatars, Element X image errors
Symptoms on matrix.reilly.asia: sync felt slow, profile and room pictures were missing, and Element X
showed "Sorry, an error occurred" for every image. Four separate faults, plus host load.
The general fault guide (checks and fixes for any Spindle operator) is in the Spindle repo:
docs/troubleshooting.md.
What was wrong and where it was fixed
| Fault | Symptom | Fix |
|---|---|---|
Element X 26.11 requests media at /_matrix/client/unstable/org.matrix.msc3916/media/…; Spindle served only /v1/media | Every image and avatar 404s (route="unmatched" in metrics); Element X error dialog | Caddy rewrite (live now, also in this kit's Caddyfile); Spindle #940 serves the prefix itself |
Federation media answered with a Location header was cached as 0 bytes | Beeper avatars blank forever | #933, in v0.1.5 |
CDN labels images application/octet-stream | M_UNSUPPORTED: no thumbnails for application/octet-stream | #937, in v0.1.6 |
| Partial-state (faster-join) rooms never finished resync: one bad member signature, or one auth event the resident refuses with 403, failed the whole room; keys were fetched one server at a time | Newly joined big rooms have no member list or member avatars; /joined_members hangs | #935 (v0.1.5), #938 (v0.1.6) |
Not Spindle bugs, still open:
- A browser tab looping on a dead token. Firefox at
https://hanthor.github.iosent sync/whoami/keys ~20 req/s, all401 M_UNKNOWN_TOKEN, after the MAS → built-in OIDC switch on 2026-11-16. Sign out and back in (or close it); the client should have logged itself out. - Host contention. 2 OCPU / 12 GB, no swap; the School and Reef Hives use ~2.6 GB and heavy CPU/disk. Spindle's first sync after a restart takes up to a minute while caches warm. Moving the Hives off this host is the biggest remaining speed-up.
synctapstill proxies sliding sync for@james(0.5 CPU cap). Remove the service and the@ssyncCaddy handle when that diagnosis is done.
Live state left behind (as of 2026-11-17)
compose.yamlrunsghcr.io/tuna-os/spindle:v0.1.5(backupcompose.yaml.bak-pre-v0.1.5-*). Bump to the release that contains #940, then delete the@msc3916rewrite.Caddyfilehas two temporary additions, each with a comment:- the
@msc3916rewrite (keep until #940 is deployed); - an access log to
/data/access-diag.loginside the Caddy container (./caddy/access-diag.logon the host). Headers andaccess_tokenare stripped. Remove thelogblock when done; the file grows.
- the
@claude-test:reilly.asiais a non-admin test account, joined to#fedora:fedoraproject.org. Its password is in~/.spindle-testuser-pass(mode 600) on the VPS. Deactivate it when it is no longer needed.
Diagnosing on this host
- Metrics:
docker exec spindle-caddy-1 wget -qO- http://spindle:9100/metrics.spindle_http_requests_total{route="unmatched"}counts paths Spindle does not serve. docker logs spindle-caddy-1shows nothing new: the container's json log file was deleted from disk. Log to a file in/datainstead (above).- Editing
Caddyfileorspindle.toml: both are single-file bind mounts.sed -iand most editors replace the inode, and the container keeps reading the old file. Write in place (cat new > Caddyfile, or Pythonopen(p, "r+")+truncate), or restart the container. Check withdocker exec spindle-caddy-1 cat /etc/caddy/Caddyfile. - Seeing what a client requests:
sudo tcpdump -i any -A "tcp port 8008 and host <spindle container IP>"shows plaintext proxy→Spindle traffic (it includes bearer tokens; do not save it). - Partial-state rooms:
docker logs spindle-spindle-1 | grep -E "resync|partial-state". - Test account while registration is closed: add
shared_secretunder[registration], restart, register via/_synapse/admin/v1/register(Synapse HMAC), remove the secret, restart. Details in the Spindle fault guide.
Releasing a Spindle fix to this host
release.yml (dispatch with bump=patch) tags main only after every check on the commit has finished,
including the Matrix compliance suite (~1 h after merge). Dispatching earlier logs "CI is still running" or
"The Rust quality gate has not passed" and builds nothing. Then:
cd /srv/spindle
cp compose.yaml compose.yaml.bak-pre-vX.Y.Z-$(date -u +%Y%m%dT%H%M%S)
sed -i 's#tuna-os/spindle:vOLD#tuna-os/spindle:vNEW#' compose.yaml # compose.yaml is not bind-mounted
docker compose pull spindle && docker compose up -d spindle