Runbook

docker compose up fails with set POSTGRES_PASSWORD in .env

Copy .env.example to .env and set a password. This is required on purpose; there is no default database password.

A permanent “reconnecting” banner

The WebSocket origin check is rejecting your browser. Set BASE_URL to the exact address in your address bar — scheme, host and port — and restart. Behind nginx, confirm the Upgrade and Connection headers from the reverse proxy config.

You set a name, but every refresh forgets you

BASE_URL is https while you’re browsing over plain http, so the browser drops the Secure cookie. Serve over HTTPS, or set an http BASE_URL.

A pod is Running but never Ready (0/1)

The container is up, so this is /readyz saying no. Ask it which of the two things it checks is failing — the body is the answer:

kubectl port-forward pod/<the-pod> 8080:8080 &
curl -s localhost:8080/readyz

(The image is distroless — no shell and no curl inside it — so port-forward rather than kubectl exec. kubectl describe pod quotes the same body on the failing probe event.)

Port-forward reaches the probes, not the app

/healthz, /readyz and /version need no session, so they answer over a port-forward. The browser app will not work through one, and this is not a bug to chase:

• A production BASE_URL is https, which makes the session cookie Secure, so the browser never sends it to http://localhost:8080 — every API call comes back unauthenticated.
• The WebSocket fails before the origin check even runs: /ws reads the principal first and answers 404 when there is none.

To exercise a real session over a port-forward, drive it as a raw client with a session cookie copied from a real browser session and the deployment’s own origin:

kubectl port-forward pod/<the-pod> 8080:8080 &
websocat "ws://localhost:8080/ws?session=<session-id>" \
  -H "Origin: https://parley.example.com" \
  -H "Cookie: parley_session=<copied-from-the-browser>"

That confirms fanout end to end. Treat the cookie as a live credential: it is one.

• database unreachable — Postgres is not answering this pod within 3 seconds. Check the database, the NetworkPolicy or security group between them, and that the sslmode on DATABASE_URL matches what your Postgres offers — boot echoes the resolved mode as db_sslmode, and refuses a plaintext-capable one unless DATABASE_ALLOW_PLAINTEXT=true.
• not listening for session changes — Postgres is fine, but this replica has lost the LISTEN it hears other replicas on. It reconnects with backoff on its own and logs each attempt, so a pod that clears on its own after a database blip is expected; one that never clears is a connection the database is refusing (max_connections, a pooler that does not pass LISTEN through — PgBouncer in transaction mode does not).

Readiness covers the listener deliberately: a replica that cannot hear the others still serves requests and still holds its WebSockets, it just never learns that anything changed elsewhere — invisible from outside unless the probe says so.

A pod that is not Running is a different question: CrashLoopBackOff with a FATAL: log line is a database prerequisite, and Kubernetes lists them.

/readyz fails but /healthz is fine

Usually the app is up and Postgres isn’t. Check the database container and DATABASE_URL. This split is deliberate: liveness never touches the database, so a blip can’t restart the process and drop every open session.

If Postgres is healthy, the other cause is that this instance lost the Postgres LISTEN it uses to hear about session changes made by other instances. It reconnects with backoff on its own, and logs each attempt. Readiness covers it because an instance that cannot hear the others still serves requests and still holds its WebSockets — it simply never learns that anything changed elsewhere, which is invisible from outside unless the probe says so.

Which version is this instance running?

Ask it. /version is unauthenticated and does not touch the database, so it answers even while Postgres is down:

$ curl -s https://parley.example.com/version
{"version":"0.10.0"}

The same string is on the container’s command line — parley -version — and in the startup log line. dev means the binary was built without a version stamp, which is what a local go build produces; a released image reports the tag it was built from, with no v prefix. Use this after an upgrade to confirm the rollout actually replaced the pods, rather than trusting that it did. See the HTTP API reference.

“That passcode doesn’t match this space”

Codes are six characters and case-insensitive; spaces and hyphens are ignored. After eight wrong tries from one address, wait a minute.

Everyone gets locked out of a space after a few wrong guesses

The throttle is counting your whole team as one client because TRUST_PROXY_HEADERS is false behind a proxy. See Configuration.

Nobody can start the standup

The rotation is built from whoever has the session open. Everyone joins first, then the facilitator starts.

A pod hangs at boot before serving

Every pod takes a blocking advisory lock before running migrations, so pods that boot together serialize. A pod waiting there is normal for as long as another pod’s migration takes. A pod waiting indefinitely means something else is holding that lock — look for a stuck backend:

select pid, state, query, age(now(), state_change)
from pg_stat_activity
where pid in (select pid from pg_locks where locktype = 'advisory');

Parley releases the lock on every exit path by closing the connection that holds it, so a survivor here is a backend the database never reaped. See Scaling and limits.

A revocation during a listener reconnect

A token revocation (logout, session kick) notifies other replicas over pg_notify, and Postgres does not queue a notification for a session that is not currently listening. If a replica’s session-notification listener is mid-reconnect — see Observability for the log line — when the revocation fires, that replica simply never sees it. This is expected, not a bug to chase: the affected socket stays connected until its next revalidate poll, at most hub’s maxRevalidate (30 seconds), which closes it as if the notification had arrived. If a revoked session outlives 30 seconds on some replica, look for a listener stuck reconnecting on that replica rather than a broken revocation path.

A migration is half-applied

Each migration file’s DDL and its row in the migrations table commit together in one transaction (internal/db/migrate.go), so a pod killed mid-migration rolls that file back entirely — the next pod to take the advisory lock finds it still unapplied and reruns it cleanly, rather than finding a half-built schema. internal/db/migrations deliberately avoids create index concurrently for exactly this reason: it cannot run inside a transaction, so a migration that needed it would fall outside this guarantee (see the comment in 0024_org_members_user_idx.sql). What can leave the fleet stuck is a migration that fails partway through on one replica and never rolls forward — check select version, name, applied_at from migrations order by version desc limit 5 against the actual schema before assuming a stalled boot is anything other than the advisory lock being held by a slow, still-running migration on another pod.