Observability

Verified against v0.10.0 · cmd/parley/main.go, internal/api/router.go, internal/api/metrics.go, internal/api/client_address.go, internal/api/secevent.go, internal/store/users.go, internal/api/ics.go

This is a short page, and honestly so.

Parley writes structured JSON logs to stdout. There is no tracing. Every response carries an X-Request-Id (generated, or the inbound value from a trusted proxy). There is no HTTP access log either — the one exception is an opt-in LOG_LEVEL=debug line that reports how each request’s client address was resolved, described below. It is a proxy diagnostic, not an access log and not an audit trail: no method, path, status, user or credential appears in it, and nothing is retained or queryable.

A Prometheus exposition at /metrics is off by default. Turn it on with METRICS_ENABLED=true when you need capacity numbers a blackbox probe cannot see. The endpoint is unauthenticated; keep it off any public ingress.

Logs

One line per event, JSON, at a level set by LOG_LEVEL (debug, info, warn, error; default info). Collect them the way you collect any container’s stdout.

The single most useful line is emitted at boot:

{"time":"...","level":"INFO","msg":"boot settings","base_url":"https://parley.example.com",
 "cookie_secure":true,"allowed_ws_origin":"https://parley.example.com","port":"8080",
 "auth_mode":"oidc","trust_proxy_headers":true,"metrics_enabled":false,"trusted_proxy_cidrs":["10.42.0.0/16"]}

trusted_proxy_cidrs appears only when TRUST_PROXY_HEADERS=true, and lists the CIDRs this process actually parsed — the check that the value you set was understood, rather than quietly reduced to an empty list.

With AUTH_MODE=oidc, one more line follows shortly after boot, from a probe that runs in the background and never delays the listener:

{"level":"INFO","msg":"identity provider is reachable — discovery succeeded; the client ID
 and secret are not checked until a sign-in reaches token exchange",
 "issuer":"https://idp.example.com/realms/parley"}

An unreachable or wrong issuer produces the same line at WARN with the error attached. Either way it is a diagnostic only: it does not gate startup and it does not affect /readyz, because an identity provider that is briefly down should fail sign-ins rather than stop a standup already in progress. Note the scope — discovery proves the issuer answers. A wrong client ID or secret still surfaces only at token exchange, when someone signs in.

Diagnosing a proxy with LOG_LEVEL=debug

LOG_LEVEL=debug is otherwise quiet; its one substantial addition is a line per request from the client-address middleware:

{"level":"DEBUG","msg":"client address resolved","peer":"10.42.0.9:41234",
 "peer_trusted":true,"forwarded_for":["203.0.113.7","10.42.0.9"],"resolved":"203.0.113.7"}

• peer — the socket address the connection actually arrived from, before any rewrite.
• peer_trusted — whether that peer fell inside TRUSTED_PROXY_CIDRS. false means the X-Forwarded-For header was ignored entirely, which is the usual cause of “everyone shares one address” in the room-code throttle.
• forwarded_for — the parsed chain, empty when the header was absent, duplicated, empty or malformed.
• resolved — the address the rest of Parley will use.

Those four fields are the whole of it. Cookies, Authorization, session tokens and other headers are never logged, at any level. Turn it on, make one request, read the line, and turn it off again — it is per-request output, not something to leave on in production.

Almost every “it deploys but does not work” report — cookies not sticking, the WebSocket refusing to connect, sign-in redirecting to the wrong host — is visible in that one line. Read it first, before reading anything else.

Beyond boot and database errors, the application logs deliberately little: a handful of lines around the sign-in flow, session errors, and one security event line per action listed below. Parley never logs request paths, user names, cookies, tokens, passcodes, or session content at any level.

Security events

Each of the following actions writes one INFO line with a fixed schema. Ship stdout the way you already do; that is the audit trail. org_audit_log still records custody and plugin-admin actions in Postgres for an operator who queries it (#399 is a reader for that table), and those same writes also emit this line.

{"level":"INFO","msg":"security event","event":"space.create",
 "actor_user_id":"…","actor_subject":"open","org":"default","space":"platform-team",
 "target":"platform-team","outcome":"ok","client_addr":"203.0.113.7",
 "request_id":"edge-1/abc-000001"}

Field	Meaning
event	The action. See the table below.
actor_user_id	The signed-in user’s id, empty only when there is no principal.
actor_subject	The identity-provider subject in oidc mode; open for an anonymous account; guest for a redeemed signed link. Never an email.
org	Org slug when the route is org-scoped.
space	Space slug when the action names one.
target	The thing acted on (a slug, a user id, a link id, a plugin action). Never a token, passcode, cookie or body.
outcome	ok when the action completed.
client_addr	The same address the rest of Parley uses for the client — after TRUST_PROXY_HEADERS rewrite, never the raw socket when a trusted proxy is in front.
request_id	The X-Request-Id this request carried or was assigned.

event	When
auth.signin	Open-mode identity creation, or a completed OIDC callback
auth.signout	DELETE /api/me
space.create	A space is created
space.delete	An owner deletes a space
space.passcode.rotate	A member issues a new passcode
space.passcode.remove	A member opens the space
space.member.add	Someone joins a space
space.member.remove	An owner removes a member
space.member.role	An owner promotes or demotes a member
link.mint	A facilitator mints a guest link
link.revoke	A facilitator revokes a guest link
ics.mint, ics.revoke	A personal calendar feed is minted or revoked. The token is not on the line
ics.feed	A calendar fetch. target is /ics/[redacted]; the token is not on the line
space.claim, space.delete, space.add_owner, org.purge	The matching org_audit_log write
plugin.install, plugin.upgrade, plugin.upgrade_requested, plugin.upgrade_approved, plugin.enable, plugin.disable, plugin.uninstall, plugin.action, theme.install, theme.reset	The matching org_audit_log write

An inbound X-Request-Id is honoured only when the socket peer is inside TRUSTED_PROXY_CIDRS and the value is at most 128 bytes of printable ASCII (no newlines). Anything else is discarded and Parley generates an id. The response always echoes the id that was used.

What is missing, and what to do instead

Capability	Status	What to do instead
Structured JSON logs	Built in	stdout, one level, no rotation needed by the process itself — the single-server compose file caps the on-disk driver instead, see Deployment.
Liveness and readiness endpoints	Built in	/healthz and /readyz — see Architecture.
Metrics endpoint	Built in	/metrics is off by default. METRICS_ENABLED=true mounts an unauthenticated Prometheus exposition on the same port as the app — Go runtime, pgxpool stats, WebSocket count, listener reconnects, and the passcode and identity throttles. Keep it off a public ingress; scrape from the cluster network. The metrics themselves are listed below.
Distributed tracing	Not built	Single process, single database — a trace would have two spans. Proxy access logs give you the same latency picture.
Request IDs	Built in	Every response echoes X-Request-Id. An inbound id is honoured only from a peer in TRUSTED_PROXY_CIDRS, capped at 128 printable ASCII characters. Correlate proxy access logs with security-event lines on that field.
HTTP access log	Not built	middleware.Logger is not mounted. Use the proxy's access log — it is the same data, and you already ship it. LOG_LEVEL=debug logs only how the client address was resolved: no method, path or status.
Audit log	Built in	One stdout JSON security event line per sign-in, sign-out, space create and delete, passcode rotate and remove, member add, remove and role change, guest-link mint and revoke, calendar-feed mint, revoke and fetch, and every org_audit_log write. Schema: event, actor_user_id, actor_subject, org, space, target, outcome, client_addr, request_id. Cookies, tokens, passcodes and request bodies are never logged. A calendar fetch's path is rewritten to /ics/[redacted] before the line is written. The Postgres table remains write-only — no read API or screen (#399).
Error reporting integration	Out of scope	Panics are recovered and logged; scrape them from stdout.

Background goroutines

Parley does run a handful of background goroutines, started from cmd/parley/main.go and internal/api/router.go. With METRICS_ENABLED=true, the listener’s reconnects are also a counter; the others are still judged by their log line.

• Session notification listener (router.go, a.listen) — holds a dedicated Postgres connection open for LISTEN/NOTIFY fan-out between replicas. Healthy: silent. Stuck: repeated "session notification listener dropped, reconnecting" at ERROR with a growing backoff, which means the replica cannot re-establish its listen connection to Postgres. parley_listener_reconnects_total counts each of those drops.
• Presence sweeper (router.go, a.sweepPresence) — clears stale presence rows on a fixed interval. Stuck: "could not sweep stale presence rows" at ERROR on every tick.
• Plugin retention (cmd/parley/main.go, plugins.RunRetention) — prunes old plugin events and reconciles quota counters hourly. Healthy: an occasional "plugin retention pass" INFO line when it actually prunes or corrects something. Stuck: "plugin retention failed; will retry on the next tick" at ERROR.
• Session token sweeper (cmd/parley/main.go, sessionSweeper) — deletes expired session_tokens on the same hourly cadence. A row stops resolving the moment it lapses; the sweep is what actually removes it. Stuck: "session token sweep failed" at ERROR ("session token sweep failed; will retry on the next tick").
• Plugin outbox worker (internal/plugin/outbox.go, started from runtime.Start when PLUGIN_DIR is set) — drains plugin_deliveries and hands events to installed plugins. Stuck: "plugin outbox failed; will retry on the next tick" at ERROR, or "plugin outbox has no Deliver handler configured; deliveries will accumulate undelivered" at WARN if the plugin host never wired a handler.
• Plugin job worker (internal/plugin/jobs.go, started alongside the outbox worker) — claims and runs queued plugin jobs. Stuck: "plugin job queue failed; will retry on the next tick" at ERROR, or "plugin job queue has no Run handler configured; jobs will accumulate unrun" at WARN.

The plugin outbox and job workers only start when PLUGIN_DIR is configured; an instance with no plugins never runs them.

/metrics

Off until METRICS_ENABLED=true. Then GET /metrics returns Prometheus text format, with no session cookie and no CSRF check — the same posture as /healthz. Anyone who can reach the process can scrape it.

The chart’s Ingress uses path: / with pathType: Prefix, so turning metrics on without a further deny publishes /metrics on the public hostname. Do not do that. Scrape the Service from inside the cluster, and keep the path off the ingress with a proxy deny (see Reverse proxy) or by not routing it.

There is no dedicated metrics port. A NetworkPolicy cannot filter by path, so the way to admit a scraper is to admit its pods to port 8080 and nobody else’s. With networkPolicy.enabled=true:

networkPolicy:
  extraIngress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app.kubernetes.io/name: prometheus
      ports:
        - protocol: TCP
          port: http

An extraIngress entry with no from admits every source on that port and defeats the default-deny policy. Always name the scraper.

Process and pool

Metric	Kind	What it is
go_goroutines	gauge	Go runtime goroutines, from the standard Go collector. The rest of that collector (go_memstats_*, go_gc_*, go_sched_*, …) is exported too.
parley_pgxpool_total_conns	gauge	Connections the pool currently holds.
parley_pgxpool_idle_conns	gauge	Idle connections in the pool.
parley_pgxpool_acquired_conns	gauge	Connections currently checked out.
parley_pgxpool_max_conns	gauge	Configured pool cap.
parley_pgxpool_constructing_conns	gauge	Connections the pool is opening.
parley_pgxpool_acquires_total	counter	Successful acquires.
parley_pgxpool_acquire_seconds_total	counter	Time spent waiting for a connection.
parley_pgxpool_canceled_acquires_total	counter	Acquires canceled before a connection was available.
parley_pgxpool_empty_acquires_total	counter	Acquires that waited because the pool was empty.
parley_pgxpool_empty_acquire_wait_seconds_total	counter	Time spent waiting on an empty pool.
parley_pgxpool_new_conns_total	counter	New connections the pool has opened.
parley_pgxpool_max_lifetime_destroys_total	counter	Connections closed for exceeding max lifetime.
parley_pgxpool_max_idle_destroys_total	counter	Connections closed for exceeding max idle time.

Pool numbers are read from pgxpool.Stat() on scrape, not sampled on a timer. They describe this replica’s pool only.

Application

Metric	Kind	What it is
parley_ws_connections	gauge	WebSocket connections this process currently holds. Per replica, not cluster-wide.
parley_listener_reconnects_total	counter	Times this replica’s LISTEN connection dropped and began reconnecting.
parley_passcode_throttled_total	counter	Times a room-code, invite-handle or link-token guess was refused because the caller’s budget was spent.
parley_identity_throttled_total	counter	Times open-mode identity creation or link redemption was refused by the hourly identity quota.

Request rate and latency still come from the reverse proxy, which already sees every request. /readyz remains the blackbox availability probe.

Is that enough?

For this shape of service, mostly yes. Availability is still is the process up and can it reach Postgres, answered by probing /readyz from outside — a stuck background goroutine still lets /readyz pass, so a person also has to be watching the log lines above, or parley_listener_reconnects_total for the fan-out listener.

What /metrics adds is the capacity picture that used to require querying the database: WebSocket count on this replica, pool saturation, and whether the room-code or identity throttles are firing. If you are running Parley for a team, you can leave it off. If you are running it for an organisation, turn it on and scrape it from inside the cluster.