Troubleshooting

Everything here is something you can act on from the product or the API. Deeper operator material — alert thresholds, database-level recovery, deploy rollback — is in docs/operations.md in the repository.

Two pages answer most questions before you go any further:

• /healthz — what this deployment is running and how the queue is doing. Public, no authentication.
• GET /v1/org/queue and GET /v1/org/extraction — the same picture, fenced to your org.

Ingest is returning 429

There are two, and they mean opposite things. Read the error field before changing anything.

backlog_full

{ "error": "backlog_full", "pending": 51234, "retryAfterSeconds": 6 }

The deployment is saying "not right now". More is arriving than extraction is draining, and refusing is the designed response — accepting into an unbounded backlog is how a busy afternoon becomes tomorrow's outage. The response carries a Retry-After header.

What to do: honour Retry-After and resend. The event is not lost; it was never accepted, so resending the same idempotencyKey is correct and safe.

If it does not clear:

1. GET /v1/org/queue — is drainedPerMinute above zero? If it is, you are ahead of the drain rate and it will clear on its own.
2. If drainedPerMinute is near zero while pending climbs, extraction is stalled rather than slow. That is an operator problem — the model endpoint, the worker processes, or an org over its token budget. Escalate with the queue numbers.
3. If you are backfilling history, slow down and use POST /v1/ingest/batch (up to 500 events per request) rather than one request per message. A connector that opens a socket per message spends its time on sockets.

quota_exceeded

{ "error": "quota_exceeded", "reason": "monthly ingested events limit reached: 200,000 of 200,000" }

The account is saying "not this month". There is deliberately no Retry-After, because retrying changes nothing until the limit is raised or the month rolls over.

What to do: check GET /v1/org/usage for month-to-date against limits, and raise the limit or wait. Do not retry in a loop. Treating these two 429s alike is how a connector either hammers a limit it can never pass or abandons a backlog it should have waited out.

The same code appears on /v1/ask and /v1/answer against the monthly answer limit. Agents query far harder than people do, so this is usually the one that trips first.

too_many_attempts

{ "error": "too_many_attempts", "retryAfterSeconds": 240 }

Not ingest — this is the login, signup, invite or password-reset rate limiter. Wait out the window.

An answer is missing something we know is in there

This is the important one, because there are two completely different causes and they look identical from the outside. Establish which it is before changing anything.

Cause 1: it was never extracted

The knowledge never entered the memory. Check GET /v1/org/extraction:

{
  "claims": { "asserted": 812, "hedged": 44, "ended": 17 },
  "messages": {
    "withClaims": 903, "noClaims": 2140, "memories": 388,
    "gated": 5510, "quarantined": 2, "budgeted": 0, "errors": 11
  }
}

Also check whether it has been processed yet. GET /v1/org/queue reports behind as a human string:

{ "pending": { "live": 0, "bulk": 12400, "total": 12400 }, "active": 8, "behind": "2.4h" }

A backfill that is two hours behind is not missing data. It has not arrived.

Cause 2: it is not visible to you

The claim exists and your principal cannot see it. This is the system working, and it is by design that it looks the same as an absence. Anything else leaks the existence of what it is hiding.

Check, in this order:

1. Is your identity linked? A person's containers come from their source identity — the Slack, Teams or custom-source user their principal is linked to. An unlinked principal reads nothing, and reads nothing quietly. GET /v1/auth/session shows your principal id; GET /v1/org/overview reports visibility.containers — the number of containers you can read. Zero there is the whole explanation.
2. Was the permission ever sent? Content and permissions travel the same ingest path. A connector that streams messages but never sends membership.change events produces a corpus nobody can see. For a custom source, check that each event carries permissions.containerTuples.
3. Is the permission subject namespaced? Tuple subjects must be user:<connector>/<sourceUserId> — user:slack/U0442, not user:U0442. An unprefixed subject is accepted by SpiceDB and then never matched by the readable-set projection, which reads exactly like a missing permission.
4. Is the tuple synced? GET /v1/org/overview reports sync.unsynced and sync.oldestUnsyncedSeconds. Unsynced tuples fail closed. A number that stays above zero for minutes rather than seconds is an operator problem.

Do not widen visibility to close a support ticket. It is the one change that cannot be undone quietly. If someone should be able to see a container, add them to it at the source and let the permission flow through.

A third case: the claim is there but no longer current

Not missing — superseded or expired. GET /v1/entities/:name/timeline shows the full visible history including struck rows, and ?asOf= on /v1/entities/:name/claims and /v1/answer answers as of an instant.

Two behaviours that surprise people and are correct:

• A claim can be current for you and superseded for a colleague. If the superseding claim lives in a container you cannot read, your version stays current and unstruck — showing it struck would reveal that a replacement exists.
• Speculation never wins. "We might move them to closed lost" is stored as hedged, stays visible, and does not displace an asserted value.

Answers are slow

Typical shape: retrieval is milliseconds and the model call is seconds. A cold question that the planner cannot route costs a model call plus a retry.

1. Check /healthz. reader tells you whether the model path is even on. queue.pending tells you whether the box is busy extracting.
2. Is it every answer or one question? Consistent slowness across all questions points at the model endpoint. One slow question usually means the planner could not compile it and it fell through to retrieval plus a read.
3. /v1/answer is the faster surface. It runs no model for questions it can compile into a plan. /v1/ask always calls a model. If you are building an agent, /v1/answer is the one to build on — let your own model write the prose.
4. Long questions with many entities cost more. The evidence set grows and the read grows with it.

If you are seeing 502s rather than slow answers, that is not a louvain refusal — it is a proxy or edge timeout in front of the API cutting the connection while the answer is still being composed. Report the elapsed time; the API's own request budget is derived from the reader timeout, so a 502 at a suspiciously round number of seconds is an edge configuration, not the answer path.

Operators have one immediate lever: LOUVAIN_READER=off degrades to deterministic planner answers. Coverage drops, correctness does not.

A connection stopped

Start at GET /v1/org/connections. Each row carries status, last_event_at, last_error, a runner state and stats.

{
  "id": "c1d2…", "connector": "slack", "name": "Acme workspace",
  "status": "error",
  "last_event_at": "2026-08-14T22:41:00Z",
  "last_error": "runner exited 1 after 4 attempts",
  "runner": { "running": false },
  "stats": { "events": 18420, "claims": 903, "errors": 2 }
}

status: "error"

The runner crashed three times in a row. The connection flips to error loudly rather than crash-looping, last_error carries the reason, and the manager keeps retrying it on a doubling schedule — 1, 2, 4 … minutes, capped at an hour — until a run stays up for five minutes or you pause it. next_attempt_at and attempts say where it is in that schedule, and the page shows it as "Error · retrying in 12 min". A transient failure (broker outage, API restart, a token the broker refreshes later) clears itself; the causes below need you.

Most common causes, in order:

• A revoked or expired credential. Re-enter it: PATCH /v1/org/connections/:id with new credentials. Credentials are sealed and cannot be read back, so re-entering is the only path.
• A rotated ingest token. POST /v1/org/connections/:id/rotate-token invalidates the previous one immediately. Anything still pushing with the old token gets 401 invalid_ingest_token.
• A path that no longer exists, for one-shot import connectors.

Set status back to enabled after fixing it; the manager reconciles every few seconds and will start the runner again.

status: "disabled" on a one-shot connector

Expected. An import runs once and returns to paused — not running is shown as not running rather than as a permanently "enabled" connection doing nothing.

running: true but last_event_at is old

The runner is alive and the source is quiet, or the runner is connected to the wrong scope. For Slack specifically: the bot only sees private channels it has been invited to (/invite @louvain). Public channels are joined automatically unless auto-join was turned off at setup.

An http inbound connection that shows no runner

Correct — mode: "inbound" connectors have no runner by design. They are a credential, an org fence and a rate limit; your system pushes to /v1/ingest. runner: { "running": false } is not a fault here. Look at last_event_at and stats.events instead, and at whatever is doing the pushing.

Events accepted but nothing appearing

Check GET /v1/org/activity, or the Activity page. It shows recent ingest with its outcome and claim count — metadata only, never message text. An event with a terminal outcome and zero claims was processed and yielded nothing typed; an event still pending has not been processed yet.

To see what an event actually wrote, expand its row on the Activity page (or call GET /v1/org/activity/:id): the claims, connections, memories and source passages behind it, fenced by the same container permissions as the Browse page. A member who cannot read the event's room is told so rather than shown an empty list, so "nothing under this row" and "nothing you may see" are never the same message.

Events that failed extraction

curl https://louvain.example.com/api/v1/org/queue/failures \
  -H "authorization: Bearer louvain_EXAMPLE_TOKEN_NOT_REAL"

Returns up to 100 events that exhausted their retries, each with attempts, the error, and a 140-character preview so you can tell which message it was.

The usual reason a batch of these exists is an outage that has since been fixed. Put them back on the queue:

curl -X POST https://louvain.example.com/api/v1/org/queue/retry \
  -H "authorization: Bearer louvain_EXAMPLE_TOKEN_NOT_REAL"

{ "requeued": 214 }

Both routes require owner or admin. The retry is org-scoped — it never touches another tenant's events.

API errors that are configuration, not faults

404 sso_not_configured on any /v1/auth/sso/* route means this deployment has no identity provider configured, not that your provider is broken.

Something changed after a deploy

/healthz reports release and environment, so "is the fix deployed" is answered by asking the process rather than by reading a dashboard. It also reports extractor, reader, sealer, signup and any loaded extensionRoutes — an enterprise module that silently failed to load would otherwise look exactly like everything being fine.

If extractor reads rule, that deployment is running the test fixture, not a model. It captures roughly one fact in eight and reports no error at all.

Extraction is using a model you did not configure

/healthz reports model: the model actually extracting, read from the running process rather than from any file. Compare it with what you believe you set.

A process keeps the environment it started with, and a shell export beats the env file — so a long-lived process can be running a model that no file on disk mentions. Restarting is what makes an edited env file take effect; until then the file describes an intention and /healthz describes reality.

This matters beyond tidiness. Different models extract differently, so a corpus ingested across a model change is not one corpus — comparing a before against an after measures the model swap as much as whatever you were trying to measure. GET /v1/org/extraction breaks usage down per model, which is how you tell whether a workspace was ingested by one model or several.

Still stuck

Collect these before asking for help. They contain no message content, so they are safe to paste:

• the full /healthz body,
• GET /v1/org/queue and GET /v1/org/extraction,
• the status, last_error and runner fields of the affected connection,
• for an answer problem: the question, the plan and via fields from /v1/answer, and whether retrievalDegraded was present.

If the conclusion is "this person cannot see it", that is an answer, not a bug.