/Docs · Louvain Engine

Limits and known defects

Every bound, and the defects the tests record

In short: the table lists every bound and the setting that moves it. A leg or sidecar that misses its bound costs only that leg, and the page names it in degraded.

Bounds#

WhatBoundSetting
A file a reader is handed32 MB, regular files onlyLOUVAIN_FILE_MAX_BYTES
Text a reader returns1,000,000 charactersLOUVAIN_FILE_MAX_CHARS
Time a reader takes on one file60 secondsLOUVAIN_FILE_READ_DEADLINE_MS
An item's key1,000 characters
An item's text2,000,000 characters
An item's title, container, author, app, revision1,000, 500, 200, 60 and 200 characters
An item's file path, URL4,096 characters each
A collection's name100 characters: letters, digits, dashes, underscores
An HTTP ingest request8 MB and 500 items; one at a time, another is answered 429
A token (read or ingest)at least 32 charactersLOUVAIN_ENGINE_TOKEN; the http() source's token
A folder run's removalsmore than 10 items and more than half the collection need --yes
A collection's ingest lease5 minutes, renewed while a run works
A records or request tombstone90 days
A tool call's reply60 secondssettings.server.toolDeadlineMs in engine.config.ts
A ranked page24 documents, the first 10 in full
A compact hit's excerpt200 characters
A list-view page1 to 100 documents, 50 by default
exclude200 ids
Documents that lead as what a search names20
Keys of one search looked up3
outsideWindow5 documents
A person's question beside a search2,000 characters, the rest cut
How deep a leg reads3 passages per document asked for
A page of louvain_read_source40,000 characters
louvain_lookup_source20 documents that are it, 1 to 50 mentions (20 by default), an identifier of 4,096 characters
A read statement15 seconds
A keyword sidecar call3 seconds
An embedding call30 seconds, 180 on the bulk lane
A rerank call4 seconds (30 on the shipped, commented local reranker line)timeoutMs in engine.config.ts, which reads LOUVAIN_RERANK_TIMEOUT_MS

To move a bound, set its setting in .env and run ./engine down, then ./engine up.

Limits of this build#

  • Grants are whole collections. A reader reads the collections engine.config.ts grants it, never a single document of one, and the engine's own token reads everything (Permissions).
  • A reader that blocks cannot be stopped. The bounds run in the engine's process; a reader that loops without yielding holds it.
  • The match floor is known for one embedder. A document matches by meaning only above a floor measured for Xenova/bge-base-en-v1.5 (0.57). With another model, documents match by their words, identifiers and figures alone.
  • A changed embedding model needs a fresh database. Vectors from different models share no space.
  • Loading has no bulk mode yet. Documents load in batches, but each passage is added to the meaning index as it is written (about 1.2 ms a passage) and each record is accepted on its own (about 4 ms), measured on an Apple M1 Max: about 80 records a second with embedding left aside, 5 a second with the CPU embedder (Configuration). Building the meaning index once after a large first load, and accepting a batch's records together, would load faster.

Known defects#

Three defects are recorded as todo tests, each stating what will hold once fixed. Run the search ones with pnpm --filter @louvain/engine test:db (Operations) and the third with pnpm --filter @louvain/engine test; each reports as todo, not a failure.

DefectTest
A superseded revision can show as a hit. Words only an earlier revision holds can rank the document with that revision's date and excerpt, unmarked. This happens with revisions written as events with no record key; a file revised in a folder and loaded again has its earlier revision withdrawna superseded revision is never a ranked hit; case a revised document's earlier words alone (packages/engine/test/db/search.cases.ts)
A fact deep in a long document, asked in other words, falls out of the first ten. A document ranks by its best passage, so a lone passage that shares no word with the search can rank lowa fact deep in a long document is in the first ten; case a fact deep in a long document, in other words (packages/engine/test/db/search.deep.cases.ts)
An unauthenticated caller can tell which of the app's paths exist. The sign-in route and the app's public page answer before a token is checkedtells an unauthenticated caller nothing about which paths exist (packages/engine/test/open-surfaces.test.ts)

We use Google Analytics, and Google’s and LinkedIn’s advertising tags, to see how this site is used and which ads bring people here. They set cookies only if you accept. Cookie Notice.