Curation loop design
This page is the design of the curation loop, which turns what people ask into definitions
an organization stands behind. Searches that find nothing become knowledge gaps. A gap, or a
query a team runs often on its own ClickHouse, becomes a drafted definition in the learnings
queue. A definition an admin promotes into the shared layer is governed: when a query names
its term, memory.search serves it above every other hit, a derived aggregate’s count
aside. The parts land in this order: knowledge gaps (recording, grouping and the list),
governed definitions (the shared-layer changes and the term leg), then
knowledge.gaps.draft and query history, which queue definition drafts and so need the
review path the second part builds. Knowledge gaps are built and described in knowledge
gaps. A definition’s write-time rules are in
conflicts and refusal; its promotion and
term-search rules are in the shared layer
and governed definitions in search.
Definition drafting, query-history sources and private review are described in
definition drafts.
Governed definitions
A definition is a memory row with memory_type = 'definition' under a definition/ key.
The key does not encode the term: the review path picks it (below), a direct caller
chooses it, and it stays the same through every revision. At promotion, names decide
whether two definitions clash: the name check below keeps two shared definitions from
holding one name. At write time, rows stay told apart by key, as for any key-identified
type.
A name is compared in its folded form: format characters (Unicode category Cf, such as a
pasted byte-order mark) dropped, then canonical_document_text (server/core/hashing.py:
NFKC, trimmed, every run of whitespace made one space), then str.casefold. One helper in
server/memory/domain/definitions.py folds a name, and its unit tests pin each step with
a table. The folded form decides whether two names are one (SAT, Sat and sat are);
matching against a query keeps the case rule of the term leg (governed definitions in
search).
A definition’s value is a JSON object, which memory.set stores whole in value and in
value_json (the embedding reads its excerpt, as for any row):
| Field | Meaning |
|---|---|
term |
The name as people write it. |
aliases |
Other names for the same thing (an abbreviation, an older name). |
body |
The definition’s text. |
rests_on |
What it is based on: for query history, the SQL and its tables and columns. |
The admin who promotes a definition answers for it (promoted_by, stamped from the
token); the value names no owner, so it holds no person’s name for an erasure to miss.
The term leg serves a definition only while it is governed (the shared
layer,
server/memory/domain/definition_check.py), checked when the leg reads it. The source is
read from the memory authority (Postgres when it holds live memory, storage
authority), and a source whose head cannot be read
fails closed: the definition is not served. A read-time check covers every path into that
state that it can see, rows promoted before this design and through N-1 included; what it
cannot see is named under rolling upgrades. The shared layer already is the namespace’s
admin-approved copy of a fact (the shared
layer):
memory.promote and memory.demote need admin, the shared row records its source
(promoted_from) and the candidate it came from (promoted_from_candidate_id),
memory.lineage walks it back, and every build knows it. So governance needs no table or
state of its own. An edit, a relabel to medium or high or a tombstone of the source,
by any writer or worker on any build, ends governance while the source differs from what
the admin approved; a copy that keeps the text (the entity pass, a touch) keeps it, and a
source that returns to the approved state (a restore, an edit reverted byte for byte) is
governed again with no new promotion. Search withholds a copy whose source is gone or
labelled medium or high (the shared
layer);
a relabel back serves it again. memory.demote ends governance; the demoted source stays
a plain namespace-wide row, which ordinary search still serves until the admin deletes it,
and a later draft of its term is a new term under a new key. Only memory.promote
promotes into loom.shared (the op and learnings.review’s path through it; demote,
erasure and a redact only retire or scrub there, as the changes below say), and no worker
promotes on its own. memory.definitions.list shows each definition’s verdict; a
definition that loses a name to another is still served for the names it keeps, and one
past the ceiling is not read. A changed text is promoted again. On N no automatic verdict
retires or marks a definition’s source (the exemptions below, and the negative-outcome
quarantine of memory.outcome, which skips a definition’s namespace-wide row as it skips
a locked one; memory.outcome_run records harm events and quarantines nothing), so a
source that is no longer active was rejected by a writer, or left that way by N-1 or
before this design. The sensitivity sweep can still relabel a source, which lapses it as a
relabel does. The admin brings it back by the way it lapsed:
- retired automatically:
memory.restore(memory.auto_retirementslists such retirements); - marked
conflicted:conflicts.resolve; - marked
duplicate:memory.dedup_resolve; - rejected with
memory.reject, or replaced by a writer’smemory.copy_to(which resets the row’s pin and lock), a writer’s explicit act like an edit: written again; - past its
expires_atorvalid_to(a row written before this design, or by a writer): written again withclear_expiryand novalid_to, or by reviewing a pending draft of the term. Review uses the retained source’s content hash as a write precondition; unpin or unlock the source before review if needed.
It is governed again when its text is the approved one, and is promoted again otherwise.
Namespace erasure reaches loom.shared: its shared.tombstone step takes every promoted
row of the namespace and its census counts them, beside the user erasure’s match by
provenance.
A redact of a past version of a source (memory.redact refuses the live head) reaches its
shared copies: every loom.shared version promoted from the key whose value is the
version’s text (by the content_sha256 the redact keeps) is scrubbed, and the key’s search
rows below the next shared version after it (all of them when the copy is the newest) are
deleted, older shared rows it replaced included. A redact that reaches a copy marks the
namespace’s memory plane (namespace_write_marks.bump), since a mutation moves no mark. A
copy is left as it is while the live head holds the redacted text too (an edit reverted byte
for byte), as memory.redact leaves a live copy of a text it scrubs from history. A live
copy keeps its row, holding "[REDACTED]", until an admin demotes it; a definition’s copy
counts toward the ceiling until then, and the read-time check does not govern it.
A copy an earlier redact scrubbed is reached again: a run cut off part way is finished by the next one, and each later redact of the key purges its search rows and cached answers again and moves the mark.
Writes that race the redaction are listed on the history page.
Private definition review and candidate redaction follow definition drafts.
Should two shared rows match one name despite the name check (one promoted through a build
without the check, or one whose claim on the namespace ran out), the term leg serves,
among the rows the read-time check passes, the one with the latest promoted_at, a tie
within its second settled by promotion_event (the same order picks the rows the term leg
reads past the ceiling): one fixed winner, not necessarily the later of the two. A rewrite
of a shared row that is not a promotion, such as a scrub, does not count.
A caller may already write rows of type definition; they are governed only once an admin
promotes them. A definition is key-identified and is judged against no row under another
key, and only a write of its namespace-wide row revises that row
(conflicts and refusal). Conflict surfacing
compares a row only with rows of its own memory_type, so a learned fact that contradicts
a definition is not flagged against it. The term leg serves the governed
definition above such a fact; a cross-type check would add a candidate read to every write,
and is not part of this design.
Knowledge gaps
Recording misses, grouping them and knowledge.gaps.list are built: knowledge
gaps. What the later parts add to them follows.
Once a document can be hidden from some readers (the mirrored access of document
sources), a knowledge.search or loom_search that
left out a document its reader may not see records no miss.
Drafting, query history and review
These paths are implemented in definition drafts, which owns their access rules, durable claims, cost bounds, promotion, erasure and upgrade requirements. The Library has a Draft definition action for each gap and a Query history source list for namespace admins.
Erasure and audit
Definitions follow memory and shared-layer erasure, history and audit paths. Namespace erasure tombstones promoted rows. Historical source redaction also scrubs shared copies of that text, unless a live source still holds it. See the shared layer.
Knowledge gaps owns misses and gap builds. Definition drafts owns private candidate scrubbing, source registration erasure and review audits.
Cost bounds
Shared-layer reads have a fixed ceiling. Definition drafts owns model-call and query-history limits; configuration lists their defaults.
The shared definition ceiling is memory_shared_definitions_max (default 500).
Term-search bounds follow governed definitions in
search.
- A definition’s promotion, through
memory.promoteorlearnings.review, reads the namespace’s definition rows once, at mostmemory_shared_definitions_maxplus one (past it the promotion is refused), to check its term and aliases; the review path runs the read three times: at candidate selection, during preflight and at the promotion write. memory.definitions.listreads the definition rows once and returns at most twice the ceiling, with values for the ceiling’s most recent only; it reads their sources in one bulk read and grades the value of each source nothing else lapses.
Rolling upgrades
Private candidate and registration rollout and rollback follow definition drafts. Gap-table rollout follows knowledge gaps.
Governed definitions are memory and shared rows, which
N-1 knows: it serves them as plain rows with no term leg. The read-time check grades the
source’s value on every read, so a later build whose detector grades differently changes
which definitions are governed, and two builds serving side by side may disagree on one
row until the older one stops; the build that changes the detector names this in its
rollout notes, as does a build whose Python ships another Unicode version, which can
change whether two names fold to one. Keys do not depend on the fold, so no definition
lapses for it. What N-1 does to them is caught by the read-time check: a promotion from
another source, or an edit, relabel or tombstone of the source, leaves a row the term leg
does not serve, and the term leg’s last-promoted rule covers a promotion that skipped the
term check. A promotion on N-1 also skips the approved-hash check and takes promoted_by
from the body, which no read-time check can tell. Private review uses the existing
conditional-set precondition (if_match_sha256="absent" or the observed source hash);
its separate fleet gate requires every replica to understand private candidates.
Direct callers rely on the approved-hash and token-stamped promotion checks only once
no N-1 replica serves. An admin then promotes again every definition promoted before then, through N-1 or
before this design (loom.shared rows of type definition by promotion time), since none
of those carried the hash check or a token-stamped promoted_by; that promotion on N also
moves the write mark, which N-1’s promotions and demotions never move, so a stored
memory.ask answer from the rollout is not reused past it (until then such an answer
lives at most memory_ask_precedents_fast_max_age_hours, with no bound when it is 0).
N-1, which does not know definition as key-identified, can merge, supersede or retire
one definition’s source while writing another. Once N serves, each is recovered and
promoted again:
- a source it retired (
memory.auto_retirementsreasonsconflict_resolver,newest_wins_contradiction,value_updateandsame_key_newest_wins):memory.restore; - a source it marked
conflicted:conflicts.resolve; one markedduplicate:memory.dedup_resolve; - a write it merged: never stored, so written again;
- a source its negative-outcome quarantine of
memory.outcomerejected (memory.outcome_runquarantines nothing): written again; - a source its
memory.consolidatetombstoned:memory.restorewithin 14 days; past them, the definition is written again from its shared copy’s text.
An N-1 relabel of a definition’s namespace-wide source to medium or high leaves its
shared copy served by N-1’s search; N’s search leaves it out by that row’s label. A copy N-1
promoted from another row, which it does not refuse (a medium one included), is judged by
the namespace-wide row under its key, not by the row it came from: the re-promotion step
above replaces it with one promoted from that row. N-1’s redact does not reach the shared
row’s text: one run during the rollout or after a rollback is run again once N serves. A
namespace erasure an older build without the shared step ran is cleared as the
rolling-upgrades rule “A namespace’s promoted rows” says (rolling
upgrades).
Invariants
- A gap shows only queries that at least
knowledge_gaps_min_askersaskers asked in that form; a draft is written from nothing but such queries, or, for query history, the normalised text of queriesknowledge_query_history_min_usersaccounts ran. - A definition is governed only while an admin’s promotion of its namespace-wide row into
the shared layer is live and the source still holds that text, active, live, within its
validity and neither labelled nor graded
mediumorhigh; the term leg serves at most one definition per name, chosen by the rule above; only governed definitions are placed above the served list. - A query naming no governed term is served exactly as without the term leg.
Source entry points
server/memory/domain/memory_types.py(KEY_IDENTIFIED_TYPES)server/memory/app/search/handler.py(the legs ofmemory.search)server/memory/app/promote.py(handle_memory_promote)server/memory/app/learnings.py(handle_learnings_review)server/identity/domain/read_scope.py(honours_read_scope)server/observability/budget.py(namespaces_over_cap)