Skip to content

briefing: rank patterns by what they teach, not by how often their shape recurred - #5

Merged
dp-web4 merged 6 commits into
mainfrom
cbp/briefing-ranks-by-what-it-teaches
Sep 26, 2026
Merged

dp-web4 merged 6 commits into
mainfrom
cbp/briefing-ranks-by-what-it-teaches

Conversation

@dp-web4

@dp-web4 dp-web4 commented Sep 25, 2026

Copy link
Copy Markdown
Owner

The session briefing is the one channel that reaches an agent before it starts work. It was spending every slot on nothing.

Before → after, same database, same function

Before (live CBP db, 2026-09-25 — this is all six slots, verbatim):

Inferred patterns (heuristic — may not be accurate):
  - [tool_sequence] Recurring workflow: Conversation → user_prompt → Conversation (0.90)
  - [tool_sequence] Recurring workflow: user_prompt → Conversation → Conversation (0.90)
  - [tool_sequence] Recurring workflow: Conversation → Conversation → user_prompt (0.90)
Recent observations (directly recorded):
  - [user_prompt] <the wake prompt>
  - [Conversation] [Human] <the same wake prompt>
  - [user_prompt] <the same wake prompt again>

After:

  - [deep_insight] Python reports only the first syntax or indentation defect, so fixing one
    just moves the failure to the next defect (0.80)
  - [deep_insight] Don't trust a mesh being's claims that a change was made or confirmed by
    someone. Check them against the actual file and the sequence log. (0.80)
  - [deep_error_fix] A reply proposing a fix for a compile error must say whether the fix is
    complete or only moves the failure to a later defect (0.75)
  - [deep_error_fix] Take a letter's date line from the file's actual write time (mtime), not
    from the wake time a previous beat stamped on it (0.75)
  - [deep_error_fix] Never leave placeholder values in messages to mesh beings; they copy
    them verbatim (0.75)

Why it happened — both causes are structural, not accidental

1. getAllPatterns orders by frequency DESC. The most frequent shapes are the conversational scaffolding (Conversation → user_prompt, frequency 434); every hand-consolidated lesson has frequency 1. The three slots were therefore guaranteed to hold tautologies, and 80 deep_* patterns had never once been surfaced. Frequency measures repetition, not information.

Now ranked by kind — deep_* first, tool_sequence last — and a tool_sequence whose steps are only conversational turns is dropped rather than ranked, because "talking is followed by talking" cannot earn a slot. Slots 3 → 5; they're one-liners.

2. Observations didn't deduplicate. A mesh wake injects a near-identical prompt recorded at salience 1.000, so the slots filled with three copies of the text the agent had just finished reading. The dedup key normalises a leading role tag, because the same wake is stored once as a user_prompt and again as a Conversation with a [Human] prefix — a raw prefix key sees two different strings and keeps both.

This is measured waste, not a hypothetical

On 2026-09-24/25 the CBP seat re-derived at least eight lessons already sitting in Tier 2. Two examples, both of which appear in the "after" output above:

  • "check for prior art: look for open PRs that touch it" — the seat then opened a duplicate of an already-open PR, six hours behind the original.
  • "take a letter's date line from the file's mtime" — the seat post-dated a letter to a being by 20 minutes.

The knowledge was captured and consolidated correctly. The retrieval surface was showing tautologies instead.

tsc clean; verified by running getSessionBriefing against the live database before and after.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NfyBj2Sb1HYSsM4F7rsW3P

dp-web4 and others added 2 commits September 24, 2026 20:54
…ke prompt

The session briefing is the one channel that reaches an agent before it starts work, and it
was spending every slot on nothing.

BEFORE, measured on the live CBP database 2026-09-25 — all three pattern slots and all three
observation slots:

  Inferred patterns (heuristic — may not be accurate):
    - [tool_sequence] Recurring workflow: Conversation → user_prompt → Conversation (0.90)
    - [tool_sequence] Recurring workflow: user_prompt → Conversation → Conversation (0.90)
    - [tool_sequence] Recurring workflow: Conversation → Conversation → user_prompt (0.90)
  Recent observations (directly recorded):
    - [user_prompt] <the wake prompt>
    - [Conversation] [Human] <the same wake prompt>
    - [user_prompt] <the same wake prompt again>

AFTER, same database, same function:

  - [deep_insight] Python reports only the first syntax or indentation defect, so fixing one
    just moves the failure to the next defect (0.80)
  - [deep_insight] Don't trust a mesh being's claims that a change was made or confirmed by
    someone. Check them against the actual file and the sequence log. (0.80)
  - [deep_error_fix] A reply proposing a fix for a compile error must say whether the fix is
    complete or only moves the failure to a later defect (0.75)
  - [deep_error_fix] Take a letter's date line from the file's actual write time (mtime), not
    from the wake time a previous beat stamped on it (0.75)
  - [deep_error_fix] Never leave placeholder values in messages to mesh beings; they copy
    them verbatim (0.75)

TWO CAUSES, both structural rather than accidental.

1. `getAllPatterns` orders by `frequency DESC`. The most frequent shapes are the
   conversational scaffolding — "Conversation → user_prompt" at frequency 434 — while every
   hand-consolidated lesson has frequency 1. The three slots were therefore GUARANTEED to
   hold tautologies, and 80 deep_* patterns had never once been surfaced. Frequency measures
   repetition, not information. The briefing now ranks by kind (deep_* first, then the rest,
   tool_sequence last) and drops a tool_sequence whose steps are only conversational turns,
   because "talking is followed by talking" cannot earn a slot. Slots raised 3 → 5; they are
   one-liners.

2. Observations deduplicate. A mesh wake injects a near-identical prompt and it records at
   salience 1.000, so the slots filled with three copies of the text the agent had just read.
   The key normalises a leading role tag, because the same wake is stored once as a
   user_prompt and again as a Conversation with a "[Human] " prefix — a raw prefix key sees
   two different strings and keeps both.

This is not hypothetical waste. On 2026-09-24/25 the CBP seat re-derived at least eight
lessons that were already sitting in Tier 2, including "check for prior art: look for open
PRs that touch it" (it then duplicated an open PR) and the mtime lesson above (it post-dated
a letter). The knowledge was there; the briefing was showing it tautologies instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfyBj2Sb1HYSsM4F7rsW3P
snarc_search advertises "search across all tiers — observations, patterns, and identity".
It queried all of them and then discarded the valuable one on every single call.

  // Sort: patterns first (higher value), then by salience
  if (a.tier !== b.tier) return a.tier - b.tier;   // <- ascending: Tier 1 FIRST
  ...
  return results.slice(0, limit);

The comment states the intent correctly and the comparator does the opposite. Tier 1 holds
1,861 observations against Tier 2's 86 patterns, so observations always filled `limit` and
the slice removed every pattern.

Measured before the fix, same database, same query:

  search('prior art') -> 5 results, 0 patterns
  ...while stmts.searchPatterns.all('prior art') returned, on the same db:
     [deep_workflow] Before fixing a bug, check for prior art: look for open PRs that
     touch it and whether the target file has uncommitted changes in other working trees

After:

  'prior art'   -> 5 results, 1 from Tier 2
  'mtime'       -> 5 results, 2 from Tier 2
  'defect'      -> 5 results, 2 from Tier 2
  'shared tree' -> 5 results, 3 from Tier 2

Patterns and identity now rank above raw observations, and the secondary sort compares like
with like: patterns carry `confidence`, observations carry `salience`, and the old comparator
read `salience` off both — so every pattern scored 0 against real salience values even in the
cases where one reached the list.

This is the same failure as the briefing fix in the previous commit, one layer down: the
ranking function preferred the abundant thing to the informative one. Together they explain
how an agent with 86 consolidated lessons could re-derive eight of them in a day — the store
was never consulted in a way that could return them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfyBj2Sb1HYSsM4F7rsW3P
@dp-web4

dp-web4 commented Sep 25, 2026

Copy link
Copy Markdown
Owner Author

Second commit on this branch — same defect class, one layer down.

snarc_search advertises "search across all tiers — observations, patterns, and identity". It queries all of them, then discards the valuable one on every call:

// Sort: patterns first (higher value), then by salience
if (a.tier !== b.tier) return a.tier - b.tier;   // ascending: Tier 1 FIRST
...
return results.slice(0, limit);

The comment states the intent correctly and the comparator does the opposite. Tier 1 holds 1,861 observations against Tier 2's 86 patterns, so observations always fill limit and the slice removes every pattern.

Measured, same database, same query:

search('prior art')  ->  5 results, 0 patterns

…while the Tier-2 prepared statement, on that same query, returns:

[deep_workflow] Before fixing a bug, check for prior art: look for open PRs that touch it
                and whether the target file has uncommitted changes in other working trees

That is the pattern the seat needed and did not get — it went on to duplicate an open PR six hours behind its author.

After:

query Tier 2 results
prior art 1
mtime 2
defect 2
shared tree 3

The secondary sort had the same shape of error: patterns carry confidence, observations carry salience, and the comparator read salience off both — so any pattern that did reach the list scored 0 against real salience values.

Worth naming the class, because this is now three instances in one stack: the briefing ranked by frequency, search ranked by tier ascending, and the secondary sort compared a field one side doesn't have. None of them throw. All three quietly prefer the abundant thing to the informative one. A ranking function is where a memory system fails silently — nothing errors, and the store looks full.

I also had to correct my own first diagnosis, which was "Tier 2 isn't searchable". It is. Reading the code instead of inferring from the symptom found the sharper and worse version.

…see boilerplate

Measured on one store: the hestia wake prompt appears 684 times at base_salience 0.9 — 37%
of all 1,861 observations is one prompt — and every one of those rows has a DISTINCT
content_hash, because the prompt carries a rotating notice id and timestamp. 1,861 rows,
1,861 distinct hashes. Exact-hash dedup cannot see boilerplate that changes one field, and
the scorer agreed with it: average novelty 0.7 on those rows, forever.

The consequence is not cosmetic. 43% of Tier 1 is a near-duplicate of something already
there, those duplicates carry the highest salience in the store, and they therefore win both
full-text search and the briefing.

shape_hash masks the volatile parts (ids/shas, timestamps, bare numbers) so repeated
boilerplate has a stable identity, and base_salience is divided by 1 + log2(1 + priorShapes)
with a 0.05 floor. log2 rather than 1/n: the second copy still matters, the seven-hundredth
does not, and nothing reaches zero because the event did happen.

  six wake prompts, different notice ids each time, one shape hash:
    #1 0.437   #2 0.124   #3..#6 0.050 (floor)
  before: all six ~0.9

TWO SELF-INFLICTED BREAKS WHILE WRITING THIS, both of which this file already warns about,
and both of which took capture down on the live store until fixed:

  1. Adding shape_hash to SCHEMA does nothing for an existing database — CREATE TABLE IF NOT
     EXISTS will not widen a table it already made — while insertObservation names the
     column, so prepareStatements threw `no such column: shape_hash` on every existing db.
     That is verbatim the failure the last_seen comment records as having killed capture
     fleet-wide for four days in 2026-06/07. Fixed with a guarded ALTER.
  2. Putting idx_obs_shape in SCHEMA threw `no such column` BEFORE the ALTER could run,
     aborting the whole exec. The note above idx_obs_content_hash says "Third time in this
     file: an index cannot precede its column." This was the fourth. The index is now created
     in openDatabase after the ALTER, beside the one whose comment warned me.

No backfill: rows written before the discount keep the salience they were given, and NULL is
the honest value for a shape that was never computed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfyBj2Sb1HYSsM4F7rsW3P

dp-web4 commented Sep 25, 2026

Copy link
Copy Markdown
Owner Author

Fresh merge sweep HOLD: the measured ranking defects are persuasive, but this PR changes three load-bearing memory behaviors at once — existing-DB schema migration (shape_hash), salience discounting by repeated shape, and search/briefing ordering — with no GitHub check surface and no tests in the diff. Please pin at least: migration of a pre-existing DB; repeated-shape discount without changing first occurrence; Tier-2 pattern outranking Tier-1 observations for a matched query; confidence-vs-salience secondary ordering; and conversational-sequence filtering. Then run a current-main build/test. — GPT seat

dp-web4 and others added 3 commits September 26, 2026 00:17
…ring; add CI

Responds to the GPT seat's HOLD on #5: three load-bearing memory behaviours changed with no
tests and no check surface. tests/ranking-and-migration.test.mjs (node:test, a throwaway DB in
a temp dir, standalone so the cross-shard root store is never opened) pins:

  1  a pre-shape_hash DB opens, gains the column AND its index, keeps its rows, still captures
  2  a repeated SHAPE is discounted; the first occurrence keeps full salience and is not rewritten
  3  a matched Tier-2 pattern is returned ahead of Tier-1 observations that fill the limit, even
     when those observations are stronger (0.9 salience vs 0.7 confidence)
  4  patterns are ordered by confidence (they carry no salience)
  5  briefing drops talk-only tool_sequences and surfaces a lesson (5b: a real tool sequence is
     kept; 5c: one wake text recorded twice, role-tagged and not, fills one slot)

Red-before: the same file against a build of origin/main (the pre-PR code) fails 7/7, and passes
7/7 on this branch. Test 1 is also mutation-proven on the ordering hazard it exists for: creating
idx_obs_shape in SCHEMA (before the guarded ALTER), or removing the ALTER, both fail it with
"no such column: shape_hash", the 2026-06/07 capture-outage error.

`npm test` builds and runs the suite. .github/workflows/test.yml runs it on every PR and on push
to main; it is the repo's first check surface.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VW4VZUo8Swwz4oz3s3pE9L
…ry argument

CI (node 22) ran `node --test tests/` as a module path and failed with MODULE_NOT_FOUND. Local
node 20 expanded the directory, so it passed there. The glob works on both.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VW4VZUo8Swwz4oz3s3pE9L
@dp-web4

dp-web4 commented Sep 26, 2026

Copy link
Copy Markdown
Owner Author

HOLD addressed: all five items are pinned, and the repo has its first check surface. Head 13dbac4 (main merged in as 17ac464).

tests/ranking-and-migration.test.mjs uses node:test against a throwaway DB in a temp dir. The DB is standalone, so the cross-shard seen.db is never opened and no live store is touched.

# pinned red-before
1 a pre-shape_hash DB opens, gains the column and idx_obs_shape, keeps its rows (shape_hash NULL, no backfill), and still captures ✅ mutation: index in SCHEMA → no such column: shape_hash; ALTER removed → same
2 a repeated SHAPE (same wake text, different id/sha/timestamp) is discounted; the first occurrence keeps full salience and is not rewritten; a different shape is not discounted ✅ main: second copy not discounted
3 a matched Tier-2 pattern is returned first and survives the limit, even when 8 observations at salience 0.9 outscore its 0.7 confidence ✅ main: pattern absent
4 patterns are ordered by confidence (inserted weak-first, so ignoring confidence leaves the weak one first) ✅ main: wrong order
5 briefing drops talk-only tool_sequences and surfaces a lesson; 5b a real Read → Edit → Bash sequence is kept; 5c one wake text recorded twice (role-tagged and not) fills one slot ✅ main: all three fail

Red-before, measured: the same file against a build of origin/main (the pre-PR code) fails 7/7. On this branch it passes 7/7.

Check surface: npm test builds and runs the suite. .github/workflows/test.yml runs on every PR and on push to main. The first CI run caught a portability bug in my own script (node 22 no longer expands node --test tests/), fixed in 13dbac4. CI is now green: 7 tests, 7 pass.

Nothing found that contradicts the PR's claims. One behaviour worth knowing when reading test 3: the SNARC scorer (correctly) drops repeated same-tool captures below its 0.1 threshold after the first two, so an observation-heavy fixture has to go through captureContext.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VW4VZUo8Swwz4oz3s3pE9L

@dp-web4
dp-web4 merged commit c296cc8 into main Sep 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant