Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
70bfcff
chore(deps): bump hono in the npm_and_yarn group across 1 directory
dependabot[bot] May 8, 2026
d0b0e4c
Run ContextBench codebase-memory recovery on CI
PatrickSys May 10, 2026
a76ed36
Fix codebase-memory readiness query cwd
PatrickSys May 10, 2026
e924a09
Run codebase-memory readiness retry
PatrickSys May 10, 2026
ed46c22
Fix codebase-memory readiness project queries
PatrickSys May 10, 2026
ae5ac57
Harden codebase-memory readiness fallback selection
PatrickSys May 10, 2026
c588cbd
Record codebase-memory partial readiness blockers
PatrickSys May 10, 2026
66a9190
Diagnose codebase-memory Go evaluator row
PatrickSys May 10, 2026
0cb6eba
Run real ContextBench gpt-5.4-mini matrix
PatrickSys May 10, 2026
3074da8
Run bounded core ContextBench gpt-5.4-mini rows
PatrickSys May 10, 2026
f6e9303
Run one bounded CBM ContextBench row
PatrickSys May 10, 2026
d6414b3
Add authenticated one-row ContextBench run
PatrickSys May 10, 2026
3cbb64b
Try ContextBench auth from Actions variables
PatrickSys May 10, 2026
4b00271
Probe GitHub Models for ContextBench model auth
PatrickSys May 10, 2026
ba7f2f3
Generate ContextBench candidate pack for one task
PatrickSys May 10, 2026
f4e5bf9
Generate CBM candidate pack for ContextBench
PatrickSys May 10, 2026
3d6ac27
Make CBM ContextBench candidate pack logs compact
PatrickSys May 10, 2026
2048fee
Extract CBM ContextBench candidates from script file
PatrickSys May 10, 2026
66a522b
Call CBM ContextBench candidate extractor from script
PatrickSys May 10, 2026
aab3abe
Add scorer for selected ContextBench prediction
PatrickSys May 10, 2026
d3d2790
Run official ContextBench score for selected CBM prediction
PatrickSys May 10, 2026
f30797f
Select ContextBench scorer task by id
PatrickSys May 10, 2026
5950bb2
Materialize only selected ContextBench score task
PatrickSys May 10, 2026
f005195
Add five-lane ContextBench candidate pack script
PatrickSys May 10, 2026
786b807
Add five-lane ContextBench readiness workflow
PatrickSys May 10, 2026
3ecc4d5
Add five-lane ContextBench scoring script
PatrickSys May 10, 2026
b1c3f8d
Harden five-lane ContextBench scorer gates
PatrickSys May 10, 2026
d675cda
Report five-lane ContextBench phase costs explicitly
PatrickSys May 10, 2026
0f36a30
Add single-lane ContextBench readiness script
PatrickSys May 10, 2026
992f2ff
Add lane-parallel ContextBench readiness workflow
PatrickSys May 10, 2026
24aa862
Install ripgrep for raw ContextBench lane
PatrickSys May 10, 2026
b5344b6
Add raw-native ContextBench readiness rerun
PatrickSys May 10, 2026
1e8f1a3
Add CodeGraphContext content readiness script
PatrickSys May 10, 2026
cfbbd33
Add CodeGraphContext ContextBench readiness rerun
PatrickSys May 10, 2026
831bfb1
Add bounded ContextBench rescue lane readiness packer
PatrickSys May 10, 2026
072f82b
Add bounded ContextBench rescue readiness workflow
PatrickSys May 10, 2026
f2b25d2
Add parallel ContextBench tool rescue readiness workflow
PatrickSys May 10, 2026
7a99d13
Add ContextBench third-party tool diagnostics workflow
PatrickSys May 10, 2026
2263513
Add bounded final ContextBench lane pack generator
PatrickSys May 10, 2026
b8159dc
Add final ContextBench lane readiness workflow
PatrickSys May 10, 2026
345c030
Add relevant CodeGraphContext readiness pack
PatrickSys May 10, 2026
551d78e
Add relevant CodeGraphContext readiness workflow
PatrickSys May 10, 2026
b69cc59
Allow scoring selected ready ContextBench lanes
PatrickSys May 10, 2026
8059f86
Add ready lane selections for ContextBench scoring
PatrickSys May 10, 2026
5a7a747
Add ready lane ContextBench scoring workflow
PatrickSys May 10, 2026
0d46c80
Add infra-backed ContextBench lane readiness
PatrickSys May 11, 2026
0c617e7
Add infra-backed ContextBench readiness workflow
PatrickSys May 11, 2026
b964a58
Fix grepai readiness loop logging
PatrickSys May 11, 2026
eb65285
Use infra readiness script for CodeGraphContext lane
PatrickSys May 11, 2026
8865e8d
Add focused GrepAI readiness pack script
PatrickSys May 11, 2026
43fb64a
Add focused GrepAI readiness workflow
PatrickSys May 11, 2026
58a2728
Add focused CodeGraphContext readiness pack script
PatrickSys May 11, 2026
d145b7b
Use focused CodeGraphContext readiness pack
PatrickSys May 11, 2026
05f1385
Trigger focused CodeGraphContext readiness run
PatrickSys May 11, 2026
7665d32
Add ContextBench ripgrep readiness pack generator
PatrickSys May 11, 2026
71467e8
Add ContextBench ripgrep readiness workflow
PatrickSys May 11, 2026
f440d4c
Add artifact-aware ContextBench five-lane scorer
PatrickSys May 11, 2026
73c071e
Install ripgrep during readiness when missing
PatrickSys May 11, 2026
5f0f1ea
Trigger ripgrep readiness after setup fix
PatrickSys May 11, 2026
0135600
Add fixed ripgrep readiness workflow
PatrickSys May 11, 2026
3c30e56
Add five-lane ContextBench selections
PatrickSys May 11, 2026
c5da98a
Keep all ripgrep query groups in readiness pack
PatrickSys May 11, 2026
555d3ee
Add balanced ripgrep readiness workflow
PatrickSys May 11, 2026
f073bc2
Add five-lane ContextBench score workflow
PatrickSys May 11, 2026
f1abc72
Add publishable benchmark goal prompt
PatrickSys May 11, 2026
da23692
Remove tracked local-only goal prompt
PatrickSys May 11, 2026
e6abab1
Add publishable ContextBench lane metrics
PatrickSys May 11, 2026
b578547
Attach candidate metrics to ContextBench selections
PatrickSys May 11, 2026
68b4fe1
Add publishable ContextBench report validator
PatrickSys May 11, 2026
f8ccb0f
Publish validated ContextBench pilot artifacts
PatrickSys May 11, 2026
5f6f2cd
Tighten publishable report validation
PatrickSys May 11, 2026
d4ad1eb
Add compact ContextBench report printer
PatrickSys May 11, 2026
0795131
Print compact ContextBench pilot report in CI
PatrickSys May 11, 2026
de3d70b
Preserve unavailable benchmark metrics as null
PatrickSys May 11, 2026
5349e5d
Keep null benchmark metrics out of numeric tables
PatrickSys May 11, 2026
bbd3a83
Tighten ContextBench score artifact output
PatrickSys May 11, 2026
86644d2
Document ContextBench implementation pilot evidence
PatrickSys May 11, 2026
55b97b9
fix(reranker): Score passages with the supported tokenizer pair API
aolin480 Jul 5, 2026
b51adeb
test(release): Keep current version checks from going stale
aolin480 Jul 5, 2026
6e77a04
Upgrade Transformers.js to v4
aolin480 Jul 15, 2026
cff0760
Keep test runs independent of shell credentials
aolin480 Jul 15, 2026
58a4cc4
Merge pull request #125 from aolin480/bugfix/huggingface-fix
PatrickSys Jul 25, 2026
3be23d6
test: keep scoped memory fixtures fresh
PatrickSys Jul 25, 2026
7ad2fb6
test: keep scoped memory fixtures fresh
PatrickSys Jul 25, 2026
d009d8e
Merge pull request #126 from PatrickSys/agent/fix-scoped-memory-fixtures
PatrickSys Jul 25, 2026
9ed8577
fix(analyzers): Recognize current React and Next patterns
aolin480 Jul 5, 2026
754130d
feat(analyzers): Recognize NestJS projects
aolin480 Jul 5, 2026
1612414
feat(indexing): Make chunk limits configurable per project
aolin480 Jul 15, 2026
96c6d91
chore(memory): Record recent repository fixes
aolin480 Jul 15, 2026
9d62f8c
style(indexing): Format project index overrides
aolin480 Jul 15, 2026
6093099
fix(analyzers): Prefer NestJS provider analysis
aolin480 Jul 26, 2026
0154f45
fix(analyzers): Tie React use metadata to imports
aolin480 Jul 26, 2026
a8e354e
Merge pull request #127 from aolin480/feature/nestjs-analyzer
PatrickSys Jul 26, 2026
2c56b10
feat(review): add bounded review-context packets
PatrickSys Aug 22, 2026
cc54fb5
docs: mark review-context G0 passed
PatrickSys Aug 22, 2026
10ba0a6
Publish historical benchmark and released setup docs (#130)
PatrickSys Oct 6, 2026
a9cd0bd
fix: support Node 22 and 24 and explain first-run indexing
PatrickSys Oct 16, 2026
0d733f1
fix: show usage when you run the help command
PatrickSys Oct 16, 2026
5021b4f
docs: make setup instructions clear before and after the release
PatrickSys Oct 16, 2026
cfaa90b
fix: set up Claude Code and Codex from your project folder
PatrickSys Oct 16, 2026
7d0e65e
docs: keep release instructions accurate and easy to follow
PatrickSys Oct 16, 2026
1928b26
docs: make install commands work against published npm versions (#141)
PatrickSys Oct 20, 2026
e9289c1
Merge remote-tracking branch 'origin/pr123' into night/cbc-20261029-i…
PatrickSys Oct 29, 2026
d7ea9ba
fix(test): bound Windows fixture concurrency
PatrickSys Oct 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 180 additions & 0 deletions .codebase-context/memory.json
Original file line number Diff line number Diff line change
Expand Up @@ -569,5 +569,185 @@
"scope": {
"kind": "global"
}
},
{
"id": "cbe8bd1377ba",
"type": "gotcha",
"category": "conventions",
"memory": "fix(analyzers): Recognize current React and Next patterns",
"reason": "Auto-extracted from git commit history",
"date": "2026-07-05T07:33:25.000Z",
"source": "git"
},
{
"id": "e3b009c312e8",
"type": "gotcha",
"category": "conventions",
"memory": "fix(reranker): Score passages with the supported tokenizer pair API",
"reason": "Auto-extracted from git commit history",
"date": "2026-07-05T14:43:29.000Z",
"source": "git"
},
{
"id": "449b60830c9b",
"type": "gotcha",
"category": "conventions",
"memory": "fix(eval): deduplicate blocked ContextBench rows",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-30T15:45:12.000Z",
"source": "git"
},
{
"id": "9dfaf21a0bc8",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): harden ContextBench schema cleanup",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T19:14:49.000Z",
"source": "git"
},
{
"id": "b8776a500ef5",
"type": "gotcha",
"category": "conventions",
"memory": "fix(eval): preserve ContextBench executor model provenance",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T19:04:04.000Z",
"source": "git"
},
{
"id": "a8b0f8664253",
"type": "gotcha",
"category": "conventions",
"memory": "fix(eval): harden ContextBench manifest checks",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T19:03:51.000Z",
"source": "git"
},
{
"id": "f84080651a19",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): relax zombie guard timeout jitter",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T18:24:56.000Z",
"source": "git"
},
{
"id": "3d7776af8a23",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): tolerate ContextBench runner cleanup races",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T18:21:18.000Z",
"source": "git"
},
{
"id": "6428cb02706e",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): tolerate ContextBench schema cleanup races",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T18:13:07.000Z",
"source": "git"
},
{
"id": "c1bd14e88de2",
"type": "gotcha",
"category": "conventions",
"memory": "fix(eval): align ContextBench harness evidence contracts",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T18:08:08.000Z",
"source": "git"
},
{
"id": "2f3db0023789",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): relax slow Windows integration timeouts",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T17:59:20.000Z",
"source": "git"
},
{
"id": "da039b1d0ff8",
"type": "gotcha",
"category": "conventions",
"memory": "fix(eval): harden ContextBench fixture verification",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T17:55:12.000Z",
"source": "git"
},
{
"id": "5e3bdcaa984c",
"type": "gotcha",
"category": "conventions",
"memory": "fix(git): scope local artifact ignores",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T17:50:47.000Z",
"source": "git"
},
{
"id": "861eb0bfb69f",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): relax slow Windows search timeouts",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T16:47:38.000Z",
"source": "git"
},
{
"id": "3614711e00a4",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): tolerate ContextBench temp cleanup races",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T16:29:24.000Z",
"source": "git"
},
{
"id": "c0cfb6599bcc",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): isolate ContextBench baseline Git env",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T16:25:08.000Z",
"source": "git"
},
{
"id": "084a0540558c",
"type": "gotcha",
"category": "conventions",
"memory": "fix(format): format ContextBench harness sources",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T16:06:53.000Z",
"source": "git"
},
{
"id": "0153743292c4",
"type": "gotcha",
"category": "conventions",
"memory": "fix(test): isolate ContextBench git fixtures",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-29T15:56:57.000Z",
"source": "git"
},
{
"id": "a5397b2ee20d",
"type": "gotcha",
"category": "conventions",
"memory": "fix(cli): lazy-load mcp runtime for direct commands",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-17T14:34:22.000Z",
"source": "git"
},
{
"id": "377e7c7c37b1",
"type": "gotcha",
"category": "conventions",
"memory": "fix(ci): unblock functional tests",
"reason": "Auto-extracted from git commit history",
"date": "2026-04-17T20:05:19.000Z",
"source": "git"
}
]
127 changes: 127 additions & 0 deletions .github/workflows/contextbench-candidate-pack-one.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
name: ContextBench Candidate Pack One

on:
push:
branches: [master]
paths:
- .github/workflows/contextbench-candidate-pack-one.yml
workflow_dispatch:

permissions:
contents: read

jobs:
pack:
runs-on: ubuntu-latest
timeout-minutes: 90
env:
ROOT: /tmp/contextbench-candidate-pack-one
TASK_PAYLOADS: /tmp/contextbench-candidate-pack-one/task-payloads.json
CHECKOUT_ROOT: /tmp/contextbench-checkouts
CBM_BIN: /tmp/contextbench-candidate-pack-one/tool/codebase-memory-mcp
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v2
with:
version: 10
- uses: actions/setup-node@v4
with:
node-version: '24'
cache: pnpm
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install tools and materialize frozen Go task
run: |
set -euxo pipefail
mkdir -p "$ROOT" "$CHECKOUT_ROOT" "$ROOT/tool"
pnpm install --frozen-lockfile
pnpm run build
python -m pip install "tree-sitter==0.20.4" "tree-sitter-languages==1.10.2" datasets pyarrow uv codegraphcontext kuzu
curl -fsSL "https://github.com/DeusData/codebase-memory-mcp/releases/download/v0.6.1/codebase-memory-mcp-linux-amd64.tar.gz" -o "$ROOT/tool/cbm.tar.gz"
tar -xzf "$ROOT/tool/cbm.tar.gz" -C "$ROOT/tool"
chmod +x "$CBM_BIN" || true
curl -sSL https://raw.githubusercontent.com/yoanbernabeu/grepai/main/install.sh | sh || true
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
echo "$HOME/bin" >> "$GITHUB_PATH"
git clone --depth 1 https://github.com/EuniAI/ContextBench.git "$ROOT/ContextBench-official"
node scripts/contextbench-runner.mjs --validate-fixtures
node scripts/contextbench-select-slice.mjs --write-task-payloads --out "$TASK_PAYLOADS" --checkout-root "$CHECKOUT_ROOT"
node scripts/contextbench-select-slice.mjs --materialize-checkouts --payloads "$TASK_PAYLOADS" --max-tasks 3
- name: Generate candidate pack
run: |
cat > "$ROOT/pack.mjs" <<'NODE'
import { spawnSync } from 'node:child_process';
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { basename, relative, sep } from 'node:path';
const root = process.env.ROOT;
const payloads = JSON.parse(readFileSync(process.env.TASK_PAYLOADS, 'utf8'));
const task = payloads.tasks[2];
const repo = task.repo_checkout_path;
const outDir = root + '/pack';
mkdirSync(outDir, { recursive: true });
function run(cmd, args, opts = {}) { const started = Date.now(); const r = spawnSync(cmd, args, { cwd: opts.cwd || process.cwd(), env: opts.env || process.env, encoding: 'utf8', timeout: opts.timeoutMs || 300000, maxBuffer: 96 * 1024 * 1024 }); return { command: [cmd, ...args].join(' '), cwd: opts.cwd || process.cwd(), status: r.status, signal: r.signal, error: r.error?.message || null, durationMs: Date.now() - started, stdout: r.stdout || '', stderr: r.stderr || '' }; }
function queryOf(text) { const stop = new Set(['that','this','with','from','when','then','into','should','would','could','there','where','which','about','after','before','have','will','been','than','also']); return String(text || '').replace(/[`*_#>\[\](){},.;:!?/\\]/g, ' ').split(/\s+/).filter((w) => w.length >= 4 && !stop.has(w.toLowerCase())).slice(0, 12).join(' '); }
function jsonish(text) { const raw = String(text || '').trim(); if (!raw) return null; try { return JSON.parse(raw); } catch {} for (const [a,b] of [['{','}'],['[',']']]) { const i = raw.indexOf(a), j = raw.lastIndexOf(b); if (i >= 0 && j > i) { try { return JSON.parse(raw.slice(i, j + 1)); } catch {} } } return null; }
function norm(file) { let f = String(file || '').replace(/^file:\/\//, ''); if (!f) return ''; if (f.startsWith(repo)) f = relative(repo, f); f = f.replaceAll('\\\\', '/').replace(/^\/+/, '').replace(/^\.\//, ''); if (f.startsWith('tmp/') || f.includes('://') || f.includes('..')) return ''; return f; }
function add(locs, file, start = 1, end = start, source = 'tool') { const clean = norm(file); if (!clean) return; const s = Math.max(1, Number(start) || 1); locs.push({ file: clean, start: s, end: Math.max(s, Number(end) || s), source }); }
function walk(value, locs, source) { if (!value || typeof value !== 'object') return; if (Array.isArray(value)) { for (const item of value) walk(item, locs, source); return; } add(locs, value.file || value.path || value.file_path || value.relative_path || value.filename || value.source_path || value.uri, value.start_line || value.startLine || value.line || value.line_number || value.start || 1, value.end_line || value.endLine || value.end || value.line || 1, source); for (const item of Object.values(value)) walk(item, locs, source); }
function collect(text, locs, source) { const parsed = jsonish(text); if (parsed) walk(parsed, locs, source); const re = /([A-Za-z0-9_.\/-]+\.(?:js|jsx|ts|tsx|py|go|rs|java|c|cc|cpp|h|hpp|rb|php|cs|kt|swift|vue|svelte|json|yml|yaml|md))(?::|#L|\s+line\s+)?(\d+)?/g; let m; while ((m = re.exec(String(text || ''))) !== null) add(locs, m[1], m[2] || 1, m[2] || 1, source); }
function uniq(locs, max = 60) { const seen = new Set(), out = []; for (const loc of locs) { const key = `${loc.file}:${loc.start}:${loc.end}`; if (!seen.has(key)) { seen.add(key); out.push(loc); if (out.length >= max) break; } } return out; }
function laneResult(lane, commands, locs, setupStatus = 'completed', indexStatus = 'completed') { for (const [i,c] of commands.entries()) writeFileSync(`${outDir}/${lane}-command-${i + 1}.json`, JSON.stringify(c, null, 2)); return { lane, setupStatus, indexStatus, toolCallable: commands.some((c) => c.status === 0), costs: { totalMs: commands.reduce((a,c) => a + c.durationMs, 0) }, candidates: uniq(locs) }; }
const query = queryOf(task.problem_statement);
const lanes = [];
{
const commands = [], locs = [];
for (const term of query.split(/\s+/).slice(0, 6)) { const r = run('rg', ['-n', '-i', '--glob', '!.git', '--glob', '!vendor/**', '--glob', '!node_modules/**', term, '.'], { cwd: repo, timeoutMs: 60000 }); commands.push(r); collect(r.stdout, locs, 'raw-native'); collect(r.stderr, locs, 'raw-native'); }
lanes.push(laneResult('raw-native', commands, locs));
}
{
const commands = [], locs = []; const env = { ...process.env, CODEBASE_ROOT: repo, CODEBASE_CONTEXT_ASCII: '1' };
const v = run('node', ['dist/index.js', '--version'], { env, timeoutMs: 60000 }); commands.push(v);
const idx = run('node', ['dist/index.js', 'reindex'], { env, timeoutMs: 1200000 }); commands.push(idx);
const s = run('node', ['dist/index.js', 'search', '--query', query, '--intent', 'edit', '--limit', '30', '--json'], { env, timeoutMs: 300000 }); commands.push(s); collect(s.stdout, locs, 'codebase-context'); collect(s.stderr, locs, 'codebase-context');
lanes.push(laneResult('codebase-context', commands, locs, 'completed', idx.status === 0 ? 'completed' : 'index_failed'));
}
{
const commands = [], locs = []; const env = { ...process.env, CBM_CACHE_DIR: outDir + '/cbm-cache', CBM_DIAGNOSTICS: '1' };
const v = run(process.env.CBM_BIN, ['--version'], { env, timeoutMs: 60000 }); commands.push(v);
const idx = run(process.env.CBM_BIN, ['cli', 'index_repository', JSON.stringify({ repo_path: repo })], { cwd: repo, env, timeoutMs: 2700000 }); commands.push(idx);
const project = (jsonish(idx.stdout) || jsonish(idx.stderr) || {}).project || basename(repo);
const g = run(process.env.CBM_BIN, ['cli', 'search_graph', JSON.stringify({ project, query, limit: 30 })], { cwd: repo, env, timeoutMs: 120000 }); commands.push(g);
const c = run(process.env.CBM_BIN, ['cli', 'search_code', JSON.stringify({ project, pattern: query.split(/\s+/)[0] || '.', mode: 'compact', limit: 30 })], { cwd: repo, env, timeoutMs: 120000 }); commands.push(c);
for (const r of [g, c]) { collect(r.stdout, locs, 'codebase-memory-mcp'); collect(r.stderr, locs, 'codebase-memory-mcp'); }
lanes.push(laneResult('codebase-memory-mcp', commands, locs, 'completed', idx.status === 0 ? 'completed' : 'index_failed'));
}
{
const commands = [], locs = [];
const v = run('grepai', ['version'], { timeoutMs: 60000 }); commands.push(v);
const init = run('grepai', ['init', '--yes', '--provider', 'synthetic', '--backend', 'gob'], { cwd: repo, timeoutMs: 120000 }); commands.push(init);
const watch = run('grepai', ['watch', '--background'], { cwd: repo, timeoutMs: 120000 }); commands.push(watch);
const status = run('grepai', ['watch', '--status'], { cwd: repo, timeoutMs: 60000 }); commands.push(status);
const s = run('grepai', ['search', query, '--json', '--compact'], { cwd: repo, timeoutMs: 180000 }); commands.push(s); collect(s.stdout, locs, 'grepai'); collect(s.stderr, locs, 'grepai');
commands.push(run('grepai', ['watch', '--stop'], { cwd: repo, timeoutMs: 60000 }));
lanes.push(laneResult('grepai', commands, locs, init.status === 0 ? 'completed' : 'setup_failed', (watch.status === 0 || status.status === 0) ? 'completed' : 'index_failed'));
}
{
const commands = [], locs = [];
const v = run('cgc', ['--help'], { timeoutMs: 60000 }); commands.push(v);
const idx = run('cgc', ['index', repo], { cwd: repo, timeoutMs: 1200000 }); commands.push(idx);
const a = run('cgc', ['analyze', 'complexity', '--limit', '30'], { cwd: repo, timeoutMs: 180000 }); commands.push(a);
const d = run('cgc', ['analyze', 'dead-code', '--limit', '30'], { cwd: repo, timeoutMs: 180000 }); commands.push(d);
const call = run('cgc', ['analyze', 'callers', query.split(/\s+/)[0] || 'main'], { cwd: repo, timeoutMs: 120000 }); commands.push(call);
for (const r of [a, d, call]) { collect(r.stdout, locs, 'codegraphcontext'); collect(r.stderr, locs, 'codegraphcontext'); }
lanes.push(laneResult('codegraphcontext', commands, locs, 'completed', idx.status === 0 ? 'completed' : 'index_failed'));
}
const pack = { createdAt: new Date().toISOString(), task: { instance_id: task.instance_id, repo: task.repo, base_commit: task.base_commit, problem_statement: task.problem_statement }, query, lanes };
writeFileSync(`${outDir}/candidate-pack.json`, JSON.stringify(pack, null, 2));
console.log(JSON.stringify({ task: pack.task, query, lanes: lanes.map((l) => ({ lane: l.lane, setupStatus: l.setupStatus, indexStatus: l.indexStatus, toolCallable: l.toolCallable, candidateCount: l.candidates.length, candidates: l.candidates.slice(0, 30) })) }, null, 2));
NODE
node "$ROOT/pack.mjs"
- name: Upload candidate pack
if: always()
uses: actions/upload-artifact@v4
with:
name: contextbench-candidate-pack-one
path: /tmp/contextbench-candidate-pack-one/pack
retention-days: 14
59 changes: 59 additions & 0 deletions .github/workflows/contextbench-cbm-candidates-one.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: ContextBench CBM Candidates One

on:
push:
branches: [master]
paths:
- .github/workflows/contextbench-cbm-candidates-one.yml
- scripts/contextbench-cbm-candidate-pack.mjs
workflow_dispatch:

permissions:
contents: read

jobs:
cbm-candidates:
runs-on: ubuntu-latest
timeout-minutes: 20
env:
ROOT: /tmp/contextbench-cbm-candidates-one
TASK_PAYLOADS: /tmp/contextbench-cbm-candidates-one/task-payloads.json
CHECKOUT_ROOT: /tmp/contextbench-checkouts
CBM_BIN: /tmp/contextbench-cbm-candidates-one/tool/codebase-memory-mcp
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v2
with:
version: 10
- uses: actions/setup-node@v4
with:
node-version: '24'
cache: pnpm
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install and materialize Go task quietly
shell: bash
run: |
set -euo pipefail
mkdir -p "$ROOT" "$CHECKOUT_ROOT" "$ROOT/tool" "$ROOT/pack" "$ROOT/logs"
pnpm install --frozen-lockfile > "$ROOT/logs/pnpm-install.log" 2>&1
python -m pip install "tree-sitter==0.20.4" "tree-sitter-languages==1.10.2" datasets pyarrow > "$ROOT/logs/pip-install.log" 2>&1
curl -fsSL "https://github.com/DeusData/codebase-memory-mcp/releases/download/v0.6.1/codebase-memory-mcp-linux-amd64.tar.gz" -o "$ROOT/tool/cbm.tar.gz"
tar -xzf "$ROOT/tool/cbm.tar.gz" -C "$ROOT/tool" > "$ROOT/logs/cbm-extract.log" 2>&1
chmod +x "$CBM_BIN" || true
git clone --depth 1 https://github.com/EuniAI/ContextBench.git "$ROOT/ContextBench-official" > "$ROOT/logs/contextbench-clone.log" 2>&1
node scripts/contextbench-runner.mjs --validate-fixtures > "$ROOT/logs/validate-fixtures.log" 2>&1
node scripts/contextbench-select-slice.mjs --write-task-payloads --out "$TASK_PAYLOADS" --checkout-root "$CHECKOUT_ROOT" > "$ROOT/logs/write-payloads.log" 2>&1
node scripts/contextbench-select-slice.mjs --materialize-checkouts --payloads "$TASK_PAYLOADS" --max-tasks 3 > "$ROOT/logs/materialize.log" 2>&1
echo "quiet_setup_completed"
- name: Print compact CBM candidates
shell: bash
run: node scripts/contextbench-cbm-candidate-pack.mjs
- name: Upload CBM candidate pack
if: always()
uses: actions/upload-artifact@v4
with:
name: contextbench-cbm-candidates-one
path: /tmp/contextbench-cbm-candidates-one/pack
retention-days: 14
Loading
Loading