From 556e4a5b24cddfbdca3da4ae608db034ad0bc571 Mon Sep 17 00:00:00 2001 From: ysyneu Date: Thu, 10 Sep 2026 23:41:21 -0700 Subject: [PATCH] fix(cli): expose list-page truncation in the payload MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An oversize structured list page is emitted as its leading rows, but the reduction was announced only on stderr while total / has_next_page / search_after_ctx kept describing the page the server returned — so a script that discards stderr and stops on has_next_page silently loses every withheld row. A reduced list envelope now carries truncated: true plus emitted_rows: N when rows were withheld (the case paging can repair); when long values inside rows were clipped instead, truncated rides alone, because re-requesting cannot restore them. Bare top-level arrays have nowhere to carry the marker and keep the stderr note as their only signal. Docs: README / README_zh bounded-pages section; a page-level note in the flashduty skill so agents recognise the two keys. --- README.md | 11 ++++- README_zh.md | 4 +- internal/cli/gen_support.go | 17 ++++++++ internal/cli/gen_support_test.go | 69 +++++++++++++++++++++++++++++++- skills/flashduty/SKILL.md | 2 + 5 files changed, 99 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3c87627..91b0514 100644 --- a/README.md +++ b/README.md @@ -321,18 +321,25 @@ inc_def456 High memory usage Warning Processing Staging 2026 Showing 2 results (page 1, total 2). ``` -**JSON (`--json` / `--output-format json`):** Machine-parseable, full data, no truncation. +**JSON (`--json` / `--output-format json`):** Machine-parseable output for `jq` and scripts. ```bash flashduty incident list --json | jq '.[].title' ``` -**TOON (`--output-format toon`):** Token-Oriented Object Notation — full data, no truncation, but drops the per-row repeated keys that JSON emits for uniform arrays, so list output costs materially fewer tokens. Preferred for LLM/agent consumption. Not directly `jq`-able; use `--json` when you need to pipe into `jq`. +**TOON (`--output-format toon`):** Token-Oriented Object Notation — drops the per-row repeated keys that JSON emits for uniform arrays, so list output costs materially fewer tokens. Preferred for LLM/agent consumption. Not directly `jq`-able; use `--json` when you need to pipe into `jq`. ```bash flashduty incident list --output-format toon ``` +**Bounded list pages.** Every structured list page is capped at 16 KiB: an oversize page is emitted as the leading rows that fit, and the reduction is announced on stderr. A reduced **list envelope says so in the payload** too — scripts routinely discard stderr — and the marker's shape tells you which reduction happened: + +- `"truncated": true` **with** `"emitted_rows": N` — the page carries its first N rows and withheld the rest. The envelope's `total` / `has_next_page` / `search_after_ctx` still describe the page as the server returned it, so a page cut to 7 of 100 rows reads as complete. To collect everything, re-request with a `--limit` no larger than the rows you received (or, where the command documents its cursor as a row id, pass the last received row's id back as `--search-after-ctx`) and repeat until the rows you hold reach `total`; stopping on `has_next_page=false` alone silently drops the withheld rows. +- `"truncated": true` **alone** — every row was emitted, but long values inside them were clipped (stderr names the fields). Paging cannot restore them; narrow `--fields` and re-request. + +A bare top-level array has nowhere to carry the marker, so it announces a reduction only on stderr — page it with a lower `--limit`, or switch to a page-envelope command (`alert event-list`, `insight incident-list`) when a script needs completeness. + **No truncation (`--no-trunc`):** Table with full field content. --- diff --git a/README_zh.md b/README_zh.md index 7cdc250..b4f736d 100644 --- a/README_zh.md +++ b/README_zh.md @@ -317,12 +317,14 @@ inc_def456 High memory usage Warning Processing Staging 2026 Showing 2 results (page 1, total 2). ``` -**JSON（`--json`）：** 机器可解析，完整数据，不截断。 +**JSON（`--json`）：** 机器可解析，可直接管道给 `jq`。 ```bash flashduty incident list --json | jq '.[].title' ``` +**列表页有 16 KiB 上限。** 结构化列表的一页超出上限时，只输出能装下的前若干行，并在 stderr 说明。如果该页是分页信封（形如 `{items, total, has_next_page, …}`），载荷内也会带上标记：`"truncated": true` 与 `"emitted_rows": N`（保留了前 N 行，其余被丢弃）。`total` / `has_next_page` / `search_after_ctx` 仍是服务端原值，因此一页被裁到"100 行里只发 7 行"时，看起来与完整页无异。脚本要完整翻页时，请从**实际收到的最后一行**之后继续（用不大于已收到行数的 `--limit` 重新请求，再跟随该响应的游标），不要只依赖 `has_next_page`；若只有 `"truncated": true` 而没有 `emitted_rows`，说明行内长值被裁剪——翻页无法恢复，应收窄 `--fields` 后重新请求。 + **不截断（`--no-trunc`）：** 表格显示完整字段内容。 --- diff --git a/internal/cli/gen_support.go b/internal/cli/gen_support.go index 0c83d62..db1439f 100644 --- a/internal/cli/gen_support.go +++ b/internal/cli/gen_support.go @@ -359,6 +359,8 @@ func printGenericResult(ctx *RunContext, data any) error { // byte-identical output. Only an over-cap list payload detours through the // bounding machinery, and only there does the output change (fewer rows; a // rebuilt envelope, so key order is no longer the struct's field order). +// A reduced list envelope also carries truncated/emitted_rows in the payload +// itself, so a machine consumer sees the reduction without reading stderr. func printBoundedGenericResult(ctx *RunContext, data any) error { encoded, err := marshalStructured(data) if err != nil || len(encoded)+1 < compactListOutputLimit { @@ -382,6 +384,8 @@ func printBoundedGenericResult(ctx *RunContext, data any) error { if err != nil { return err } + // A bare array has nowhere to carry the marker the envelope branch + // adds; the stderr note is its only signal. noteProjectionBound(ctx.Cmd.ErrOrStderr(), note) return ctx.Printer.Print(bounded, nil) case map[string]any: @@ -407,6 +411,19 @@ func printBoundedGenericResult(ctx *RunContext, data any) error { return err } value[key] = bounded + if note != "" { + // In-payload, not just on stderr: scripts discard stderr, and + // the pagination siblings keep describing the server page, so a + // reduced page would otherwise read as complete. emitted_rows + // is set only when rows were WITHHELD (a prefix was dropped), + // which is the case paging can repair; when instead long values + // were clipped, truncated rides alone — re-requesting cannot + // restore them, narrowing --fields can. + value["truncated"] = true + if len(bounded) < len(rows) { + value["emitted_rows"] = len(bounded) + } + } out, err := marshalStructured(value) if err != nil { return err diff --git a/internal/cli/gen_support_test.go b/internal/cli/gen_support_test.go index 6ba00bf..b31b1d1 100644 --- a/internal/cli/gen_support_test.go +++ b/internal/cli/gen_support_test.go @@ -90,14 +90,36 @@ func TestPrintGenericResultBoundsListEnvelope(t *testing.T) { t.Errorf("bounded %s output lost envelope key %q:\n%s", format, key, out) } } + // The reduced page says so in the payload, not only on stderr — + // with the row-withheld shape: emitted_rows present alongside + // truncated. A values-clipped reduction carries truncated alone. + if format == "toon" { + // TOON renders the marker as envelope-level lines beside the + // items block. + if !strings.Contains(out, "truncated: true") || !strings.Contains(out, "emitted_rows:") { + t.Errorf("bounded toon output lost the in-payload truncation marker:\n%s", out) + } + } if format == "json" { var envelope map[string]any if err := json.Unmarshal([]byte(strings.TrimSpace(out)), &envelope); err != nil { t.Fatalf("bounded json is not an object: %v", err) } - if _, ok := envelope["items"].([]any); !ok { + items, ok := envelope["items"].([]any) + if !ok { t.Fatalf("bounded json lost the items array: %v", envelope) } + if envelope["truncated"] != true { + t.Errorf("bounded json truncated = %v, want true", envelope["truncated"]) + } + emitted, ok := envelope["emitted_rows"].(float64) + if !ok { + t.Fatalf("bounded json lost emitted_rows: %v", envelope) + } + if int(emitted) != len(items) || int(emitted) >= len(rows) { + t.Errorf("bounded json emitted_rows = %v, want len(items)=%d and < %d requested rows", + envelope["emitted_rows"], len(items), len(rows)) + } } }) } @@ -227,4 +249,49 @@ func TestPrintGenericResultShortenedRowStaysUTF8(t *testing.T) { if !strings.Contains(stderrText, "were shortened to fit") { t.Errorf("shortened row should announce the clipped fields on stderr, got:\n%s", stderrText) } + // Clipped values are data loss too: the marker appears even though every + // row was emitted — and it rides WITHOUT emitted_rows, because paging + // cannot restore a clipped value (only a narrower --fields can), so the + // withheld-rows continuation must not be read into this payload. + var envelope map[string]any + if err := json.Unmarshal([]byte(strings.TrimSpace(out)), &envelope); err != nil { + t.Fatalf("shortened single row is not a JSON object: %v", err) + } + if envelope["truncated"] != true { + t.Errorf("shortened single row lost the in-payload truncation marker: %v", envelope) + } + if _, ok := envelope["emitted_rows"]; ok { + t.Errorf("a values-clipped page must not carry emitted_rows (it misreads as withheld rows): %v", envelope) + } + if items, _ := envelope["items"].([]any); len(items) != 1 { + t.Errorf("clipping must not drop rows, got %d items", len(items)) + } +} + +// TestPrintGenericResultCompleteEnvelopeUnmarked guards the marker's negative +// case: a page that fits carries no truncated/emitted_rows keys — the marker +// means "this page was reduced", not "this command supports reduction". +func TestPrintGenericResultCompleteEnvelopeUnmarked(t *testing.T) { + saveAndResetGlobals(t) + stub := newGFStub(t) + stub.data = map[string]any{ + "items": []any{ + map[string]any{"incident_id": "inc-1", "title": "small", "severity": "Info"}, + map[string]any{"incident_id": "inc-2", "title": "smaller", "severity": "Info"}, + }, + "total": 2, + "has_next_page": false, + } + + out, stderrText, err := execCommandSplit("insight", "incident-list", + "--start-time", "7d", "--end-time", "now", "--output-format", "json") + if err != nil { + t.Fatalf("execCommandSplit: %v", err) + } + if strings.Contains(out, "truncated") || strings.Contains(out, "emitted_rows") { + t.Errorf("within-budget envelope must not carry the truncation marker:\n%s", out) + } + if strings.Contains(stderrText, "note: emitted") { + t.Errorf("within-budget envelope must not announce a reduction, got:\n%s", stderrText) + } } diff --git a/skills/flashduty/SKILL.md b/skills/flashduty/SKILL.md index fb19224..d3afbec 100644 --- a/skills/flashduty/SKILL.md +++ b/skills/flashduty/SKILL.md @@ -29,6 +29,8 @@ Append `--output-format toon` to read commands: it drops the per-row repeated ke **Shape the payload before you fetch it.** For ID scans, counts, or "find the matching row" tasks, prefer `--fields` projections and compact list verbs over full detail dumps. Huge raw JSON dumps are a last resort, not a default. +**A structured list page is capped at 16 KiB.** An oversize page is emitted as its leading rows, and the envelope carries `"truncated": true` with `"emitted_rows": N`; `total` / `has_next_page` / `search_after_ctx` still describe the page as the server returned it, so a reduced page reads as complete unless you check the marker. `truncated` WITHOUT `emitted_rows` means long values inside the rows were clipped — narrow `--fields` instead of paging. When a walk must cover everything (a script, a full export), resume after the last row you actually received: re-request with a `--limit` no larger than the rows you got, then follow that response's cursor. + **Empty result = authoritative not-found.** A filter returning `[]` means no such entity in scope — report it (optionally the 1–2 closest names) and stop. Do **not** brute-force (no shifted-keyword re-queries, no widening past caps, no full-dump grep). Never infer "feature not enabled" from an empty list, and never fabricate data absent from tool output. **A result you did not fetch is "unknown", never "empty" — and "fetched" means the same scope, not just the same verb.** You may report a command's result for a given window, entity, or aspect — including "returned empty" or any count/list/finding — **only if a call covering that exact scope appears in your tool-call history this turn**. A wider or different time window, or a sibling entity's result, does not transfer: extrapolating from what you did fetch is the same fabrication as skipping the fetch. If the scope wasn't queried, the honest answer is "未查询 — 可运行 ", not a filled-in number or a generalized claim.