Skip to content

Add initial performance investigation tooling - #122

Open
lightspeed-internal-1[bot] wants to merge 13 commits into
mainfrom
perf/initial-profiling-findings
Open

lightspeed-internal-1[bot] wants to merge 13 commits into
mainfrom
perf/initial-profiling-findings

Conversation

@lightspeed-internal-1

Copy link
Copy Markdown
Contributor

Original task — issue #121

Implement the initial runtime performance-investigation milestone from the published perf/initial-profiling-findings starting point: a standalone OpenAI Chat Completions simulator, a TypeScript driver using the public runtime API, minimum queue/execution instrumentation, and stepped experiments for rapid agent loops, idle/active event readers, and mature histories. Publish measured findings/data on issue #121 and stop before environment profiling or a general optimization campaign. Use isolated synthetic data only; no production endpoints/data or billable model calls.

Delivered

  • Standalone crates/llm-simulator serving native Chat Completions responses with deterministic scenario/tool sequences, response sizes/usage, delay/jitter, scripted failures, and per-request metrics.
  • scripts/perf/ driver, fixtures, scheduler/accounting tests, run scripts and observations; minimal optional Temporal SDK metrics and SQLx pool-acquisition diagnostics.
  • Three requested scenario families plus simulator headroom/repeatability checks, run on October 19, 2026 on the exclusively assigned 16-logical-CPU VM using release builds, loopback synthetic traffic, local PostgreSQL/Temporal/MinIO and the real runtime/provider/CAS/Temporal paths.
  • Initial measured signals: rapid-loop throughput flattened near 3.9 completed runs/s on this local Temporal development topology; the 2,048 reads/s step exposed gateway EMFILE errors at a 1,024 soft file-descriptor limit; the simulator sustained 1,000 small requests/s. These are not production or bank-capacity claims.
  • Roadmap implementation progress and limitations are recorded. No tuning, environment profiling, billable calls or production traffic.

Validation

Passed: cargo test -p llm-simulator --locked (4); OpenAI endpoint/client integration (6); explicitly selected local simulator/runtime-adapter/CAS integration (1); runtime config tests (9); driver scheduler/accounting tests (7); TypeScript typecheck; cargo fmt --check; npm run check:docs; local roadmap links; git diff --check; public API smoke (4/4 completed); scripted context-limit failure classification; and the exact required gate cargo clippy --workspace --all-targets --locked -- -D warnings.

No ignored Temporal live suite, runs_live_slow, credentialed provider test or environment-capacity test was run. Platform-authenticated reads, hardware CPU/allocation profiles and some CAS/context attribution remain explicit gaps. Full tables, exact commands, raw result files and diagnostic limitations will be published in the issue report with an accessible artifact link.

Handoff / remaining

Implementation commit: f5f8ae6f985b0bab1fb3bd4e7302cc5980d32dcd. This PR is for human review against main; do not merge automatically. The issue report and accessible raw-data asset are the final delivery and then this workstream stops pending follow-up. Luke's latest task clarification was to use the exclusively assigned VM for these experiments within the stated scope and safety limits; that was followed.

lukebuehler and others added 2 commits October 19, 2026 13:53
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
@lightspeed-internal-1

Copy link
Copy Markdown
Contributor Author

No description provided.

@lightspeed-internal-1 lightspeed-internal-1 Bot added the engineer-managed Internal routing marker for an issue-owned Engineer pull request; does not authorize work. label Oct 19, 2026
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>
Signed-off-by: Engineer <310755840+lightspeed-internal-1[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engineer-managed Internal routing marker for an issue-owned Engineer pull request; does not authorize work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant