Repository navigation
Performance investigation: initial runtime findings and data #121
Description
Task
Engineer: implement and run the first performance-investigation milestone, then publish the initial findings and data as a report comment on this issue and stop. We will choose the next work from that evidence. Do not proceed into environment profiling or a general optimization campaign.
Published starting point
- Branch:
perf/initial-profiling-findings. - Published commit:
41e702fff814b6296c85a2838a037b95de6bdf51. - Plan: Performance profiling and load testing, especially First handoff: initial findings and data.
Base the implementation on this published branch/commit and preserve the roadmap. No local writer is active on this workstream. Open a PR against main for the implementation when ready; leave it for human review and do not merge. This issue owns the handoff and report; do not independently enroll a second workstream on its PR.
First milestone
- Add one standalone workspace binary crate,
crates/llm-simulator. Start with the OpenAI-compatible Chat Completions API,POST /v1/chat/completions(openai:completions), not the Responses API. Use normal provider endpoint configuration; keep the realllm-clients,llm-runtime, CAS and Temporal execution paths. No second higher-level load-testing LLM fake or production simulation branches. The existingFakeLlmcan keep its current correctness-test role. - Support configurable delay, valid response sizes/usage, deterministic text/tool-call sequences and enough scripted failures to validate the setup. Keep concurrent scenarios isolated. Reuse native provider types where useful; the simulator has no harness, Temporal or Lightspeed-store dependency.
- Add the lightweight
scripts/perf/TypeScript load driver and reproducible scenario fixtures, using the generated API client where applicable. Drive the public HTTP API, follow events, distinguish admission from completion and account for scheduled/submitted/completed/failed/unfinished work. Measure generator and simulator headroom so their limits are visible. Use enough independent sessions to apply pressure; reproducing hundreds of users/sessions is not a prerequisite. - Add the minimum instrumentation needed to explain queueing versus execution: request/run latency and throughput, worker/Temporal observations, database pool/query costs, CPU/memory and relevant CAS/context costs. Avoid making a broad observability redesign a prerequisite. Record any unavailable measurement explicitly.
- On one isolated, fixed-resource deployment with release builds, run stepped-load baseline/saturation experiments for rapid agent loops, idle/active event readers, and large/mature session histories. Use text turns and admitted VFS/blob tools as needed. Separate fixture creation and warm-up from measurement; retain the effective configuration and exact commands. Check drain behavior after stopping arrivals. Profile the implicated paths around the observed limits, using CPU/allocation/query evidence where available.
Start through the public runtime API. Include the Platform-authenticated read path where practical; if it cannot be measured in this first pass, explicitly report that coverage gap instead of claiming to have tested it.
Stopping point and scope limits
- Stop after publishing the first report on this issue. Do not interpret the broader roadmap checklist as permission to implement the entire program.
- No environment profiling: no envd/environment-gateway capacity tests, Incus/VM experiments, environment-backed process/tool capacity tests or file-transfer profiling.
- Independent code-interpreter profiling, broader fan-out/mixed-load campaigns, extensive recovery/soak tests and bank-scale capacity qualification are follow-up work.
- Make only changes necessary to implement and validate the measurement setup. Report candidate optimizations and recommended next experiments; do not launch a tuning campaign before we review the first data.
- Use isolated test data/infrastructure. Do not run experiments against production or make billable model calls. Keep normal endpoint validation, tenant boundaries and review/CI requirements.
- Update implementation progress in the roadmap. Do not modify the root README or unrelated documentation.
- Do not merge or auto-close this planning issue. After the report, await explicit follow-up instructions before expanding scope.
If infrastructure or a scenario is blocked, report the concrete blocker and all findings/data already obtained on this issue. Do not silently skip it, fabricate zero-cost measurements, or substitute environment work.
Required report on this issue
Publish a substantive issue comment with compact tables directly in the comment, plus accessible links to retained raw data/profiles where needed. A local report path or PR description alone is not delivery.
Include:
- Published implementation SHA and PR, scoped test/check results, resource limits, role/replica placement, dependency topology/versions, worker/pool/cache settings, seed, scenario definitions and exact reproduction commands. Exclude credentials.
- For every load step: offered/submitted/completed throughput, p50/p95/p99 latency, errors and unfinished work, queue growth/drain behavior, CPU/memory and available PostgreSQL/Temporal/CAS measurements. Distinguish run acceptance, first visible progress and terminal completion where measured.
- Generator/simulator capacity evidence, profiles/query statistics supporting findings, repeatability/variation, and warm/cold-state assumptions.
- Measured bottlenecks separated from hypotheses, gaps and limits on extrapolation. Do not infer bank capacity from a small synthetic setup.
- A ranked recommendation for the next optimizations or experiments, including whether each appears to need code, configuration, shared-service capacity or more workers.
Validation
Follow repository AGENTS.md: meaningful simulator/client-adapter integration coverage and concurrent scenario-isolation tests; driver scheduling/accounting/deadline tests; appropriate scoped Rust/TypeScript checks. Run the exact Rust PR gate before publishing Rust changes:
cargo clippy --workspace --all-targets --locked -- -D warningsExternal suites must be explicitly selected; serialize existing ignored Temporal live suites under the repository rules. Do not run runs_live_slow as part of this milestone. Run documentation checks when updating the roadmap.
Already done locally: the roadmap is committed and pushed; npm run check:docs, local-link validation and whitespace checks passed. No simulator, instrumentation or load experiment has been implemented or run yet.
Activity
lightspeed-internal-1 commentedon Oct 19, 2026
Issue #121 — initial performance investigation report
Implementation: code commit
f5f8ae6f985b0bab1fb3bd4e7302cc5980d32dcd; latest PR heade2fa59adf366f11b6547f343e4b2a70383330901adds a roadmap publication-status correction. Branchperf/initial-profiling-findings; open, unmerged PR #122 againstmain.Raw evidence: download the complete 5.7 MiB raw-data bundle or download this report. The archive contains per-step JSON/configs, raw timestamps, PostgreSQL snapshots, SDK/process/container samples, query plans, runtime/generator limits, validation logs and a source snapshot. Verify the bundle using its SHA-256 checksum.
Setup and reproducibility
The current VM was exclusively assigned to this implementer (Luke's direct clarification on October 19, 2026). It had 16 logical CPUs, 49,254,480 KiB physical RAM (~46.97 GiB),
cpu.max=max 100000,memory.max=max, no swap; the local containers had no additional CPU/memory quotas. This was a fixed whole-VM resource budget with a single-host, shared-resource topology—not isolated service quotas. Generator, runtime and dependencies competed on the same host; measurements record their respective process/container use.One release
lightspeed-runtimeprocess served gateway+sessions, one releasellm-simulator, and one Node/tsx driver. Synthetic traffic used loopback only, single auth mode, a synthetic default universe, task queueperf-121, CAS prefixperf-121; no environment/code worker, production service, credentialed provider or billable model call was used. PostgreSQL 17.11 (runtime DBlightspeed, max connections 100, shared buffers 128 MiB,pg_stat_statements, I/O timing disabled), Temporal CLI 1.9.0 / server 1.32.0 / UI 2.54.1 (localstart-devwith SQLite persistence), and MinIORELEASE.2026-09-22T19-25-18Zwere local dependencies. Image digests and exact effective settings are inresources.txtin the archive. These are development dependencies, not a production-like Temporal persistence topology.Runtime settings: SQLx default pool 10 connections / 30 s acquisition timeout; Temporal SDK 1.0.0 defaults of 100 workflow and activity slots, cache 1,000, five workflow and activity pollers (sticky/nonsticky ratio 0.2); one universe-cache entry, maximum 1,024 and 4 h idle eviction; CAS inline threshold 64 KiB; checkpoint thresholds 512 events or 2 MiB tail, with ten terminal runs also advancing checkpoints. Provider debug dumps were disabled. SDK histogram values are milliseconds and can round sub-millisecond observations to zero.
Seed was 121. Rapid/active scenarios used 64 independent sessions; each step prepared fresh sessions and one warm-up turn per session, then offered arrivals for 30 s with at most one in-flight run per session and a 64-session cap. The simulator used 10 ms delay plus 0–2 ms deterministic jitter, 32-byte user text, 256-byte output and synthetic usage (100 input/64 output tokens). History fixtures used 16 sessions, 32 measured/preparatory history turns plus warm-up as configured, and 8 KiB input/output text per turn with compaction disabled. Their repetitive payloads favor CAS deduplication and warm caches; data was retained between steps. No cold-worker/OS-cache result is claimed. Idle readers used 1 s long-polls; reader loads increased through 2,048 offered reads/s. Generator dropped arrivals explicitly at concurrency/session bounds; it did not build an unbounded queue.
acceptance, first run-associated visible event and terminal latency are measured relative to scheduled arrival. First visible progress is an upper bound and may coincide with completion. Completed-through-arrival and including-drain rates are both reported. Latency quantiles cover submitted work; dropped work has no run latency. A 60 s drain was allowed. Initial idle-reader attempts withlimit=1000were invalid (API maximum 512); their errors are retained underinvalid-reader-attempts/and excluded from reported capacity rows; the sweep was rerun withlimit=512.Reproduce from the implementation checkout. Local dependency start/PG setup is in
reproduce.sh; use separate terminals for the runtime, simulator, observer and driver. The captured run usedOUT=/tmp/issue-121-perf.For the separately labeled pool diagnostic only, restart the runtime with
RUST_LOG=info,sqlx::pool::acquire=debug, then runnpx tsx scripts/perf/run.ts "$OUT/pool-diagnostic.json" "$OUT/pool-diagnostic". It adds logging overhead and is not the 2,048-reader failure measurement. Full effective fixtures, source/binary hashes and exact per-step settings are in the bundle. The scripts reject non-loopback targets.Measured results
The following tables are the generated
report-tables.mdoutput from the archivedanalysis.jsonand retained raw samples. All main steps use 30 s arrivals. Rates are per second; latencies/drain are milliseconds.F/U/D= failed/unfinished/dropped. Completed rates show during-arrival / including-drain; reader rows count HTTP reads, not agent runs.All main steps use 30s arrivals. Rates are per second; latencies/drain are milliseconds. F/U/D = failed/unfinished/dropped. Completed rates show during-arrivals / including-drain. Read rows count HTTP reads, not agent runs.
Every submitted run in these tables was admitted. Read admission is not a run admission. Latencies below describe terminal results, excluding transport failures and dropped work. First visible means first run-associated event observed after admission; it may be the acceptance event, not first output.
CPU is average cores (1.0 = one core). Runtime RSS is sampled peak; PG/Temporal RSS is the final container sample. Sampling is approximate at window boundaries.
SQL times are query execution sums, not full request or transaction latency. Temporal times are SDK mean milliseconds; idle steps have no execution observations. CAS column is the subset of SQL calls/time involving CAS tables, not all CAS I/O.
Simulator headroom (10s arrivals, max concurrency 256). All requests completed; zero errors, drops or unfinished work. Large input is 540,672 bytes plus an identifying prefix, with 8KiB output.
Additional reads of the prepared mature fixture, one page of at most 512 events per request (no model calls):
Findings and attribution
Measured: rapid loops flattened near 3.9 completed runs/s on this local Temporal topology. At 10 offered runs/s, 149/300 arrivals were submitted under the 64-session bound; all submitted runs completed, median terminal latency was 15.05 s, and drain was 8.42 s. A repeat completed 151 vs 149 runs (+2.0% including-drain throughput; p50 -1.9%, p95 -2.1%). Runtime averaged roughly 0.23 CPU cores. Workflow schedule-to-start averaged about 989 ms while activity execution averaged about 3.37 ms; SDK
append_eventswas ~2.75 ms and LLM activity ~14 ms. A retained history sample had 56 workflow tasks and 51 activities; mean workflow schedule-to-start 830 ms, server-observed workflow task execution 93 ms and activity execution 92 ms. Interpretation: queuing/Temporal completion round trips, not simulated inference delay, dominate this rapid-loop sample. SQLite persistence/dispatch is a hypothesis, not a proven root cause. A normal-queue describe snapshot at zero backlog does not cover sticky queues and does not invalidate the measured wait.Measured: adding 64 active event readers at 2 runs/s increased query calls from 5,483 to 29,065 (~5.3×); at 10 runs/s, calls rose from 31,414 to 64,809. These deltas exclude fixture setup/warm-up. At 2,048 offered idle reads/s the gateway logged
EMFILEaccept errors with a 1,024 soft / 524,288 hard FD limit. At 512 reads/s all 15,360 requests succeeded with ~1,006 ms p95 (near the configured 1,000 ms wait). At 2,048, 37,525 requests were submitted, 28,824 completed, 8,701 transport-failed, and 23,915 arrivals were dropped; 22,675 drops hit the concurrency bound and 1,240 missed the arrival window. Driver p99 dispatch lag was 51 ms. This is a measured connection-admission/resource limit, not evidence of PostgreSQL CPU saturation. Limits were intentionally not tuned.Simulator headroom reached 1,000 small requests/s and 100 large-input requests/s with zero errors/drops/unfinished work; at the largest small request step it used ~0.092 CPU cores, while the generator used ~0.40 cores. The measured runtime plateau was ~3.9 runs/s, so neither was near that limit. Additional reads on the mature fixture completed 3,000/3,000 at 100 reads/s (p95 17 ms, no model calls). Query-statistics deltas, indexed EXPLAIN plans, SDK activity metrics, process samples and drain counts are in the archive. A separate 512-read/s pool diagnostic recorded 61,440 acquisition samples with mean/p50/p95/p99/max 0.139/0.129/0.212/0.310/4.118 ms; it is not evidence about the 2,048/s failure point.
Gaps/limits: Platform-authenticated reads were not measured.
perfsampling was unavailable (perf_event_paranoid=4, no sudo under no-new-privileges); allocation/heap profiles and dedicated Temporal server metrics were unavailable. SDK metrics became unreachable during FD exhaustion and later samples are marked unavailable, not zero. Full CAS byte/cache-hit counters and separate context-materialization spans were unavailable; CAS SQL calls/time are only a subset of CAS costs. Small inline/repetitive payloads barely exercise MinIO. History fixture began at 333 events / 33 completed turns and 528 KiB nominal text; no cold-worker, 10,000-event, heterogeneous payload, VFS-tool capacity or production-auth result is implied. Two Temporal continue-as-new events occurred without run failures. One scripted context-length failure was admitted and classified failed, not completed. Two repeat samples do not establish confidence intervals. These small local results make no bank-capacity, production SLO, scaling-efficiency or leak/soak claim.Ranked next experiments/recommendations — no tuning performed
Validation and publication state
cargo test -p llm-simulator --lockedcargo test -p llm-clients --test openai_endpoint_override --lockedcargo test -p llm-runtime --test simulator_local --locked -- --ignored --test-threads=1cargo test -p temporal-runtime config::tests --lockednpx tsc -p scripts/perf/tsconfig.jsonnpx tsx --test scripts/perf/scheduler.test.tscargo fmt --check -p llm-simulator -p llm-runtime -p temporal-runtimecargo clippy --workspace --all-targets --locked -- -D warningsnpm run check:docs; local roadmap links;git diff --checkNo ignored Temporal live suite,
runs_live_slow, credentialed provider test or environment-capacity test was run. Rust 1.99.0, Node 24.20.0 and TypeScript 6.0.3 were used; Node is below the repository's documented 24.21.0 minimum. On initial implementation headf5f8ae6, GitHubchangesanddocschecks passed whilerustandconsumerswere pending. The later documentation-only heade2fa59adpassed localnpm run check:docs; its GitHub CI run 39256038535 was pending at the time of this report update. The exact Rust Clippy gate passed on the unchanged code commit. The PR is for human review; no merge or issue closure occurred. Measurement processes have stopped. Local Docker services/synthetic data remain available in the VM, while the raw archive contains retained evidence. Stop here for human review and explicit follow-up.lightspeed-internal-1 commentedon Oct 19, 2026
Follow-up to the original issue #121 performance-investigation task — Luke (user), October 19, 2026:
The first report is delivered. Continue this same issue/PR workstream for a focused follow-up: adapt the reproducible local performance setup so Temporal persistence uses PostgreSQL as well as Lightspeed's own PostgreSQL database; run the next measurements on that topology to learn more about where improvement is warranted; and publish an updated report with a few evidence-backed top hotspots and proposed improvements.
Use the current exclusively assigned VM and synthetic isolated data only; no production endpoints/data, billable model calls, environment profiling, or broad optimization/tuning campaign. Preserve the previous SQLite-backed results for comparison and label the two topologies clearly. Re-run the relevant rapid-loop, reader, and mature-history steps with release builds where feasible; collect PostgreSQL/Temporal queue/execution and resource evidence, profiles/query evidence where available, and identify what remains a hypothesis. Make only the setup/instrumentation changes needed to run the PostgreSQL-backed tests. Add the results as an update report comment on this issue with accessible raw-data/profile links. Continue using PR #122; leave it open for human review and do not merge. No second PR/workstream.
lightspeed-internal-1 commentedon Oct 19, 2026
Issue #121 — Temporal PostgreSQL follow-up report
Rapid-loop saturation completed 16.30–16.32 runs/s including drain, versus 3.88/s in the retained SQLite topology. The Temporal version/configuration difference prevents a PostgreSQL-only causal claim. The reader FD ceiling reproduced. Profiles and query data point to workflow queueing/durable round-trip volume, reader polling/descriptor costs, and transient Temporal allocations as the next focused investigation targets.
Follow-up to the initial report: Temporal history and visibility now both use PostgreSQL. This is a synthetic local topology comparison on the same exclusively assigned VM, not a production capacity claim. No optimization, environment profiling, production traffic or billable model calls were performed.
Implementation: PostgreSQL setup/measurement commit
8869e08246d2df94de42cd239016a31a9696bf93; PR #122 is open and unmerged againstmainfor human review.Raw evidence: Download the follow-up report, download the 82 MiB raw data/profile bundle, or verify its SHA-256. The original SQLite baseline remains unchanged in the initial artifact release. Paths below are relative to the extracted follow-up archive.
Topology and comparability.
max 100000, memorymax, CPUs 0–15postgres12plugin for history and visibilitytemporalhistory schema 1.18 andtemporal_visibilityschema 1.9, dedicated named volumelightspeedlightspeed_perf_postgres; prior data preservedperf-postgres; single auth modeperf-postgresobject prefix; same inline/cache/checkpoint defaults127.0.0.1:17233, metrics:19093, profile endpoint container-local127.0.0.1:7936; runtime:18080, simulator:19090, SDK metrics:19092The matching
temporalio/auto-setup:1.32.0image was unavailable. The available image reports 1.29.3, pinned by digest inscripts/perf/temporal-postgres/compose.yaml. This changes server version and dev-server configuration as well as persistence. Therefore performance differences cannot be attributed to PostgreSQL alone. A version-matched comparison remains a follow-up experiment. The PostgreSQL image is also pinned by digest. Full image IDs, effective redacted Temporal configuration, schema/row verification, resource limits and runtime settings are retained.Neither container stack adds a CPU or memory quota inside the VM. PostgreSQL defaults include
max_connections=100,shared_buffers=128MB; the new Temporal PostgreSQL enablespg_stat_statementsandtrack_io_timing. Temporal SQL pool limits are 20 history / 10 visibility per configured store, with corresponding idle limits and one-hour connection lifetime. Lightspeed SQLx pool remains 10 with 30-second acquisition timeout. SDK workflow/activity slots remain 100, workflow cache 1,000, pollers five each, sticky ratio 0.2. Runtime soft/hard FD limits remain 1,024/524,288. The existing SQLite container was stopped, without deleting its volume, while the PostgreSQL measurements ran. The original PostgreSQL, MinIO and idle pgAdmin services remain on this VM.Workloads and measurement. Seed 121, unchanged fixture definitions, 30 seconds of measured arrivals, 60-second request deadline. Each step uses newly created independent sessions; preparation and warm-up are excluded. Rapid loops use 64 sessions/in-flight bound, 32-byte input, 256-byte output and 10–12 ms simulated delay. Active-reader steps add 64 event followers. Idle readers use 16 warmed sessions, one-second long polls and no run arrivals; high-rate reader steps use the same 4,096 concurrency bound as the initial reader sweep. Mature histories use 16 sessions/in-flight bound, 32 history turns plus one warm-up, 8 KiB input/output, and disabled compaction (333 events / nominal 528 KiB text at measurement start). These are warm, highly repetitive synthetic payloads favorable to CAS deduplication; they are not cold storage or diverse production histories. The 16-session history limit means its latency cannot be directly compared with the 64-session rapid-loop limit.
Requests use the generated public runtime client and normal configured Chat Completions endpoint; actual adapters, CAS and Temporal paths remain in use. First visible progress is the first run-associated event seen after admission and can be the acceptance event, not model output. Dropped arrivals were never submitted; they are reported separately from failures and unfinished work. PostgreSQL statement snapshots occur after fixture preparation and after drain, filtered by database. Background Temporal maintenance can contribute to these database-wide counters. Process/SDK/server metrics are sampled approximately each second and containers/PG waits approximately every five seconds. Statement execution sums exclude network, transaction commit and application wait time. Wait snapshots are sparse counts, not time attribution.
Startup-only attempts failed before fixture creation because background processes did not survive their launching shell; those logs are retained under
startup-failed-attempts/. Overlapping startup observers produced invalid files, explicitly retained as*-startup-overlap.invalidand excluded from analysis. The first rapid 2/s and 10/s steps retain valid request/query data but lack clean resource/SDK coverage; clean repeats are included. No missing measurements are represented as zero.All load steps.
All main steps use 30s arrivals. Rates are per second; latencies/drain are milliseconds. F/U/D = failed/unfinished/dropped. Completed rates show during-arrivals / including-drain. Read rows count HTTP reads, not agent runs.
Every submitted run in these tables was admitted. Read admission is not a run admission. Latencies below describe terminal results, excluding transport failures and dropped work. First visible means first run-associated event observed after admission; it may be the acceptance event, not first output.
CPU is average cores (1.0 = one core). Runtime RSS is sampled peak; PG/Temporal RSS is the final container sample. Sampling is approximate at window boundaries.
SQL times are query execution sums, not full request or transaction latency. Temporal times are SDK mean milliseconds; idle steps have no execution observations. CAS column is the subset of SQL calls/time involving CAS tables, not all CAS I/O.
Simulator headroom (10s arrivals, max concurrency 256). All requests completed; zero errors, drops or unfinished work. Where present, large input is 540,672 bytes plus an identifying prefix, with 8KiB output.
Temporal persistence evidence and comparison.
Topology comparison: completion rates include drain; terminal latencies describe successful requests. Version/configuration differences prevent persistence-only attribution.
Findings, attribution and recommendations.
Workflow queueing and durable round-trip volume are the main rapid-loop target. At 40/s the unprofiled runs completed 16.30 and 16.32 runs/s including drain (about 0.15% variation), versus 3.88/s in the original topology. Terminal p50 varied 3.56–3.74 s and p95 3.97–4.47 s, so stable throughput does not imply stable tails. At 10/s both trials completed all 300 runs; p50 was 1.35/1.34 s, but drain varied 1.29/3.04 s. Low-load p50 varied 1.05/0.80 s and did not improve over the original 0.60 s. These are short trials, not confidence intervals.
The rapid-40 SDK means were 172 ms workflow schedule-to-start, 85 ms activity schedule-to-start and 4.13 ms activity execution (LLM activity 14.75 ms). A single retained workflow history during the same window contained 120 workflow tasks and 110 activities: workflow queue/execution means 154/13.7 ms; activity queue/execution 79.7/17.9 ms. Server-observed execution includes round trips omitted by local SDK execution. The server reported approximately 4.27 ms mean UpdateWorkflowExecution persistence and 20 ms RespondWorkflowTaskCompleted RPC latency. The sampled per-series matching backlog peaked at 38 tasks, while client outstanding runs reached 64; these are different populations. All submitted work drained. Peak sampled SDK activity/workflow slots were only 9/10 out of 100, and sampled history SQL connections in use peaked at 4 out of its 20-connection pool. Sampling can miss short peaks, but this does not support generic worker-slot or pool exhaustion.
Rapid-40 used about 0.85 runtime cores, 3.08 Temporal cores, 1.56 Temporal-PG cores and 0.47 Lightspeed-PG cores. It did not exhaust the 16-CPU VM. Temporal history executed 307,640 SQL statements in the measurement/drain window, including 26,928 workflow-execution updates and 6,732 history-node reads. The latter consumed 1.65 s of statement execution (~0.245 ms/call). Sparse PG observations saw five active
WalSyncbackend observations and other WAL waits across six samples, alongside many idle connections; these counts are not a time share. Buffer read counters were zero in this warm rapid window, but WAL/commit cost is not zero and is excluded from statement execution sums. Storage latency, matching/poller configuration and serial durable transitions remain attribution hypotheses.Readers retain a separate gateway descriptor ceiling and measurable polling cost. At 512/s, all 15,360 reads completed, runtime used ~0.73 cores and Lightspeed PG ~0.78 cores. The window executed 184,327 SQL statements (~12/read), taking 3.24 s of SQL execution. The read implementation polls at up to 250 ms intervals. At 2,048/s, 8,916 submitted reads failed with transport errors and the runtime logged 34
Too many open filesaccept errors at the unchanged 1,024 soft FD limit. Of 24,092 unsubmitted arrivals, 24,083 hit the concurrency bound and nine missed the arrival window. Failed-request settled latency p50/p95/p99 was 10.49/10.86/10.91 s; successful-request latency is in the tables. There were 3,948 requests outstanding when arrivals stopped, with 5.55 s final drain and zero unfinished work. The SDK metrics endpoint became unavailable (34 failed samples during this step; final curl exit 7) and was not treated as a zero-cost observation. During-arrivals read completion was 915/s versus 921/s previously; the including-drain difference is not evidence that PostgreSQL fixed reader capacity. Generator dispatch p99 rose to 78.7 ms and event-loop p99 to 64.3 ms here, so the generator was also less punctual at overload. It is not an unlimited-rate source.At ordinary run rates, generator dispatch p99 stayed around 1–3 ms. The simulator repeated 1,000 requests/s for 10 s: 10,000 completed, no errors/drops/unfinished work, p50/p95/p99 14/16/19 ms, mean handler 11.94 ms, ~0.095 simulator cores / 11.7 MiB peak RSS, and 4.13 generator CPU-seconds. This establishes headroom above these runtime rates, not the simulator's maximum capacity. The previous large-payload simulator result remains in the SQLite archive; it was not rerun in this follow-up.
Temporal allocations and growing history reads are concrete secondary costs, not yet demonstrated capacity limits. The separately profiled rapid-40 trial completed 548 runs (16.93/s including drain); its approximately 30-second CPU profile recorded 86.21 CPU-seconds, or 2.86 cores.
UpdateWorkflowExecutionAsActiveaccounted for 25.24% cumulative CPU; syscalls accounted for 27.28% flat CPU. Cumulative stacks overlap, and retry-wrapper names do not establish actual retries or isolate disk from network cost. The allocation delta profile, explicitly analyzed with-sample_index=alloc_space, estimates 4,793.84 MB allocated in 30.05 s. Metrics tag-map creation and Tally subscopes account for 326/270 MB (12.4% combined flat allocation); lib/pq write buffers and bytea encoding for 275/152 MB (8.9%). These are sampled transient allocations, not RSS growth or a leak. The initial default pprof rendering selectedinuse_space; those separately named files are retained to avoid confusing it with allocation volume. Profiling/metrics overhead is part of this instrumented deployment, and the profiled result is not substituted for the unprofiled baseline.In mature-history-40, history-node reads took 5.20 s for 4,040 calls (~1.29 ms/call, versus 0.245 ms in rapid-40) and dominated that store's 8.76 s SQL execution sum. Nevertheless, workflow queueing averaged only 36 ms, LLM activity 18 ms, and runtime ~0.67 cores; the sixteen-session limit bounded offered pressure. A post-run warm Lightspeed event-page EXPLAIN used
session_events_pkey, 413 shared-buffer hits, no shared-buffer reads and 1.11 ms execution. This does not measure full API decoding/context reconstruction or cold-history I/O. Runtime peak RSS grew across the sweep from ~162 MiB at the first saturation step to ~659 MiB after reader overload as fixtures/caches accumulated; no leak conclusion follows from this short sequence.No proposed optimization was implemented or benchmarked as a tuning change. More workers, larger pools, more database CPU and index changes are not currently evidence-backed prescriptions. No bank-scale inference is justified.
Coverage and retained evidence. Both PostgreSQL persistence stores are verified by effective
postgres12configuration, schema versions, live connections/query snapshots and 786 final execution rows in each store (including Temporal's internal workflows). Agent runs reuse long-lived session workflows; most visibility writes occur during excluded fixture creation. The low measured visibility query count is mostly observation/background work and is not a visibility capacity test. The comparison also changes server version, standard-server versus dev-server defaults, persistence placement and server-metrics instrumentation. SQLx acquisition debug logging was not enabled in this follow-up; the previous pool diagnostic remains historical evidence only. Server pprof is available, but Rust CPU/allocation profiles remain unavailable (perf_event_paranoid=4, no usable perf installation/privilege path). Complete CAS/cache hit counters, per-request distributed traces, cold-state histories and Platform-authenticated reads remain unmeasured. Single-auth public runtime reads are the measured path. The current Node v24.20.0 is below the repository's stated v24.21 minimum; the listed TypeScript checks passed on that installed version.Raw data include every effective scenario JSON,
fixture.json, all per-request timestamps and accounting insweep/*/results.json, database-specific*-before.csv/*-after.csv,observations.jsonl(process, simulator, SDK and server metrics),container-stats.jsonl,pg-observations.jsonl,analysis.json,temporal-analysis.json, the retained workflow history/timing summary, event-page query plan, effective redacted configuration, schema proofs, image/resource metadata and binary/source hashes. CPU/allocation profiles are insweep/rapid-40-profile/;temporal-*-top.txtandtemporal-*-cumulative.txtprovide readable summaries. Server logs retained after the profiling restart are explicitly namedtemporal-server-profile.log; the pre-recreation server logs were not retained. Baseline service metrics and query data are retained. Invalid startup files are labeled and excluded. The original SQLite archive is unchanged and its checksum was revalidated.Reproduction and checks.
commands.shin the archive retains exact commands; setup is inscripts/perf/temporal-postgres/{compose.yaml,up.sh,prepare-runtime.sh,env.sh,reproduce.sh}. The main sequence is:cargo build --release -p temporal-runtime -p llm-simulator --lockedcargo clippy --workspace --all-targets --locked -- -D warningsnpx tsc -p scripts/perf/tsconfig.jsonnpx tsx --test scripts/perf/scheduler.test.tsnpm run check:docsgit diff --checkruns_live_slowSimulator/native adapter correctness and isolation coverage from the initial implementation remains unchanged; this follow-up exercised the real runtime/adapter/CAS against PostgreSQL-backed Temporal rather than introducing another fake. The new setup was brought up repeatedly against its existing volume without resetting it. No shared statistics reset is performed by the sweep now. Test processes and observers have stopped, both new containers are stopped with volumes preserved, and the original SQLite Temporal service is restored. The implementation is committed at
8869e08246d2df94de42cd239016a31a9696bf93and pushed to PR #122; this update report and its raw-data asset are published for review. PR CI run 39267039768 (changes,docspassed;rust,consumerspending) was still running on this head when this report was prepared. No merge or optimization was performed. Stop here pending the next explicit instruction.lightspeed-internal-1 commentedon Oct 19, 2026
Follow-up from Luke (user), October 19, 2026, to the already-published PostgreSQL Temporal report:
Treat the most recent PostgreSQL-backed benchmarks as the working baseline; use the observed gains as PostgreSQL-related for the purpose of prioritizing what to explore. Do not spend the next round re-litigating the SQLite-versus-PostgreSQL comparison unless a concrete hypothesis requires it. Explore low-hanging performance limits and costs across PostgreSQL, Temporal, and Lightspeed. Run controlled experiments for configuration changes and small, focused code changes where evidence suggests a likely win, and validate any measured improvement against the PostgreSQL baseline.
Do not make complicated code/runtime/architecture changes before checking back with Luke. If a promising direction requires broad or risky implementation, stop at a concise evidence-backed thesis and ask before proceeding. At the end, publish an update report here with any real improvements, or a clear evidence-backed thesis, the top hotspots, exact reproduction details, and accessible raw data/profiles. Keep measurements isolated and synthetic: no production endpoints/data, billable provider calls, environment profiling, merge, or second PR. Continue on issue #121 / existing PR #122; leave it open for human review.
lightspeed-internal-1 commentedon Oct 19, 2026
Addendum to Luke's follow-up in issue comment #6270039524 — Luke's direct instruction on October 19, 2026: Let Codex choose the concrete A/B experiments and what to try; do not over-specify a test plan. Use the PostgreSQL-backed runs as the baseline and assume their gains were PostgreSQL-related for prioritizing exploration. Focus on low-hanging configuration or small code changes; check with Luke before any complicated runtime/architecture change. This is a direction for the same scoped issue/PR workstream, not authorization to merge or enter environment profiling.
lightspeed-internal-1 commentedon Oct 19, 2026
Focused PostgreSQL performance follow-up — issue #121 / PR #122
Outcome: Two configuration wins are retained: Temporal
pgximproved rapid-loop throughput by 16–27% across two driver pairs, and an 8,192 runtime file-descriptor limit eliminated submitted-reader failures at the tested high rate. A reader fast-path prototype reduced SQL/CPU work but consistently lengthened rapid-loop drain, so it was not adopted.Implementation commit:
4548388432fbe4091c27489cca9b34368d750b71; PR #122 is open and unmerged for human review. This update report and all-step tables are published on issue #121. The data-only prerelease includes the report, all-step tables, raw data/profiles bundle, and SHA-256 checksum.The previous PostgreSQL and SQLite artifacts remain separate and unchanged. Base revision, final implementation patch, source/binary hashes, and the non-adopted candidate prototype are retained in the raw bundle.
Method and scope
This follow-up treats the latest PostgreSQL-backed deployment as the working baseline. It tests a supported Temporal PostgreSQL driver, the runtime descriptor limit, and one small caught-up event-reader fast path. No durability setting, worker count, pool size, hardware allocation, or architecture was changed. All workloads use synthetic local data, the real public runtime API and provider adapter, and the local simulator. No environment work or billable model calls were performed.
Each arm starts a fresh runtime and simulator; each step creates independent sessions and finishes fixture creation and warm-up before measurement. PostgreSQL volumes, caches and accumulated history are preserved. Driver switches restart Temporal; repeated same-driver arms leave the Temporal service warm. These are short warm-state tests on one VM, not cold-start, soak, or bank-capacity claims. Raw timestamps, query snapshots, service/process samples, effective configurations, commands and profiles accompany the report.
The first control arm's four workload steps and profile completed, but its outer script exited 2 after the last step because the running script was edited. Its successful workload results are retained with an explicit orchestration note, not reported as a clean orchestration run. Subsequent arms execute an unchanged runner. A reverse driver repeat and persisted-session continuity checks provide independent validation.
Deployment and reproduction
Base PostgreSQL setup commit:
8869e08246d2df94de42cd239016a31a9696bf93; this follow-up's retained configuration/setup changes are committed in4548388432fbe4091c27489cca9b34368d750b71, the current head of existing PR #122. The final change defaults only the performance setup to Temporal'spostgres12_pgxdriver and launches its runtime with an 8,192 soft FD limit; it does not alter production runtime defaults. The original runtime binary was reused for each configuration/control arm. Candidate fast-path measurements used a separate release binary; its prototype and tests are retained incandidate-fastpath.patchbut are excluded from the PR because the rapid-loop tradeoff was negative. The final rebuilt binary is separately identified and has no performance claim; no compilation or profile reduction ran during load windows.max 100000, memorymax, CPUs 0–15; no additional container quotastemporalhistory andtemporal_visibilitydatabases, schema 1.18/1.9; both use the arm's drivermax_connections=100,shared_buffers=128MB,pg_stat_statements,track_io_timing; Temporal SQL pools 20 history/10 visibility per configured store, matching idle limits, one-hour lifetime; durability unchangedperf-postgrestask queue, single-auth public runtime API; SQLx pool 10, 30s acquisition timeout; SDK slots 100 each, pollers five each, workflow cache 1,000lightspeed_perf_postgres; existing MinIO,perf-postgresobject prefix; same inline/cache/checkpoint settingsImage digests, current resource limits and redacted configuration are in
topology/and each arm. Full reproduction commands are incommands.sh, with selected checks inchecks/run.sh. Use fresh output directories; no command resets shared databases or removes volumes. To recreate the original control binary, build the recorded base revision in a separate checkout and setPERF_RUNTIME_BINARYto that release executable. To reproduce the candidate, applycandidate-fastpath.patchas well asimplementation.patchto that base and runcargo build --release -p temporal-runtime -p llm-simulator --locked. Prepare local dependencies with the existingup.sh/prepare-runtime.sh; never direct these tests at production.Workloads retain seed 121, 30s arrivals and 60s deadlines. Rapid loops: 64 independent sessions/in-flight bound, 32-byte input, 256-byte output, 10–12ms simulator delay. Active-reader controls add 64 event followers at 10 runs/s. Idle readers: 16 warmed sessions, one-second long polls, concurrency bound 4,096. Mature histories: 16 sessions, 32 history turns plus warm-up, 8KiB input/output, compaction disabled (333 starting events / nominal 528KiB text). Repeated synthetic text favors deduplication. Read counts are HTTP reads, not agent runs. First progress can be the acceptance event; it is not time to first model token.
All raw scheduled/submitted/accepted/completed/failed/dropped/unfinished samples are retained. CPU means use sampled windows; container memory is a sampled final value, process RSS a sampled peak. Query deltas include background maintenance and measure SQL execution, excluding network, client encoding, transaction commit and application wait. Sparse PostgreSQL wait observations are not wait-time attribution. Profile windows are separately labeled and do not define the unprofiled throughput win.
Demonstrated changes
postgres12_pgx; retain explicitpostgres12control.The two unprofiled driver pairs improved throughput by 16.3% and 27.1%. Their ranges do not overlap, but two pairs do not establish a confidence interval. Mature-history driver gains were smaller (10.90 → 11.40 runs/s). The 16-session bound and 500ms admission polling quantization influence that workload. No claim is made that every workload improves by the rapid-loop percentage.
At 2,048 reads/s, the candidate code completed 61,428 of 61,440 scheduled reads, with zero failures/unfinished and 12 generator-bound drops, versus 60,898/542 in its old-binary control. Terminal p99 was 1.46s versus 2.27s. Generator pressure makes this latency/capacity observation less conclusive than the deterministic query-count reduction. One-second long-poll wait remains intentional; the fast path is chiefly a CPU/query saving, not a claim to remove that wait.
Temporal evidence and remaining hotspots
postgres12pgxpostgres12pgxActivity execution stayed roughly 4.1–4.3ms, far below queue times. Sampled worker slot use peaked at 7–9 workflow and 8–11 activity slots out of 100. History SQL pool active connections peaked at four/five out of twenty in the first driver pair. There were no sampled database deadlocks, temp spills or rollback growth; data was predominantly buffer-resident. These observations do not justify increasing worker count, pool sizes or shared buffers as the next default action.
The first driver pair recorded 312,609 / 357,452 history SQL calls (about 554 / 547 per completed run) and 7.62 / 9.21s SQL execution sums. History-node reads were the largest SQL execution contributor: 6,768 / 7,848 calls, 1.71 / 2.36s aggregate, exactly twelve calls per completed run. Current-execution selects/updates and execution updates each occur 48 times per run. Visibility had only eight/seven calls in these measurement windows; its steady-session cost is not a bottleneck here, and session-creation-heavy visibility capacity was not tested. The substantially lower PostgreSQL CPU with
pgxis measured; binary encoding/preparation/transaction behavior as its explanation is a hypothesis, since statement execution sums omit several of those costs.Separately profiled 30s repeats retained native CPU and delta
alloc_spaceprofiles. The control recorded 94.30 CPU-seconds / 5.42GB allocated (pprof units) andpgx100.62 CPU-seconds / 6.16GB while processing more work. These are total transient allocations, not retained RSS, and do not establish a large total allocation reduction. Control flat allocation sites included lib/pq write buffers (4.82%) and bytea encoding (2.70%);pgxshowed its binary bytes encoder (2.38%) and protocol Bind encoding (1.61%). Common remaining sites were history executable-tracker splitting (11.27% / 10.21% flat) and metrics tags + Tally subscopes (11.19% / 11.76% combined flat). Nested cumulative percentages are not additive. Profiles and top/cumulative text are under each driver'ssweep/rapid-40-profile/.Ranked next work, after review
The demonstrated
pgxand FD-limit wins are implemented narrowly. The reader fast path is not adopted despite its measured reader savings. No complex runtime or architecture changes were made. Pool growth, adding workers, larger PostgreSQL buffers and durability reductions are not recommended from this evidence.Gaps and limits
Platform-authenticated reads remain unmeasured; this uses public runtime single-auth mode. Rust CPU/allocation profiles remain unavailable under the VM's performance-counter restrictions, and full CAS object I/O/cache/context timing is not available. CAS SQL is reported separately but is not total CAS cost. No new SQLx acquisition tracing was enabled; the earlier 512-read/s diagnostic is separate evidence, not a measurement in these arms. Temporal queue means and sampled approximate-backlog gauges are retained, but sparse gauges do not constitute complete queue-length timelines. Outstanding client work at arrival stop and exact drain are reported for every step; all admitted agent runs drained to completion.
Simulator code/configuration is unchanged. The retained previous PostgreSQL report's standalone 100/500/1,000 requests/s and large-input capacity tests remain supporting evidence; this follow-up retains actual per-step simulator request/error/handler counters and sampled process CPU/RSS. Rapid-loop generator dispatch lag is small; at 2,048 reads/s, generator CPU/event-loop lag and its 4,096 concurrency bound materially limit extrapolation. This is a shared-VM synthetic setup, not an independently provisioned client or production capacity qualification. Profiles are one diagnostic repeat per driver, not a statistical allocation study. Warm caches, persistent database growth, short trials and workload/session bounds limit generalization.
Reader resource A/B detail
All rows here use
pgxand 8,192 runtime FDs. At 512/s the reduction is 41.7% in SQL calls, 34–37% runtime CPU, and 36–44% runtime-PG CPU across two pairs. At 2,048/s the old binary performs fewer poll cycles per delayed request, so its calls/read are already below twelve; the percentage saving must not be extrapolated mechanically. Generator CPU was 33.14 → 30.76 seconds over the high-rate windows, event-loop p99 88.8 → 41.2ms, and RSS 633 → 563MiB. This supports the explicit generator-headroom caveat.Reader prototype not adopted: rapid-loop tradeoff
The candidate's including-drain throughput was 5.5–8.5% lower across these three pairs; the difference during arrivals was much smaller. All submitted runs completed. Ordering reversal did not remove the longer candidate drain. We have not established the mechanism or isolated a source-level effect from release-binary layout/timing effects. The stronger reader query/CPU result is therefore not sufficient to call this an unqualified application win. The five-line prototype and its passing integration-test additions remain in
candidate-fastpath.patchfor review, and are absent from the final working-tree production changes. No further tuning was attempted to compensate for the tradeoff.Complete data, validation and handoff
There are 28 measured steps. All 9,690 submitted agent runs completed; none failed or remained unfinished. The deliberately overloaded 1,024-FD idle-reader control produced 8,919 failed reads. Scheduled arrivals dropped at session/generator bounds are accounted for separately, never counted as admitted failures or hidden as completed work. See all-step tables for every step's offered/submitted/completed rates, counts, acceptance/first-progress/terminal p50/p95/p99, outstanding work, drain, CPU/memory, generator lag, SQL/CAS and Temporal measurements.
all-analysis.jsonsupplies the machine-readable combined results; each arm includes raw samples, query snapshots, effective configuration and logs. Native profiles and readable top/cumulative summaries are in the two profiled driver arms.commands.sh,summarize.pyand repository analysis scripts reproduce the tables.Validation logs and exact invocations are retained under
checks/: release builds, eleven store unit tests, four explicitly selected serialized local storage tests (tenant isolation; fork/history/subsequent append; range/checkpoint fencing; soft-delete history), seven driver scheduling/accounting tests, scoped TypeScript checking, shell/Python checks, documentation checks, local roadmap-link validation and whitespace checks. The exact Rust gate was run both on the prototype and again on the final tree:The four live tests were selected individually with
--ignored --test-threads=1against synthetic local data; no ignored Temporal live suite, credentialed provider suite orruns_live_slowwas run. Prototype test additions are in its separate patch, not in the final implementation patch. The final release rebuild removes the prototype from the executable as well as the source. The final launcher health/FD check passed. Both prior artifact archives passed SHA-256 verification unchanged. The isolated PostgreSQL Temporal services were stopped, their volumes preserved, and the original SQLite service restored. Final launch/cleanup records are retained alongside the checks.Publication status: Implementation commit
4548388432fbe4091c27489cca9b34368d750b71is pushed to existing PR #122, which remains open and unmerged for human review. This report and raw artifacts are linked in the issue #121 update comment and the data-only prerelease above. At report preparation, current-head CI run 39278695493 was queued. The only adopted changes are the isolated Temporalpgxperformance default and 8,192-FD launcher; do not applycandidate-fastpath.patchwithout resolving its measured rapid-loop tradeoff. No broad code/runtime redesign or tuning campaign was performed. Stop here and await explicit follow-up.lightspeed-internal-1 commentedon Oct 19, 2026
Further direction from Luke (user), October 19, 2026, for the already-authorized performance follow-up:
Please include a focused investigation of workflow design—especially the session workflow—to understand which measured round-trip/queue costs are intrinsic to durable Temporal execution and which may come from Lightspeed's workflow structure or repeated transitions. Use the PostgreSQL-backed results as the working baseline and consider the other already-measured hotspots too. Choose the smallest useful diagnostics/experiments; do not over-plan. If a small, safe, evidence-backed change is clearly worthwhile, validate it; check with Luke before any complex workflow/runtime redesign. If the cost is inherent or no safe change is established, say so clearly with data and the next best experiment. Publish the findings in an update report on issue #121 with accessible evidence. Keep the existing PR #122 open/unmerged and all prior isolation/no-production/no-billable/no-environment guardrails.
lightspeed-internal-1 commentedon Oct 19, 2026
Session workflow bookkeeping A/B — issue #121
Bottom line: No clear rapid-loop win; do not adopt the batching change.
The test combined
turn.plannedandturn.generation_requestedin one append. It removed one activity and one append per run, reducing SQL calls. All 3,652 admitted runs completed. But the rapid results were mixed, one variant had a longer drain, and event visibility became atomic rather than separately observable.Why the waits: A saturated sampled run cycle averaged 2.46s: 0.86s in activity queues, 1.25s between workflow tasks, and 0.35s in server-observed execution intervals. Queue waits were about 86% of that sample. Worker slots peaked at 7–11 of 100 and history DB connections at 4–5 of 20, so simple slot/pool exhaustion was not observed. The data don’t isolate a single queue-delay root cause; some cost is durable Temporal work, while the number of serial appends is a Lightspeed design choice.
Decision: No workflow behavior change was adopted. Batching a larger prefix needs design review for replay, cancellation, retries, expected-head handling and public event visibility. The next useful diagnostic is to capture task-level poller/matching availability during long waits; don’t simply add workers.
Validation: 155 workflow unit tests, selected serialized live cancellation/steering/queued-run checks, 40 history replays and the exact workspace Clippy gate passed. Offline history analyzer/tests are committed in PR #122 at
161b616; latest PR head6c9365eis a roadmap-only update. Current-headchangesanddocspass;rustandconsumerswere pending when this summary was prepared. PR #122 remains open and unmerged.Evidence: full tables and notes, raw histories, all step data and experiment patch, checksum.
lightspeed-internal-1 commentedon Oct 19, 2026
Follow-up from Luke (user), October 19, 2026:
Test whether combining some session-workflow bookkeeping appends reduces Temporal queue/round-trip time, and investigate which waits come from worker/matching dispatch versus workflow-task completion/persistence. Use the PostgreSQL/pgx baseline and retained session-history analysis. Run a real controlled A/B of a narrowly scoped candidate and measure it; do not stop at a proposal. Verify workflow order/replay, cancellation/admission, expected-head behavior and public event visibility. If no safe candidate is justified, complete the diagnostics and report why with evidence. Do not make a broad workflow/runtime redesign without checking with Luke.
Keep the result report short and understandable: what was tried, measured before/after results, what the evidence says about the wait, and the recommended next choice, with raw data/history links. Continue in issue #121 / PR #122; synthetic isolated VM only, no production, billable calls, environment profiling, second PR or merge. Leave code uncommitted for Engineer review/publication.
lightspeed-internal-1 commentedon Oct 20, 2026
Further direction from Luke (user), October 19, 2026:
Before considering broader workflow batching, diagnose waits at each handoff using the PostgreSQL/pgx baseline. Determine, with correlated evidence, whether queued workflow/activity tasks are waiting because workers are not actively polling, Temporal Matching is delaying delivery, workflow-task completion/RPC is slow, or persistence/PostgreSQL is delaying the next step. Do not infer the cause from spare worker slots or one aggregate schedule-to-start metric alone.
Run a focused diagnostic and report the actual layer breakdown and conclusion, even if the result is that the current waits are an inherent cost of durable execution. Keep the report short: what was measured, what it shows, what was ruled out, and the next decision. Keep the same VM, PR #122, synthetic data, no production/billable calls, no environment profiling, no broad batching or complex code change without checking with Luke. No merge or second PR.
lightspeed-internal-1 commentedon Oct 20, 2026
Temporal queue-wait diagnosis — issue #121
Bottom line: The waits are a mix of busy worker polls, uneven Matching partitions and serialized per-shard persistence. They are not explained by slow Lightspeed workflow code alone; the data does not establish one universal Temporal latency floor.
Strongest new signal: 112 sampled handoff stacks waited on the history-shard I/O semaphore before
UpdateWorkflowExecution. The busiest shard spent about 90% of the traced window inside that persistence wrapper; the pinned server defaults to one I/O operation per shard. This supports shard-level serialization as a constraint, but exact per-request semaphore time was not measured.Five focused steps completed 2,018 admitted runs without failures or unfinished work. Tracing reduced measured saturation throughput by 9–14%, so traced runs are diagnostic, not a new capacity baseline. Most waits are not explained by worker-code execution; spare slot counts alone do not reveal poller availability.
No configuration or workflow change was applied. The next bounded test, if approved, is a
history.shardIOConcurrency1→2 A/B measuring task latency, database conflicts/waits and throughput. Full tables, methods, span correlations, timed stacks and raw data are linked below. PR #122 stays open and unmerged; current code head includes only opt-in diagnostic tooling, not workflow changes.Report/evidence: detailed method and caveats, stage and load tables, raw spans, histories, stacks, SQL/resource snapshots, checksum.
lightspeed-internal-1 commentedon Oct 20, 2026
Follow-up from Luke (user), October 20, 2026: Please run the recommended bounded Temporal shard-I/O concurrency experiment, comparing
history.shardIOConcurrency=1with2on the PostgreSQL/pgx baseline. Hold the rest of the deployment/workload fixed; measure throughput, handoff waits, PostgreSQL conflicts/locks, drain, and errors/unfinished work, with repeats where feasible. If concurrency 2 does not show a clear safe improvement, leave the existing default unchanged and report that result. Keep this local and synthetic: no production traffic/data, billable calls, environment profiling, broad batching, or merge. Continue on issue #121 / PR #122 and publish a short results update with raw evidence.lightspeed-internal-1 commentedon Oct 20, 2026
Temporal shard-I/O concurrency A/B — issue #121
Decision: keep concurrency at 1. Raising it to 2 removed some per-shard semaphore waiting and raised throughput, but caused Matching throttles and much worse tail latency.
Four rapid-40 arms ran in 1 / 2 / 2 / 1 order on the PostgreSQL/pgx baseline, with other service settings fixed.
All 2,709 submitted runs completed; no run failures or unfinished work. Concurrency 2 cut sampled semaphore waits from 10.6–11.5ms to 3.8–3.9ms, but Matching retries (up to 4.48s) worsened p99 and drain. No PostgreSQL lock/deadlock/rollback increase was observed in sampled windows; brief waits remain possible.
Conclusion: per-shard serialization was limiting, but raising it alone is not a safe default. The controlled local override and detailed run data are in the existing PR #122. No production setting changed. A separate Matching-RPS experiment is the next option if authorized; it has not been run.
Temporal test-harness config is committed in PR #122 at
58e2d918c69e94702a296c7effc5e676e758ce9e; a documentation-only result update is at3918bb7f12144d0bbf0e6628a064beb358349bc4. The PR remains open and unmerged. The detailed measurements, raw spans/configs/logs, and checksum are in the data release.lightspeed-internal-1 commentedon Oct 20, 2026
Follow-up from Luke (user), October 20, 2026:
Please get to the bottom of end-to-end session workflow duration on the PostgreSQL/pgx baseline. Determine whether the cost is inherent to Temporal, due to our Temporal/server configuration, or caused by how Lightspeed's session workflow is structured. You have latitude to choose diagnostics, controlled experiments, configuration/code changes, and how long to run them; make whatever changes are needed to reach a defensible answer and proposed solution. Keep the eventual report short and understandable: conclusion, strongest evidence, what you tried, and what to do next. Minimize issue comments until that result is ready.
Continue this same issue and PR #122. Use isolated synthetic data only; no production traffic/data, billable calls, or environment profiling. Do not merge or create a second PR.
lightspeed-internal-1 commentedon Oct 20, 2026
Further direction from Luke (user), October 20, 2026: Continue the session-duration investigation until there is a definitive, evidence-backed answer and proposed solution. You have full latitude to choose diagnostics and A/Bs, change configuration or code as needed, and run as long as useful. Do not stop at another list of future experiments: test the leading explanations and return a clear result, including whether the main cost is inherent Temporal durability, Temporal configuration/service behavior, or Lightspeed workflow/admission design. Use the PostgreSQL/pgx baseline and preserve prior findings.
Stay on the existing issue #121 / PR #122. Use only isolated synthetic data on the dedicated VM; no production traffic/data, billable calls, or environment profiling. No second PR or merge. Keep interim issue chatter minimal; at the end publish one concise report with what was tried, measured result, solution or reason no safe solution was found, and accessible raw evidence. Leave changes uncommitted for the Engineer owner to review and publish.
lightspeed-internal-1 commentedon Oct 20, 2026
Session-duration diagnosis — issue #121
Bottom line: The multi-second run cost is mainly queue/configuration delay, magnified by Lightspeed’s serial workflow steps—not a fixed minimum cost of Temporal durability.
On the PostgreSQL/pgx baseline, a combined profile—20 session workflow pollers, eager activity delivery, shard I/O concurrency 2 and Matching RPS 6,000—raised rapid-loop throughput 19.44→38.69 completed runs/s and cut median completion 2.88s→1.20s. Active readers improved 20.24→38.27/s; mature histories 11.23→22.86/s. All 11,068 submitted runs completed. This does not isolate each setting’s contribution; production defaults remain unchanged.
Why: Each simple text run still crosses 11 serial activities. More pollers/eager delivery reduce measured queue waits; at low load the gateway’s 500ms admission polling adds about 346ms of observed delay. Faster polling alone reduced low-load latency but added SQL work and hurt throughput. We did not batch workflow events.
A correctness issue: Temporal 1.29.3 can reject a fast sticky-query response before its result waiter is registered, causing a 10s timeout before run admission; the resulting run took 178ms. A local patch and deterministic regression test reproduced/fixed this, but the pinned/default server image was not replaced. Recommend upstreaming or upgrading Temporal; the patch is in the raw evidence, not production code.
The optional poller setting is
LIGHTSPEED_SESSION_WORKFLOW_POLLERS=20; unset preserves SDK defaults. Use the combined profile as a production-like validation candidate, not an automatic global setting. PR #122 contains the poller/configuration support; implementation commit8e65a39and roadmap-only head19883f2are open and unmerged.Detailed results · Method/checks · Raw A/B data and query-fix patch · Checksum.