Problem. In single-slot deployments, one long request (a ~35s prefill)
queues all other clients. Downstream bridges only survive this via aggressive
timeouts (the cc-bridge adapter uses a 10s connect / 600s read timeout, and
cancellation propagation) — throughput is still zero during the prefill.
Request. Confirm and document the recommended deployment for multiple
concurrent clients (--continuous-batching where available), and consider
queueing/diagnostics so clients can distinguish "queued" from "hung".
Impact. Documented behavior would let bridges size their timeouts from the
engine contract instead of empirical measurement.
Problem. In single-slot deployments, one long request (a ~35s prefill)
queues all other clients. Downstream bridges only survive this via aggressive
timeouts (the cc-bridge adapter uses a 10s connect / 600s read timeout, and
cancellation propagation) — throughput is still zero during the prefill.
Request. Confirm and document the recommended deployment for multiple
concurrent clients (
--continuous-batchingwhere available), and considerqueueing/diagnostics so clients can distinguish "queued" from "hung".
Impact. Documented behavior would let bridges size their timeouts from the
engine contract instead of empirical measurement.