diff --git a/skills/dstack-prototyping/SKILL.md b/skills/dstack-prototyping/SKILL.md index 552d00dad..d054d8cb4 100644 --- a/skills/dstack-prototyping/SKILL.md +++ b/skills/dstack-prototyping/SKILL.md @@ -101,19 +101,33 @@ command, resources, cache, or model behavior needs to change, go back to a task. If the tested serving setup is still right and only the dstack service configuration is wrong, fix the configuration and submit the service again. -## PD disaggregation +## Router -If the intention is to use PD disaggregation: +If a fleet has `placement: cluster` and a CPU-only instance, it is generally +recommended to use a router configuration, regardless of whether the workers +are aggregated or PD disaggregated, with the router on the CPU-only instance. +Whenever possible, connect the workers over gRPC, not HTTP: with a gRPC +router, request parsing, serialization, and tokenization move from the +serving engine to the router, so latency improves just by introducing it. + +When using a router: - Use node groups for the task and replica groups for the service: tasks' node groups are the equivalent of services' replica groups. -- In both cases, you run a router and prefill/decode workers separately, and - you need to use a fleet with an interconnect (`placement: cluster`). - With tasks, still use `sleep infinity` even when using `groups` (set it in each group's `commands`; top-level `commands` is not allowed with `groups`), and run the actual commands on each node interactively over SSH. - When testing inference, call the router endpoint, not the workers directly (unless you want to test if they are alive). +- Look for "Prototyping services" in `https://dstack.ai/docs/concepts/tasks.md` + and "Router" in `https://dstack.ai/docs/concepts/services.md`. + +## PD disaggregation + +If the intention is to use PD disaggregation: + +- Follow `## Router`: the router and the prefill/decode workers run as + separate groups, and the fleet needs an interconnect (`placement: cluster`). - Look for "Node groups" and "PD disaggregation" in `https://dstack.ai/docs/concepts/tasks.md` and "Replica groups" and "PD disaggregation" in `https://dstack.ai/docs/concepts/services.md`. diff --git a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md index 101191b94..1a11b74ab 100644 --- a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md +++ b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md @@ -260,9 +260,9 @@ mindful of which specific change was the root cause. {"resources": {...}, "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...} ``` -2. In case the task is using node groups (e.g. for PD disaggregation), - instead of a single `resources` it includes `groups`, and the fields are - these and no others: +2. In case the task is using node groups (a router group plus worker + groups, see `## Cluster Placement`), instead of a single `resources` it + includes `groups`, and the fields are these and no others: ``` {"groups": [[{...}], [{...}, {...}], [{...}]], "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...} @@ -321,9 +321,9 @@ serving engine: use `dataset` and `concurrency``co `shared_prefix_tokens` from `constraints.json` and measure all trials the same way so that their results are comparable with each other. -In case the task is using PD disaggregation, run benchmarks via SSH inside -the router node, directly against the router engine. Never benchmark prefill -or decode workers — each handles only part of a request. +In case the task is using a router (see `## Cluster Placement`), run benchmarks +via SSH inside the router node, directly against the router engine, never against a +worker, so that results stay comparable across trials. Before any benchmark, reset the serving engine's prefix cache, or restart the @@ -506,6 +506,13 @@ Use these offers when selecting fleet, backend, and hardware. To classify each backend's capabilities, fetch `https://dstack.ai/docs/concepts/backends.md` and classify from the fetched document, not from memory. VM-based backends are listed under `## VM-based` (they support idle instances and instance volumes). Kubernetes backend is listed under `## Container-based`, but supports instance volumes and thus is preferred over other container-based backends. SSH fleets can be treated as VM-based backends as they support both idle instances (its equivalent) and instance volumes. +## Cluster Placement + +If a fleet has `placement: cluster`, decide before trial 1 whether to use a +router, following `## Router` in the +`$/dstack-prototyping` skill, and +report the decision via `progress`. + # Final Service Once the trials are over, pick the best trial that has not been verified yet @@ -565,9 +572,9 @@ trial benchmarks so that the results are comparable with each other. Attach to the service with `dstack attach `, which enables `ssh ` into the replica. -In case the service is using PD disaggregation, run the final benchmark -inside the router replica, directly against the router engine. Attach to it -via `dstack attach --replica --job `. +In case the service is using a router (see `## Cluster Placement`), run the +final benchmark inside the router replica, directly against the router engine. +Attach to it via `dstack attach --replica --job `. If the service or its benchmark cannot be completed, stop that service, pick the next-best trial, and repeat, until a service is verified or there