Skip to content

feat(inference): pin each conversation to its own llama-server slot - #202

Merged
ganisback merged 1 commit into
mainfrom
feat/llama-slot-affinity
Sep 27, 2026
Merged

ganisback merged 1 commit into
mainfrom
feat/llama-slot-affinity

Conversation

@ganisback

Copy link
Copy Markdown
Collaborator

Summary

  • With two or more parallel slots, each conversation stays on one llama-server slot, so agent sessions that share a system prompt stop evicting each other's prompt cache.
  • The conversation key now covers the whole first turn, plus Anthropic system and Responses API input, so sessions that share a preamble stay distinct across chat, Claude, and Codex requests.
  • A single slot is unchanged and does not send id_slot.

Test plan

  • Set parallel slots to 2 or more, run two agent sessions on the same local model, and confirm each keeps its prompt cache across turns.
  • With one slot, confirm requests do not send id_slot.
  • Confirm a follow-up tool turn stays on the same slot, and two sessions that share a system preamble get different slots.
  • Check the parallel-slots setting description in English and Chinese.

Agent sessions share a system prompt, so llama-server was parking every request in one slot and evicting the others. With two or more slots, keep each conversation on the same slot so its prompt cache survives.
@ganisback
ganisback merged commit 7320d1a into main Sep 27, 2026
4 checks passed
@ganisback
ganisback deleted the feat/llama-slot-affinity branch September 27, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant