A max_tokens value is usually sent from the client to indicate how many tokens are to be expected maximally in a prompt. Models can use it to allocate space on device for the prompt processing. If ommited, the current behavior of OpenArc is to set a default of 16k, which might be too high.
This feature request is about providing a parameter --max-tokens to the openarc add ... command to configure the default max_tokens value if clients ommit it. With this option the value can be set to a lower value, i.e. 1024, if the prompts are expected to never be larger than 1024, which leads to a more gentle space allocation on device.
Note:
I have implemented this in my fork here. Unfortunately I mixed it up with another story (which I will write shortly) in the same commit... but will provide a PR shortly.
A
max_tokensvalue is usually sent from the client to indicate how many tokens are to be expected maximally in a prompt. Models can use it to allocate space on device for the prompt processing. If ommited, the current behavior of OpenArc is to set a default of 16k, which might be too high.This feature request is about providing a parameter
--max-tokensto theopenarc add ...command to configure the default max_tokens value if clients ommit it. With this option the value can be set to a lower value, i.e. 1024, if the prompts are expected to never be larger than 1024, which leads to a more gentle space allocation on device.Note:
I have implemented this in my fork here. Unfortunately I mixed it up with another story (which I will write shortly) in the same commit... but will provide a PR shortly.