HI馃槉, we are reproducing your experimental results as the baseline of our method.
We are confused by the following questions. 馃
First, why running the following code needs more than 1 minutes? Since the prompt is short, it is expected to finish very quickly.
In [8]: prompt = "TESLA company is found by"
In [9]: output = model(prompt=prompt)
...(a long time)...
Second, after waiting for over 1 min, we finally got the results looks like this:
In [10]: output
Out[10]: {'text': ['Nik']}We think this output is not reasonable and want to know if there are some improper configurations in the following scripts?
@dataclass
class MagicpigConfig:
server_type: str = 'hf'
server_host: str = '127.0.0.1'
server_port: str = '5000'
ssh_server: Optional[str] = None
ssh_key_path: Optional[str] = None
model_name_or_path: str = 'meta-llama/Llama-2-7b-chat-hf'
temperature: float = 0.0
top_k: int = 32
top_p: float = 1.0
random_seed: int = 0
stop_words: list = field(default_factory=list)
sliding_window_size: int = None
threads: int = 1
K: int = 10
L: int = 150
S: float = 4.0
W: int = 64
Q: int = 0
QR: float = 0.0
max_seq_length: int = 4096
max_new_tokens: int = 128If there are improper configurations for short prompt generation, we want to further know what is the most suitable configuration under different prompt length? e.g. 1K, 2K, 4K, 8K.
HI馃槉, we are reproducing your experimental results as the baseline of our method.
We are confused by the following questions. 馃
First, why running the following code needs more than 1 minutes? Since the prompt is short, it is expected to finish very quickly.
Second, after waiting for over 1 min, we finally got the results looks like this:
We think this output is not reasonable and want to know if there are some improper configurations in the following scripts?
If there are improper configurations for short prompt generation, we want to further know what is the most suitable configuration under different prompt length? e.g. 1K, 2K, 4K, 8K.