Skip to content

types: keep the server's bytes field on logprobs - #782

Open
Uneadlight wants to merge 1 commit into
ollama:mainfrom
Uneadlight:fix/750-logprob-bytes
Open

Uneadlight wants to merge 1 commit into
ollama:mainfrom
Uneadlight:fix/750-logprob-bytes

Conversation

@Uneadlight

Copy link
Copy Markdown

What

TokenLogprob now keeps the bytes field the server already sends.

Why

With logprobs=True, /api/generate and /api/chat return a bytes array (the
token's raw UTF-8 bytes) on every entry of logprobs and top_logprobs:

{"token": "Hello", "logprob": -0.20118224620819092, "bytes": [72, 101, 108, 108, 111]}

TokenLogprob only declared token and logprob, and since
SubscriptableBaseModel does not set extra='allow', pydantic's default
extra='ignore' dropped the field without any error:

>>> r = client.generate(model='qwen2.5:3b', prompt='Say hi in three words.', logprobs=True)
>>> r.logprobs[0].model_dump()
{'logprob': -0.2524, 'token': 'Hello', 'top_logprobs': None}
>>> r.logprobs[0].bytes
AttributeError: 'Logprob' object has no attribute 'bytes'

After the change:

>>> r.logprobs[0].bytes
[72, 101, 108, 108, 111]

How

Add one optional field to TokenLogprob (Logprob inherits it):

bytes: Optional[Sequence[int]] = None
'Raw UTF-8 bytes of the token. Useful when a token is not a standalone valid string.'

It is kept as a list of ints rather than a bytes object so model_dump()
stays JSON-serializable.

Older servers that do not send the field still parse: the attribute is None and
model_dump(exclude_none=True) omits it, which is the backward-compatible
behavior requested in #750.

Tests

4 added to tests/test_type_serialization.py:

  • test_generate_response_keeps_logprob_bytes
  • test_chat_response_keeps_logprob_bytes
  • test_logprob_bytes_is_optional
  • test_top_logprobs_keep_bytes

pytest tests: 113 passed, 8 failed. The 8 failures are pre-existing and
unrelated (test_client_create_blob* hits PermissionError on this machine and
the image serialization tests fail with Invalid image data due to a broken
%TEMP%); the same 8 fail on main without this change.

ruff check and ruff format --check: clean.

Fixes #750

When logprobs=True the server returns `bytes` (the token's raw UTF-8
bytes) on every entry of `logprobs` and `top_logprobs`. `TokenLogprob`
only declared token/logprob, so pydantic's default `extra='ignore'`
dropped the field without any error.

Add `bytes: Optional[Sequence[int]] = None` to `TokenLogprob`, which
`Logprob` inherits. It stays a list of ints (not `bytes`) so
`model_dump()` remains JSON-serializable.

Verified against a live server with qwen2.5:3b:

  /api/generate (logprobs=True)
    -> {token: Hello, logprob: -0.20, bytes: [72, 101, 108, 108, 111]}
  client before this change
    -> AttributeError: 'Logprob' object has no attribute 'bytes'
  client after this change
    -> [72, 101, 108, 108, 111]

Older servers that omit the field still parse, with None by default and
no key in `model_dump(exclude_none=True)`, matching the backward
compatible behavior requested in ollama#750.

Tests: 4 added (generate, chat, optional field, top_logprobs).
Fixes ollama#750

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TokenLogprob/Logprob drop server bytes field

1 participant