Preserve structured output stop reasons

[sylvesterkaczmarek]

Summary

Preserve structured-output responses when generation terminates with stop_reason="refusal" or "max_tokens" instead of replacing the response with a schema-validation exception.

messages.parse() and the beta equivalent currently attempt to validate every text block against output_format unconditionally. Refusal text and max-token-truncated JSON are not guaranteed to satisfy the requested output schema, so those terminal responses can fail inside Pydantic/JSON parsing before the caller can inspect the model's actual stop_reason.

For example, a refusal containing ordinary refusal text is currently fed to TypeAdapter.validate_json(), and a response truncated at {"value": is treated as malformed structured output. In both cases the SDK loses the more important information: why generation stopped.

Fix

Skip structured-output validation only for:

• refusal
• max_tokens

The text remains available unchanged, parsed_output is None, and the original stop_reason is preserved.

Normal completed responses continue through the existing validation path, so malformed structured output on an ordinary completed turn still raises as before.

The behavior is applied symmetrically to GA and beta parsed messages.

Regression coverage

Adds tests verifying:

• refusal text returns with parsed_output=None for GA and beta;
• truncated JSON from max_tokens returns with parsed_output=None;
• the original stop reason is preserved;
• invalid structured output on a normally completed response still raises a validation error.

[HardMax71]

@sylvesterkaczmarek the streaming path has the same problem, and this PR doesn't touch it. The stream accumulator parses at content_block_stop (_messages.py#L510-L513, same in _beta_messages.py#L544-L547), but stop_reason only arrives with the following message_delta. So client.messages.stream(..., output_format=Model) raises a raw pydantic.ValidationError mid-stream when the reply is cut at max_tokens, and the caller never gets to see the stop reason.

Offline repro (stub transport, the reply is cut at {"lane": "A-B", "pri with stop_reason: max_tokens):

repro

# Offline: messages.stream(output_format=Model) where the reply is cut off at
# max_tokens. Does the caller get stop_reason="max_tokens", or a raw pydantic
# ValidationError before stop_reason is known? Stub transport, synthetic SSE.
import json

import anthropic
try:
    import httpx2 as httpx  # anthropic>=1.0 ships on httpx2
except ImportError:
    import httpx
import pydantic
from pydantic import BaseModel


class Rate(BaseModel):
    lane: str
    price: float


def sse(events):
    out = []
    for ev in events:
        out.append(f"event: {ev['type']}\ndata: {json.dumps(ev)}\n\n")
    return "".join(out).encode()


EVENTS = [
    {"type": "message_start", "message": {"id": "msg_1", "type": "message", "role": "assistant",
        "model": "claude-test", "content": [], "stop_reason": None, "stop_sequence": None,
        "usage": {"input_tokens": 10, "output_tokens": 1}}},
    {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}},
    {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": '{"lane": "A-B", "pri'}},
    {"type": "content_block_stop", "index": 0},
    {"type": "message_delta", "delta": {"stop_reason": "max_tokens", "stop_sequence": None},
        "usage": {"output_tokens": 8}},
    {"type": "message_stop"},
]


def handler(request: httpx.Request) -> httpx.Response:
    return httpx.Response(200, headers={"content-type": "text/event-stream"}, content=sse(EVENTS))


client = anthropic.Anthropic(api_key="sk-test", http_client=httpx.Client(transport=httpx.MockTransport(handler)), max_retries=0)
print("anthropic", anthropic.__version__)
try:
    with client.messages.stream(model="claude-test", max_tokens=8, output_format=Rate,
                                messages=[{"role": "user", "content": "x"}]) as stream:
        msg = stream.get_final_message()
    print("stop_reason:", msg.stop_reason, "parsed_output:", msg.parsed_output)
except pydantic.ValidationError as e:
    print("pydantic.ValidationError raised mid-stream, stop_reason never reached:", e.errors()[0]["type"])

anthropic 1.8.0
pydantic.ValidationError raised mid-stream, stop_reason never reached: json_invalid

Same on 0.87.0. Could this PR move that parse to message_stop, where stop_reason is known, with the same refusal / max_tokens check? Or I can open a separate PR for the stream side if you'd rather keep this one small.

[sylvesterkaczmarek]

@HardMax71 Confirmed. The streaming path had the same event-ordering issue: content_block_stop runs before message_delta, so structured-output validation could fail before stop_reason was available.

Updated in signed commit 32d3d1f:

• successful structured output can still parse when the content block closes;
• validation failures are deferred until message_stop, after the stop reason is known;
• refusal and max_tokens now preserve the text and return parsed_output=None instead of masking the terminal reason;
• invalid structured output on a normal completed response still raises;
• the behavior is covered in both GA and beta streaming paths.

Focused validation: 66 streaming/parsing tests passed, including public-stream regressions for max_tokens.