Repository navigation
Assessment: Fix schema for Anthropic #1331
Description
Is your feature request related to a problem?
The assessment output schema is failing with Anthropic while it works fine on OpenAI. This results in batch failures due to errors caused by overly large compiled grammars, leading to incomplete runs.
Describe the solution you'd like
- Reproduce the failure with the problematic schema (production submission, Sonnet).
- Identify schema features that exacerbate the grammar size issue (size, nesting, enums, long property lists).
- Implement fixes or validate schemas upon config save to prevent failures after submission.
Original issue
Describe the bug
The assessment output schema works reliably on OpenAI but not on Anthropic. With Sonnet (assessor peepul v1, v10) the whole batch failed: every request came back errored, so the run ended FAILED with "Batch ended with no successful requests (errored=4, canceled=0, expired=0)".
Per-request error from the Anthropic batch:
{
"custom_id": "row_2",
"result": {
"type": "errored",
"error": {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "The compiled grammar is too large, which would cause performance issues. Simplify your tool schemas or reduce the number of strict tools."
},
"request_id": "req_011CgfvbfjWPDMGoRkCHba6c"
}
}
}Likely cause
map_kaapi_to_anthropic_params (backend/app/services/llm/mappers.py) sends the output schema as output_config.format (json_schema) after running it through _ensure_openai_strict_schema, which makes every property required and adds additionalProperties: false at every level. Anthropic compiles that schema into a grammar and rejects it when it is too large; OpenAI accepts the same schema.
Expected
Output schemas that work on OpenAI should work on Anthropic, or be rejected up front with a clear message, rather than failing every request in the batch.
To do
- Reproduce with the failing schema (production submission, Sonnet).
- Find which schema features blow up the grammar (size, nesting, enums, long property lists).
- Fix or validate at config save, so a run does not fail after submission.
Activity
[-]Assessment: Anthropic output schema fails with "compiled grammar is too large"[/-][+]Assessment: Fix schema for Anthropic[/+]