[Prajna1999]

Summary

Target issue is #171

Moves every LLM-backed validator gpt-4o / gpt-4o-mini to gpt-5.6-sol.
Move issue formatter model in github/workflows/issue-formatter.yml` are updated to match.

What changed

• ANSWER_RELEVANCE_LLM_MODEL and DEFAULT_LLM_CALLABLE default to gpt-5.6-sol; the JSON-capable model allowlist in llm_utils.py and `.
• answer_relevance_custom_llm fix: gpt-5.6-sol is a reasoning model. LiteLLM sends max_tokens=10 as max_completion_tokens=10, which also has to cover hidden reasoning tokens, so the YES/NO answer would usually come back empty. The budget is now 512 tokens, and reasoning_effort="medium" is sent when the model supports reasoning (matching topic_relevance_llm). The check uses a shared supports_reasoning() helper in llm_utils.py, so non-reasoning models like gpt-4o don't receive an unsupported param; topic_relevance_llm now imports this helper instead of keeping its own copy from fix(topic_relevance_llm): label topic config in prompt, support reasoning models #173. An empty reply now returns the "unexpected response" failure instead of being reported as a failed LLM call.
• topic_relevance_llm token ceiling: fix(topic_relevance_llm): label topic config in prompt, support reasoning models #173 removed max_tokens because a 300-token cap shared with hidden reasoning produced empty replies, which left no ceiling on cost or latency. It's restored at 1024 tokens (2048 with a fallback message), about 4x the highest usage seen in the live eval, so it only stops runaway calls.
• Docs updated throughout. Tips that compared mini vs. full gpt-4o were rewritten rather than renamed. Also fixed the evaluation README's reference to a DEFAULT_CONFIG that doesn't exist.

Checklist

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

• Tests: 168 passed in app/tests/test_llm_validators.py and app/tests/validators/, run with --noconftest because the test Postgres wasn't available locally. No calls were made against the live OpenAI API.
• The two "llm_callable is forwarded" tests now pass openai/gpt-5.6-sol. Using plain gpt-5.6-sol would equal the new default, so they'd pass even if forwarding broke.
• A new test covers the empty-reply path. test_no_max_tokens_cap_is_sent (from fix(topic_relevance_llm): label topic config in prompt, support reasoning models #173) is replaced by tests for the 1024 and 2048 caps.
• Live eval: see Live eval results below. gpt-5.6-sol is more accurate on both validators but about 2x slower per call; this is accepted. The cap and lower reasoning effort don't change that.
• Not changed, worth a look:
  • ANSWER_RELEVANCE_LLM_MODEL and DEFAULT_LLM_CALLABLE now hold the same value and could be merged.
  • The nsfw_text docs give the default model as textdetox/xlmr-large-toxicity-classifier, but the code default is michellejieli/NSFW_text_classifier.

Live eval results

About 100 examples per validator against the live OpenAI API. No breakage: across 600 gpt-5.6-sol calls there were no empty replies, cut-off replies, errors or parse failures. gpt-5.6-sol is more accurate on both validators but about 2x slower per call; the slower speed is accepted.

answer_relevance_custom_llm

100 query/answer pairs (healthcare and education, English and Hinglish): 50 with the correct answer (expect YES) and 50 with another question's answer (expect NO).

Run	Accuracy	F1	Wrong	Latency typical / slowest 5%	Tokens, highest (limit 512)
gpt-5.6-sol, medium effort	0.92	0.93	8	1.2s / 2.8s	179 (169 reasoning)
gpt-5.6-sol, low effort	0.89	0.90	11	1.2s / 2.8s	173
gpt-4o-mini	0.84	0.86	16	0.7s / 0.9s	1

• Both models caught every mismatched answer (50/50). Every wrong result was a correct answer being rejected, never a wrong answer passing.
• The rejected correct answers look like the prompt's "fully satisfy" wording being read strictly, not a model problem.
• The old max_tokens=10 would have cut off many gpt-5.6-sol replies; the highest use was 179 tokens.

topic_relevance_llm

100 rows sampled from the education and healthcare datasets (50 each, 32 out of scope), prompt schema v2.

Run	Accuracy	Out of scope caught	Blocks correct	F1	Latency typical / slowest 5%	Tokens, highest
gpt-5.6-sol, medium effort, no cap	0.90	0.81	0.87	0.84	2.65s / 4.16s	263
gpt-5.6-sol, medium effort, cap 1024	0.89	0.78	0.86	0.82	2.67s / 4.05s	194
gpt-5.6-sol, low effort, cap 1024	0.89	0.81	0.84	0.83	2.61s / 3.85s	167
gpt-4o-mini	0.89	0.66	1.00	0.79	1.34s / 1.59s	69

• gpt-5.6-sol catches more out-of-scope inputs (about 26 vs 21 of 32) but also blocks some in-scope ones, e.g. "where can I get my baby's vaccination near my village?". Expect a few new false blocks on borderline in-scope questions.

Latency

• Neither the token cap nor lower reasoning effort changes typical latency much. With reasoning switched off (reasoning_effort="none"), a trivial YES/NO call still took about 1.1s, so that's roughly this model's floor. (minimal isn't supported by this model.)
• Expect about 1–3s more per LLM validator per request than with gpt-4o-mini. The guardrails code sets no timeouts, but callers with tight timeouts may need to raise them.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one): 1

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: ProjectTech4DevAI/kaapi-guardrails
• Branch: chore/gpt-5.6-sol-model
• File: .coderabbit.yaml
• Commit: 62ac0a3
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Advanced
• Run ID: 4b84ce86-9803-44c9-9065-eb7250b5a097

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[AkhileshNegi]

can we use this in topic_relevance_llm.py:52since similar function is used there and use utils for that

[Prajna1999]

Done in 2b7509a. topic_relevance_llm now imports supports_reasoning from llm_utils, and its local copy plus the direct litellm import are removed.

[AkhileshNegi]

can we update this to "medium"

[Prajna1999]

Done in 2b7509a. _REASONING_EFFORT is now "medium", the same as topic_relevance_llm. The 512-token budget should still leave room for the one-word YES/NO.

[AkhileshNegi]

remove these

[Prajna1999]

Removed the budget and effort tests in 2b7509a, since they only restated the constants. I kept test_empty_content_returns_unexpected_response_fail: an empty reply (e.g. when reasoning uses up the budget) used to raise on .strip() and get reported as "LLM call failed". It now returns the unexpected-response failure, and this test covers that. Happy to drop it too if you prefer.

[AkhileshNegi]

do we need this change?

[Prajna1999]

gpt-4o-mini is going to be deprecated in January 2027. So we could make this toggle now instead of doing it closer to the deprecation date.

[Prajna1999]

Live eval: gpt-5.6-sol vs gpt-4o-mini

About 100 examples per validator, run against the live OpenAI API on this branch (2b7509a for the first round, the cap from a94c7a0 for the cap and effort runs).

Result: no breakage. Across 600 gpt-5.6-sol calls there were no empty replies, no cut-off replies (finish_reason=length), no errors and no parse failures. gpt-5.6-sol is more accurate on both validators, but about 2x slower per call. We're accepting the slower speed.

answer_relevance_custom_llm

100 query/answer pairs (healthcare and education, English and Hinglish): 50 with the correct answer (expect YES) and 50 with another question's answer (expect NO).

Run	Accuracy	F1	Wrong	Latency typical / slowest 5%	Tokens, highest (limit 512)
gpt-5.6-sol, medium effort	0.92	0.93	8	1.2s / 2.8s	179 (169 reasoning)
gpt-5.6-sol, low effort	0.89	0.90	11	1.2s / 2.8s	173
gpt-4o-mini	0.84	0.86	16	0.7s / 0.9s	1

• Both models caught every mismatched answer (50/50). Every wrong result was a correct answer being rejected, never a wrong answer passing.
• The rejected correct answers look like the prompt's "fully satisfy" wording being read strictly, not a model problem.
• The old max_tokens=10 would have cut off many gpt-5.6-sol replies; the highest use was 179 tokens.

topic_relevance_llm

100 rows sampled from the education and healthcare datasets (50 each, 32 out of scope), prompt schema v2.

Run	Accuracy	Out of scope caught	Blocks correct	F1	Latency typical / slowest 5%	Tokens, highest
gpt-5.6-sol, medium effort, no cap	0.90	0.81	0.87	0.84	2.65s / 4.16s	263
gpt-5.6-sol, medium effort, cap 1024	0.89	0.78	0.86	0.82	2.67s / 4.05s	194
gpt-5.6-sol, low effort, cap 1024	0.89	0.81	0.84	0.83	2.61s / 3.85s	167
gpt-4o-mini	0.89	0.66	1.00	0.79	1.34s / 1.59s	69

• gpt-5.6-sol catches more out-of-scope inputs (about 26 vs 21 of 32) but also blocks some in-scope ones, e.g. "where can I get my baby's vaccination near my village?". Expect a few new false blocks on borderline in-scope questions.

Latency

• Neither the token cap nor lower reasoning effort changes typical latency much. With reasoning switched off (reasoning_effort="none"), a trivial YES/NO call still took about 1.1s, so that's roughly this model's floor. (minimal isn't supported by this model.)
• Expect about 1–3s more per LLM validator per request than with gpt-4o-mini. The guardrails code sets no timeouts, but callers with tight timeouts may need to raise them.

_{Run with a one-off script outside the repo; the 100 answer-relevance pairs were written for this eval, and nothing was added to the repo's datasets.}