SelfAskGeneralFloatScaleScorer and SelfAskGeneralTrueFalseScorer crash when the judge returns a category and the scorer also has one configured.
Both docstrings say a category in the response is used, and the configured one only applies when the response has none. But they build a JsonSchemaResponseHandler that raises ValueError("Category is present in the response and an argument") when both are present. That isn't an InvalidJsonException, so there's no retry and scoring fails after one call:
category="harm", response {"score_value": "true", "rationale": "r", "category": "violence"}
-> RuntimeError ... ValueError: Category is present in the response and an argument (target calls: 1)
Fix: let these two scorers opt into "response category wins, configured is the fallback". Other handlers and scorers keep the current error.
SelfAskGeneralFloatScaleScorerandSelfAskGeneralTrueFalseScorercrash when the judge returns acategoryand the scorer also has one configured.Both docstrings say a category in the response is used, and the configured one only applies when the response has none. But they build a
JsonSchemaResponseHandlerthat raisesValueError("Category is present in the response and an argument")when both are present. That isn't anInvalidJsonException, so there's no retry and scoring fails after one call:Fix: let these two scorers opt into "response category wins, configured is the fallback". Other handlers and scorers keep the current error.