Skip to content

FIX: preserve rationale-less float scores - #2886

Open
Barry (barry166) wants to merge 1 commit into
microsoft:mainfrom
barry166:fix/2880-float-rationale
Open

Barry (barry166) wants to merge 1 commit into
microsoft:mainfrom
barry166:fix/2880-float-rationale

Conversation

@barry166

Copy link
Copy Markdown
Contributor

Description

Fixes #2880.

Float-scale aggregate rationales now include every constituent score, including scores without a rationale. This keeps the output consistent with the true/false and undetermined aggregation paths and preserves which scores contributed to the result.

Tests and Documentation

  • uv run --offline pytest -q tests/unit/score/test_float_scale_score_aggregator.py (39 passed)
  • uv run --offline ruff check pyrit/score/float_scale/float_scale_score_aggregator.py tests/unit/score/test_float_scale_score_aggregator.py
  • git diff --check

# Only include scores with non-empty rationales
rationale_parts = [format_score_for_rationale(s) for s in scores if s.score_rationale]
rationale_parts = [format_score_for_rationale(s) for s in scores]
rationale = "\n".join(rationale_parts) if rationale_parts else ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new lines still don't identify which Azure harm category produced each value. For example, rationale-less AzureContentFilterScorer scores for Hate=0.2 and Violence=0.8 produce AzureContentFilterScorer 0.2: and AzureContentFilterScorer 0.8:. The aggregate lists both categories, but cannot tell a reader which category caused the 0.8 MAX or a threshold verdict. Could we include each score's category in its line (ideally in the shared formatter) and test this multi-category case? That would make the newly preserved entries useful for tracing the result.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A float-scale aggregate drops rationale-less constituents from its rationale

2 participants