Repository navigation
fix(benchsdk-cli): per-command option allowlists + reliability table gaps - #564
Merged
Merged
Conversation
…gaps - parseSubcommandOptions takes an optional allowlist; every command now rejects flags it does not understand. Previously --from/--to/--bucket/ --metrics/--breakdown were silently accepted (and ignored) everywhere, so e.g. 'bench results --from' printed unfiltered results. - printReliability now renders the requested --metrics stats and --breakdown tables in the range view. - A blank --source-org now errors instead of silently falling back to the active org, where a shared slug could return another benchmark's reliability data. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Devin resolved the finding on ad5ed92
Fixed by Devin (1)
- Run details ignore accepted filters
View all findings in Devin Review
The runs subcommand union let --participant/--source-org/... parse on 'runs show', where handleRunsShow ignored them — a shared slug could show another org's run silently. 'show' now rejects any option. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #563 (merged before the fixes could be pushed): the three Devin Review findings on that PR.
parseSubcommandOptions(args, allowed?)rejects a flag the command doesn't understand. The option schema is global, so--from/--to/--bucket/--metrics/--breakdownwere silently accepted — and ignored — on every subcommand;bench results --fromprinted unfiltered results.--metrics/--breakdownnow print:printReliabilityrenders metric stats (p50/p95/p99 per participant) and per-key outcome breakdowns in the range view's table output.--source-orgerrors:--source-org ' 'previously trimmed to empty and fell back to the active org, where a shared slug could return another benchmark's reliability data.Testing
npm run typecheckclean;benchsdk-cli50/50 tests pass.benchbinary:results foo --from→Unknown option for this command;reliability foo --source-org ' '→--source-org must name an organization slug.Link to Devin session: https://app.devin.ai/sessions/fdf951a9b9184f11a6d65e3c5056e907
Open in Devin Desktop: https://app.devin.ai/desktop/session/fdf951a9b9184f11a6d65e3c5056e907?variant=devin
Requested by: @dtice25