Repository navigation
✨ feat: opt-in multi-model task routing for agent runs - #4332
Draft
FiveFingersMountain wants to merge 1 commit into
Draft
FiveFingersMountain wants to merge 1 commit into
FiveFingersMountain wants to merge 1 commit into
Conversation
Route agent runs to the best-matching model of the agent's own configured pool (quick QA / long context / deep reasoning tiers). - Opt-in via MODEL_ROUTING_ENABLED (default false; pre-routing behaviour fully preserved when off) - Domain vocabularies live in configs/model_routing_rules.json (medical terms shipped as example; engine is domain-agnostic) - Explicit request model selection always wins over routing - No-signal and no-capable-candidate cases keep the existing selection; routing never fails a run - Every decision emits a structured audit log with matched rules, candidate scores and fallback tier - Capacity guard prefers the provider input limit and rejects candidates that cannot hold the estimated tokens - 43 unit tests plus create_agent_config integration coverage
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in routing step to agent run preparation: when
MODEL_ROUTING_ENABLED=trueand the request did not pin a model, the run isclassified (task tier / risk / estimated size) and dispatched to the
best-matching model within the agent's own configured
model_idspool.With the flag off (default) the model selection path is byte-for-byte the
pre-routing behavior.
Tiers:
quick_qa(lookups, short factual),long_context(record/documentprocessing),
deep_reasoning(analysis, multi-step judgement). Tiervocabularies and thresholds come from a rules JSON
(
backend/configs/model_routing_rules.json, medical example shipped); theengine is domain-agnostic.
Driven by a production deployment in the medical domain at a large
top-tier hospital in China; see #4331 for the demand context.
Behavior contract (enforced by tests)
model_idalways wins — routing skipped.MODEL_ROUTING_ENABLED=false(default) — routing skipped entirely.None, caller keeps theexisting selection (routing never guesses).
None, caller keeps theexisting selection (routing never fails a run).
quick_qa → long_context → deep_reasoningrecorded asfallback_fromin the audit record.How
backend/management/services/model/routing.py— routing service:rules loading (graceful degradation), classification rule engine,
capability-based tier detection from existing
model_record_tcolumns(
reasoning_supported,context_window_tokens, name heuristics),scoring with a hard capacity guard (prefers
max_input_tokens; rejectscandidates that cannot hold the estimated tokens), tier relaxation,
structured audit record.
backend/configs/model_routing_rules.json— example rules (medicalvocabulary; replace the file to adapt to another domain).
backend/management/services/agent/preparation/compiler.py— wiring:_route_agent_modelbehindMODEL_ROUTING_ENABLED;create_agent_configgains optional
run_history_length/run_has_attachments(defaults keepbackward compatibility; both recursive sub-agent call sites pass through).
backend/management/services/agent/preparation/run.py— passes realhistory length / attachment flag.
backend/contracts/constants/environment.py—MODEL_ROUTING_ENABLED,MODEL_ROUTING_RULES_JSON_PATH.deploy/env/.env.example— operator-facing flag documentation.test/backend/management/test_service_directory_architecture.py—canonical-location whitelist registration (
model/routing.py).Test evidence
test/backend/services/test_model_task_routing.py— 43 unit tests:rules loading/degradation, classification priority ladder, tier detection,
scoring/capacity guard, fallback + audit, contract locks (no-signal keeps
selection; exhausted fallback keeps selection; input records not mutated).
test/backend/agents/test_create_agent_info.py— 253 integration testsgreen (4 exact-call assertions extended with the two new kwargs).
git stashdiff on the surrounding suites: zero new failures(pre-existing failures documented in the PR discussion).
Compatibility & rollout
for the context-budget feature; NULL columns degrade to name heuristics.
Limitations / follow-ups (tracked in the issue)
filter + preference scoring) is a deliberate follow-up.
(
context_policyisextra="forbid"; candidates listed in the issue).review_requiredcurrently lands in the audit log only; surfacing it inthe session API is follow-up work, not claimed as implemented.