Skip to content

✨ feat: opt-in multi-model task routing for agent runs - #4332

Draft
FiveFingersMountain wants to merge 1 commit into
ModelEngine-Group:developfrom
FiveFingersMountain:feature/multi-model-task-routing
Draft

FiveFingersMountain wants to merge 1 commit into
ModelEngine-Group:developfrom
FiveFingersMountain:feature/multi-model-task-routing

Conversation

@FiveFingersMountain

@FiveFingersMountain FiveFingersMountain commented Nov 11, 2026 •

Copy link
Copy Markdown

What

Adds an opt-in routing step to agent run preparation: when
MODEL_ROUTING_ENABLED=true and the request did not pin a model, the run is
classified (task tier / risk / estimated size) and dispatched to the
best-matching model within the agent's own configured model_ids pool.
With the flag off (default) the model selection path is byte-for-byte the
pre-routing behavior.

Tiers: quick_qa (lookups, short factual), long_context (record/document
processing), deep_reasoning (analysis, multi-step judgement). Tier
vocabularies and thresholds come from a rules JSON
(backend/configs/model_routing_rules.json, medical example shipped); the
engine is domain-agnostic.

Driven by a production deployment in the medical domain at a large
top-tier hospital in China; see #4331 for the demand context.

Behavior contract (enforced by tests)

  1. Explicit request model_id always wins — routing skipped.
  2. MODEL_ROUTING_ENABLED=false (default) — routing skipped entirely.
  3. Candidates = agent-configured + tenant-available models only.
  4. No classification signal (DEFAULT source) → None, caller keeps the
    existing selection (routing never guesses).
  5. No capable candidate or exhausted fallback → None, caller keeps the
    existing selection (routing never fails a run).
  6. Fallback chain quick_qa → long_context → deep_reasoning recorded as
    fallback_from in the audit record.
  7. Stable tie-break by input order.

How

  • backend/management/services/model/routing.py — routing service:
    rules loading (graceful degradation), classification rule engine,
    capability-based tier detection from existing model_record_t columns
    (reasoning_supported, context_window_tokens, name heuristics),
    scoring with a hard capacity guard (prefers max_input_tokens; rejects
    candidates that cannot hold the estimated tokens), tier relaxation,
    structured audit record.
  • backend/configs/model_routing_rules.json — example rules (medical
    vocabulary; replace the file to adapt to another domain).
  • backend/management/services/agent/preparation/compiler.py — wiring:
    _route_agent_model behind MODEL_ROUTING_ENABLED; create_agent_config
    gains optional run_history_length / run_has_attachments (defaults keep
    backward compatibility; both recursive sub-agent call sites pass through).
  • backend/management/services/agent/preparation/run.py — passes real
    history length / attachment flag.
  • backend/contracts/constants/environment.py — MODEL_ROUTING_ENABLED,
    MODEL_ROUTING_RULES_JSON_PATH.
  • deploy/env/.env.example — operator-facing flag documentation.
  • test/backend/management/test_service_directory_architecture.py —
    canonical-location whitelist registration (model/routing.py).

Test evidence

  • test/backend/services/test_model_task_routing.py — 43 unit tests:
    rules loading/degradation, classification priority ladder, tier detection,
    scoring/capacity guard, fallback + audit, contract locks (no-signal keeps
    selection; exhausted fallback keeps selection; input records not mutated).
  • test/backend/agents/test_create_agent_info.py — 253 integration tests
    green (4 exact-call assertions extended with the two new kwargs).
  • Architecture/contracts guards: 14 passed.
  • Cross-file stub-pollution verified fixed (both import orders, 296 passed).
  • Baseline git stash diff on the surrounding suites: zero new failures
    (pre-existing failures documented in the PR discussion).
  • Copyright header check: OK on all changed files.

Compatibility & rollout

  • Default off; no DB migration; no SDK change; no HTTP contract change.
  • Routing reuses capacity/capability columns that operators already maintain
    for the context-budget feature; NULL columns degrade to name heuristics.
  • Rollback = revert flag to false (code path skipped) or revert the commit.

Limitations / follow-ups (tracked in the issue)

  • Tier model is mutually exclusive; capability composition (hard-constraint
    filter + preference scoring) is a deliberate follow-up.
  • Per-agent explicit task tag needs a config-carrier decision
    (context_policy is extra="forbid"; candidates listed in the issue).
  • review_required currently lands in the audit log only; surfacing it in
    the session API is follow-up work, not claimed as implemented.

Route agent runs to the best-matching model of the agent's own
configured pool (quick QA / long context / deep reasoning tiers).

- Opt-in via MODEL_ROUTING_ENABLED (default false; pre-routing
  behaviour fully preserved when off)
- Domain vocabularies live in configs/model_routing_rules.json
  (medical terms shipped as example; engine is domain-agnostic)
- Explicit request model selection always wins over routing
- No-signal and no-capable-candidate cases keep the existing
  selection; routing never fails a run
- Every decision emits a structured audit log with matched rules,
  candidate scores and fallback tier
- Capacity guard prefers the provider input limit and rejects
  candidates that cannot hold the estimated tokens
- 43 unit tests plus create_agent_config integration coverage

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant