Skip to content

Topic Relevance: Improve prompt labeling #174

Closed

Description

@AkhileshNegi

Is your feature request related to a problem?
The topic_relevance_llm builds its system prompt without labeling the tenant's configuration, leading to bare keywords being misinterpreted as noise. This results in inappropriate scoring for questions related to forbidden topics.

Describe the solution you'd like

  • Clearly label the config in the prompt using <forbidden_topics> / <allowed_topics> blocks.
  • Ensure None content from the LLM fails gracefully with a specific error message.
  • Remove fixed max_tokens cap and send reasoning_effort only for models that support reasoning.
  • Move config to a {{TOPIC_CONFIGURATION}} placeholder and restructure prompts as per OpenAI's guidelines.
Original issue

Describe the bug
topic_relevance_llm builds its system prompt by concatenating the tenant's configuration text directly onto the scoring rules (system_prompt = config + "\n\n" + rules in backend/app/core/validators/topic_relevance_llm.py). Nothing tells the model what that text is. With a keyword-style config such as Cricket scores\ncricket\nglific, gpt-4o-mini reads the bare words as noise rather than a forbidden-topic list, so "How do I build a Glific flow?" scores 3 and passes.

This surfaced after #166 made the forbidden schema (v2) the default and kaapi-backend #1245 started sending inline specs, so tenants now hit this path with bare keyword lists and no descriptive stored config.

The validator also breaks on reasoning models (#171 moves the default to gpt-5.6-sol):

  • content.strip() raises AttributeError when the completion content is None, reported as LLM call failed.
  • The fixed max_tokens=300 cap counts hidden reasoning tokens, so the JSON verdict can be truncated.

Expected behavior

  • The config is clearly labelled in the prompt (e.g. inside a <forbidden_topics> / <allowed_topics> block placed after the instructions), and bare keywords are treated as topics.
  • None content from the LLM fails gracefully with LLM returned an unparseable response.
  • No max_tokens cap; reasoning_effort is sent only when the model supports reasoning.

Steps to reproduce

  1. Create a topic_relevance_llm config with prompt_schema_version=2, threshold=2, configuration Cricket scores\ncricket\ncricket match\nglific\nGlific flows.
  2. Validate How do I build a Glific flow? with gpt-4o-mini.
  3. Observe score 3 / passed; expected score 1 / blocked.

Proposed fix

  • Move the config into a {{TOPIC_CONFIGURATION}} placeholder at the end of v1.md, v2.md, v3.md and restructure the prompts per OpenAI's prompting guides (headers, explicit rules, few-shot examples, XML boundary for injected content).
  • (content or "").strip() before parsing.
  • Drop _MAX_TOKENS; add reasoning_effort="medium" when litellm.supports_reasoning(model) is true.

Activity

  1. changed the title [-]Topic Relevance LLM: Label config in prompt, support reasoning models[/-] [+]Topic Relevance: Improve prompt labeling[/+] on Oct 26, 2026
  2. self-assigned this
    on Oct 26, 2026
  3. moved this from To Do to In Progress in Kaapi-devon Oct 26, 2026
  4. moved this from In Progress to Closed in Kaapi-devon Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or request

Type

No type

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions