Repository navigation
Topic Relevance: Improve prompt labeling #174
Description
Is your feature request related to a problem?
The topic_relevance_llm builds its system prompt without labeling the tenant's configuration, leading to bare keywords being misinterpreted as noise. This results in inappropriate scoring for questions related to forbidden topics.
Describe the solution you'd like
- Clearly label the config in the prompt using
<forbidden_topics>/<allowed_topics>blocks. - Ensure
Nonecontent from the LLM fails gracefully with a specific error message. - Remove fixed
max_tokenscap and sendreasoning_effortonly for models that support reasoning. - Move config to a
{{TOPIC_CONFIGURATION}}placeholder and restructure prompts as per OpenAI's guidelines.
Original issue
Describe the bug
topic_relevance_llm builds its system prompt by concatenating the tenant's configuration text directly onto the scoring rules (system_prompt = config + "\n\n" + rules in backend/app/core/validators/topic_relevance_llm.py). Nothing tells the model what that text is. With a keyword-style config such as Cricket scores\ncricket\nglific, gpt-4o-mini reads the bare words as noise rather than a forbidden-topic list, so "How do I build a Glific flow?" scores 3 and passes.
This surfaced after #166 made the forbidden schema (v2) the default and kaapi-backend #1245 started sending inline specs, so tenants now hit this path with bare keyword lists and no descriptive stored config.
The validator also breaks on reasoning models (#171 moves the default to gpt-5.6-sol):
content.strip()raisesAttributeErrorwhen the completion content isNone, reported asLLM call failed.- The fixed
max_tokens=300cap counts hidden reasoning tokens, so the JSON verdict can be truncated.
Expected behavior
- The config is clearly labelled in the prompt (e.g. inside a
<forbidden_topics>/<allowed_topics>block placed after the instructions), and bare keywords are treated as topics. Nonecontent from the LLM fails gracefully withLLM returned an unparseable response.- No
max_tokenscap;reasoning_effortis sent only when the model supports reasoning.
Steps to reproduce
- Create a
topic_relevance_llmconfig withprompt_schema_version=2,threshold=2, configurationCricket scores\ncricket\ncricket match\nglific\nGlific flows. - Validate
How do I build a Glific flow?withgpt-4o-mini. - Observe score 3 / passed; expected score 1 / blocked.
Proposed fix
- Move the config into a
{{TOPIC_CONFIGURATION}}placeholder at the end ofv1.md,v2.md,v3.mdand restructure the prompts per OpenAI's prompting guides (headers, explicit rules, few-shot examples, XML boundary for injected content). (content or "").strip()before parsing.- Drop
_MAX_TOKENS; addreasoning_effort="medium"whenlitellm.supports_reasoning(model)is true.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsClosed
Activity
[-]Topic Relevance LLM: Label config in prompt, support reasoning models[/-][+]Topic Relevance: Improve prompt labeling[/+]