Skip to content

SQLiteMemory cannot filter message pieces on integer prompt_metadata values (get_seeds can) #2891

Description

@feiiiiii5

Describe the bug

SQLiteMemory stringifies every prompt_metadata value before comparing it with JSON_EXTRACT, so a filter on an integer value matches nothing — and the same query with the value as a string does not match either, because SQLite will not compare an integer against text.

MemoryInterface.get_message_pieces types the filter as dict[str, str | int], and PyRIT itself stores an integer there: pyrit/prompt_target/common/prompt_target.py:179 sets piece.prompt_metadata[RequestTraceContext.REQUEST_METADATA_KEY] = 1 on every outgoing request. So asking for the pieces of a request returns an empty list on SQLite, while the same call on AzureSQLMemory returns them.

The fix looks local to sqlite_memory.py:174. Two neighbouring implementations already do the right thing:

  • sqlite_memory.py:186-188, the seed path in the same file, deliberately does not stringify: # Note: We do NOT convert values to string here, to allow integer comparison in JSON
  • azure_sql_memory.py:318-319 stringifies because of a SQL Server premise that does not hold in SQLite: JSON_VALUE always returns nvarchar in SQL Server, so we must convert all values to strings

SQLite's json_extract is JSON-type-preserving (select json_extract('{"a":5}','$.a') = 5 → 1, = '5' → 0), so the SQL Server rule appears to have been carried over without its premise.

Steps/Code to Reproduce

from pyrit.memory import SQLiteMemory
from pyrit.models.messages.message_piece import MessagePiece
from pyrit.models.seeds.seed_prompt import SeedPrompt
import uuid

memory = SQLiteMemory()
memory.add_message_pieces_to_memory(
    message_pieces=[
        MessagePiece(
            conversation_id=str(uuid.uuid4()),
            role="user",
            original_value="hi",
            prompt_metadata={"pyrit_target_request": 1},
        )
    ]
)
await memory.add_seeds_to_memory_async(
    seeds=[SeedPrompt(value="hi", metadata={"pyrit_target_request": 1})], added_by="repro"
)

print(len(memory.get_message_pieces(prompt_metadata={"pyrit_target_request": 1})))   # pieces, int
print(len(memory.get_message_pieces(prompt_metadata={"pyrit_target_request": "1"}))) # pieces, str
print(len(memory.get_seeds(metadata={"pyrit_target_request": 1})))                  # seeds, int

Expected Results

1, 0 (an int-valued filter cannot match a text bind parameter) and 1 — i.e. the seed path and the message-piece path should agree, and the integer filter should find the row.

Actual Results

0
0
1

The third line is the same JSON column and the same dict[str, str | int] contract, read through the helper that does not stringify; the first two go through the one that does.

Screenshots

Not applicable.

Versions

  • OS: macOS (Darwin arm64)
  • Python version: 3.11.15
  • PyRIT version: 1.2.0.dev0, installed from main at 9c26cd7
  • Relevant packages: sqlalchemy 2.0.51, pydantic 2.13.4

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions