Skip to content

fix(rdb): lowercase filter values for every operator on text fields - #463

Open
RaphaelFakhri wants to merge 1 commit into
vxcontrol:mainfrom
RaphaelFakhri:fix/rdb-text-filter-case
Open

RaphaelFakhri wants to merge 1 commit into
vxcontrol:mainfrom
RaphaelFakhri:fix/rdb-text-filter-case

Conversation

@RaphaelFakhri

Copy link
Copy Markdown

Description of the Change

Problem

TableQuery.Init maps text fields to LOWER(column::text), but DataFilter lowercases a string value only for like and not like. A filter such as {"field":"title","operator":"=","value":"Recon Example"} compares LOWER(title::text) = 'Recon Example' and never matches, and the same value with != matches every row. The in operator already lowercases its items, so the same value behaves differently depending on the operator.

Solution

DataFilter lowercases a string value for every operator when the field is a text field, using the same isNumbericField check that Init uses to decide whether to wrap the column in LOWER(...). Numeric, *_id and *_at fields keep their values unchanged, and like / not like still wrap the value in %.

Closes #462

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Configuration change
  • 🧪 Test update
  • 🛡️ Security update

Areas Affected

  • Core Services (Frontend UI/Backend API)
  • AI Agents (Researcher/Developer/Executor)
  • Security Tools Integration
  • Memory System (Vector Store/Knowledge Base)
  • Monitoring Stack (Grafana/OpenTelemetry)
  • Analytics Platform (Langfuse)
  • External Integrations (LLM/Search APIs)
  • Documentation
  • Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: main (843dfeb)
Docker Version: n/a (postgres:16-alpine for the test tier)
Host OS: Linux x86_64
LLM Provider: n/a
Enabled Features: none

Test Steps

  1. Start a PostgreSQL server and set PENTAGI_TEST_DSN.
  2. Run go test -tags postgres -race -count=1 -run TestTable_ ./pkg/server/rdb/ from backend.
  3. Revert table.go and run the same command.

Test Results

The new TestTable_Query_ComparesTextFieldsWhateverTheCaseOfTheValue lives in table_live_test.go (the postgres tier), because Init emits LOWER(column::text) and sqlite cannot parse the ::text cast. It filters a table holding Recon Example and Other through TableQuery.Query with =, !=, like and in.

Without the fix:

--- FAIL: TestTable_Query_ComparesTextFieldsWhateverTheCaseOfTheValue/equal_to_the_stored_title
    table_live_test.go:88: title = Recon Example counted 0 and read [], want [{1 Recon Example}]
--- FAIL: TestTable_Query_ComparesTextFieldsWhateverTheCaseOfTheValue/not_equal_to_the_stored_title
    table_live_test.go:88: title != Recon Example counted 2 and read [{2 Other} {1 Recon Example}], want [{2 Other}]

With the fix, all 4 subtests pass. go test -race ./pkg/server/rdb/ ./pkg/server/services/ passes, and golangci-lint v2.12.2 reports 0 issues on pkg/server/rdb with and without the postgres tag.

Security Considerations

None. The value is still passed as a bound parameter.

Performance Impact

None. One strings.ToLower per text filter value.

Documentation Updates

  • README.md updates
  • API documentation updates
  • Configuration documentation updates
  • GraphQL schema updates
  • Other:

Deployment Notes

None.

Checklist

Code Quality

  • My code follows the project's coding standards
  • I have added/updated necessary documentation
  • I have added tests to cover my changes
  • All new and existing tests pass
  • I have run go fmt and go vet (for Go code)
  • I have run pnpm run lint (for TypeScript/JavaScript code)

Security

  • I have considered security implications
  • Changes maintain or improve the security model
  • Sensitive information has been properly handled

Compatibility

  • Changes are backward compatible
  • Breaking changes are clearly marked and documented
  • Dependencies are properly updated

Documentation

  • Documentation is clear and complete
  • Comments are added for non-obvious code
  • API changes are documented

Additional Notes

Filters that already worked are unchanged: a lower-case value gives the same result as before, and like, not like and in were already lowercased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REST list filters with = or != ignore text values that contain upper case

1 participant