Update indexing and quantization docs - #344
Conversation
| | `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | | ||
| | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. | | ||
| | `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | | ||
| | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. | |
There was a problem hiding this comment.
I'm not sure if changing this was in scope, but I find these descriptions pretty weak. They would be better if they gave some kind of comparison between each other. For example if we are saying these are all useful because they fall at different points on the compression/speed/accuracy plane, we should describe how they actually relate. Why would I choose PQ over RQ, or SQ over PQ, is not really answered by this table.
Another way to do this is something like,
If you need X [petabyte scale low-latency search on static dataset], choose Y because x, y, z
If you need A, choose B because a, b, c
etc. I have no idea what the real values to plug in here are though.
There was a problem hiding this comment.
"I think these descriptions are pretty weak" I agree. Unfortunately, until recently many technical descriptions were written similar to this, and I didn't really know how to approach them (I also didn't have a clearly defined scope for this documentation task) so I just largely took my best guess. I agree that there could be further improvements. Maybe I'll make a note of this and add it to a future commit?
| @@ -0,0 +1,172 @@ | |||
| 9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 1) --- | |||
There was a problem hiding this comment.
no. oops. you're right to delete the whole file
2 participants
Substantial changes to indexing documentation, as part of Jason's internship project. Reorganized and rewrote "overview", "vector index", and "quantization" tabs in https://docs.lancedb.com/indexing . Added descriptions of algorithms and improved API references, usage examples, and overall readability/flow. Msg me on slack with questions