Repository navigation
ci: index the docs on Context7 and refresh on push - #178
Conversation
Context7 has no classifier library yet, so agents that use it cannot retrieve the current API. context7.json limits indexing to the README and docs/, leaves out the maintainer files, and gives agents rules taken from the docs. It also carries the url and public_key that Context7 reads to verify the maintainer's claim on the library. The refresh workflow asks Context7 to refresh the index after a push to master that changes the README, context7.json, or docs/. Context7 allows one refresh every 10 days for one library and answers too-early before then, so that answer ends the job with a notice. Any other non-200 answer fails the job. No CHANGELOG entry: the change does not reach gem users.
|
Review feedback on #178. The rule listed train_batch as a way to train on data larger than memory, but docs/streaming.md says that train_batch takes documents already in memory. The rule now names train_from_stream for data larger than memory, fit_from_stream for TF-IDF, and train_batch, or add_batch for LSI and KNN, for documents in memory, as docs/streaming.md describes.
|
@greptileai Please review the latest commit 4f8e225. The streaming rule now matches docs/streaming.md, including fit_from_stream for TF-IDF and add_batch for LSI and KNN. |
| "Start with Classifier::Bayes for text classification. Use Classifier::LogisticRegression for calibrated probabilities, Classifier::LSI for semantic search and related documents, and Classifier::KNN for nearest-example votes.", | ||
| "Classifier::TFIDF must be fitted on a corpus before it transforms text; transform raises Classifier::NotFittedError before a fit. The keywords command fits a vocabulary first and reads that model in later commands.", | ||
| "Save and load trained models with the JSON persistence API. Prefer it to Marshal, and load a marshalled model only from a source you trust.", | ||
| "For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch." |
There was a problem hiding this comment.
TF-IDF guidance omits memory limits
The rule now presents fit_from_stream as the Classifier::TFIDF option for data larger than memory. However, it collects every document in an array before calling fit. Agents following this advice can generate code that runs out of memory on large corpora. State that fit_from_stream still requires the full corpus to fit in memory.
| "For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch." | |
| "For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream, but it collects all documents in memory before fitting, so the corpus must fit in memory. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch." |
Knowledge Base Used: Streaming processing
Prompt To Fix With AI
This is a comment left during a code review.
Path: context7.json
Line: 20
Comment:
**TF-IDF guidance omits memory limits**
The rule now presents `fit_from_stream` as the `Classifier::TFIDF` option for data larger than memory. However, it collects every document in an array before calling `fit`. Agents following this advice can generate code that runs out of memory on large corpora. State that `fit_from_stream` still requires the full corpus to fit in memory.
```suggestion
"For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream, but it collects all documents in memory before fitting, so the corpus must fit in memory. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch."
```
**Knowledge Base Used:** [Streaming processing](https://app.greptile.com/craftsmanfounder/-/custom-context/knowledge-base/cardmagic/classifier/-/docs/streaming-processing.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Why
Context7 has no
classifierlibrary yet, so coding agents that use it cannot retrieve the current API.What changes
context7.jsonlimits indexing to the README anddocs/, leaves outAGENTS.md,CLAUDE.md, andCONTRIBUTING.md, and gives agents five rules taken from the docs. It also carries theurlandpublic_keythat Context7 reads to verify the maintainer's claim. Thepublic_keyis the maintainer's public account key, which Context7 designs to be committed..github/workflows/context7-refresh.ymlasks Context7 to refresh the index after a push tomasterthat changes the README,context7.json, ordocs/. Context7 allows one refresh every 10 days for a library and answerstoo-earlybefore then. That answer ends the job with a notice. Any other non-200 answer fails the job. It reads theCONTEXT7_API_KEYrepository secret, which is set.No CHANGELOG entry: the change does not reach gem users.
Validation
actionlintpasses, andcontext7.jsonparses.docs/tfidf.mddocumentsClassifier::NotFittedError,docs/persistence.mdsays to prefer JSON over Marshal, andtrain_from_streamandtrain_batchexist.