Skip to content

ci: index the docs on Context7 and refresh on push - #178

Merged
cardmagic merged 2 commits into
masterfrom
ci/context7
Oct 8, 2026
Merged

cardmagic merged 2 commits into
masterfrom
ci/context7

Conversation

@cardmagic

Copy link
Copy Markdown
Owner

Why

Context7 has no classifier library yet, so coding agents that use it cannot retrieve the current API.

What changes

  • context7.json limits indexing to the README and docs/, leaves out AGENTS.md, CLAUDE.md, and CONTRIBUTING.md, and gives agents five rules taken from the docs. It also carries the url and public_key that Context7 reads to verify the maintainer's claim. The public_key is the maintainer's public account key, which Context7 designs to be committed.
  • .github/workflows/context7-refresh.yml asks Context7 to refresh the index after a push to master that changes the README, context7.json, or docs/. Context7 allows one refresh every 10 days for a library and answers too-early before then. That answer ends the job with a notice. Any other non-200 answer fails the job. It reads the CONTEXT7_API_KEY repository secret, which is set.

No CHANGELOG entry: the change does not reach gem users.

Validation

  • actionlint passes, and context7.json parses.
  • Each rule was checked against the docs and code: the class names exist, docs/tfidf.md documents Classifier::NotFittedError, docs/persistence.md says to prefer JSON over Marshal, and train_from_stream and train_batch exist.
  • The same workflow script runs in cardmagic/solid-objects-ruby and cardmagic/solid-objects-js, where it passed against the live API.

Context7 has no classifier library yet, so agents that use it cannot
retrieve the current API.

context7.json limits indexing to the README and docs/, leaves out the
maintainer files, and gives agents rules taken from the docs. It also
carries the url and public_key that Context7 reads to verify the
maintainer's claim on the library.

The refresh workflow asks Context7 to refresh the index after a push
to master that changes the README, context7.json, or docs/. Context7
allows one refresh every 10 days for one library and answers
too-early before then, so that answer ends the job with a notice. Any
other non-200 answer fails the job.

No CHANGELOG entry: the change does not reach gem users.
@greptile-apps

greptile-apps Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High risk] Adds CI workflow that calls external API with a secret.

The PR appears safe to merge, with one non-blocking correction to the TF-IDF guidance.

Findings

  1. P2 TF-IDF guidance omits memory limits ▶
Fix with agent prompt
### Issue 1
context7.json:20
The rule now presents `fit_from_stream` as the `Classifier::TFIDF` option for data larger than memory. However, it collects every document in an array before calling `fit`. Agents following this advice can generate code that runs out of memory on large corpora. State that `fit_from_stream` still requires the full corpus to fit in memory.

```suggestion
    "For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream, but it collects all documents in memory before fitting, so the corpus must fit in memory. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch."
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds Context7 indexing settings and a workflow that requests a refresh after documentation changes on master.

  • The latest commit fixes the previous advice about train_batch.
  • The new TF-IDF advice still needs a note that fitting retains the whole corpus in memory.

Reviews (2) · Last reviewed commit: "ci: describe batch training as in-memory" · Reviewed by Greptile

Comment thread context7.json Outdated
Review feedback on #178. The rule listed train_batch as a way to
train on data larger than memory, but docs/streaming.md says that
train_batch takes documents already in memory. The rule now names
train_from_stream for data larger than memory, fit_from_stream for
TF-IDF, and train_batch, or add_batch for LSI and KNN, for documents
in memory, as docs/streaming.md describes.
@cardmagic

Copy link
Copy Markdown
Owner Author

@greptileai Please review the latest commit 4f8e225. The streaming rule now matches docs/streaming.md, including fit_from_stream for TF-IDF and add_batch for LSI and KNN.

Comment thread context7.json
"Start with Classifier::Bayes for text classification. Use Classifier::LogisticRegression for calibrated probabilities, Classifier::LSI for semantic search and related documents, and Classifier::KNN for nearest-example votes.",
"Classifier::TFIDF must be fitted on a corpus before it transforms text; transform raises Classifier::NotFittedError before a fit. The keywords command fits a vocabulary first and reads that model in later commands.",
"Save and load trained models with the JSON persistence API. Prefer it to Marshal, and load a marshalled model only from a source you trust.",
"For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 TF-IDF guidance omits memory limits

The rule now presents fit_from_stream as the Classifier::TFIDF option for data larger than memory. However, it collects every document in an array before calling fit. Agents following this advice can generate code that runs out of memory on large corpora. State that fit_from_stream still requires the full corpus to fit in memory.

Suggested change
"For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch."
"For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream, but it collects all documents in memory before fitting, so the corpus must fit in memory. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch."

Knowledge Base Used: Streaming processing

Prompt To Fix With AI
This is a comment left during a code review.
Path: context7.json
Line: 20

Comment:
**TF-IDF guidance omits memory limits**

The rule now presents `fit_from_stream` as the `Classifier::TFIDF` option for data larger than memory. However, it collects every document in an array before calling `fit`. Agents following this advice can generate code that runs out of memory on large corpora. State that `fit_from_stream` still requires the full corpus to fit in memory.

```suggestion
    "For data larger than memory, use train_from_stream, which reads one document per line from an IO stream. Classifier::TFIDF uses fit_from_stream, but it collects all documents in memory before fitting, so the corpus must fit in memory. Use train_batch only for documents already in memory; Classifier::LSI and Classifier::KNN name it add_batch."
```

**Knowledge Base Used:** [Streaming processing](https://app.greptile.com/craftsmanfounder/-/custom-context/knowledge-base/cardmagic/classifier/-/docs/streaming-processing.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@cardmagic
cardmagic merged commit c24efd5 into master Oct 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant