Skip to content

fix: drop stale topic aspects when topics are recalculated - #2550

Open
raashish1601 wants to merge 1 commit into
MaartenGr:masterfrom
raashish1601:fix/2531-merge-aspects
Open

raashish1601 wants to merge 1 commit into
MaartenGr:masterfrom
raashish1601:fix/2531-merge-aspects

Conversation

@raashish1601

Copy link
Copy Markdown

What does this PR do?

Fixes #2531

update_topics() sets representation_model to whatever is passed in, so after update_topics(docs, vectorizer_model=...) it is None. A later merge_topics() (or topic reduction) then recalculates the main representation but not the aspects. The old entries in topic_aspects_ stay as they were and keep pointing at the old topic ids, so after a merge every topic past the merged one shows the aspects of a different topic.

_extract_topics() now clears topic_aspects_ before recalculating representations for a new set of topics. Aspects that are still in representation_model get recalculated as before. Aspects whose model is gone are dropped instead of being left attached to the wrong topics.

Calling update_topics() by itself doesn't change the topics, so the aspects it keeps still line up and that path is unchanged.

Two tests in tests/test_representation/test_representations.py:

  • merging topics recalculates the MMR aspect for the new topics (already worked, now covered)
  • merging after update_topics() with a custom vectorizer leaves no aspect keyed by old topics. This one fails on master.

I ran tests/test_representation, tests/test_bertopic.py, tests/test_variations and tests/test_reduction on Linux (Python 3.12), plus ruff check and ruff format --check.

Before submitting

After update_topics() without a representation_model, merge_topics()
and topic reduction recalculated the main representation but left the
old topic_aspects_ in place, keyed by the old topic ids. Clear the
aspects before recalculating so that only aspects from the current
representation_model remain.

Fixes MaartenGr#2531

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Topic aspects are not updated when merging topics and the model uses a custom vectorizer

1 participant