Conversation
|
Hi @MaartenGr — my July 8 batch was part of the influx you're describing, so I've closed 12 of them. One correction, for what it's worth: the code isn't AI-written. It predates my use of these tools and are patches I wrote by hand for a commercial deployment of BERTopic, running in production, that I wanted to contribute back rather than sit on. What was generated is the packaging: the PR and issue descriptions, the titles, the [stacks on #NNNN] markers. This was an attempt at splitting my contributions in reviewable pieces. But I can see why it read the way it did. The volume objection stands either way, though. 13 PRs at once is un-reviewable for a single maintainer, and I should have opened one issue and waited for a yes. I left #2496 open as it's a bug fix rather than a feature: _extract_representative_docs samples with replace=True and maps indices by text matching, so duplicate documents cause the wrong representative docs to be selected. The issues I've left open: please close any that aren't useful. #2498 and #2499 are the two I'd flag: MaximalMarginalRelevance makes 2N embedding calls where 1 would do, and _extract_representative_docs recomputes from scratch on every call. Both were real costs at our data volume. The closed branches are still on my fork if you ever want one re-opened individually. On the process: I did read Thanks for your work on this, and have a good day. |
|
@pidefrem Thank you for the explanation. Do note though that I wasn't specifically referencing those PRs. It's just that I'm seeing an increase in PRs across all my repos that I have to manage somehow with many being lower effort (including issues and broken comments). Thank you again though for clarifications on your stacked PRs and let's move this conversation else since it's not related to a potential v1.0. |
2 participants
[HIGHLY EXPERIMENTAL] / [UNLIKELY TO MERGE]
This PR is a branch off of #2467 and something that I have been thinking about for a while now.
BERTopic has been a fun project, but it's time to take it to the next level. A big part of that is applying standardization in a way that allows for the development of new features. In general, I want to:
All of the above is meant to speed up development in a way that makes my life easier, that of contributors, and even that of Agents.
Standardization
The idea of modularity in BERTopic has turned out to be a compelling point for using the package, but did leave it with several bugs that were hard to squash. The code was also written in the early days of my career and I would do some things differently now.
In particular, I think I can improve stability and allow for a lot of cool new features by introducing a couple of data classes:
bertopic._corpus.Corpus- This class tracks all information on a document-level and concerns the input data (e.g.,documents,images,embeddings) as well as data generated during a fit (e.g.,reduced embeddings,topic assignments,probabilities)bertopic_topics.Topic- This class tracks all information on a topic-level and can contain any number of documents or their sub-components (e.g.,topic_id, varioustopic_representations,topic_embeddings)bertopic_topics.Topics- This class tracks all information on a multi-topic-level and contains references to multiple topics that are somehow related to each other (e.g.,hierarchical_topics,dynamic_topics,topic_taxonomy)Features
Alongside the above standardization, I want to make sure that BERTopic is ready for the LLM/Agentic era where we do much more than just name the clusters. Creating summarizations, sentiment-based clusters, agent-driving clusters, etc. should all be possible with BERTopic if it is truly modular.
A big part of that is rewriting the topic representation classes as they currently are. I made it simple for myself at the start by assuming keywords and hacking labels on top of that, but it should change.
There is much more that could be enabled with this standardization, but it will depend on how quickly (and how stable) this new standardization will be.
AI-only PRs
There are an increasing number of AI-written PRs that I have a lot of issues keeping up with. They are often out of scope, too large, create a lot of unnecessary code, or do not adhere to the practices of this repo. That's not to say that we shouldn't have AI write PRs, it just means that it is my responsibility to create clear guidelines on what will be expected. Due to my exceedingly limited bandwidth at the moment, that turns out to be quite the challenge.