More modalities

[MaartenGr]

Add audio, video, and code as modalities

Yes, even code. I'm thinking I would like to treat code as a separate modality. The reason for that is mostly recognition. If it's treated as text, then it would be difficult to recognize and potentially create different tokenizers/processors for that entity. Not sure if it's a good idea, but let's do it anyway 😉

I'm trying to separate this by looking at if from two perspectives:

• embedding -- Allow different media to be embedded (I'm calling multimodal data "media")
• labeling -- Much like images, "media" should be labeled through some method. This can be with text by captioning them or through representative items (much like images!)

Embeddings

• audio=, video= and code= are parameters added to fit, fit_transform, transform, update_topics, merge_topics, reduce_topics and reduce_outliers. documents is now optional.
• When documents and media come in equal numbers, they are paired and their embeddings averaged, as documents= with images= always did. Otherwise, each document and each media item is its own row.
  • This was a tricky one. I didn't want to add yet another flag, so I'm keeping this behavior until users find it annoying.
• MultiModalBackend takes a model per modality (image_model, audio_model, video_model, code_model), with embedding_model as the fallback. This will make it possible to use multimodal embedding models on various modalities (typically image + text) or choose a single modality to be embedded (like using semble / potion models, which I love)
  • I'm trying to also differentiate between "rows". This essentially means that I think a row could be several modalities related to each other (image with captions) or independent entities that span multiple rows rather than a single one. This is implicit in BERTopic to simplify things.

Labeling topics

• MultiModalRepresentation replaces VisualRepresentation because I think it should capture all modalities and have standard strategies for each modality supported. It captions images and video frames with any image-text-to-text model and transcribes audio with a speech-recognition model such as Whisper, or with any callable, so a corpus of media alone still gets keywords.
• Only a sample of each topic's media is described, and captions passed in by the user are kept as they are, much like was done with images!
• Each topic keeps its representative media as a Media representation. get_topic_info() shows them as Representative_Items, representative_items_ sits beside representative_docs_, and representative_images_ holds the collages <-- Not sure about this one yet and will have to test it out quite a bit.

Breaking changes

• VisualRepresentation becomes MultiModalRepresentation(model=...).
• MultiModalBackend.embed_images becomes embed_media(images, "image"). embed(docs, images) no longer averages the two; fit_transform(documents=..., images=...) does that.
• A topic's media is now in Representative_Items instead of in a column named after its aspect.

New extras: bertopic[audio] (librosa, soundfile) and bertopic[video] (torchcodec).

Note: media and text passed as separate rows tend to form separate topics, because embedding models place each modality in its own region (the modality gap). Pairs don't have this problem, since each pair is one averaged row. This was an interesting experiment but differs highly between models. Something to document though.