Vector based topics tagging for videos

[shanbady]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/4981

Description (What does it do?)

This PR switches us to using vector embeddings to tag videos with topics.

The changes also include:

• the creation of a new "topics" collection in qdrant that stores subtopics
• the addition of a "sync_topic_embeddings" management command that syncs topics from the database into qdrant
• a periodic (daily) vector_search.tasks.sync_topics celery task to keep topics in sync

How can this be tested?

1. checkout this branch and restart celery
2. run python manage.py sync_topic_embeddings to create the topics collection in qdrant
3. verify that the topics collection exists via the qdrant dashboard - there should be topics populated (just sub-topics aka topics that have a parent)
4. you can test the topic tagging for any resource manually via the following shell script:

from learning_resources.models import *
from learning_resources.utils import similar_topics_action
from learning_resources.models import *
lrs = LearningResource.objects.filter(id__in=ContentFile.objects.filter(file_extension__in=[".srt", ".vtt"]).values_list("run__learning_resource__id", flat=True))
for lr in lrs:
	print(f"topics for '{lr.title}' --- ", [topic['name'] for topic in similar_topics_action(lr)])

5. additionally, you can set YOUTUBE_DEVELOPER_KEY and YOUTUBE_CONFIG_URL settings and fetch youtube videos via python manage.py backpopulate_youtube_data --fetch and then inspect the tags it assigns via the following:

from learning_resources.models import *
for video in Video.objects.all().order_by("-created_on")[:100]:
	lr = video.learning_resource
	print(f"topics for '{lr.title}' --- ", [topic.name for topic in lr.topics.all()],"\n\n")

Additional Context

the topics look much better than the previously used opensearch topic assignment however - there is a minor issue (which manifests differently depending on the embedding model) where sometimes broader topics end up being preferred over more specific ones - "hubness problem" or "centroid bias" in embeddings - we can consider a follow up change to blacklist/de-rank more general topics within the subtopics or some other solution. You may see topics like "philosophy" or "pedagogy" being preferred over a more specific options.

[github-actions]

OpenAPI Changes

Show/hide No detectable change.

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[mbertrand]

Works great, just a couple minor suggestions

[mbertrand]

Minor nitpick, this could be a fixture since it returns the same 2 topics in all the above tests

[shanbady]

good call. I made it a fixture.

[mbertrand]

Move to top of the file

[shanbady]

I moved the dense_encoder import to the top but the imports from vector_search.utils were causing a circular import error

[mbertrand]

Move to the top of the file

[shanbady]

see comment about circular import above

[mbertrand]

👍