Adopting LLMs txt file for latest package docs

[oraNod]

Opening this as a draft so we can discuss it at the next DaWGs.

The sphinx-llms-txt extension generates an llms.txt file and a combined llms-full.txt file. These files support the llms.txt standard that aims to make docsites more LLM-friendly.

It's kind of hacky right now but, for the purposes of POC, try it out by passing LLM_DEFINES="-D html_copy_source=True" with your make invocation.

Attaching generated files for preview:

• llms.txt
• llms-full.txt

Note that llms-full.txt is 329.79 MB when generated so I've truncated it after line 1000.

[gundalow]

@jamestalton mentioned that while this will help, having it reference markdown files is even better as they are more easily injested for training.

I know updating the Makefile to generate markdown is more involved, though Sphinx can export Markdown

James: If you can provide some extra context, that would be great

[oraNod]

@jamestalton mentioned that while this will help, having it reference markdown files is even better as they are more easily injested for training.

I know updating the Makefile to generate markdown is more involved, though Sphinx can export Markdown

James: If you can provide some extra context, that would be great

llms.txt actually is a markdown file. I know that's not immediately obvious though: https://llmstxt.org/#format

[gundalow]

Sorry, my fault for typing to fast.
I mean the files that llms.txt links to should be Markdown, as they are more easily parsed.

ie https://base-ui.com/react/overview/about is also published as https://base-ui.com/react/overview/about.md and the Markdown version is the one linked in https://base-ui.com/llms.txt