build: generate llms.txt and markdown files of the documentation for optimal AI consumption

INSTUI-4701

Author: ToMESSKa

docs/contributor-docs/AI features.md

@@ -0,0 +1,77 @@
+---
+title: AI features
+category: Contributor Guides
+order: 8
+---
+
+## Generating AI agent friendly markdown files for components and guides
+
+InstUI's JSON metadata is used to create a markdown version of the documentation for easier use by AI tools and coding assistants.
+
+These markdown files are created during the build process then are made available for users using specific links.
+
+This collection includes structured files for almost everything under the 'Getting Started', 'Guides', 'Patterns' and 'Components' (including Components/AI Components and Components/Utilities) sections of the documentation.
+
+How it works:
+The core logic resides in **packages/**docs**/buildScripts/ai-accessible-documentation/generate-ai-accessible-markdowns.mts**.
+
+1. **Input and output**  
+   Input: The script processes the JSON files found under **packages/**docs**/**build**/docs** if they are belong to the corresponding sections mentioned above.  
+   Output: A directory of generated .md files under **packages/**docs**/**build**markdowns**, which are also compressed into a documentation.zip archive as well which also can be found in this directory.
+
+2. **Data loading and classification**  
+   The script reads all JSON files from the specified docs folder.
+   Identifies the necessary components and guides based on data.category value.
+
+3. **Markdown generation**  
+   A markdown file is generated using the data found under the 'description' key. As for components, an addtional table of props and a 'Usage' section is generated for showing how to import the component.
+   As for the table of props, the props of a child component (e.g. Tabs.Panel) of a parent component (e.g. Tabs) is also included in the table.
+
+4. **Index file creation and packaging**  
+   After all individual markdown files are written, INDEX.md file is generated, which acts as a master list of contents for the entire documentation set.
+   Finally, the script uses the system's zip command to package all .md files in the output directory into the final documentation.zip file, which can be downloaded here: https://instructure.design/markdowns/documentation.zip  
+   The AI-friendly markdown files are now avaliable either downloading the documentation.zip file or through a link that follows this pattern: https://instructure.design/markdowns/Alert.md
+
+## Generating llms.txt for LLMs
+
+In addition to the markdown files, a llms.txt file is also generated. This file is specifically formatted for optimal consumption by Large Language Models (LLMs) and AI coding agents.
+This file contains a catalog of links pointing to the online markdown files of InstUI components and guides generated above and short description of each component/guide
+
+How it works:
+The core logic resides in **packages/**docs**/buildScripts/ai-accessible-documentation/ggenerate-ai-accessible-llms-file.mts**.
+
+1. **Input and output**  
+   Input: The script processes the following JSON file: **packages/**docs**/**build**/markdown-and-sources-data.json**  
+   Output: An llms.txt file is placed into **packages/**docs**/**build\*\*\*\* which can be found here: https://instructure.design/llms.txt
+
+2. **Data loading and classification**  
+   The script indenftifies the necessary elements looking for "Getting Started", "Guides", "Patterns", and "Components" section names.
+
+3. **Markdown generation**  
+   It creates a documentation structure like this:
+
+   ```md
+   ---
+   type: code
+   ---
+
+   # Instructure UI (InstUI) - React Component Library
+
+   ## Documentation
+
+   ### User Guides
+
+   #### Getting Started
+
+   #### Guides
+
+   #### Patterns
+
+   ### Components
+
+   #### Utilities
+
+   #### AI components
+   ```
+
+   It generates links for the components/guides and appends a brief summary of each of them using the **packages/**docs**/buildScripts/ai-accessible-documentation/summaries-for-llms-file.json** file, searching for matching component/guide title.

docs/contributor-docs/contributing.md

@@ -102,6 +102,7 @@ Please update the documentation and examples with any changes.
 11. Visit [http://localhost:9090](http://localhost:9090) in a browser. You should see your component listed in the docs.
 12. Start making changes to your component, and watch it update in the browser automatically.
 13. Resolve all `FIXME` comments in the generated code (except in the `MyComponentLocator.ts`).
+14. Add a short description of the new component to the **packages/**docs**/buildScripts/ai-accessible-documentation/summaries-for-llms-file.json file**. This will optimize the component's documentation for consumption by AI agents.
 
 ### Adding a new dependency
 

docs/getting-started/usage.md

@@ -74,6 +74,20 @@ Congrats, you have now a (very) basic app that uses Instructure UI :)
 
 Just add the `@instructure/ui` dependency as shown above and wrap the part of your app that will use InstUI in `<InstUISettingsProvider>` and start using InstUI components.
 
+## Using InstUI with AI coding agents
+
+InstUI provides a compressed, downloadable collection of all component documentation and user guides in an AI-agent-friendly markdown format.
+
+You can download the archive from the following link:
+https://instructure.design/markdowns/documentation.zip
+
+The download includes an INDEX.md file that references all available documentation within the compressed folder.
+
+These files are designed to be used as context for AI coding agents.
+
+Additionally, an llms.txt file is available. This file contains a catalog of links pointing to the online markdown files for InstUI components and guides, which are also accessible to AI agents. It can be found at:
+https://instructure.design/llms.txt
+
 ## Further reading
 
 - To use a different theme or customize one read about [Using theme overrides](#using-theme-overrides)

packages/__docs__/buildScripts/ai-accessible-documentation/generate-ai-accessible-llms-file.mts

@@ -0,0 +1,140 @@
+/**
+ * Processes InstUI JSON metadata to generate a file
+ * formatted for Large Language Models (LLMs) and AI agent consumption
+ * - Contains a catalog of links pointing to ai-accessible markdown files about InstUI components and guides
+ * - Contains a summary of each component/guide next to the links
+*/
+
+import { writeFileSync, readFileSync } from 'fs'
+
+interface SectionData {
+  docs?: string[]
+  sections?: string[]
+  title?: string
+}
+
+interface GenerateLLMSOptions {
+  summariesFilePath?: string
+  baseUrl?: string
+  outputFilePath?: string
+}
+
+function generateAIAccessibleLlmsFile(
+  sourceFilePath: string,
+  options: GenerateLLMSOptions = {}
+): string {
+  const { summariesFilePath, baseUrl = '', outputFilePath } = options
+
+  const fileContent = readFileSync(sourceFilePath, 'utf-8')
+  const jsonData = JSON.parse(fileContent)
+  const { sections, library } = jsonData
+
+  const version = library?.version
+
+  let summaries: Record<string, string> = {}
+  // Generate summaries object from the summaries file for later lookup
+  if (summariesFilePath) {
+    try {
+      const summariesContent = readFileSync(summariesFilePath, 'utf-8')
+      const data = JSON.parse(summariesContent)
+
+      data.forEach((item: { title: string; summary?: string }) => {
+        summaries[item.title] = item.summary || ''
+      })
+    } catch (error) {
+      console.warn('Could not load summaries file:', error)
+    }
+  }
+
+  let LlmsMarkdownContent = `# Instructure UI (InstUI) - React Component Library\n\n- version ${version} \n\n`
+  LlmsMarkdownContent += `- Instructure UI (InstUI) is a comprehensive React component library.\n\n`
+
+  // Add main Documentation section
+  LlmsMarkdownContent += `## Documentation\n\n`
+
+  // Add User Guides section
+  LlmsMarkdownContent += `### User Guides\n`
+
+  Object.entries(sections as Record<string, SectionData>).forEach(
+    ([sectionKey, sectionData]) => {
+      const sectionTitle = sectionData.title || sectionKey
+
+      // Only process these specific sections
+      if (!['Getting Started', 'Guides', 'Patterns'].includes(sectionTitle)) {
+        return
+      }
+
+      LlmsMarkdownContent += `#### ${sectionTitle}\n\n`
+
+      if (sectionData.docs && sectionData.docs.length > 0) {
+        const uniqueDocs = [...new Set(sectionData.docs)]
+        uniqueDocs.forEach((doc) => {
+          // Skip unnecessary documents
+          if (
+            doc === 'CODE_OF_CONDUCT' ||
+            doc === 'LICENSE' ||
+            doc.includes('upgrade-guide')
+          ) {
+            return
+          }
+          const displayName = doc
+            .replace(/([A-Z])/g, ' $1')
+            .replace(/-/g, ' ')
+            .trim()
+            .toLowerCase()
+            .replace(/^\w/, (char) => char.toUpperCase())
+
+          const summary = summaries[doc]
+          LlmsMarkdownContent += `- [${displayName}](${baseUrl}${doc}.md)${
+            summary ? `: ${summary}` : ''
+          }\n`
+        })
+      }
+      LlmsMarkdownContent += '\n'
+    }
+  )
+
+  // Add Components section under Documentation
+  LlmsMarkdownContent += `### Components\n\n`
+
+  const componentsSection = sections['components'] as SectionData | undefined
+  if (componentsSection) {
+    const allComponents = [...new Set(componentsSection.docs || [])]
+    allComponents.forEach((component) => {
+      const summary = summaries[component]
+      LlmsMarkdownContent += `- [${component}](${baseUrl}${component}.md)${
+        summary ? `: ${summary}` : ''
+      }\n`
+    })
+  }
+
+  // Process component subsections like Components/AI Components and Components/Utilities, skip deprecated
+  if (componentsSection?.sections && componentsSection.sections.length > 0) {
+    const subsections = componentsSection.sections
+    subsections.forEach((subsectionPath: string) => {
+      if (subsectionPath.toLowerCase().includes('deprecated')) return
+
+      const subsection = sections[subsectionPath] as SectionData | undefined
+      if (subsection && subsection.docs && subsection.docs.length > 0) {
+        const subsectionTitle = subsection.title
+        LlmsMarkdownContent += `\n#### ${subsectionTitle}\n\n`
+        // Avoid duplicates
+        const uniqueSubDocs = [...new Set(subsection.docs)]
+        uniqueSubDocs.forEach((doc) => {
+          const summary = summaries[doc]
+          LlmsMarkdownContent += `- [${doc}](${baseUrl}${doc}.md)${
+            summary ? `: ${summary}` : ''
+          }\n`
+        })
+      }
+    })
+  }
+
+  if (outputFilePath) {
+    writeFileSync(outputFilePath, LlmsMarkdownContent)
+  }
+
+  return LlmsMarkdownContent
+}
+
+export { generateAIAccessibleLlmsFile }