feat: AI client user-agent detection for automatic markdown serving #15059
+128,007
−4,194
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Detect AI/LLM tools (Claude, Cursor, Copilot, etc.) via user-agent - Automatically redirect to .md versions for better LLM consumption - Add manual override with ?format=md query parameter - Preserve existing redirect functionality 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Log all user agents for non-static requests - Log when redirects to markdown occur - Helps identify what user agents AI tools are sending
- Add /got/i pattern to detect sindresorhus/got library - Common HTTP client used by Node.js tools and automation - Ensures markdown serving for tools using got for requests
- Show 📄 MARKDOWN or 🌐 HTML indicators in logs - Determine content type before logging for better debugging - Makes it clear which requests get markdown vs HTML content
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![semgrep-code-getsentry[bot]](https://avatars.githubusercontent.com/u/1396951?s=60&v=4){kind=small}
Comment on lines
+107
to
+118
| return NextResponse.redirect(newUrl, { | ||
| status: redirectStatusCode, | ||
| }); |
There was a problem hiding this comment.
Critical severity vulnerability may affect your project—review required:
Line 107 lists a dependency (next) with a known Critical severity vulnerability.
ℹ️ Why this matters
Affected versions of next are vulnerable to Improper Authorization / Incorrect Authorization. Next.js middleware-based authorization checks can be bypassed by an attacker who forges external requests that include a specific header, tricking the application into treating the request as a trusted internal call and thus bypassing access controls.
To resolve this comment:
Check if you perform authorization in the middleware.
- If you're affected, upgrade this dependency to at least version 15.2.3 at yarn.lock.
- If you're not affected, comment
/fp we don't use this [condition]
💬 Ignore this finding
To ignore this, reply with:
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
You can view more details on this finding in the Semgrep AppSec Platform here.
Comment on lines
24
to
31
| return NextResponse.json( | ||
| { | ||
| query, | ||
| limit, | ||
| error: error instanceof Error ? error.message : "Unknown error", | ||
| }, | ||
| {status: 500} | ||
| ); |
There was a problem hiding this comment.
Critical severity vulnerability may affect your project—review required:
Line 24 lists a dependency (next) with a known Critical severity vulnerability.
ℹ️ Why this matters
Affected versions of next are vulnerable to Improper Authorization / Incorrect Authorization. Next.js middleware-based authorization checks can be bypassed by an attacker who forges external requests that include a specific header, tricking the application into treating the request as a trusted internal call and thus bypassing access controls.
To resolve this comment:
Check if you perform authorization in the middleware.
- If you're affected, upgrade this dependency to at least version 15.2.3 at yarn.lock.
- If you're not affected, comment
/fp we don't use this [condition]
💬 Ignore this finding
To ignore this, reply with:
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
You can view more details on this finding in the Semgrep AppSec Platform here.
Comment on lines
+17
to
+22
| return NextResponse.json({ | ||
| query, | ||
| limit, | ||
| count: results.length, | ||
| results, | ||
| }); |
There was a problem hiding this comment.
Critical severity vulnerability may affect your project—review required:
Line 17 lists a dependency (next) with a known Critical severity vulnerability.
ℹ️ Why this matters
Affected versions of next are vulnerable to Improper Authorization / Incorrect Authorization. Next.js middleware-based authorization checks can be bypassed by an attacker who forges external requests that include a specific header, tricking the application into treating the request as a trusted internal call and thus bypassing access controls.
To resolve this comment:
Check if you perform authorization in the middleware.
- If you're affected, upgrade this dependency to at least version 15.2.3 at yarn.lock.
- If you're not affected, comment
/fp we don't use this [condition]
💬 Ignore this finding
To ignore this, reply with:
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
You can view more details on this finding in the Semgrep AppSec Platform here.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| const {searchParams} = new URL(request.url); | ||
| const query = searchParams.get('q') ?? ''; | ||
| const limitParam = searchParams.get('limit'); | ||
| const limit = limitParam ? Math.min(25, Math.max(1, Number(limitParam))) : 10; |
There was a problem hiding this comment.
Bug: Invalid Query Parameter Causes Empty Search Results
When the limit query parameter is a non-numeric value, it's parsed as NaN. This NaN is then passed to searchIndex, causing matches.slice(0, NaN) to return an empty array, effectively yielding no search results.
Bundle ReportChanges will increase total bundle size by 1.75MB (7.52%) ⬆️
Affected Assets, Files, and Routes:view changes for bundle: sentry-docs-edge-server-array-pushAssets Changed:
view changes for bundle: sentry-docs-client-array-pushAssets Changed:
App Routes Affected:
view changes for bundle: sentry-docs-server-cjsAssets Changed:
Files in
Files in
App Routes Affected:
|
|
Closing this PR to create a clean version without unrelated commits. The new PR will only include the AI client detection changes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements automatic markdown serving for AI/LLM tools and development environments by detecting user agents. When AI clients access documentation pages, they are automatically redirected to
.mdversions for better LLM consumption and processing.Key Features
generate-md-exports.mjs)?format=mdquery parameter for manual markdown requestsCurrently Configured User Agents
The middleware detects and redirects the following user agent patterns to markdown:
AI/LLM Tools
/claude/i) - Claude Desktop/Code/chatgpt/i) - ChatGPT clients/openai/i) - OpenAI tools and APIs/anthropic/i) - Anthropic toolsDevelopment Tools & IDEs
/cursor/i) - Cursor IDE/copilot/i) - GitHub Copilot/vscode/i) - VS Code extensions/intellij/i) - IntelliJ plugins/sublime/i) - Sublime Text pluginsHTTP Libraries
/got/i) - sindresorhus/got HTTP library for Node.jsHow It Works
src/middleware.tsintercepts requests and checks user agents.mdversions (302 redirect)Example Usage
Automatic Detection
Manual Override
Logging Output
Enhanced middleware logging shows content type and user agent:
Benefits
Breaking Changes
None. This is purely additive functionality that doesn't affect existing behavior.
Note
Add UA-based markdown redirects in middleware and introduce a simple docs search API with generated index.
src/middleware.tsto auto-serve markdown (redirect to.md), with optional query override.app/api/search/route.tsusingapp/api/search/searchIndex.ts.app/scripts/generate-search-index.mjsand bundled datapublic/search-index.json.app/api/[transport]/route.ts.app/shared/docs-utils.tshelpers.src/@types/remark-mdx-images/index.d.tstypes.tsconfig.jsonandpackage.jsonfor new scripts/types.Written by Cursor Bugbot for commit a725164. This will update automatically on new commits. Configure here.