Add tokenizer tests and fix auto-compact calculation to account for max_tokens

[sestinj]

Summary by cubic

Fix auto-compact to properly account for model max_tokens and prevent context overflow. Add unit tests for tokenizer functions and compaction logic.

• Bug Fixes

  • Reserve output tokens using maxTokens; default to 35% when unset.
  • Compute usage against available input (contextLength - reservedForOutput) and respect the 80% threshold.
  • Throw when maxTokens >= contextLength.
  • Improve debug logging with input, limits, and decision details.

• Tests

  • Add tokenizer.test.ts covering message/chat token counts, usage %, and compaction with/without maxTokens, including edge cases.
  • Include real-world scenarios (e.g., Claude 200k with 4k max, GPT-4 128k with 4k) and mocks for logger/tokenizer.

[github-actions]

⚠️ PR Title Format

Your PR title doesn't follow the conventional commit format, but this won't block your PR from being merged. We recommend using this format for better project organization.

Expected Format:

<type>[optional scope]: <description>

Examples:

• feat: add changelog generation support
• fix: resolve login redirect issue
• docs: update README with new instructions
• chore: update dependencies

Valid Types:

feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert

This helps with:

• 📝 Automatic changelog generation
• 🚀 Automated semantic versioning
• 📊 Better project history tracking

This is a non-blocking warning - your PR can still be merged without fixing this.

[cubic-dev-ai]

1 issue found across 3 files

_{React with 👍 or 👎 to teach cubic. You can also tag @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

Error message is misleading when triggered by small context limits with default 35% reservation

Prompt for AI agents

Address the following comment on extensions/cli/src/util/tokenizer.ts at line 140:

<comment>Error message is misleading when triggered by small context limits with default 35% reservation</comment>

<file context>
@@ -117,20 +117,37 @@ export function calculateContextUsagePercentage(
 /**
  * Check if the chat history exceeds the auto-compact threshold
  * @param chatHistory The chat history to check
- * @param modelName The model name
+ * @param model The model configuration
  * @returns Whether auto-compacting should be triggered
  */
 export function shouldAutoCompact(
   chatHistory: ChatCompletionMessageParam[],
</file context>