Implement pluggable heuristic pipeline for XML tool-call parsing

[Copilot]

Refactors XML parsing logic from hardcoded sequential repairs into a composable three-phase heuristic pipeline (pre-parse, fallback-reparse, post-parse), enabling protocol-level configuration while maintaining backward compatibility.

Architecture

Core Engine (heuristic-engine.ts)

• Phase-based orchestrator applying ToolCallHeuristic[] with conditional gating via applies(ctx)
• IntermediateCall context carries toolName, schema, rawSegment, parsed, errors, meta
• Heuristics return HeuristicResult with optional rawSegment, parsed, reparse, stop flags

Default Pipeline (default-heuristics.ts)

• Pre-parse: normalizeCloseTags + escapeInvalidLt (applied before pipeline entry)
• Fallback-reparse: balanceTags (safety-gated: skips if no malformed tags AND balancing adds content), dedupeShellStringTags (gated by schema.properties.command.type === "array")
• Post-parse: repairAgainstSchema (array/object coercion)

Integration (heuristic-integration.ts)

• parseWithHeuristics() wraps pipeline application with sensible defaults
• Stores original content in ctx.meta for heuristics needing pre-normalization state

API

// Use defaults
const protocol = morphXmlProtocol();

// Add custom heuristics
const protocol = morphXmlProtocol({
  heuristics: [customHeuristic]  // Merged with defaults by phase
});

// Override entire pipeline
const protocol = morphXmlProtocol({
  pipeline: {
    preParse: [...],
    fallbackReparse: [...],
    postParse: [...]
  }
});

Changes to morphXmlProtocol

• processToolCall and handleStreamingToolCallEnd now route through parseWithHeuristics()
• Removed tryParseSecondaryXml (logic migrated to pipeline heuristics)
• Pipeline configuration threaded through buffer processing for streaming

Testing

Added 51 unit tests covering engine mechanics (phase execution, gating, error recovery) and individual heuristics. All 255 existing integration tests pass unchanged.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>RFC: Pluggable Heuristic Pipeline for XML Tool-Call Parsing</issue_title>
<issue_description>## Summary
Introduce a pluggable heuristic pipeline to make XML tool-call parsing more robust, testable, and configurable. Instead of hardcoding recovery logic (e.g., duplicate string tag dedupe) inside a specific protocol, provide a composable pipeline of text/object heuristics that run around the core parse step.

Motivation

• Current heuristics (e.g., last-win duplicate string tags for shell-like tools) are embedded in morph-xml and hard to reuse.
• We want to keep model output untouched while making the parser resilient.
• A pipeline improves separation of concerns, reusability, and unit testing.

Goals

• Decouple heuristics from a concrete protocol.
• Provide distinct phases:
  • pre-parse (text normalization),
  • fallback-reparse (text repair after initial parse failure),
  • post-parse (object repair/coercion).
• Unify streaming and non-stream processing via the same engine.
• Allow opt-in gating by tool/schema (preserve backward compatibility).

Non-Goals

• Changing the underlying RXML parser.
• Broadly altering error semantics (defaults should remain backward compatible).
• Enabling all heuristics for all tools by default (opt-in remains the default for sensitive rules).

Proposed Design

• Heuristic phases: pre-parse, fallback-reparse, post-parse.
• Intermediate representation carries toolName, schema, rawSegment, parsed, errors, and meta.
• Orchestrator (engine) applies a list of ToolCallHeuristics per phase, controlling re-parse and stop conditions.
• Default pipeline composes existing logic:
  • pre-parse: normalizeCloseTags, balanceTags
  • fallback-reparse: shell-only duplicate string tag dedupe (last-win, gated by command: array)
  • post-parse: repairParsedAgainstSchema, array-item/object coercions
• Streaming uses the same engine at tool-call boundaries.

API Sketch

export type HeuristicPhase = 'pre-parse' | 'fallback-reparse' | 'post-parse';

export type IntermediateCall = {
  toolName: string;
  schema: unknown;
  rawSegment: string; // inner XML
  parsed: unknown | null;
  errors: unknown[];
  meta?: Record<string, unknown>;
};

export type HeuristicResult = {
  rawSegment?: string;   // updated text for reparse
  parsed?: unknown;      // updated object
  reparse?: boolean;     // request a reparse now
  stop?: boolean;        // stop further heuristics
  warnings?: string[];
};

export type ToolCallHeuristic = {
  id: string;
  phase: HeuristicPhase;
  applies(ctx: IntermediateCall): boolean;
  run(ctx: IntermediateCall): HeuristicResult;
};

export type PipelineConfig = {
  preParse?: ToolCallHeuristic[];
  fallbackReparse?: ToolCallHeuristic[];
  postParse?: ToolCallHeuristic[];
};

export function morphXmlProtocol(opts?: { heuristics?: ToolCallHeuristic[]; pipeline?: PipelineConfig })

Migration Plan

1. Introduce the engine and move current logic into heuristics with identical defaults.
2. Integrate into parseGeneratedText and streaming end-of-call path.
3. Add unit tests per heuristic+phase and E2E regressions.
4. Document configuration and gating.

Acceptance Criteria

• All existing tests pass unchanged.
• New unit tests cover the engine and each default heuristic.
• Defaults preserve behavior (e.g., non-shell duplicate string tags still fall back to text).
• Streaming/non-stream both use the same pipeline successfully.

Risks & Mitigations

• Complexity: keep default pipeline small and well documented.
• Performance: limit reparses to 1 per call; heuristic gating by schema/tool.
• Backward compatibility: defaults mirror current behavior; opt-in for broader heuristics.

Open Questions

• Naming/placement of the engine API (protocols/ vs internal/)
• Default enabled heuristics beyond shell-like duplicate string tags?
• Configuration surface: protocol-level vs global registry.

Tasks

• Introduce heuristic engine and types
• Port normalizeCloseTags and balanceTags to pre-parse
• Port shell-only duplicate string dedupe to fallback-reparse
• Port repairParsedAgainstSchema and coercions to post-parse
• Expose morphXmlProtocol({ heuristics?, pipeline? })
• Add unit tests per heuristic and streaming/non-stream integration tests
• Documentation: architecture and examples</issue_description>

<agent_instructions>Make sure both pnpm check and pnpm test pass.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes RFC: Pluggable Heuristic Pipeline for XML Tool-Call Parsing #108

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 78f705e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR