✨ Consolidate MLIR CI to LLVM 21 and align with upstream cadence

[burgholzer]

Problem Statement

Maintaining CI across multiple LLVM versions (19/20) increases build time and maintenance while providing little practical value for plugin interoperability. Recent work on the Catalyst plugin (#881) and CUDA-Q exploration (#924) showed that cross-framework exchange via binary-compatible plugins is brittle and not feasible at scale. As MLIR development is not tied to LLVM releases, tracking multiple released LLVM versions does not reflect upstream practice and slows iteration. With a shared exchange format on top of IRs under consideration (e.g., Jeff: https://github.com/unitaryfoundation/jeff), broad binary compatibility across LLVM versions becomes less relevant. CI performance also suffers without proper caching (#1124).

Proposed Solution

• Switch CI to a single, latest LLVM release (LLVM 21) and drop 19/20 from the matrix.
• Document a clear support policy: supported LLVM == latest release; optional periodic job against llvm-project main to detect breakage early.
• Keep 🚀 MLIR - Provide Pre-Built MLIR/LLVM Binaries for CI/CD #1124 for build caching improvements and moving to more recent versions
• Update build scripts, Dockerfiles, and docs to default to LLVM 21; provide a variable to override locally if needed.
• Add a separate issue to evaluate a shared exchange format (Jeff) for cross-framework compiler pass exchange.
• Benefits:
  • Faster CI and simpler maintenance.
  • Quicker adoption of MLIR changes aligned with upstream.
  • Reduced brittleness from binary plugin dependencies.
  • Clearer support policy for users.