feature: engine extension #13718
Open
feature: engine extension #13718
+13,179
−1,645
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Begins the major architectural refactoring to enable external execution engines. Core Architecture: - Split ExecutionEngine into Discovery/Instance pattern - Introduce _discovery flag for new pattern - Create EngineProjectContext as limited ProjectContext subset - Enable loading engines from external URLs - Rename LaunchedExecutionEngine → ExecutionEngineInstance Quarto API Foundation: - Create @quarto/types package with TypeScript definitions - Start implementing QuartoAPI interface - Begin building type system for external engines Co-Authored-By: Claude <[email protected]>
Ports Julia engine to the new ExecutionEngineDiscovery/Instance pattern while continuing to build out the Quarto API. Julia Engine: - Port julia to ExecutionEngineDiscovery/Instance interfaces - Explore moving julia out of core (attempted, reverted, restored with override) - Lay groundwork for julia as bundled extension Quarto API Expansion: - Add quarto.jupyter namespace (capabilities, daemon management) - Add quarto.path.* APIs (resource, runtime, dataDir) - Add quarto.format utilities and complete Format type - Add quarto.system APIs for process execution - Add mapped string utilities - Bundle types package for distribution - Pass quarto API to init(), claimsFile, populateCommand - Rename asLaunchedEngine → asEngineInstance throughout Co-Authored-By: Claude <[email protected]>
Completes porting of core engines to ExecutionEngineDiscovery pattern and finishes core Quarto API functionality. Jupyter Engine Port: - Port jupyter to ExecutionEngineDiscovery interface (7-part series) - Update kernel management and cell execution for new pattern - Fix uses of legacy ExecutionEngine in jupyter code Knitr Engine Port: - Port knitr to ExecutionEngineDiscovery interface - Update R Markdown engine to work with new pattern API Completion: - Add system.onCleanup and path.dataDir to API - Move _discovery flag to engine implementations - Reorganize types by function for better structure - Clean up type exports and build configuration - Remove quarto.markdown, relocate asYamlText and breakQuartoMd - Port percent.ts to use quarto API - Use mapped types for FormatIdentifier Co-Authored-By: Claude <[email protected]>
Completes the architecture migration by cleaning up legacy code and making the check command extensible for external engines. Julia as Bundled Extension: - Use ported julia engine from extension (not core) - Julia now serves as reference implementation for external engines Legacy Code Removal: - Remove legacy ExecutionEngine adapter pattern - Remove deprecated ExecutionEngine interface and _discovery flags - Add quartoRequired version checking for execution engines Check Command Extensibility: - Make quarto check dynamically extensible via ExecutionEngineDiscovery - Move check logic into individual engine implementations - Load external engines in cmd.ts before validation - Add quarto.system.checkRender() API - Fix issue where check was discovering CWD as project API Polish: - Fix jupyter API message function signatures in quarto-types - Add console namespace to Quarto API (spinner, output utilities) - Add inputFilesDir and additional Jupyter methods Co-Authored-By: Claude <[email protected]>
Implements tooling and infrastructure for managing engine extensions as git subtrees, enabling them to be maintained in separate repositories while bundling them with Quarto releases. Git Subtree Infrastructure: - Add 'quarto dev-call pull-git-subtree' command to update subtree extensions from their source repositories - Support git subtree directory structure in bundled extensions - Fix pull-git-subtree to search current branch only with debug logging - Remove re-authoring logic from pull-git-subtree Julia Bundled Extension: - Move julia engine to bundled extensions (first bundled engine) - Update tests to account for julia-engine bundled subtree extension Build Fix: - Fix quarto-bld missing V8 experimental regexp engine flag Co-Authored-By: Claude <[email protected]>
Final touches to the engine architecture and improvements to Julia engine testing. Architecture Polish: - Fix Windows dynamic import by converting paths to file:// URLs - Make engineCommand dynamic to support external engines - Correct relative path handling - Refactor project initialization and fix julia test cleanup Julia Testing: - Refactor julia tests to use proper Deno test steps - Improve test structure and cleanup Co-Authored-By: Claude <[email protected]>
Single-file renders now initialize a ProjectContext with config, enabling
both engine and metadata extensions to work without requiring _quarto.yml.
Key changes:
- Single-file ProjectContext now has config = { project: {} } instead of
undefined, making it consistent with multi-file projects
- Pass full RenderOptions to singleFileProjectContext for extension services
- Export mergeExtensionMetadata() and resolveEngineExtensions() from
project-context.ts for use in single-file path
- Call both functions in singleFileProjectContext() following same order
as regular projects
- Use context.dir consistently for extension discovery
Extension output-dir handling:
- When extensions contribute output-dir metadata, set forceClean to match
the behavior of --output-dir flag, ensuring proper cleanup of the
temporary .quarto directory (addresses #9745, #13625)
- Warn users when extension contributes output-dir, showing the path and
whether cleanup will occur (respects --no-clean flag)
Co-Authored-By: Claude <[email protected]>
Completes the external engine ecosystem with user-facing tools and organizational improvements. Git Subtree Organization: - Separate git subtree extensions into extension-subtrees/ directory to distinguish from other bundled extensions - Update subtree extension directory handling and tests - Remove bundled subtree extensions from input to pandoc to avoid duplicate processing Engine Template: - Add 'engine' extension template for quarto create - Include Discovery/Instance pattern boilerplate - Fix hardcoded "example" references in engine template - Improve engine template and error handling - Add cell language prompt for engine configuration - Add engine extension type to basic creation tests Metadata Display: - Filter bundled engines from metadata display output - Suppress expected/noisy engine metadata while preserving user's custom engines Co-Authored-By: Claude <[email protected]>
this was a red herring in my investigation, but this is the correct thing to do will squash into the proper commit later.
that reaches out of quarto-cli this caused quarto-latexmk not to build correctly
yet another instance of #9737
this was causing dependencies to blow up where quarto-latexmk was trying to load all execution engines
Co-Authored-By: Claude <[email protected]>
Implements MVP of `quarto dev-call build-ts-extension` command that: - Type-checks TypeScript extensions against Quarto API types - Bundles extensions into single JavaScript files using esbuild - Provides default configuration for extensions Key features: - Auto-detects entry point (config, single file, or mod.ts) - Config resolution (user deno.json or default from share/extension-build/) - Type checking with Deno's bundled binary - Bundling with esbuild's bundled binary - --check flag for type-check only Distribution setup: - Static config templates in src/resources/extension-build/ - Copied to share/extension-build/ during packaging - Works in both dev mode and distribution mode Test extension: - Minimal test in tests/smoke/build-ts-extension/ - Implements ExecutionEngineDiscovery interface - Demonstrates type-checking and bundling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…odes Problem: import-map.json pointed to ./quarto-types.d.ts which doesn't exist in dev mode (src/resources/extension-build/). Solution: Follow Lua filter transformation pattern: - Source file in src/resources/ uses dev-mode path - Transform during packaging for distribution Changes: - import-map.json now points to ../../../packages/quarto-types/dist/index.d.ts (works in dev mode from src/resources/extension-build/) - Added updateImportMap() function in prepare-dist.ts that: - Runs after supportingFiles() copies all resources - Transforms @quarto/types path to ./quarto-types.d.ts for distribution - Updates all Deno std versions from src/import_map.json - Loops over all imports dynamically (not hardcoded list) - Updated README.md to document dev vs distribution behavior Result: Default config now works in dev mode without committing generated artifacts to src/resources/. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Implements --init-config flag that generates a minimal deno.json with
an absolute importMap path pointing to Quarto's bundled configuration.
Features:
- Checks if deno.json already exists and errors with helpful message
- Creates minimal config with compilerOptions and absolute importMap path
- Uses resourcePath() to get correct path in dev and distribution modes
- Exits without building (config generation only)
- Shows helpful customization guidance for users
Example output:
✓ Created deno.json
Import map: /usr/local/share/quarto/extension-build/import-map.json
Customize as needed:
- Add "quartoExtension" section for build options
- Modify "compilerOptions" for type-checking behavior
Generated deno.json points to Quarto's shared import-map.json which
provides @quarto/types and Deno std library imports with correct versions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Remove vendored quarto-types approach and restructure engine template to match the build-ts-extension specification. Changes: - Remove backward compatibility quarto-types copy from prepare-dist.ts - Remove vendoring logic from artifacts/extension.ts - Move TypeScript source from _extensions/ to src/ in engine template - Update _extension.yml to reference .js output instead of .ts - Update imports to use @quarto/types via import map - Add build instructions to README - Fix deno.json to include "deno.ns" in lib array (both default and --init-config) Engine extensions now follow the spec: - Source in src/ - Build to _extensions/<name>/<name>.js using build-ts-extension - Reference types via import map (no vendored copies) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Replace manual directory scanning with expandGlobSync to find _extension.yml
files, properly supporting nested organization structures like
_extensions/org-name/extension/_extension.yml.
Changes:
- Use expandGlobSync("_extensions/**/_extension.yml") pattern
- Support both flat and nested extension directory structures
- Simplify logic by removing manual directory iteration
- Improve error messages with cleaner relative path display
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Remove Deno std entries from import map since esbuild cannot bundle external URL schemes (jsr:, npm:, https:). Add externals support for marking imports as external (not bundled). Changes to quarto-cli: - Simplify import-map.json: only @quarto/types remains - Simplify updateImportMap(): just path transformation, no version syncing - Add externals?: string[] to quartoExtension interface - Implement --external flag handling in bundle function - Add default externals to deno.json: ["jsr:*", "npm:*", "https:*", "http:*"] Rationale: - esbuild binary only bundles relative file paths - External URL schemes must be resolved at runtime by Deno - Import map now serves single purpose: @quarto/types aliasing - Extension authors use full JSR URLs in source code - Default externals cover all non-bundleable schemes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Switch from esbuild to deno bundle for better import map support and
simpler tooling. deno bundle natively handles JSR/npm/https imports
and does type-checking + bundling in one step.
Changes:
- Replace esbuild with deno bundle in bundle() function
- Remove typeCheck() function (deno bundle does both)
- Restore import map entries for Deno std libraries
- Restore updateImportMap() version syncing in prepare-dist.ts
- Remove externals from quartoExtension interface (not needed)
- Add validateExtensionYml() to warn about path mismatches
- Simplify main action: --check uses deno check, normal build uses deno bundle
Benefits:
- Import maps work natively (no external URL workarounds)
- Extension authors can use bare imports ("path" vs "jsr:@std/[email protected]")
- One tool instead of two (simpler, faster)
- Native support for jsr:, npm:, https: imports
- Version consistency with Quarto maintained via import map
- ~60 lines of code removed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Adds support for specifying the entry point as a command-line argument: quarto dev-call build-ts-extension [entry-point] Entry point resolution priority (highest to lowest): 1. [entry-point] command-line argument 2. quartoExtension.entryPoint in deno.json 3. Single .ts file in src/ directory 4. src/mod.ts if multiple files exist This allows building extensions with multiple TypeScript files without requiring a deno.json just to specify the entry point. Updated error messages to suggest both CLI argument and deno.json options. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Renamed the deno.json configuration section from "quartoExtension" to
"bundle" for better semantic alignment with the underlying deno bundle
command.
Mapping to deno bundle CLI options:
- bundle.entryPoint → [file] positional argument
- bundle.outputFile → --output <output> (✅)
- bundle.minify → --minify (✅)
- bundle.sourcemap → --sourcemap[=<sourcemap>] (✅)
The "bundle" name is more intuitive and accurately reflects that these
are bundling configuration options that map to deno bundle CLI flags.
Updated all references:
- Interface definition: DenoConfig.bundle
- All config.bundle references throughout
- Documentation strings
- Error messages with example JSON
Example deno.json:
{
"bundle": {
"entryPoint": "src/main.ts",
"outputFile": "_extensions/my-ext/main.js",
"minify": true,
"sourcemap": "linked"
}
}
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Refactored output path resolution to support three modes:
1. Just filename (no path separators):
{ "bundle": { "outputFile": "julia-engine.js" } }
→ Infers directory from _extension.yml location
→ Result: _extensions/julia-engine/julia-engine.js
2. Full path (has path separators):
{ "bundle": { "outputFile": "_extensions/custom/out.js" } }
→ Uses path as-is
→ Result: _extensions/custom/out.js
3. Omitted:
{ "bundle": {} }
→ Infers both filename (from entry point) and directory
→ Result: _extensions/julia-engine/julia-engine.js
Implementation changes:
- Split into two focused functions:
- inferFilename(entryPoint): derives "foo.js" from "src/foo.ts"
- inferOutputPath(outputFilename): finds _extension.yml and joins with filename
- Updated bundle() to detect filename-only vs full-path
- Each function has single responsibility, no coupling
This provides flexibility while maintaining convention over configuration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
…rces/extension-subtrees/julia-engine'
…m commit 2fb2adf6e git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 2fb2adf6ef7b24272ff76a54ee0b51b5f6962a66
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
from
ecb4468 to
9aedc82
Compare
Moves the build-ts-extension command from the hidden dev-call namespace to the public call command namespace, making it accessible to users. Changes: - Moved src/command/dev-call/build-ts-extension/ to src/command/call/build-ts-extension/ - Removed .hidden() flag from command definition - Updated error messages to reference 'quarto call build-ts-extension' - Added command registration to src/command/call/cmd.ts - Removed command registration from src/command/dev-call/cmd.ts The command is now available as: quarto call build-ts-extension [entry-point] 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…engine-extension-3
…m 2fb2adf6e..db03cb0c5 db03cb0c5 add checkInstallation stub git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: db03cb0c58661064db5bd6c1c083b828c8276749
…e context Fixes the issue where `quarto check julia` was failing with "Invalid value 'julia'" even though Julia is available as a bundled engine extension. The problem: When running commands like `quarto check julia` outside a project directory, no project context exists, so bundled engine extensions were never loaded. The solution: Create a zeroFileProjectContext() that provides just enough structure to discover and register bundled engine extensions when no actual project or file exists. The function initializeProjectContextAndEngines() now falls back to this zero-file context when no project is found. Also changed initializeProjectContextAndEngines() to return Promise<void> instead of Promise<ProjectContext | undefined>, since the name suggests initialization rather than retrieval and both callers ignore the return value. Now `quarto check julia` works both inside and outside project directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Make error messages more helpful by: - Only showing deno.json configuration when deno.json already exists - This discourages users from creating deno.json when they don't need it - Use relative paths in examples instead of absolute paths - Order solutions from simplest to most complex (CLI arg, then rename, then config) Changes: - "No src/ directory" error: Show CLI argument first, only show deno.json config if it exists - "Multiple .ts files" error: Show CLI argument first, then rename to mod.ts, then deno.json (if exists) - "Multiple extensions" error: If no deno.json, explain this isn't supported and suggest --init-config; if deno.json exists, show how to configure outputFile - Use relative paths in outputFile example (strip absolute path prefix) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- llm-docs/quarto-api.md: Step-by-step guide for adding functionality to the Quarto API (type definitions in @quarto/types and implementation in quarto-api.ts), including usage patterns for internal and external engines - llm-docs/error-messages.md: Style guide for writing idiomatic error messages with proper newline placement (end messages with \n, never start with \n or use empty calls)
Engine extensions now require quarto call build-ts-extension to bundle TypeScript before rendering. Updated test to build before attempting to render the created extension.
When a test fails on Windows, exit immediately instead of continuing through remaining tests in the bucket. This matches Linux behavior (bash -e flag) and keeps test failures at the end of the log where they're easy to find, instead of buried under 1000+ lines of subsequent test output.
also, load builtin engine extensions
…m db03cb0c5..7eaf7bc3d 7eaf7bc3d claude: use quarto api for logging git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 7eaf7bc3d77aa439805baa9d59df96633825fa48
…engine-extension-3
…m 7eaf7bc3d..9bb6ee655 9bb6ee655 artifact git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 9bb6ee655388780db187cb48bdd8e34aed89148d
…engine-extension-3
…prove error messages Check configEntryPoint before validating src/ directory existence, allowing non-extension projects to specify custom entry points. Clarify error message when outputFile lacks a path by suggesting ./ prefix as an alternative to creating _extensions structure. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Add test that verifies build-ts-extension correctly bundles TypeScript engine extensions. Test validates that imports from the import map (like @std/path) are properly bundled into the output JavaScript file. Remove deno.json from test to verify auto-detection of entry point from src/ directory works correctly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Wrap log.info/warning/error in arrow functions instead of direct references in quartoAPI object. This prevents the bundler from creating top-level async initialization code.
When functions are assigned directly at module level (e.g., `{ info, error }`), the bundler assumes they might be accessed immediately and inserts top-level await calls to initialize underlying Deno extensions. Wrapping them in arrow functions defers access until the functions are actually called, avoiding the top-level await issue.
Fixes prepare-dist workflow error:
await init_quarto_api();
^
SyntaxError: Unexpected reserved word
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Contributor
Author
|
I think this PR has passed through bundler hell and could be ready to merge. It required some dynamic imports due to new dependencies from check and command to render modules using module level await, triggering esbuild bundler bugs in test-bundle Then those fixes caused spurious eslint variables not found, disabled in validate-bundle. So, yeah! I’m not excited about merging it with problems I fundamentally don’t understand. I’m going to sleep on it. |
Add a --dev flag to `quarto run` that uses the main development import map (src/import_map.json) instead of the user-facing run_import_map.json. This allows internal development tools (like import-report scripts) to access the full quarto development environment including deno_ral and other internal modules. Not visible for end users; description is for documentation purposes only. Usage: quarto run --dev script.ts [args...] The --dev flag should be used for internal quarto development tools that import from src/deno_ral or src/core. User-facing scripts should continue to use the standard `quarto run` without --dev. Co-Authored-By: Claude <[email protected]>
…ules If you have a dependency cycle where all of the modules are async (transitively because of their dependencies) then it would probably deadlock if you ran it. esbuild gives up and emits invalid JS. So you must either break these cycles, or prevent the async from propagating to the cycles. This tool uses ILP to show - Minimal places to break remove static imports to stop transitive dependencies before they hit cycles (minimal hitting set) - Minimal places to remove static imports to break these cycles (minimum feedback arc set) Co-Authored-By: Claude <[email protected]>
Fix async bundler issues by combining MFAS-recommended dynamic imports with architectural refactoring to eliminate circular dependency. ## Changes ### 1. Dynamic Import Fixes (MFAS recommendations) Added dynamic imports to break async propagation through cycles: - src/command/check/check-render.ts * Made render-shared.ts import dynamic in checkRender() * Prevents async from propagating into render modules ### 2. Architectural Refactoring Eliminated circular dependency between engine.ts and command-utils.ts: **Created:** src/command/call/engine-cmd.ts - Moved engineCommand definition from engine.ts - Imports core functions: executionEngine(), executionEngines() - Imports initializeProjectContextAndEngines from command-utils.ts - Follows existing command pattern (check/cmd.ts, render/cmd.ts) **Modified:** src/execute/engine.ts - Removed engineCommand export (40 lines) - Removed import of initializeProjectContextAndEngines - Now contains only core engine logic **Modified:** src/command/call/cmd.ts - Updated import to use new ./engine-cmd.ts **Modified:** src/command/command-utils.ts - Removed dynamic import workaround for reorderEngines - Restored clean static import from engine.ts ## Impact The 2-cycle between engine.ts ↔ command-utils.ts is eliminated: - engine.ts no longer imports from command-utils.ts - command-utils.ts cleanly imports from engine.ts - engineCommand now lives in proper command file ## Result - Eliminates circular dependency completely - Only 2 dynamic imports remain (essential async propagation fixes) - Better architecture: CLI separated from core engine logic - Follows existing codebase patterns - Build succeeds without async bundling errors - Net reduction: 38 lines of code 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Breaks the async propagation chain from hash.ts by converting the static import of withCriClient to a dynamic import. This prevents esbuild from marking core/handlers/base.ts as async, which was causing it to propagate async to 14 cyclic files, generating invalid bundle code. The ILP optimizer identified this single import as the optimal break point affecting all 14 async propagation chains. Changes: - Remove static import of withCriClient from core/cri/cri.ts - Add getCriClient() helper function with dynamic import - Update extractHtml() to use getCriClient() - Update createPngsFromHtml() to use getCriClient() Result: Bundle builds successfully with no async cycle errors. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Move executeResultIncludes and executeResultEngineDependencies from src/execute/jupyter/jupyter.ts to src/core/jupyter/jupyter.ts. These functions are shared utilities used by multiple engines, not engine-specific logic. Moving them to core breaks a circular dependency: - src/core/quarto-api.ts was importing from src/execute/jupyter/jupyter.ts - src/execute/jupyter/jupyter.ts was importing quartoAPI - This created a cycle: core → execute → core With this change, the dependency flow is now one-way: - Engines (execute/) import from quartoAPI (core/) - quartoAPI imports from core utilities (core/jupyter/) - Core utilities have no dependency on engines This is a prerequisite for implementing true dependency inversion via the registry pattern, which will eliminate remaining circular dependencies. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
from
14c5b2d to
0e273bd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
replaces #13701
Working branch for CI while I work out dependency and build issues.