doc: overhaul non-cargo build system docs

Author: cormacrelf

crates/rust-analyzer/src/config.rs

@@ -494,7 +494,7 @@ config_data! {
         /// }
         /// ```
         ///
-        /// ## On `DiscoverWorkspaceConfig::command`
+        /// ## Workspace Discovery Protocol
         ///
         /// **Warning**: This format is provisional and subject to change.
         ///
@@ -862,10 +862,18 @@ config_data! {
         /// (i.e., the folder containing the `Cargo.toml`). This can be overwritten
         /// by changing `#rust-analyzer.check.invocationStrategy#`.
         ///
-        /// If `$saved_file` is part of the command, rust-analyzer will pass
-        /// the absolute path of the saved file to the provided command. This is
-        /// intended to be used with non-Cargo build systems.
-        /// Note that `$saved_file` is experimental and may be removed in the future.
+        /// It supports two interpolation syntaxes, both mainly intended to be used with
+        /// [non-Cargo build systems](./non_cargo_based_projects.md):
+        ///
+        /// - If `{saved_file}` is part of the command, rust-analyzer will pass
+        ///   the absolute path of the saved file to the provided command.
+        ///   (A previous version, `$saved_file`, also works.)
+        /// - If `{label}` is part of the command, rust-analyzer will pass the
+        ///   Cargo package ID, which can be used with `cargo check -p`, or a build label from
+        ///   `rust-project.json`. If `{label}` is included, rust-analyzer behaves much like
+        ///   [`"rust-analyzer.check.workspace": false`](#check.workspace).
+        ///
+        ///
         ///
         /// An example command would be:
         ///

docs/book/src/configuration_generated.md

@@ -323,10 +323,18 @@ each of them, with the working directory being the workspace root
 (i.e., the folder containing the `Cargo.toml`). This can be overwritten
 by changing `#rust-analyzer.check.invocationStrategy#`.
 
-If `$saved_file` is part of the command, rust-analyzer will pass
-the absolute path of the saved file to the provided command. This is
-intended to be used with non-Cargo build systems.
-Note that `$saved_file` is experimental and may be removed in the future.
+It supports two interpolation syntaxes, both mainly intended to be used with
+[non-Cargo build systems](./non_cargo_based_projects.md):
+
+- If `{saved_file}` is part of the command, rust-analyzer will pass
+  the absolute path of the saved file to the provided command.
+  (A previous version, `$saved_file`, also works.)
+- If `{label}` is part of the command, rust-analyzer will pass the
+  Cargo package ID, which can be used with `cargo check -p`, or a build label from
+  `rust-project.json`. If `{label}` is included, rust-analyzer behaves much like
+  [`"rust-analyzer.check.workspace": false`](#check.workspace).
+
+
 
 An example command would be:
 
@@ -1542,8 +1550,8 @@ Default: `null`
 
 Enables automatic discovery of projects using [`DiscoverWorkspaceConfig::command`].
 
-[`DiscoverWorkspaceConfig`] also requires setting `progress_label` and `files_to_watch`.
-`progress_label` is used for the title in progress indicators, whereas `files_to_watch`
+[`DiscoverWorkspaceConfig`] also requires setting `progressLabel` and `filesToWatch`.
+`progressLabel` is used for the title in progress indicators, whereas `filesToWatch`
 is used to determine which build system-specific files should be watched in order to
 reload rust-analyzer.
 
@@ -1562,7 +1570,7 @@ Below is an example of a valid configuration:
 }
 ```
 
-## On `DiscoverWorkspaceConfig::command`
+## Workspace Discovery Protocol
 
 **Warning**: This format is provisional and subject to change.
 

docs/book/src/non_cargo_based_projects.md

@@ -220,7 +220,15 @@ interface Runnable {
 This format is provisional and subject to change. Specifically, the
 `roots` setup will be different eventually.
 
-There are three ways to feed `rust-project.json` to rust-analyzer:
+### Providing a JSON project to rust-analyzer
+
+There are four ways to feed `rust-project.json` to rust-analyzer:
+
+-   Use
+    [`"rust-analyzer.workspace.discoverConfig": … }`](./configuration.md#workspace.discoverConfig)
+    to specify a workspace discovery command to generate project descriptions
+    on-the-fly. Please note that the command output is message-oriented and must
+    follow [the discovery protocol](./configuration.md#workspace-discovery-protocol).
 
 -   Place `rust-project.json` file at the root of the project, and
     rust-analyzer will discover it.
@@ -240,19 +248,86 @@ location or (for inline JSON) relative to `rootUri`.
 You can set the `RA_LOG` environment variable to `rust_analyzer=info` to
 inspect how rust-analyzer handles config and project loading.
 
-Note that calls to `cargo check` are disabled when using
-`rust-project.json` by default, so compilation errors and warnings will
-no longer be sent to your LSP client. To enable these compilation errors
-you will need to specify explicitly what command rust-analyzer should
-run to perform the checks using the
-`rust-analyzer.check.overrideCommand` configuration. As an example, the
-following configuration explicitly sets `cargo check` as the `check`
-command.
-
-    { "rust-analyzer.check.overrideCommand": ["cargo", "check", "--message-format=json"] }
-
-`check.overrideCommand` requires the command specified to output json
-error messages for rust-analyzer to consume. The `--message-format=json`
-flag does this for `cargo check` so whichever command you use must also
-output errors in this format. See the [Configuration](#_configuration)
-section for more information.
+### Flycheck support
+
+Rust-analyzer has functionality to run an actual build of a crate when the user saves a file, to
+fill in diagnostics it does not implement natively. This is known as "flycheck".
+
+**Flycheck is disabled when using `rust-project.json` unless explicitly configured**, so compilation
+errors and warnings will no longer be sent to your LSP client by default. To enable these
+compilation errors you will need to specify explicitly what command rust-analyzer should run to
+perform the checks. There are two ways to do this:
+
+- `rust-project.json` may contain a `runnables` field. The `flycheck` runnable may be used to
+  configure a check command. See above for documentation.
+
+- Using the [`rust-analyzer.check.overrideCommand`](./configuration.md#check.overrideCommand)
+  configuration. This will also override anything in `rust-project.json`. As an example, the
+  following configuration explicitly sets `cargo check` as the `check` command.
+
+  ```json
+  { "rust-analyzer.check.overrideCommand": ["cargo", "check", "--message-format=json"] }
+  ```
+
+  Note also that this works with cargo projects.
+
+Either option requires the command specified to output JSON error messages for rust-analyzer to
+consume. The `--message-format=json` flag does this for `cargo check` so whichever command you use
+must also output errors in this format.
+
+Either option also supports two syntaxes within each argument:
+
+- `{label}` will be replaced with the `BuildInfo::label` of the crate
+  containing a saved file, if `BuildInfo` is provided. In the case of `check.overrideCommand` being
+  used in a Cargo project, this will be the cargo package ID, which can be used with `cargo check -p`.
+- `{saved_file}` will be replaced with an absolute path to the saved file. This can be queried against a
+  build system to find targets that include the file.
+
+For example:
+
+```json
+{ "rust-analyzer.check.overrideCommand": ["custom_crate_checker", "{label}"] }
+```
+
+If you do use `{label}` or `{saved_file}`, the command will not be run unless the relevant value can
+be substituted.
+
+
+#### Flycheck considerations
+
+##### Diagnostic output on error
+
+A flycheck command using a complex build orchestrator like `"bazel", "build", "{label}"`, even with
+a tweak to return JSON messages, is often insufficient. Such a command will typically succeed if
+there are warnings, but if there are errors, it might "fail to compile" the diagnostics and not
+produce any output. You must build a package in such a way that the build succeeds even if `rustc`
+exits with an error, and prints the JSON build messages in every case.
+
+##### Diagnostics for upstream crates
+
+`cargo check -p` re-prints any errors and warnings in crates higher up in the dependency graph
+than the one requested. We do clear all diagnostics when flychecking, so if you manage to
+replicate this behaviour, diagnostics for crates other than the one being checked will show up in
+the editor. If you do not, then users may be confused that diagnostics are "stuck" or disappear
+entirely when there is a build error in an upstream crate.
+
+##### Compiler options
+
+`cargo check` invokes rustc differently from `cargo build`. It turns off codegen (with `rustc
+--emit=metadata`), which results in lower latency to get to diagnostics. If your build system can
+configure this, it is recommended.
+
+If your build tool can configure rustc for incremental compiles, this is also recommended.
+
+##### Locking and pre-emption
+
+In any good build system, including Cargo, build commands sometimes block each other. Running a
+flycheck will (by default) frequently block you from running other build commands. Generally this is
+undesirable. Users will have to (unintuitively) press save again in the editor to cancel a
+flycheck, so that some other command may proceed.
+
+If your build system has the ability to isolate any rust-analyzer-driven flychecks and prevent lock
+contention, for example a separate build output directory and/or daemon instance, this is
+recommended. Alternatively, consider using a feature if available that can set the priority of
+various build invocations and automatically cancel lower-priority ones when needed. Flychecks should
+be set to a lower priority than general direct build invocations.

editors/code/package.json

@@ -1208,7 +1208,7 @@
                 "title": "Check",
                 "properties": {
                     "rust-analyzer.check.overrideCommand": {
-                        "markdownDescription": "Override the command rust-analyzer uses instead of `cargo check` for\ndiagnostics on save. The command is required to output json and\nshould therefore include `--message-format=json` or a similar option\n(if your client supports the `colorDiagnosticOutput` experimental\ncapability, you can use `--message-format=json-diagnostic-rendered-ansi`).\n\nIf you're changing this because you're using some tool wrapping\nCargo, you might also want to change\n`#rust-analyzer.cargo.buildScripts.overrideCommand#`.\n\nIf there are multiple linked projects/workspaces, this command is invoked for\neach of them, with the working directory being the workspace root\n(i.e., the folder containing the `Cargo.toml`). This can be overwritten\nby changing `#rust-analyzer.check.invocationStrategy#`.\n\nIf `$saved_file` is part of the command, rust-analyzer will pass\nthe absolute path of the saved file to the provided command. This is\nintended to be used with non-Cargo build systems.\nNote that `$saved_file` is experimental and may be removed in the future.\n\nAn example command would be:\n\n```bash\ncargo check --workspace --message-format=json --all-targets\n```\n\nNote: The option must be specified as an array of command line arguments, with\nthe first argument being the name of the command to run.",
+                        "markdownDescription": "Override the command rust-analyzer uses instead of `cargo check` for\ndiagnostics on save. The command is required to output json and\nshould therefore include `--message-format=json` or a similar option\n(if your client supports the `colorDiagnosticOutput` experimental\ncapability, you can use `--message-format=json-diagnostic-rendered-ansi`).\n\nIf you're changing this because you're using some tool wrapping\nCargo, you might also want to change\n`#rust-analyzer.cargo.buildScripts.overrideCommand#`.\n\nIf there are multiple linked projects/workspaces, this command is invoked for\neach of them, with the working directory being the workspace root\n(i.e., the folder containing the `Cargo.toml`). This can be overwritten\nby changing `#rust-analyzer.check.invocationStrategy#`.\n\nIt supports two interpolation syntaxes, both mainly intended to be used with\n[non-Cargo build systems](./non_cargo_based_projects.md):\n\n- If `{saved_file}` is part of the command, rust-analyzer will pass\n    the absolute path of the saved file to the provided command.\n    (A previous version, `$saved_file`, also works.)\n- If `{label}` is part of the command, rust-analyzer will pass the\n    Cargo package ID, which can be used with `cargo check -p`, or a build label from\n    `rust-project.json`. If `{label}` is included, rust-analyzer behaves much like\n    [`\"rust-analyzer.check.workspace\": false`](#check.workspace).\n\n\n\nAn example command would be:\n\n```bash\ncargo check --workspace --message-format=json --all-targets\n```\n\nNote: The option must be specified as an array of command line arguments, with\nthe first argument being the name of the command to run.",
                         "default": null,
                         "type": [
                             "null",
@@ -3032,7 +3032,7 @@
                 "title": "Workspace",
                 "properties": {
                     "rust-analyzer.workspace.discoverConfig": {
-                        "markdownDescription": "Enables automatic discovery of projects using [`DiscoverWorkspaceConfig::command`].\n\n[`DiscoverWorkspaceConfig`] also requires setting `progress_label` and `files_to_watch`.\n`progress_label` is used for the title in progress indicators, whereas `files_to_watch`\nis used to determine which build system-specific files should be watched in order to\nreload rust-analyzer.\n\nBelow is an example of a valid configuration:\n```json\n\"rust-analyzer.workspace.discoverConfig\": {\n        \"command\": [\n                \"rust-project\",\n                \"develop-json\"\n        ],\n        \"progressLabel\": \"rust-analyzer\",\n        \"filesToWatch\": [\n                \"BUCK\"\n        ]\n}\n```\n\n## On `DiscoverWorkspaceConfig::command`\n\n**Warning**: This format is provisional and subject to change.\n\n[`DiscoverWorkspaceConfig::command`] *must* return a JSON object corresponding to\n`DiscoverProjectData::Finished`:\n\n```norun\n#[derive(Debug, Clone, Deserialize, Serialize)]\n#[serde(tag = \"kind\")]\n#[serde(rename_all = \"snake_case\")]\nenum DiscoverProjectData {\n        Finished { buildfile: Utf8PathBuf, project: ProjectJsonData },\n        Error { error: String, source: Option<String> },\n        Progress { message: String },\n}\n```\n\nAs JSON, `DiscoverProjectData::Finished` is:\n\n```json\n{\n        // the internally-tagged representation of the enum.\n        \"kind\": \"finished\",\n        // the file used by a non-Cargo build system to define\n        // a package or target.\n        \"buildfile\": \"rust-analyzer/BUILD\",\n        // the contents of a rust-project.json, elided for brevity\n        \"project\": {\n                \"sysroot\": \"foo\",\n                \"crates\": []\n        }\n}\n```\n\nIt is encouraged, but not required, to use the other variants on `DiscoverProjectData`\nto provide a more polished end-user experience.\n\n`DiscoverWorkspaceConfig::command` may *optionally* include an `{arg}`, which will be\nsubstituted with the JSON-serialized form of the following enum:\n\n```norun\n#[derive(PartialEq, Clone, Debug, Serialize)]\n#[serde(rename_all = \"camelCase\")]\npub enum DiscoverArgument {\n     Path(AbsPathBuf),\n     Buildfile(AbsPathBuf),\n}\n```\n\nThe JSON representation of `DiscoverArgument::Path` is:\n\n```json\n{\n        \"path\": \"src/main.rs\"\n}\n```\n\nSimilarly, the JSON representation of `DiscoverArgument::Buildfile` is:\n\n```json\n{\n        \"buildfile\": \"BUILD\"\n}\n```\n\n`DiscoverArgument::Path` is used to find and generate a `rust-project.json`, and\ntherefore, a workspace, whereas `DiscoverArgument::buildfile` is used to to update an\nexisting workspace. As a reference for implementors, buck2's `rust-project` will likely\nbe useful: <https://github.com/facebook/buck2/tree/main/integrations/rust-project>.",
+                        "markdownDescription": "Enables automatic discovery of projects using [`DiscoverWorkspaceConfig::command`].\n\n[`DiscoverWorkspaceConfig`] also requires setting `progressLabel` and `filesToWatch`.\n`progressLabel` is used for the title in progress indicators, whereas `filesToWatch`\nis used to determine which build system-specific files should be watched in order to\nreload rust-analyzer.\n\nBelow is an example of a valid configuration:\n```json\n\"rust-analyzer.workspace.discoverConfig\": {\n        \"command\": [\n                \"rust-project\",\n                \"develop-json\",\n                \"{arg}\"\n        ],\n        \"progressLabel\": \"buck2/rust-project\",\n        \"filesToWatch\": [\n                \"BUCK\"\n        ]\n}\n```\n\n## Workspace Discovery Protocol\n\n**Warning**: This format is provisional and subject to change.\n\n[`DiscoverWorkspaceConfig::command`] *must* return a JSON object corresponding to\n`DiscoverProjectData::Finished`:\n\n```norun\n#[derive(Debug, Clone, Deserialize, Serialize)]\n#[serde(tag = \"kind\")]\n#[serde(rename_all = \"snake_case\")]\nenum DiscoverProjectData {\n        Finished { buildfile: Utf8PathBuf, project: ProjectJsonData },\n        Error { error: String, source: Option<String> },\n        Progress { message: String },\n}\n```\n\nAs JSON, `DiscoverProjectData::Finished` is:\n\n```json\n{\n        // the internally-tagged representation of the enum.\n        \"kind\": \"finished\",\n        // the file used by a non-Cargo build system to define\n        // a package or target.\n        \"buildfile\": \"rust-analyzer/BUILD\",\n        // the contents of a rust-project.json, elided for brevity\n        \"project\": {\n                \"sysroot\": \"foo\",\n                \"crates\": []\n        }\n}\n```\n\nIt is encouraged, but not required, to use the other variants on `DiscoverProjectData`\nto provide a more polished end-user experience.\n\n`DiscoverWorkspaceConfig::command` may *optionally* include an `{arg}`, which will be\nsubstituted with the JSON-serialized form of the following enum:\n\n```norun\n#[derive(PartialEq, Clone, Debug, Serialize)]\n#[serde(rename_all = \"camelCase\")]\npub enum DiscoverArgument {\n     Path(AbsPathBuf),\n     Buildfile(AbsPathBuf),\n}\n```\n\nThe JSON representation of `DiscoverArgument::Path` is:\n\n```json\n{\n        \"path\": \"src/main.rs\"\n}\n```\n\nSimilarly, the JSON representation of `DiscoverArgument::Buildfile` is:\n\n```json\n{\n        \"buildfile\": \"BUILD\"\n}\n```\n\n`DiscoverArgument::Path` is used to find and generate a `rust-project.json`, and\ntherefore, a workspace, whereas `DiscoverArgument::buildfile` is used to to update an\nexisting workspace. As a reference for implementors, buck2's `rust-project` will likely\nbe useful: <https://github.com/facebook/buck2/tree/main/integrations/rust-project>.",
                         "default": null,
                         "anyOf": [
                             {