Implement building GenMC C++ code

[Patrick-6]

This PR is in preparation for adding GenMC support to Miri.
It adds the genmc-sys crate, which can automatically load GenMC from a git repo (temporarily pointing to a private repo, but will point to the GenMC Github repo after everything is implemented)

This PR doesn't add any functionality for GenMC mode, only a test for calling C++ functions from Rust code through the cxx crate.

[RalfJung]

@rustbot author
Please mark it as ready when CI is green.

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[Patrick-6]

@rustbot ready

[RalfJung]

Here's my first round of comments. :)

@rustbot author

[Patrick-6]

Should the build script not download anything if cargo is invoked with --offline or --frozen (which causes cargo to not access the network)? I can't really find anything about how a build script should behave in such a case, or if it can even detect it. There is the CARGO_NET_OFFLINE env. var, but that appears to be only for changing cargo's behavior, it's not one of the env. vars set for build scripts..

[Patrick-6]

For some reason, adding this path to the rerun-if-changed if the path doesn't exist yet causes full rebuilds on every build. I think the non-existing directory is seen as a change by cargo.

Here is the output from running CARGO_LOG=cargo::core::compiler::fingerprint=info ./miri build --features=genmc after a complete build without any file modifications (as described here):

cargo::core::compiler::fingerprint: stale: missing "/path/to/miri/genmc-sys/./genmc/"
cargo::core::compiler::fingerprint: fingerprint dirty for miri v0.1.0 (/path/to/miri)

Is this intended behavior for cargo?

[RalfJung]

Can you just add GENMC_LOCAL_PATH only if you actually use the local path, not the download?

[Patrick-6]

Yes, but I think then we wouldn't trigger an automatic rebuild if a user adds the ./genmc directory. That's probably still better than constantly rebuilding, since this case should be rare.

[RalfJung]

Oh, you are using the existence of the directory as the signal for using it... yeah that doesn't work with cargo's model, it seems. I'll ask the cargo devs about it, but for now, let's use an env var to signal explicitly that you want a local build instead of a download, and then add that env var to rerun-if-changed.

[RalfJung]

Also see rust-lang/cargo#6003

[RalfJung]

Should the build script not download anything if cargo is invoked with --offline or --frozen (which causes cargo to not access the network)? I can't really find anything about how a build script should behave in such a case, or if it can even detect it. There is the CARGO_NET_OFFLINE env. var, but that appears to be only for changing cargo's behavior, it's not one of the env. vars set for build scripts..

Yeah I don't think we have a way to check for that.

Please ask on some RUst forum (URLO on Zulip) what the typical approach here is, I don't know.

[RalfJung]

Can you just add GENMC_LOCAL_PATH only if you actually use the local path, not the download?

[RalfJung]

which one, adding the warning?

Ah, sorry, stopped reading too early... Add the flag to disable SB to the tests.

[RalfJung]

Suggested change

	} else if genmc_local_path.exists() {
	// If the local repository exists, cargo should watch it for changes:
	// FIXME(genmc,cargo): We could always watch this path even if it doesn't (yet) exist, depending on how `https://github.com/rust-lang/cargo/issues/6003` is resolved.
	// Adding it here means we don't rebuild if a user creates `GENMC_LOCAL_PATH`, which isn't ideal.
	// Cargo currently always rebuilds if a watched directory doesn't exist, so we can only add it if it exists.
	println!("cargo::rerun-if-changed={GENMC_LOCAL_PATH}");
	genmc_local_path

The point of the env var was to control whether we download or not, wasn't it?

[Patrick-6]

I thought we would keep this so you can use the normal ./miri commands for development, without having to set the env var, by having your development copy of GenMC in genmc-sys/genmc/.

[RalfJung]

Normal ./miri doesn't turn on the feature. ./miri build --feature genmc should use the downloaded version -- that's the right default for most contributors.

[Patrick-6]

So for Miri-GenMC contributors the way to do it is GENMC_SRC_PATH="/some/path/" ./miri build --features=genmc ? (or export GENMC_SRC_PATH=...)

[RalfJung]

For the extremely rare contributor that needs a custom genmc, yeah.

It'd be nicer if you just had to create a folder, but that doesn't seem to be possible with cargo currently.

[Patrick-6]

The current implementation uses:

• path in GENMC_SRC_PATH if set
• genmc-sys/genmc if present
• genmc-sys/downloaded/genmc if present
• do the download into downloaded/genmc

[RalfJung]

But that is what we have if I don't remove that if

But it doesn't work: if you first build without the folder existing, and then you create the folder, it won't rebuild and will keep using the downloaded genmc.

[Patrick-6]

Nothing a little touch genmc-sys/build.rs doesn't fix ;)

But ok, we can remove it.

[RalfJung]

Nothing a little touch genmc-sys/build.rs doesn't fix ;)

If you prefer that, we can also do that, and carefully document it.

But we don't want that and the env var. Let's have one way and one way only for local builds.

[Patrick-6]

I've switched to only the env var + download ways, it is still reasonably usable by doing export GENMC_SRC_PATH=... in the current shell, then the path doesn't need to be passed every time.

[Patrick-6]

Apparently using the linkedProjects setting means that rust-analyzer auto-discovery is turned off, so you get no RA help at all in directories no listed here. Adding the genmc-sys lines fixes it for VSCode, but I can't test it for helix (no reason to believe it wouldn't work though).

There still another issue (maybe not for this PR): I still don't get any RA help in any Rust files in the miri/tests/ directory, but maybe that could be added too somehow? I'm unsure how this would work though, since the settings file here also overrides the check command to use ./miri clippy/check instead of cargo clippy/check, which may not work with the tests directory.

Also, this setting expects paths to either Cargo.toml, rust-project.json, or loose .rs files, and we definitely don't want to add all tests here.

[RalfJung]

Yeah RA doesn't have support for ui test suites like we're using them, unfortunately.

[Patrick-6]

We are now using the official GenMC release, not my cloned repo

[Patrick-6]

As mentioned on Zulip, the license changes are still missing, so this probably shouldn't be merged yet, but otherwise:

• From a technical point of view, everything should be in order now.
• The documentation might need to be improved/cleaned up/expanded.
• We don't yet have any formatting & linting rules (and automatic checks) for the C++ code that lives in the Miri repo (but that's a low priority for now).

@rustbot ready