Build C test cases via Python script to target-specific dir #112

wingo · 2025-08-15T09:44:54Z

We move all the C sources from tests/c/testsuite to tests/c/src, somewhat like the Rust tests, then treat tests/c/testsuite as a built directory.

The build script is rewritten to Python, as it is getting a bit more complicated.

Tests are compiled into target-specific dirs, currently only tests/c/testsuite/wasm32-wasip1.

The build script writes the version-specific manifest.json, so as to include the target in the test suite name.

Workflows updated appropriately.

Stacked on #110.

pchickey

LGTM. Since I don't have 100% of the context on this, I think we should wait a few days for @loganek to be able to take a look as well

loganek · 2025-08-18T20:23:21Z

Hi, sorry I was on holidays. I'll follow-up on this, and other related topics in the next few days.

loganek · 2025-08-19T10:31:41Z

tests/c/src/fopen-with-no-access.c

@@ -7,7 +7,7 @@ int main() {
  FILE *file = fopen("fs-tests.dir/file", "r");

  assert(file == NULL);
-  assert(errno == ENOTCAPABLE);
+  assert(errno == ENOTCAPABLE || errno == ENOENT);


is it expected in wasip2 to return ENOENT when file is not accessible?

Yes, in wasip2 and beyond we got rid of the rights system from p1, so ENOTCAPABLE is no longer an applicable errno. It shouldn't be used at all in wasip2 and beyond.

since those tests are also being used for p1 too, I'd suggest to either have a copy of that in the wasi version specific directory, or use a compiler's define for different WASI versions and conditionally turn this assert off or on. For this specific test my preference would be the latter given the test is almost compatible with all the WASI versions.

This change is no longer part of this PR, fwiw, as it was merged in #110.

should this be reverted and done correctly then given it's weakening the assertion for p1?

I'm merging this PR as the change is not related, but please follow-up on this particular change.

As in WebAssembly#112, we reorganize the directory structure and change to a python build script, with the goal of making space for wasip3. Before, it was: - tests/rust/: Root of cargo package - tests/rust/src: Rust source files - tests/rust/testsuite: JSON test files, fs-tests.dir; built .wasm files dumped here Now it is: - tests/rust/wasm32-wasip1: Root of cargo package for wasip1 builds - tests/rust/wasm32-wasip1/src: Rust source, but also JSON and fs-tests.dir - tests/rust/testsuite/: Removed; instead it is a built dir - tests/rust/testsuite/wasm32-wasip1: All wasm and json files copied here

We move all the C sources from tests/c/testsuite to tests/c/src, somewhat like the Rust tests, then treat tests/c/testsuite as a built directory. The build script is rewritten to Python, as it is getting a bit more complicated. Tests are compiled into target-specific dirs, currently only tests/c/testsuite/wasm32-wasip1. The build script writes the version-specific manifest.json, so as to include the target in the test suite name. Workflows updated appropriately.

wingo · 2025-08-21T09:12:30Z

Rebased and simplified build.py.

As in WebAssembly#112, we reorganize the directory structure and change to a python build script, with the goal of making space for wasip3. Before, it was: - tests/assemblyscript/: Root of npm repo - tests/assemblyscript/testsuite: AS source files, JSON test files; built .wasm files dumped here Now it is: - tests/assemblyscript/wasm32-wasip1: Root of npm package for wasip1 builds - tests/assemblyscript/wasm32-wasip1/src: Assemblyscript source, but also JSON - tests/assemblyscript/testsuite/: Removed; instead it is a built dir - tests/assemblyscript/testsuite/wasm32-wasip1: All wasm and json files copied here

As in WebAssembly#112, we reorganize the directory structure and change to a python build script, with the goal of harmonizing the various test suites, so that testsuite/ is always a built dir. Before, it was: - tests/assemblyscript/: Root of npm repo - tests/assemblyscript/testsuite: AS source files, JSON test files; built .wasm files dumped here Now it is: - tests/assemblyscript/wasm32-wasip1: Root of npm package for wasip1 builds - tests/assemblyscript/wasm32-wasip1/src: Assemblyscript source, but also JSON - tests/assemblyscript/testsuite/: Removed; instead it is a built dir - tests/assemblyscript/testsuite/wasm32-wasip1: All wasm and json files copied here

As in WebAssembly#112, we reorganize the directory structure and change to a python build script, with the goal of making space for wasip3. Before, it was: - tests/rust/: Root of cargo package - tests/rust/src: Rust source files - tests/rust/testsuite: JSON test files, fs-tests.dir; built .wasm files dumped here Now it is: - tests/rust/wasm32-wasip1: Root of cargo package for wasip1 builds - tests/rust/wasm32-wasip1/src: Rust source, but also JSON and fs-tests.dir - tests/rust/testsuite/: Removed; instead it is a built dir - tests/rust/testsuite/wasm32-wasip1: All wasm and json files copied here

As in WebAssembly#112, we reorganize the directory structure and change to a python build script, with the goal of harmonizing the various test suites, so that testsuite/ is always a built dir. Before, it was: - tests/assemblyscript/: Root of npm repo - tests/assemblyscript/testsuite: AS source files, JSON test files; built .wasm files dumped here Now it is: - tests/assemblyscript/wasm32-wasip1: Root of npm package for wasip1 builds - tests/assemblyscript/wasm32-wasip1/src: Assemblyscript source, but also JSON - tests/assemblyscript/testsuite/: Removed; instead it is a built dir - tests/assemblyscript/testsuite/wasm32-wasip1: All wasm and json files copied here

loganek · 2025-08-21T13:36:45Z

tests/c/build.py

@@ -0,0 +1,104 @@
+#!/usr/bin/env python3


just looking at the https://github.com/WebAssembly/wasi-testsuite/pull/114/files and it seems a lot of duplication, I wonder if we can refactor it in a way that the common part can be reused across different languages?

I think yes, but wdyt about me doing that in a followup? It's a bit easier when the three implementations are actually there in the source tree.

That's fine by me. Could you open a tracking issue so we don't forget about this? Thanks

loganek · 2025-08-22T10:44:01Z

.gitignore

+.DS_Store
+/tests/c/testsuite/


perhaps it's best to use relative path here?

As in WebAssembly#112, we reorganize the directory structure and change to a python build script, with the goal of making space for wasip3. Before, it was: - tests/rust/: Root of cargo package - tests/rust/src: Rust source files - tests/rust/testsuite: JSON test files, fs-tests.dir; built .wasm files dumped here Now it is: - tests/rust/wasm32-wasip1: Root of cargo package for wasip1 builds - tests/rust/wasm32-wasip1/src: Rust source, but also JSON and fs-tests.dir - tests/rust/testsuite/: Removed; instead it is a built dir - tests/rust/testsuite/wasm32-wasip1: All wasm and json files copied here

As in #112, we reorganize the directory structure and change to a python build script, with the goal of harmonizing the various test suites, so that testsuite/ is always a built dir. Before, it was: - tests/assemblyscript/: Root of npm repo - tests/assemblyscript/testsuite: AS source files, JSON test files; built .wasm files dumped here Now it is: - tests/assemblyscript/wasm32-wasip1: Root of npm package for wasip1 builds - tests/assemblyscript/wasm32-wasip1/src: Assemblyscript source, but also JSON - tests/assemblyscript/testsuite/: Removed; instead it is a built dir - tests/assemblyscript/testsuite/wasm32-wasip1: All wasm and json files copied here

As in WebAssembly#112, we reorganize the directory structure and change to a python build script, with the goal of making space for wasip3. Before, it was: - tests/rust/: Root of cargo package - tests/rust/src: Rust source files - tests/rust/testsuite: JSON test files, fs-tests.dir; built .wasm files dumped here Now it is: - tests/rust/wasm32-wasip1: Root of cargo package for wasip1 builds - tests/rust/wasm32-wasip1/src: Rust source, but also JSON and fs-tests.dir - tests/rust/testsuite/: Removed; instead it is a built dir - tests/rust/testsuite/wasm32-wasip1: All wasm and json files copied here

As in #112, we reorganize the directory structure and change to a python build script, with the goal of making space for wasip3. Before, it was: - tests/rust/: Root of cargo package - tests/rust/src: Rust source files - tests/rust/testsuite: JSON test files, fs-tests.dir; built .wasm files dumped here Now it is: - tests/rust/wasm32-wasip1: Root of cargo package for wasip1 builds - tests/rust/wasm32-wasip1/src: Rust source, but also JSON and fs-tests.dir - tests/rust/testsuite/: Removed; instead it is a built dir - tests/rust/testsuite/wasm32-wasip1: All wasm and json files copied here

wingo force-pushed the c-multi-target branch from 1d6b134 to 651dcc3 Compare August 15, 2025 09:49

wingo marked this pull request as ready for review August 15, 2025 09:53

pchickey approved these changes Aug 15, 2025

View reviewed changes

pchickey requested a review from loganek August 15, 2025 17:40

loganek reviewed Aug 19, 2025

View reviewed changes

wingo mentioned this pull request Aug 20, 2025

Update wasi-testsuite to support testing newer versions of WASI #109

Open

wingo mentioned this pull request Aug 21, 2025

Per-target rust testsuite #114

Merged

wingo force-pushed the c-multi-target branch from 651dcc3 to 7db37c0 Compare August 21, 2025 09:11

wingo mentioned this pull request Aug 21, 2025

Per-target Assemblyscript testsuite #115

Merged

loganek reviewed Aug 21, 2025

View reviewed changes

wingo mentioned this pull request Aug 21, 2025

upload_test_binaries clears testsuite before downloading artifact #116

Merged

loganek reviewed Aug 22, 2025

View reviewed changes

.gitignore

.DS_Store

/tests/c/testsuite/

Copy link

Collaborator

loganek Aug 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps it's best to use relative path here?

loganek merged commit b1978f0 into WebAssembly:main Aug 26, 2025
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build C test cases via Python script to target-specific dir #112

Build C test cases via Python script to target-specific dir #112

wingo commented Aug 15, 2025

Uh oh!

pchickey left a comment

Uh oh!

loganek commented Aug 18, 2025

Uh oh!

loganek Aug 19, 2025

Uh oh!

pchickey Aug 19, 2025

Uh oh!

loganek Aug 26, 2025

Uh oh!

wingo Aug 26, 2025

Uh oh!

loganek Aug 26, 2025 •

edited

Loading

Uh oh!

loganek Aug 26, 2025

Uh oh!

wingo commented Aug 21, 2025

Uh oh!

loganek Aug 21, 2025

Uh oh!

wingo Aug 21, 2025

Uh oh!

loganek Aug 22, 2025

Uh oh!

loganek Aug 22, 2025

Uh oh!

Uh oh!

Uh oh!

Build C test cases via Python script to target-specific dir #112

Build C test cases via Python script to target-specific dir #112

Conversation

wingo commented Aug 15, 2025

Uh oh!

pchickey left a comment

Choose a reason for hiding this comment

Uh oh!

loganek commented Aug 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

loganek Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wingo commented Aug 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

loganek Aug 26, 2025 •

edited

Loading