Detect individual kselftest build failures

[musamaanjum]

If build of one or more kselftests gets failed, the remaining kselftests are built without any problem. These errors aren't reported on KernelCI in any form. These errors can only be found out in build logs. Lets take an example, build of kselftest-futex gets failed every time it is built. This isn't reported on the dashboard until someone opens the raw kselftest build logs. Later on, kselftest try to execute futex tests, but it gets failed and gives the following errors:

14:50:43.367847  # selftests: futex: run.sh
14:50:43.638098  # ./run.sh: 29: cd: can't cd to functional
14:50:43.642534  # ./run.sh: 29: cd: can't cd to functional

The errors at run time don't give much information that why the execution has failed. Build errors should have been got caught in the build phase to avoid confusion later.

[gctucker]

This is because kselftest builds will always complete without returning any error status code except if all the kselftests failed. To detect that some have failed and others haven't, we would need to parse the build output or try to build one directory at a time.

This is more of a request for a new feature than a bug, as at the moment there's no way to keep track of individual kselftest build errors.

[gctucker]

With the new KernelCI API, tests and builds could depend on each other. So we could have each kselftest collection tests depend on the corresponding kselftest build (i.e. if one directory fails to build then it will be reported as such and the collection won't be run).

[gctucker]

On the kernel side of things, by having all the kselftests to build correctly first we could then have the option to make any build failure result in a non-zero status code from the make command. We may also consider making any kselftest build error a fatal one like for modules. This is complementary to being able to submit build results for each kselftest separately.