[GSoC2025] Add Silicon Heaven protocol adaptation into NuttX and SHV examples

[zdebanos]

This commit marks the end of my GSoC 2025 project in the NuttX section.

Currently creating as a draft pull request.
I have agreed with Pavel Pisa to separate the library's dependancy named ulut from the shv-libs4c repository (located currently here https://github.com/silicon-heaven/shv-libs4c/tree/master/submodule). This will require more time for me. Also, I have discussed with Karel Koci from Elektroline a.s. about the posibilities of the exports of external libraries - something that I somehow solved in this commit: #3154.
In the end we have agreed that this should be considered as "hacking the whole Makefile system" and I should rather come up with a better solution, and thus I'm postponing it. Since I require the export for the GSoC, I'd be glad to leave it here as draft until I solve all the issues already mentioned.

I am about to add documentation into the nuttx repo, too.

Summary

Please look into the commit's verbose message :-).

Impact

Adds the Silicon Heaven protocol implementation. A framework designed to register your callbacks into your application. The callbacks are organized into nodes and the nodes can form a tree structure.

Testing

The testing was done on my custom samv7 board. In nuttx:
tools/configure.sh myboard/myconfig - configuration OK
make - downloads the shv-libs4c repo from github, and compiles OK

DISCLAIMER: in my GSoC, I was using make export. But with the solution used here (https://github.com/zdebanos/pysimCoder/tree/shv-refactor) I succesfully compiled against automatically generated source code in pysimCoder.

[zdebanos]

I came across an issue, which must be fixed: locally, I was running against pyshv==0.7.3 (implementation of Silicon Heaven API in python) but when I added requirements and changed the imports to match with pyshv==0.10.0, it stopped working, probably by using a bad function of some sort. I'll have to look deeper into that.

All i know is that the script worked with the previous version, the target device functionality has not changed.

[simbit18]

would reformat /home/runner/work/nuttx-apps/nuttx-apps/apps/examples/shv-nxboot-updater/update-script/gui.py

Oh no! 💥 💔 💥
1 file would be reformatted.

black check failed, run following command to update the style:
  $ black <src>

[zdebanos]

@simbit18 I know about that, I'm working on a different fix and I'm gonna fix the linter error too.

[zdebanos]

I also need to address the shv-libs4c correct version. I've got two options I can implement:

1. download a correct .zip from github releases
2. clone a "specific hash" commit and compile against it

I'm in favor with the second option. That means I'll have to specify the commit hash inside the Makefile.

Currently, I'm running against master as it's still draft and I'm fixing last errors. Then I'll specify the correct version.

[michallenc]

I also need to address the shv-libs4c correct version. I've got two options I can implement:

1. download a correct .zip from github releases

2. clone a "specific hash" commit and compile against it

I'm in favor with the second option. That means I'll have to specify the commit hash inside the Makefile.

Currently, I'm running against master as it's still draft and I'm fixing last errors. Then I'll specify the correct version.

I am also for the second option. The commit hash can even be a configurable option in Kconfig to easily allow the change of a version per project.

[zdebanos]

Thanks to Pavel Pisa, the Makefile system adjusts better to the build system used in the shv-libs4c library. As discussed in the heading, we have also successfully separated the ulut dependancy from the shv-libs4c library and now it's an external dependancy.

Currently, ulut is placed in neutilts/libulut. We can't think of a better place to put this library, as it is a collection of data structures (containers) written in the C language, and as it is a direct dependency of SHV, we put it parallel to libshvc. We should probably put this somewhere else but now we are keeping it here.

Also I suppose the export of the library should be done outside of this PR as a separate commit. I have found out we can use an EXTOBJS variable to handle the job better (only for .o files, not a solution to .h files).

I have also changed the apps a bit, now the mandatory arguments are position independent.

Also the pipeline is failing due to a typo. But I do not think it's a typo the pipeline should fail at since it is used in a macro in the Makefile. But I'll consult this with Pavel Pisa.

[zdebanos]

I have unmarked this commit as draft since we have agreed upon a stable solution. The only thing that needs to be done is the documention, I'm working on it right now.

[xiaoxiang781216]

should we skip the clone if user already git clone manually like other 3rd party project?

[ppisa]

I am not sure what is the best option. If the repository is already cloned then there should be mechanism how to keep it at the correct version matching given NuttX version/commit.

I have implemented next script for integration into pysimCoder

https://github.com/zdebanos/pysimCoder/blob/gsoc-2025-shv-refactor/ExtLibs/extlib-fetch.sh

It keeps the local branch, if the repository is clean it switches it to the specified library version by reset. If there is only small local change which does not clash with switch to newly required commit, it informs user and switches. If there are conflicting changes it fails. If there is another branch and state is clean it switches and it is user's task to move changes.

I am not sure what is preferred and typical for NuttX

[zdebanos]

I think the current solution is better - you get the correct version that Is supported by the library (i.e. what we think works the best). Your approach would in my opinion overcomplicate things.

[raiden00pl]

I also think we should use a similar approach as in other 3rd party projects to be consistent in the project: download the archive and unpack it.
Example here:

nuttx-apps/wireless/bluetooth/nimble/Makefile

Lines 43 to 58 in 10cab6b

	$(NIMBLE_TAR):
	$(Q) curl -L $(NIMBLE_URL) -o $(NIMBLE_TAR)
	$(NIMBLE_UNPACKDIR): $(NIMBLE_TAR)
	$(Q) tar zxf $(NIMBLE_TAR)
	$(Q) mv mynewt-nimble-$(CONFIG_NIMBLE_REF) $(NIMBLE_UNPACKDIR)
	$(Q) touch $(NIMBLE_UNPACKDIR)
	# Download and unpack tarball if no git repo found
	ifeq ($(wildcard $(NIMBLE_UNPACKDIR)/.git),)
	context:: $(NIMBLE_UNPACKDIR)
	distclean::
	$(call DELFILE,$(NIMBLE_TAR))
	$(call DELDIR,$(NIMBLE_UNPACKDIR))
	endif

By default, if the user only wants to use a 3rd party project, the code will always be downloaded from a verified hash.
But skipping the package download when a git repository exists makes the work of developers working on a given 3rd party project much easier.
Maintaining the correct version of code when a user uses their own git repo is not a problem. The user has to clone the repository to the correct location, and if they do so, we can assume they're doing it consciously and intentionally.

Another thing is that the user may want to use custom 3rd party package with custom patches. With the current solution this is impossible.

[cederom]

In a perfect world we would like to have all external dependencies listed somewhere, fetch before build, and then build. This is important for testing automation in order to provide all dependencies prior build is executed where malicious scripts can be executed. But this is another story of the future :-)

[zdebanos]

I've agreed with @ppisa that the solution with tar.gz is possible but as we do not have any release yet, we will use the commit hash instead as an identifier. We also both think a hash identifier is a more reasonable solution.

I'll try to submit a solution as fast as I can.

[xiaoxiang781216]

should we skip the clone if user already git clone manually like other 3rd party project

[zdebanos]

Ditto