Improve distribution strategy for Nim dependencies

[igor-sirotin]

Context

We have several Go wrapper repositories that bind to Nim libraries compiled as shared libraries via CGO:

• waku-org/waku-go-bindings → requires nwaku
• waku-org/sds-go-bindings → requires nim-sds
• Future: Waku Chat SDK
• Future: Nimbus Verification Proxy

Current Approach

Binding repository has a Makefile target that clones the Nim library into a third_party/ subdirectory (e.g., third_party/nwaku), then builds the shared library from source.

This currently works because we vendor dependencies in status-go, so when users run make in the vendored bindings directory, there are no permission issues.

The Problem

We are removing vendoring from status-go (#6951). Once vendoring is removed, the current approach breaks because:

1. Go module cache is read-only: When users go get the bindings, they're downloaded to $GOPATH/pkg/mod/, which is read-only. The Makefile cannot clone into third_party/ or write build artifacts.
2. go get doesn't run Make: Go's standard workflow doesn't execute Makefiles, so users must manually discover and run build steps.

Why Git Submodules Don't Work

1. Go module incompatibility: go get doesn't fetch git submodules, breaking the standard Go workflow
2. Nested submodule issues: status-go is already a submodule of status-desktop. Adding Nim libraries as submodules of status-go creates deeply nested submodules (desktop → status-go → nwaku), which causes build failures and is difficult to maintain

Viable Options

Option 1: System Installation - the only Go-idiomatic approach

Document installation via package managers (apt, brew, etc.) or install scripts.

Pros	Cons
✅ Small repo size	❌ Doesn't exist at the moment
✅ Standard Unix approach	⚠️ Platform-specific instructions
✅ Separation of concerns	❌ Can't fix/define dependency version

Option 2: Environment Variable Configuration

Use environment variables to let users specify library paths, supporting multiple installation methods.

CGO directives in the bindings:

package wakugo

/*
#cgo CFLAGS: -I${NWAKU_INCLUDE_DIR}
#cgo LDFLAGS: -L${NWAKU_LIB_DIR} -lnwaku -Wl,-rpath,${NWAKU_LIB_DIR}
#include "nwaku.h"
*/
import "C"

Option 3: Vendor Pre-compiled Binaries

Bundle pre-compiled .so/.dylib/.dll files for common platforms directly in each wrapper repo.

Pros	Cons
✅ Works with go get out of the box	⚠️ Large repo size (and Git LFS not supported by go get)
✅ Zero installation steps for users	⚠️ Must build for multiple platforms
✅ Standard Go module experience	⚠️ Must manually support each nwaku version
✅ No build step required

Option 4: Nimble Package Manager

Distribute Nim libraries via Nimble and document installation via nimble install.

Pros	Cons
✅ Natural for Nim developers (our primary contributors)	⚠️ Doesn't work with go get alone
✅ Existing ecosystem and tooling	⚠️ Requires Nimble + Nim compiler installed
✅ Easy updates via Nimble	❌ All our dependencies are not available with Nimble yet (ETA 1-2 months?)

Option 5: Git Subtree

Embed Nim library source using git subtree into each wrapper repo.

Pros	Cons
✅ Works with go get	⚠️ Large repo size
✅ No nested submodule issues	⚠️ Requires Nim compiler during build

Option 6: Manually download libwaku artifacts

Implement a custom solution to download libwaku artifacts from Github Releases.
Example in waku-org/waku-go-bindings#92.

Pros	Cons
✅Smaller repo size (no vendored binaries)	⚠️Doesn't work with go get alone - requires running download script first
✅Fast implementation, no need to integrate with package managers	❌️Still writes to read-only module cache if triggered during go get
	⚠️Users must discover and run the download script manually

[igor-sirotin]

My personal preference order

1. Option 1 - Use system-installed libraries (as the most Go-idiomatic way)
2. Option 4 - Nimble
3. Option 2 - Environment Variable Configuration
4. Option 5 - Pre-compiled binaries (I don't want to make the repo ~100Mb heavier on each release)

And I don't like the git subtree at all. This will duplicate things too much, and will, again, make the repo heavy (though we could squash it).

[fryorcraken]

To clarify, Vac has indeed committed to help with nwaku usage via nimble, and replacing nimble-build-system with nimble in nwaku.
Once done, we should be able to have all Waku library (inc. chat sdk, sds), nimble ready. cc @kaiserd

[Ivansete-status]

We can discard Option 5: git subtree because the git subtree doesn't retrieve all the submodules automatically.

[igor-sirotin]

I was thinking about this one more time.

cc @Ivansete-status @fryorcraken @osmaczko please let me know what you think

Mix of Option 2 and Option 4

I believe this will be best for us.

1. waku-go-bindings expect nwaku to be provided externally with env vars, as per Option 2.
2. status-go get nwaku from Nimble (as per Option 4) and define the env vars accordingly, if not provided externally
3. status-desktop can :

• either also get nwaku from Nimble
  This will be quite clean, but then we will define the version of nwaku in both status-go and status-desktop
• or use status-go Nimble package definition to install dependencies
• or get nwaku as git submodule and set envvars to status-go
  This is already possible, as status-desktop doesn't use Nimble atm.

So we mostly rely on Nimble, yet:

• enable target projects to get nwaku from anywhere convenient
• don't have to wait for nwaku Nimble package to be ready. For now we can get nwaku from anywhere and define envvars accordingly.

---

Also, one thing I am not sure if we should try to achieve: fixing nwaku version in status-go.
When I am using status-go in my project (would it be status-desktop, or anything with go get status-go), should the nwaku version be also automatically defined? Or should I be able to use any version of nwaku? 🤔

[igor-sirotin]

UPD: Option 2 is not doable, because #cgo directives don't support env vars ☹️