Make libproj shared and enable pyproj [full build]

[tauroid]

See this discussion pyodide/pyodide#5887

TLDR: different instantiations of sqlite in different pyproj extension modules meant that sqlite was initialising global function pointers in one module and failing to find them in another. At least this is my take. Making libproj shared (and by extension, the statically linked sqlite inside libproj) appears to solve the issue.

This PR is experimental for now until CI passes and I've thought about it a little more.

Update: I bumped Cartopy and geopandas to check they pass too and looks good from here.

I can't see (as a new contributor) any more issues to solve. Apologies if I've missed something, let me know and I can change it.

[github-actions]

Package Build Results

Total packages built: 56
Total build time: 0:46:28

Package Build Times (click to expand)

Package	Build Time
libgdal	29m 13s
libgeos	16m 41s
scipy	11m 21s
libproj	11m 20s
libopenblas	9m 42s
libopenssl	4m 44s
numpy	4m 19s
pandas	4m 14s
matplotlib	3m 42s
libiconv	3m 23s
libboost	3m 1s
Pillow	2m 15s
sqlite3	1m 47s
libtiff	1m 26s
libwebp	1m 24s
contourpy	1m 15s
liblzma	1m 8s
pyproj	1m 7s
test	46s
shapely	45s
kiwisolver	41s
libf2c	35s
fiona	33s
Cartopy	32s
regex	23s
hashlib	15s
ssl	10s
MarkupSafe	7s
pydoc_data	7s
lzma	6s
atomicwrites	6s
pydecimal	4s
packaging	4s
pytz	2s
setuptools	2s
exceptiongroup	2s
py	2s
micropip	2s
geopandas	2s
fonttools	1s
cycler	1s
certifi	1s
six	1s
cligj	1s
tblib	1s
click	1s
python-dateutil	1s
iniconfig	1s
pyshp	1s
pytest-asyncio	1s
more-itertools	1s
Jinja2	1s
pyparsing	1s
attrs	1s
pytest	0s
pluggy	0s

Longest build: libgdal (29m 13s)
Packages built in more than 10 minutes: 4

[ryanking13]

Hmm, so do you think the problem is that sqlite3 is linked both in libproj and pyproj?

Then what happens if we just keep libproj static and remove sqlite3 from pyproj?

[ryanking13]

Anyway, I am happy if this solves the problem. I don't fully understand why... but there are always unknown unknowns in package build, so if it works, I am happy to merge it.

I triggered full build (building all packages) to see if there is any regression.

[tauroid]

Hmm, so do you think the problem is that sqlite3 is linked both in libproj and pyproj?

Not quite, I think it's that pyproj has 10 different extension modules (like _context.cpython-313-wasm32-emscripten.so, _crs.cpython-313-wasm32-emscripten.so etc), which each have a copy of libproj and sqlite3 statically linked inside, and it looks like libproj expects instead to be talking to one single "persistent" (in memory) sqlite library. In that it doesn't aggressively reinitialise sqlite whenever it calls it. So when pyproj initialises libproj from one cython module, then calls other functions from other different cython modules, the initialisation somehow doesn't carry over and you get the null function error.

Though I'm not actually sure what the consequences of linking sqlite in libproj and then again in pyproj are 🤔 I guess the 10 extension modules could then also each have had some conflicting sqlite stuff inside? But I haven't personally seen any sign of that in what I've looked at so far.

[tauroid]

Then what happens if we just keep libproj static and remove sqlite3 from pyproj?

I can give this a quick try though, and revert if it fails (as I think it will)

[tauroid]

Though I'm not actually sure what the consequences of linking sqlite in libproj and then again in pyproj are 🤔

I think this is a place my lack of knowledge about compilers is showing - keeping the static build of libproj and removing the link flag in pyproj causes sqlite not to be included at all. Testing locally, I get cannot resolve symbol sqlite3_snprintf. So I guess in the static build of libproj, sqlite3 is referred to, but not actually included (which I think makes sense). Only in the link step of pyproj is the sqlite code added to the library (so only added once per cython module).

Whereas if libproj is a shared library build, I suppose it gets a link step, so sqlite3 is included at that point.

I'll let the CI finish now it's started, but I think the cannot resolve symbol is what will happen. If so I'll revert after. I think I might have cancelled the initial full build, sorry, though I thought I saw it complete.

[tauroid]

PythonError: Traceback (most recent call last):
  File "/lib/python313.zip/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/lib/python3.13/site-packages/micropip/logging.py", line 123, in ctx_level
    yield self.logger
  File "/lib/python3.13/site-packages/micropip/package_manager.py", line 235, in install
    await asyncio.gather(*(wheel.install(wheel_base) for wheel in wheels))
  File "/lib/python3.13/site-packages/micropip/wheelinfo.py", line 139, in install
    await self._install(target)
  File "/lib/python3.13/site-packages/micropip/wheelinfo.py", line 238, in _install
    await install(
    ...<5 lines>...
    )
pyodide.ffi.JsException: Error: Failed to load dynamic library /lib/python3.13/site-packages/pyproj/_context.cpython-313-wasm32-emscripten.so: 'Could not load dynamic lib: /lib/python3.13/site-packages/pyproj/_context.cpython-313-wasm32-emscripten.so
Error: Dynamic linking error: cannot resolve symbol sqlite3_snprintf

locally, vs

pytest_pyodide.runner.JavascriptException: PythonError: Traceback (most recent call last):
E     File "/lib/python313.zip/_pyodide/_base.py", line 597, in eval_code_async
E       await CodeRunner(
E       ...<9 lines>...
E       .run_async(globals, locals)
E     File "/lib/python313.zip/_pyodide/_base.py", line 411, in run_async
E       coroutine = eval(self.code, globals, locals)
E     File "<exec>", line 1, in <module>
E     File "/lib/python3.13/site-packages/pyproj/__init__.py", line 34, in <module>
E       import pyproj.network
E     File "/lib/python3.13/site-packages/pyproj/network.py", line 10, in <module>
E       from pyproj._context import _set_context_ca_bundle_path
E   ImportError: dynamic module does not define module export function (PyInit__context)

on CI

Not having PyInit seems a bit extreme, I can't really explain that.

[tauroid]

Hmm okay, I did want to use rasterio, I'll look into it. It created an issue with libgdal I think

[ryanking13]

Not quite, I think it's that pyproj has 10 different extension modules (like _context.cpython-313-wasm32-emscripten.so, _crs.cpython-313-wasm32-emscripten.so etc), which each have a copy of libproj and sqlite3 statically linked inside, and it looks like libproj expects instead to be talking to one single "persistent" (in memory) sqlite library.

I see. I don't understand why it worked before and why it doesn't anymore, but I am fine with adopting your original approach if other packages work fine. Thanks for the investigation!

[ryanking13]

Hmm okay, I did want to use rasterio, I'll look into it. It created an issue with libgdal I think

I think you need to add libproj as a run dependency of libgdal. As libproj is now shared library, libproj need to be loaded runtime when loading libgdal.

[tauroid]

I see. I don't understand why it worked before and why it doesn't anymore, but I am fine with adopting your original approach if other packages work fine. Thanks for the investigation!

Same, it's still a mystery to me. The only thing I have to go on is that from PROJ 9.3.1 to 9.4.0, they changed from a custom FindSqlite3 to the default cmake one. But I have no idea why that would affect anything. Maybe something changed in sqlite itself as well.

I'm getting round to this, probably tomorrow.

[tauroid]

Same issue :/ rasterio and rioxarray are working locally, I'll have to unpick the differences in slower time. Compiling just libgdal still takes about 30 mins a pop. Only thing I do have to go on is I'm compiling with lots of debug flags, and maybe need to double check versions (though I think those are the same).

Locally, it seems to even work statically linking libproj (to libgdal, not pyproj), I guess because the .a is still created