WIP: embed media files

[illwieckz]

Work in progress: embed media files from daemon_src.dpkdir.

I deeply reworked the GLSL include generator to make it reusable, I also renamed DaemonBuildInfo.cmake as DaemonGeneratedSource.cmake and put everything about generating sources there, and revamped it a lot.

It doesn't work yet because:

• The current code uses strings and then files are truncated on first 0, but the sound.wav file contains some bytes like that and this is valid.
• Files aren't listed, so there is no way the engine.shader file gets parsed.

[illwieckz]

So it now loads the whole file.

For a funny reason, it works because since the unvanquished dpk lists those names, the engine looks for them, and then the code to read them works. Of course we should be able to list them even if they aren't in a dpk already.

[illwieckz]

It almost works but…

• This error:

Warn: libpng error: PNG file corrupted by ASCII conversion 
Warn: PNG image 'gfx/2d/bigchars.png' has first exception handler called [libpng v.'1.6.50'] 

• The files are embedded in all engines (including the server and the tty client).
• I guess the server now requires the client to have that pak loaded just because the engine loaded it.

[illwieckz]

This error

Now fixed, there was a source beautifier stripping \r, now that we not only embed GLSL files, we better drop that.
Also we better beautify the sources in the repository directly.

[slipher]

Now fixed, there was a source beautifier stripping \r, now that we not only embed GLSL files, we better drop that.

Don't drop it. Add an option to specify text or binary mode.

[illwieckz]

Hmm, right now there is something corrupting something, the engine is slow, and the console has some things printed everyframe, actually it prints some PK string so it looks like the console is attempting to print the content of a zip every frame… that's very weird.

[illwieckz]

Don't drop it. Add an option to specify text or binary mode.

I added an option.

[illwieckz]

I now should have filtered out the builtin pak name from the list of required paks sent to clients.

I still don't know from where comes that weird bug slowing down the engine and polluting the console…

[illwieckz]

So I stop there for now, for people around, don't hesitate to look at it, especially for tracking bugs! 😁️

[slipher]

Hmm, in #71 you said we "don't have to think about some generic dpk embedding mechanism", but it looks like you have implemented precisely that. I was thinking we would avoid that and just use the byte arrays generated by the existing embed mechanism directly where needed. For example if we wanted the "bigchars" image, we could add that to the builtin images: create a *consoleFallbackFont image which would upload in the embedded byte array as an image. No need to make it a VFS file. Similarly, for q3shaders, we could just have a string in the source code and feed it into the shader parsing function.

[illwieckz]

Hmm, in #71 you said we "don't have to think about some generic dpk embedding mechanism", but it looks like you have implemented precisely that.

Yes, life is funny. 😄

I started wiring the VFS to test the loading of files, because in fact it's also a good way to make sure the path is exposed and that all the decoding functions are used. For example the PNG image has to be decoded by PNG, etc.

I don't mind the implementation, what I wanted to do there is to plumb the actual file embedding, which seems to now work.

I don't know how to properly make those files usable once they are embedded, and that's a task anyone can pick up.

[illwieckz]

Another thing funny is that I tried to create the smallest null.wav file possible, but then I get this error:

Warn: Unhandled exception (St12length_error): cannot create std::vector larger than max_size() 

My “smaller file” is using such format: signed 16-bit mono 8000 Hz, likely one sample only.

[illwieckz]

Ok, I had to keep enough samples, something like 0.3ms, the file now weights 90 bytes instead of 20KB, here is the generated file:

const unsigned char sound_null_wav[] = {
0x52,0x49,0x46,0x46,0x52,0x00,0x00,0x00,0x57,0x41,0x56,0x45,0x66,0x6d,0x74,
0x20,0x10,0x00,0x00,0x00,0x01,0x00,0x01,0x00,0x40,0x1f,0x00,0x00,0x80,0x3e,
0x00,0x00,0x02,0x00,0x10,0x00,0x64,0x61,0x74,0x61,0x2e,0x00,0x00,0x00,0x00,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,};

Edit: I also made sure it was truly zeroed.

[illwieckz]

ASAN catches nothing. EletricFence catches nothing. I don't know what to do.

[illwieckz]

] /devmap plat23
…
Debug: SVC_DirectConnect ()  
Client 0 connecting  
Debug: Going from CS_FREE to CS_CONNECTED for illwieckz [granger]  
Debug: CL packet loopback: connectResponse   
]PK

And then the console is unresponsive, and sometime filled with garbage (including filling the input!).

I have to quit the engine by doing daemon +quit in another terminal (and it takes time to process the command).

[illwieckz]

I don't know if that's related, but Valgrind says that:

==00:00:00:02.953 378687== Warning: set address range perms: large range [0x9bfb080, 0x29bfb080) (undefined)
==00:00:00:36.484 378687== Syscall param sendmsg(msg.msg_iov[3]) points to uninitialised byte(s)
==00:00:00:36.484 378687==    at 0x4E8D06B: __libc_sendmsg (sendmsg.c:28)
==00:00:00:36.484 378687==    by 0x4E8D06B: sendmsg (sendmsg.c:25)
==00:00:00:36.484 378687==    by 0x35C7C4: NaClSendDatagram (nacl_imc.cc:132)
==00:00:00:36.484 378687==    by 0x26D5B7: IPC::InternalSendMsg(int, bool, IPC::FileDesc const*, unsigned long, void const*, unsigned long) (Primitives.cpp:288)
==00:00:00:36.484 378687==    by 0x26D78C: IPC::Socket::SendMsg(Util::Writer const&) const (Primitives.cpp:309)
==00:00:00:36.484 378687==  Address 0x3015f2e1 is 17 bytes inside a block of size 3,088 alloc'd
==00:00:00:36.484 378687==    at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==00:00:00:36.484 378687==    by 0x192F58: std::__new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:151)
==00:00:00:36.484 378687==    by 0x18C9F0: allocate (alloc_traits.h:482)
==00:00:00:36.484 378687==    by 0x18C9F0: std::_Vector_base<char, std::allocator<char> >::_M_allocate(unsigned long) (stl_vector.h:381)
==00:00:00:36.484 378687==    by 0x18AE9E: void std::vector<char, std::allocator<char> >::_M_range_insert<char const*>(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, char const*, char const*, std::forward_iterator_tag) (vector.tcc:805)
==00:00:00:36.484 378687==  Uninitialised value was created by a stack allocation
==00:00:00:36.484 378687==    at 0x1ACBF8: CGameVM::CGameRocketFrame() (cl_cgame.cpp:1061)
==00:00:00:36.484 378687== 
==00:00:05:17.384 378687== Warning: set address range perms: large range [0x9bfb068, 0x29bfb098) (noaccess)

[illwieckz]

If I modify those lines this way, the bug disappears, just by adding the logger:

void CGameVM::CGameRocketFrame()
{
	cgClientState_t state = {};
	state.connectPacketCount = clc.connectPacketCount;
	state.connState = cls.state;
Log::Warn("=1===[%s]===", cls.servername);
	Q_strncpyz( state.servername, cls.servername, sizeof( state.servername ) );
Log::Warn("=2===[%s]===", cls.updateInfoString);
	Q_strncpyz( state.updateInfoString, cls.updateInfoString, sizeof( state.updateInfoString ) );
Log::Warn("=3===[%s]===", clc.serverMessage);
	Q_strncpyz( state.messageString, clc.serverMessage, sizeof( state.messageString ) );
	state.clientNum = cl.snap.ps.clientNum;
	this->SendMsg<CGameRocketFrameMsg>(state);
}

And the log prints nothing:

Warn: =1===[]===
Warn: =2===[]===
Warn: =3===[]===

[illwieckz]

Ah no I forgot to do /devmap… 🤦🏻

This is what it first prints:

Warn: =1===loopback===
Warn: =2======
Warn: =3======

…and then the console becomes broken:

Warn: =1===loop
Warn: =
Warn: =
Warn: =1===loo
Warn:
Warn:
Warn: =1===lo
Warn:
Warn:
Warn: =1===l
Warn
Warn
Warn: =1===
War
War
Warn: =1==
Wa
Wa
Warn: =1=
W
W
Warn: =1


Warn: =


Warn:


Warn:


Warn


War


Wa


W

[slipher]

Well, I tried embedding a font file and it worked for me: #1848

[illwieckz]

I FOUND THE BUG, IT WORKS.

The bug was that I forgot to set loadedPak.fd = -1 for the loaded builtin pak, meaning later some code was attempting to close the zip for real! I don't know how it did not crashed, and why what looks like some other pak got printed in the terminal (I got PK strings…), but that was that!

[illwieckz]

[FS] Loading builtin pak '*daemon_0.55.5'...
[FS] Loading builtin pak '*daemon-client_0.55.5'...
[FS] Loading pakdir '…/UnvanquishedAssets/build/_pakdir/pkg/unvanquished_test.dpkdir/'...
[FS] Loading pakdir '…/UnvanquishedAssets/build/_pakdir/pkg/tex-common_test.dpkdir/'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-players_0.55.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-legacy_0.55.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-weapons_0.55.3.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-weapons_0.55.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-buildables_0.55.4.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-buildables_0.55.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-voices_0.55.3.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-voices_0.54.dpk'...
[FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-soundtrack_0.54.dpk'...

]/which fonts/unifont.ttf
File "fonts/unifont.ttf" found in "*daemon-client_0.55.5"
]/which sound/null.wav
File "sound/null.wav" found in "*daemon-client_0.55.5"

[illwieckz]

One can actually run a custom dll game code without any dpk by doing:

./daemonded -set fs_basepak '*daemon' -set vm.cgame.type 3 -set vm.sgame.type 3

[illwieckz]

The annoying downside of embedding a pak in the engine is that, unless I'm wrong, files from a builtin pak are loaded from that builtin pak, i.e. copied in memory. So you get the file taking twice the amount of memory: first in the executable itself, and then as a loaded file.