Performace Benchmark on Large Data Volume

[SemAiSoft]

I just want to know if there is any sort of performance metrics on a given volume of data. (Nedb did this). Especially since the whole of Collection is being loaded into memory in this lib (?).
What happens with enough number of collection, e.g. 50+, each with quite lots of data?

What do you advice on using it nodejs/electronjs as a full-fledged db, though with backup/pack-up to a standard db, say mongodb?

[maxnowack]

I haven't run any benchmarks yet. Feel free to contribute!
In general the performance depends a lot on the system where signaldb is running on. If there is more data in signaldb than the system has memory, the system starts swapping or even crashes.

SignalDB is not a replacement for mongodb. I depends on your use case if it's a good idea to use signaldb in an electron app. If you're dealing with a lot of data, then there are maybe some other, more performant solutions.

[Cmacu]

@maxnowack what would be your general guideline for web applications?

Would you say that something like several hundred MBs is all right?

Also what about strategies where different collections are loaded in and out of memory as needed?

For example dashboard page might use collection A, B and C, while other pages interact with them individually.

I guess we can start and pause the sync as needed, but my main concern is what happens with registerRemoteChange.

I think the introduction of IndexedDB is step in scalability direction, but some divide and conquer might be necessary to make it even more robust.

[SemAiSoft]

Yes, different collections are loaded in and out of memory as needed!
I have thought about that too, when we are sure a particular collection is not currently need by views, it would be dispose().

That would help, right?

[Cmacu]

Yep and when we have a view that needs the collection again it can be loaded back in memory from IndexedDB to enable reactivity. Also not sure how the registerRemoteChange would work, but maybe it's OK to just pull changes.
Also this would still be an issue for very large collections. A strategy of offloading/expiring/archiving transactional records could be nice, but perhaps it's up to dev land how to handle these.