feat: Smoother Data Loading with Web Workers & Better Validation #774
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Problem
Currently, when you load a large event file (like a heavy
.igarchive), the entire application freezes for a few seconds. You can't click anything or move the camera until it's done. This happens because the "heavy lifting" of reading the data is being done on the same track that draws the screen.The Solution
I've moved this heavy work to a "background thread" (Web Worker). This means the data loads in the background while the main screen remains smooth and responsive. I also added a safety check to ensure the data we receive is actually correct, preventing weird crashes later on.
Key Improvements
Non-Blocking UI: The application no longer freezes when loading files. You can continue to use the interface smoothly (60fps) even while heavy data is being processed.
Data Reliability: We now strictly check the data structure when it arrives. If a file is broken, we catch it immediately and show a clear error instead of letting the 3D viewer crash silently.
Implementation Details
Web Worker (
cms-loader.worker.ts):Moved the expensive
JSON.parseand string sanitization logic off the main thread.Implements a message-based interface to receive raw data strings and return validated objects.
Schema Validation (
validation-schemas.ts):Defined comprehensive
Zodschemas for CMS Event types (Types,Collections,Associations).Ensures strict typing for
eventDatawhich was previously typed asany.Loader Refactor (
CMSLoader.ts):Refactored
readIgArchiveto asynchronously delegate parsing tasks to the worker.Maintains backward compatibility with the existing
EventDataLoaderinterface.Verification
Unit Tests: Added
src/tests/loaders/cms-loader.worker.test.tsto verify worker message handling and parsing logic.Static Analysis: Code compiles successfully with
tsc.Manual: Verified that
CMSLoadercorrectly initializes the worker and handles.igarchive extraction.Dependencies
Added
zodfor runtime schema validation.