feat: Implement memfs for file system management #2281

irfanfaraaz · 2025-06-26T05:20:56Z

Description

Migrated the sandbox file system from custom file/content maps to a proper Virtual File System (VFS) implementation using memfs. This change significantly improves performance, reduces complexity, and provides enhanced file system capabilities while maintaining 100% backward compatibility.

Replaced FileSyncManager with VFSSyncManager and virtual-fs.ts

Related Issues

fixes #2280

Type of Change

Testing

`VirtualFileSystem`: 22 tests covering core file system functionality

`VFSSyncManager`: 21 tests ensuring compatibility and new features

Updated existing tests: All 82 sandbox tests passing

Screenshots (if applicable)

Additional Notes

Files added:

virtual-fs.ts - Core VFS implementation
vfs-sync-manager.ts - Drop-in replacement for FileSyncManager

Important

Replaces FileSyncManager with VFSSyncManager using memfs for improved file system management, adding new VFS implementation and extensive testing.

Behavior:
- Replaces FileSyncManager with VFSSyncManager in sandbox/index.ts.
- Introduces virtual-fs.ts for core VFS implementation.
- Adds vfs-sync-manager.ts as a drop-in replacement for FileSyncManager.
- Ensures backward compatibility with existing file system operations.
Testing:
- Adds 22 tests for VirtualFileSystem covering core functionality.
- Adds 21 tests for VFSSyncManager ensuring compatibility and new features.
- Updates existing tests; all 82 sandbox tests passing.
Dependencies:
- Adds memfs to package.json for VFS support.

^{This description was created by}^{for 07692dd. You can customize this summary. It will automatically update as commits are pushed.}

vercel · 2025-06-26T05:21:01Z

@irfanfaraaz is attempting to deploy a commit to the Onlook Team on Vercel.

A member of the Team first needs to authorize it.

Kitenite · 2025-06-26T18:02:16Z

This is awesome! What made you choose memfs out of the other solutions? Just curious if there were tradeoffs you considered.

irfanfaraaz · 2025-06-26T18:55:17Z

When I researched VFS libraries, memfs was the most recommended because of it's focus on in-memory use and works as a lightweight, near drop-in replacement for Node's fs. Other libraries are more focused on disk storage

Kitenite · 2025-06-26T21:52:00Z

That makes sense. Seems consistent with a deep research run since I haven't used these libraries
https://chatgpt.com/s/dr_685dc0d031e48191a010568d16408ba2

Is all the behavior the same for the code tab, AI and assets management or are you seeing any differences there?

irfanfaraaz · 2025-06-27T19:01:32Z

Yes the behavior is the same. I used Gemini for testing since i ran out of Anthropic credits, so can’t fully comment on performance. There’s a bit of inconsistency in tool calling, but it was present both before and after my changes.

spartan-vutrannguyen · 2025-06-28T04:49:59Z

apps/web/client/src/components/store/editor/sandbox/virtual-fs.ts

+    }
+
+    // Path utilities
+    normalizePath(path: string): string {


It’d be great to double-check these utility functions like normalizePath, dirname, and basename to ensure they align with our existing ones in the files utility. Currently, these functions are returning different paths compared to the existing ones.

https://github.com/irfanfaraaz/onlook/blob/vfs/apps/web/client/src/components/store/editor/sandbox/helpers.ts#L9

https://github.com/irfanfaraaz/onlook/blob/vfs/apps/web/client/src/components/store/editor/pages/helper.ts

spartan-vutrannguyen · 2025-06-28T04:52:12Z

It looks good overall! But could you kindly rebase with main and verify if Asset Management still works with this implementation?
Right now, it seems this change is breaking asset management, so that’ll need to be addressed as well.

irfanfaraaz · 2025-06-28T12:13:55Z

Thanks for pointing it out. Updated the code to use existing utility functions and added separate logic for memfs, since memfs requires absolute paths.
I’ve been trying to test this for 4–5 hours, but I keep getting errors when starting the app.Same issues happen on main as well.

spartan-vutrannguyen · 2025-06-28T16:57:14Z

Thanks for pointing it out. Updated the code to use existing utility functions and added separate logic for memfs, since memfs requires absolute paths. I’ve been trying to test this for 4–5 hours, but I keep getting errors when starting the app.Same issues happen on main as well.

Could you share more details about the errors you're encountering?
Any error messages, steps to reproduce, or context would be really helpful!

irfanfaraaz · 2025-06-28T18:02:52Z

Initially i was facing this error

Then i made some changes in trpc.ts to bypass it

and now i randomly get either of the below 2

Kitenite · 2025-06-29T02:30:43Z

@irfanfaraaz could you try pulling from latest and run bun run db:push?

irfanfaraaz · 2025-06-29T02:34:25Z

Just fixed that issue.. now facing this (in main)

irfanfaraaz · 2025-06-29T02:41:07Z

Oh finally it's working.. guess morphllm was down

above screenshot is from vfs branch
@Kitenite

Kitenite · 2025-06-29T02:45:18Z

Nice! Glad it's working. We're using relace for the prod version but interesting that morph went down

irfanfaraaz · 2025-06-29T03:48:56Z

I’ve synced with main, retested and pushed my changes. everything is working as expected, finally, after debugging for the whole day.

apps/web/client/src/components/store/editor/sandbox/virtual-fs.ts

spartan-vutrannguyen · 2025-06-30T04:02:17Z

apps/web/client/src/components/store/editor/sandbox/virtual-fs.ts

@@ -439,20 +458,23 @@ export class VirtualFileSystem implements VirtualFileSystemInterface {
            for (const sandboxPath of allFiles) {
                try {
                    const vfsPath = this.toVFSPath(sandboxPath);
-                    // Try to read as text first
-                    const textContent = this.volume.readFileSync(vfsPath, {


The issue is that readFileSync with encoding: 'utf8' will attempt to decode binary files as UTF-8 text, which often succeeds but produces garbled/corrupted text instead of throwing an error. This means the catch block never executes for binary files.

spartan-vutrannguyen · 2025-06-30T04:02:58Z

apps/web/client/src/components/store/editor/sandbox/virtual-fs.ts

-                        const vfsPath = this.toVFSPath(sandboxPath);
+
+                    // Determine if file is binary based on extension
+                    const isBinary = this.isBinaryFile(sandboxPath);


So I wrote an utility function to check

Oh, I wasn’t aware of that. Got to learn something. Thanks for the changes, mate! @spartan-vutrannguyen

Kitenite · 2025-07-02T04:26:27Z

apps/web/client/src/components/store/editor/sandbox/virtual-fs.ts

+    ctime: Date;
+}
+
+export interface VirtualFileSystemInterface extends FileOperations {


Not sure I understand why we need to re-implement this interface instead using the direct fs api from memfs

https://github.com/streamich/memfs/blob/master/docs/node/usage.md

Here’s what I’m doing and why:
Using Volume from memfs instead of (fs,vol):
I'm using Volume so each project/sandbox has its own isolated file system.
Wrapper Methods:
The wrappers handle path normalization (so callers can use relative paths for memfs) with the underlying this.volume instance
Persistence (new commit):
I switched to memfs’s toJSON()/fromJSON() for saving/restoring the file system.

The goal I was hoping to achieve was to potentially reduce the amount of code we have implementing a file system. To potentially have a better way to mirror the files system from the sandbox without implementing file system interfaces. Looks like this would not be feasible even with using memfs?

Seems like it's essentially the same amount of work and complexity with just a different persistence layer. I was hoping we would remove this class entirely and use memfs' interface directly.

feat: Implement memfs for file system management

07692dd

spartan-vutrannguyen reviewed Jun 28, 2025

View reviewed changes

Merge branch 'main' of https://github.com/onlook-dev/onlook into vfs

8948bb7

Merge branch 'main' of https://github.com/irfanfaraaz/onlook into vfs

c89153e

update read binary files

b508dbf

spartan-vutrannguyen reviewed Jun 30, 2025

View reviewed changes

apps/web/client/src/components/store/editor/sandbox/virtual-fs.ts Show resolved Hide resolved

spartan-vutrannguyen reviewed Jun 30, 2025

View reviewed changes

spartan-vutrannguyen and others added 3 commits June 30, 2025 11:13

Merge branch 'main' into vfs

2c36648

merge from main

1012e27

remove legacy file-sync-manager

d4e6573

Kitenite reviewed Jul 2, 2025

View reviewed changes

irfanfaraaz and others added 3 commits July 2, 2025 19:54

refactor: simplify persistence using memfs toJSON/fromJSON snapshot

0574ec3

Merge branch 'main' into vfs

57c8a5b

merge from main

018dc10

feat: Implement memfs for file system management #2281

Are you sure you want to change the base?

feat: Implement memfs for file system management #2281

Uh oh!

Conversation

irfanfaraaz commented Jun 26, 2025 • edited by ellipsis-dev bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Type of Change

Testing

VirtualFileSystem: 22 tests covering core file system functionality

VFSSyncManager: 21 tests ensuring compatibility and new features

Updated existing tests: All 82 sandbox tests passing

Screenshots (if applicable)

Additional Notes

Files added:

Uh oh!

vercel bot commented Jun 26, 2025

Uh oh!

Kitenite commented Jun 26, 2025

Uh oh!

irfanfaraaz commented Jun 26, 2025

Uh oh!

Kitenite commented Jun 26, 2025

Uh oh!

irfanfaraaz commented Jun 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spartan-vutrannguyen commented Jun 28, 2025

Uh oh!

irfanfaraaz commented Jun 28, 2025

Uh oh!

spartan-vutrannguyen commented Jun 28, 2025

Uh oh!

irfanfaraaz commented Jun 28, 2025

Uh oh!

Kitenite commented Jun 29, 2025

Uh oh!

irfanfaraaz commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

irfanfaraaz commented Jun 29, 2025

Uh oh!

Kitenite commented Jun 29, 2025

Uh oh!

irfanfaraaz commented Jun 29, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

irfanfaraaz commented Jun 26, 2025 •

edited by ellipsis-dev bot

Loading

`VirtualFileSystem`: 22 tests covering core file system functionality

`VFSSyncManager`: 21 tests ensuring compatibility and new features

irfanfaraaz commented Jun 29, 2025 •

edited

Loading