feat: add StreamingParser for memory-efficient CSV parsing #5087
+504
−2
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add StreamingParser for memory-efficient CSV parsing of large files
What?
This PR introduces a new
StreamingParser
class for thek6/experimental/csv
module that enables memory-efficient parsing of large CSV files without loading them entirely into memory.Key additions:
Parser
for seamless migrationWhy?
Problem: The current CSV parser causes Out-of-Memory (OOM) errors when processing large files because:
fs.open()
loads entire files into memory usingio.ReadAll()
[]byte
Root Cause: The file system module (
internal/js/modules/k6/experimental/fs/cache.go:102
) usesio.ReadAll()
which is unsuitable for large file processing.Impact: Users cannot process large CSV files (>RAM size) for load testing scenarios with extensive test data.
Before (OOM error):
After (memory-efficient):
Performance improvement:
Checklist
make check
) and all pass.Checklist: Documentation (only for k6 maintainers and if relevant)
Please do not merge this PR until the following items are filled out.
Related PR(s)/Issue(s)
Closes #5080
Testing
All tests pass including new streaming-specific tests: