feat: bypass chunking when no transformation is requested #597
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RSM upload steps are messy and hard to read, mainly because of the encryption metadata. By moving the instantiation to the data key and aad, we can separate the upload stages a bit better.
Chunking is defined by original chunk size, not original content size. This commit removed the scenario where these two sizes are conflated (e.g. passing original size zero to disable chunking, etc.).
jeqo
force-pushed
the
jeqo/upload-log-from-file
branch
from
06fff4d to
325b9d1
Compare
When producing an input stream, if no transformation is applied, then an input stream from the source file can be provided. This will further be used to pass the path directly to the storage backend in a further PR.
jeqo
force-pushed
the
jeqo/upload-log-from-file
branch
from
325b9d1 to
04dbb86
Compare
| "encryption.enabled", Boolean.toString(encryption) | ||
| )); | ||
| final SegmentEncryptionMetadataV1 encryptionMetadata; | ||
| final DataKeyAndAAD maybeEncryptionKey; |
There was a problem hiding this comment.
What about using an Optional instead of setting maybeEncryptionKey to null ?
Then when using maybeEncryptionKey you don't need to check again if encryptionEnabled, but if the value is present in the Optional then it means that encryption is enabled and you will use that encryption key. Probably it does not reduce the code but I find it more easy to reason about explicit nulls.
There was a problem hiding this comment.
Agree; though the main issue is that it would force the functions to take Optional as input parameters which is not recommended; see related discussion: #531 (comment)
I guess a context object that brings these values together could simplify this; but would leave that for another PR.
giuseppelillo
approved these changes
Oct 1, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pass directly the file input stream to the storage layer instead of calculating chunks on the fly. Optimizes for the default configuration (no encryption/compression), and prepares for further passing the path to the storage layer if uploading directly from file is preferred.
It contains a couple of meaningful refactoring to clean up RSM logic and simplify chunking. See commits for further details.