Batch trim logic and cache getAllTransactions #951
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… updates
**Cache `Block.getAllTransactions()` Results:**
Similar to the existing caching for `getTransactions()`, this change implements a lazy-loading cache for `getAllTransactions()`. Previously, every call to this method resulted in a database query. Now, the full list of transactions for a block is fetched from the database only on the first call and cached within the `Block` object for subsequent requests. This significantly reduces database load and improves the performance of any operation that repeatedly accesses all transactions of a block.
This commit refactors the database `trim` functionality in `VersionedEntitySqlTable.java` to make it more robust, memory-efficient, and complete. Previously, the `trim` implementation had a significant limitation: it used a hardcoded `LIMIT 50000` on its `SELECT` query. This was a safeguard against `OutOfMemoryError` but meant that if a table had more than 50,000 outdated versioned records to process, the trim operation would be incomplete, leaving old data behind. This made the `trim` feature superficial and unreliable for large databases. This change introduces a comprehensive batching mechanism for both the `SELECT` and `DELETE` operations within the `trim` method. Instead of fetching up to 50,000 records at once, the logic now iteratively fetches and processes records in smaller, configurable batches. Key Changes: - **Batched `SELECT`:** The `SELECT` query now runs inside a `while` loop, fetching records in batches (default size: 10,000) until no more outdated records are found. This ensures all relevant records are processed without loading them all into memory at once. - **Removal of Hardcoded Limit:** The `LIMIT 50000` clause has been removed, as the new batching strategy makes it obsolete. The `trim` operation is now thorough and will process all outdated records, regardless of quantity. - **Batched `DELETE`:** The corresponding `DELETE` operations are also executed in smaller batches (default size: 1,000) within the loop, reducing the size and impact of individual database transactions. - **Improved Logging:** Logging has been updated to provide better insight into the batching process, reporting on the progress and total number of trimmed rows. This refactoring makes the database trimming feature a complete and reliable maintenance tool, crucial for the long-term health and performance of a node.
The second `DELETE` statement in the `trim` method, which removes all non-latest (`latest = false`) rows, has been refactored to execute in batches. Previously, this was a single, unbounded `DELETE` operation. After a large fork or a long period of being offline, this could lead to millions of rows being deleted in one transaction. Such a large operation could cause long-running table locks, high database load, and potential timeouts, impacting node stability. This commit changes the operation to run inside a `while` loop, deleting rows in smaller, configurable batches (`deleteBatchSize`). This approach breaks the large transaction into many smaller, faster ones, significantly improving the stability and performance of the `trim` process under heavy load.
The previous implementation of lazy-loading for `getTransactions()` and `getAllTransactions()` had a potential race condition. While the use of `AtomicReference` ensured correctness (preventing inconsistent states), it did not prevent multiple threads from simultaneously passing the `get() == null` check and performing the expensive database query. In a high-contention scenario, this could lead to the database being hit multiple times for the same block, partially defeating the purpose of the cache. This commit refactors both methods to use a "double-checked locking" pattern with `compareAndSet`. This atomic operation guarantees that even if multiple threads attempt to initialize the cache concurrently, only the first one will succeed in setting the value. Subsequent threads will use the already-populated cache, ensuring that the database query is executed only once per block instance. This change improves the efficiency and robustness of the transaction caching mechanism without introducing the overhead of a full `synchronized` block.
This commit improves the visual experience of the metrics panel by making the progress bars more accurately and dynamically represent their underlying moving average (MA) data. Previously, some progress bars (like network speed) were scaled against peak instantaneous values rather than the peak of the moving average, leading to a less intuitive visualization where the bar could appear full even for low MA values. Additionally, progress bars for fractional metrics did not visually represent the decimal values correctly. Changes include: - **Dynamic Scaling for Network Speed:** The upload and download speed progress bars now set their maximum value based on the historical maximum of their respective moving averages. This ensures the progress bar's fill level accurately reflects the current MA speed relative to its peak MA speed. - **Fractional Value Representation:** For metrics displayed with two decimal places (e.g., Blocks/Sec, Txs/Sec), the underlying `double` values are now multiplied by 100 before being passed to the `JProgressBar`. The progress bar's maximum is also scaled, preserving the ratio and providing a correct visual representation of fractional values. - **Timing Chart Stability:** Disabled auto-ranging on the timing chart's Y-axis and set a fixed range. This prevents the chart from "flickering" or resizing when new data with a higher peak value arrives, resulting in a more stable and pleasant viewing experience. These changes result in a smoother, more accurate, and more intuitive user experience when monitoring node performance.
Inicialization of Block Height and Peers during init phase
Add block.setAtTransactions(transactions); after transactionDb.saveTransactions(transactions);
Add block.setEscrowTransactions(resultTransactions); after Signum.getDbs().getTransactionDb().saveTransactions(resultTransactions);
Add block.setSubscriptionTransactions(paymentTransactions); after transactionDb.saveTransactions(paymentTransactions);
Using: int atTransactionCount = block.getAtTransactions().size(); int subscriptionTransactionCount = block.getSubscriptionTransactions().size(); int escrowTransactionCount = block.getEscrowTransactions().size(); int systemTransactionCount = atTransactionCount + subscriptionTransactionCount + escrowTransactionCount; int allTransactionCount = userTransactionCount + systemTransactionCount; instead of: int allTransactionCount = block.getAllTransactions().size();
Add proper fields and setter/getter methods to Block.java
private List<Transaction> atTransactions = new ArrayList<>();
private List<Transaction> subscriptionTransactions = new ArrayList<>();
private List<Transaction> escrowTransactions = new ArrayList<>();
…l updates
This commit introduces two major optimizations to address these issues:
1. **Parallelized Chart Data Processing:**
* Introduced DTOs (Data Transfer Objects) to decouple data calculation from UI rendering.
* The `updateAllCharts` method now uses `CompletableFuture` to perform expensive data calculations for performance, timing, and network charts in parallel on the common ForkJoinPool.
* All UI updates (progress bars and chart series) are now batched and executed in a single `SwingUtilities.invokeLater` call, preventing the Event Dispatch Thread (EDT) from being blocked by calculations.
* Chart notifications are disabled during the batched update to prevent excessive repainting, further improving rendering performance.
2. **`MovingAverage` Rework:**
* Replaced the `LinkedList`-based implementation with a more efficient array-based circular buffer. This significantly reduces memory overhead and improves performance by providing better data locality.
* Implemented Kahan summation to minimize floating-point precision errors when calculating the sum, leading to more accurate averages over time.
* The class is now fully thread-safe.
The combined effect of these changes is a much smoother, more responsive, and less memory-intensive Metrics Panel, providing a better user experience without sacrificing the detail of the displayed information.
The Block class has been refactored to improve performance by reducing redundant computations and object allocations. This is achieved by introducing lazy-loading and caching for frequently accessed, computationally expensive properties of a block.
Key changes:
1. **Lazy Loading for Transactions:**
* The `getTransactions()` and `getAllTransactions()` methods now fetch the block's transactions from the database only on the first call.
* The resulting transaction list is then cached in an `AtomicReference` for subsequent fast retrieval, significantly reducing database load.
2. **Lazy Calculation and Caching of ID and Hashes:**
* The block's unique ID, string ID, and full hash are now computed only once when `getId()` is first called.
* These values are cached, eliminating the need for repeated and costly SHA-256 hashing operations every time the ID is requested.
3. **Cached Byte and JSON Representations:**
* The `getBytes()` and `getJsonObject()` methods now cache their respective outputs (`byte[]` and `JsonObject`).
* This avoids the overhead of re-allocating `ByteBuffer`s, re-serializing the block to JSON, and re-creating these objects on every access.
4. **Thread-Safe Implementation:**
* All caching mechanisms are implemented in a thread-safe manner using `AtomicReference`, `AtomicLong`, and synchronized blocks to ensure data integrity in the node's multi-threaded environment.
These optimizations lead to a more efficient `Block` object, reducing CPU and memory usage, which contributes to better overall node performance, especially during intense operations like synchronization or when serving API requests that frequently access block data.
This commit introduces a new "Pause/Resume Sync" button to the main GUI toolbar, providing users with direct control over the blockchain synchronization process. Key changes include: - A `syncButton` is added to the main toolbar with a pause icon. - Clicking the button toggles the `isSyncStopped` state. - When paused, it calls `setGetMoreBlocksPause(true)` and `setBlockImporterPause(true)` on the `BlockchainProcessor` to halt the block downloading and importing threads. - The button's text and icon dynamically update to "Resume Sync" (Play icon) when paused, and back to "Pause Sync" (Pause icon) when active. - The main window title is appended with "(Sync paused)" to provide clear visual feedback on the node's state. - The experimental GUI timer for sync duration is also paused and resumed accordingly.
This commit introduces a significant overhaul of the peer information display and refactors the MovingAverage utility class for better code organization. ### Peer Information Enhancements The `PeersDialog` has been completely redesigned for better usability and clarity: - Replaced the single list with a `JTabbedPane` to categorize peers into "Active", "Connected", "Blacklisted", and "All Known". - Each tab now dynamically displays the count of peers in its category. - Peers are color-coded based on their status (e.g., red for blacklisted, green for active) for quick visual identification. - The list of peers is now sorted alphabetically by address. The main `SignumGUI` window has also been updated: - The status bar now shows a more informative peer summary: "Peers: Active / All Known (BL: Blacklisted)". - The peer info panel is now clickable, opening the detailed `PeersDialog`. ### Refactoring - The `MovingAverage` class has been moved from `brs.gui` to a new `brs.gui.util` package. This improves the project structure by separating utility classes from main GUI components. - All usages of `MovingAverage` have been updated to reflect the new package location.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✨ Batch trim logic and cache getAllTransactions
📝 Description
This pull request introduces two significant improvements aimed at enhancing database performance and node stability:
1️⃣ Comprehensive Batching for Trim Logic
This change refactors the database trim functionality in
VersionedEntitySqlTable.javato make it more robust, memory-efficient, and complete.The previous trim implementation had a significant limitation: it used a hardcoded
LIMIT 50000on its SELECT query.Without batching trim database processes the following issue could accour in some cases:
✅ Solution
This change introduces a comprehensive batching mechanism for both SELECT and DELETE operations within the trim method.
⚙️ Key Changes
whileloop, fetching records in batches (default size: 10,000) until no outdated records remain.LIMIT 50000has been removed.2️⃣ Cache Block.getAllTransactions() Results
Previously, every call to
getAllTransactions()triggered a database query for the complete transaction list of a block.MetricsPanel).✅ Solution
getAllTransactions()in theBlockclass.AtomicReferencewithin the block instance.📈 Overall Impact
OutOfMemoryError.getAllTransactions()avoids redundant lookups.