HDFS-17815. Fix upload fsimage failure when checkpoint takes a long time #7845
+63
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
Hexiaoqiao
reviewed
Aug 8, 2025
| LOG.info("Triggering checkpoint because it has been {} seconds " + | ||
| "since the last checkpoint, which exceeds the configured " + | ||
| "interval {}", secsSinceLast, checkpointConf.getPeriod()); | ||
| LOG.info("Triggering checkpoint because it has been {} seconds " |
There was a problem hiding this comment.
Please keep the original format.
There was a problem hiding this comment.
@Hexiaoqiao Thanks for review. I have resolve it.
| @@ -487,8 +488,9 @@ private void doWork() { | |||
| namesystem.setCreatedRollbackImages(true); | |||
| namesystem.setNeedRollbackFsImage(false); | |||
| } | |||
| lastCheckpointTime = now; | |||
| LOG.info("Checkpoint finished successfully."); | |||
| lastCheckpointTime = monotonicNow(); | |||
There was a problem hiding this comment.
Great catch here. Thanks.
lfxy
force-pushed
the
feature/HDFS-17815
branch
from
3f9c5da to
63e6466
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
lfxy
force-pushed
the
feature/HDFS-17815
branch
from
d585b7b to
e525d98
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
ayushtkn
reviewed
Aug 10, 2025
...oop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
Outdated
Show resolved
Hide resolved
Comment on lines
652
to
656
| /** | ||
| * Test that lastCheckpointTime is correctly updated at each checkpoint | ||
| */ | ||
| @Test(timeout = 300000) | ||
| public void testLastCheckpointTime() throws Exception { |
There was a problem hiding this comment.
This test is passing with your prod change as well for me
lfxy
force-pushed
the
feature/HDFS-17815
branch
from
e525d98 to
4084cfa
Compare
|
💔 -1 overall
This message was automatically generated. |
lfxy
force-pushed
the
feature/HDFS-17815
branch
from
4084cfa to
a4e83fc
Compare
|
🎊 +1 overall
This message was automatically generated. |
lfxy
force-pushed
the
feature/HDFS-17815
branch
from
a4e83fc to
de1bdae
Compare
|
🎊 +1 overall
This message was automatically generated. |
lfxy
force-pushed
the
feature/HDFS-17815
branch
from
de1bdae to
89899c4
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
@Hexiaoqiao @ayushtkn Hi, Do you have any other suggestions? |
Hexiaoqiao
approved these changes
Aug 18, 2025
There was a problem hiding this comment.
LGTM. +1. Let's wait if @ayushtkn has anymore comments here. Thanks again.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The capacity of Our hdfs federation cluster are more then 500 PB, with one NS containing over 600 million files. Once checkpoint takes nearly two hours.
We discover checkpoint frequently failures due to fail to put the fsimage to the active Namenode, leading to repeat checkpoints. We configured dfs.recent.image.check.enabled=true. After debug, the reason is the standby NN updates the lastCheckpointTime use the start time of checkpoint, rather than the end time. In our cluster, the lastCheckpointTime of the standby node is approximately 80 minutes ahead of the lastCheckpointTime of the active NN.
When the checkpoint interval in standby NN exceeds dfs.namenode.checkpoint.period, the next checkpoint is performed. Because the active NN's lastCheckpointTime is later than standby NN's, the interval is less than dfs.namenode.checkpoint.period, and the putting fsimage is been rejected, causing the checkpoint to fail and retried.