Implements CVM disk backup #241
Conversation
| .open(BACKUP_LOCK_FILE) | ||
| .context("Failed to create backup lock file, there is another backup in progress")?; | ||
| // Run /dstack/hooks/pre-backup if it exists | ||
| let pre_backup_hook = "/dstack/hooks/pre-backup"; |
There was a problem hiding this comment.
How can we set up hooks for pre-backup and post-backup?
There was a problem hiding this comment.
Just write some shell commands into the files pre-backup/post-backup. It usaully need nothing todo. Some apps may want to flush their app data to disk before backup.
There was a problem hiding this comment.
Are they set up only by node operators? I'm wondering who can actually create these hook scripts.
There was a problem hiding this comment.
Are they set up only by node operators? I'm wondering who can actually create these hook scripts.
It's up to app defines the logic in the hooks. For example, flush their mysql database in the CVM. The hooks is put there for future use. I think we don't need to care about it at this time, until their is a use case appears.
| if backup_level == "full" { | ||
| // clear the bitmaps | ||
| let output = Command::new("qmpbackup") | ||
| .arg("--socket") | ||
| .arg(&qmp_socket) | ||
| .arg("cleanup") | ||
| .arg("--remove-bitmap") | ||
| .output() | ||
| .context("Failed to clear bitmaps")?; | ||
| if !output.status.success() { | ||
| let stderr = String::from_utf8_lossy(&output.stderr); | ||
| warn!("Failed to clear bitmaps for {id}: {stderr}"); | ||
| } | ||
| // Switch to new dir and symbol link the latest to it | ||
| let timestamp = chrono::Utc::now().format("%Y%m%dZ%H%M%S").to_string(); | ||
| let new_dir = backup_dir.join(×tamp); | ||
| fs::create_dir_all(&new_dir).context("Failed to create backup directory")?; | ||
| if fs::symlink_metadata(&latest_dir).is_ok() { | ||
| fs::remove_file(&latest_dir) | ||
| .context("Failed to remove latest directory link")?; | ||
| } | ||
| fs::os::unix::fs::symlink(×tamp, &latest_dir) | ||
| .context("Failed to create latest directory link")?; | ||
| } |
There was a problem hiding this comment.
I'm suggesting to add a checksum for each backup.
There was a problem hiding this comment.
Good point. The feature would be better to add to the qmpbackup command, since it controls the filename generation.
There was a problem hiding this comment.
Even better: can we verify the checksum before restoring?
There was a problem hiding this comment.
Even better: can we verify the checksum before restoring?
Surely should add verification in qmprestore, if checksum is added to qmpbackup.
kvinwang
force-pushed
the
disk-backup
branch
3 times, most recently
from
2ab1f2d to
f57a83b
Compare
This PR implements CVM disk backup in both built-in vmm and external script
dstack-backup.py.As the backup usually takes a very long time, it is recommended to run with the external script
dstack-backup.pyrather than vmm RPC.How to use:
dstack-backup.pyto crontab or similar task scheduler with a daily trigger. For example:It will backup the disks of running VMs to a configured directory.
Run
dstack-backup.py --helpto show the arguments.There are two kinds of backups,
fullandincremental. By default, it performs full backup weekly, and incremental backup daily.