Skip to content

Commit 915a4ca

Browse files
psafontpsafont
authored
RFC: update better VM revert proposal (#6696)
This proposal is 11 years old and the architecture of xapi has changed enough that the proposal needs changing. In particular, SMAPIv3 has been introduced which means that there's an xapi storage interface shared between the two SMAPI backend, and that the fallback the logic that was done at SMAPI level now must be done at the level of xapi. I've also elected to further restrict the fallback behaviour to VM reverts, VDI revert will require backend support: it reduces the chances of failures during revert resulting in wrong fields in the database. I'm still unsure about exposing VDI.revert, I would avoid it if at all possible, maybe as an internal or hidden API? It might be useful to exposed to use it in quicktests, like it was done in the initial prototype.
2 parents e2ba3d7 + f9b12db commit 915a4ca

File tree

2 files changed

+150
-108
lines changed

2 files changed

+150
-108
lines changed

‎doc/content/design/snapshot-revert.md‎

Lines changed: 92 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,85 +1,128 @@
11
---
2-
title: Improving snapshot revert behaviour
2+
title: Better VM revert
33
layout: default
44
design_doc: true
5-
revision: 1
5+
revision: 2
66
status: confirmed
77
---
88

9-
Currently there is a XenAPI `VM.revert` which reverts a "VM" to the state it
10-
was in when a VM-level snapshot was taken. There is no `VDI.revert` so
11-
`VM.revert` uses `VDI.clone` to change the state of the disks.
9+
## Overview
1210

13-
The use of `VDI.clone` has the side-effect of changing VDI refs and uuids.
14-
This causes the following problems:
11+
XenAPI allows users to rollback the state of a VM to a previous state, which is
12+
stored in a snapshot, using the call `VM.revert`. Because there is no
13+
`VDI.revert` call, `VM.revert` uses `VDI.clone` on the snapshot to duplicate
14+
the contents of that disk and then use the new clone as the storage for the VM.
1515

16-
- It is difficult for clients
17-
such as [Apache CloudStack](http://cloudstack.apache.org) to keep track
18-
of the disks it is actively managing
19-
- VDI snapshot metadata (`VDI.snapshot_of` et al) has to be carefully
20-
fixed up since all the old refs are now dangling
16+
Because `VDI.clone` creates new VDI refs and uuids, some problematic
17+
behaviours arise:
2118

22-
We will fix these problems by:
19+
- Clients such as
20+
[Apache CloudStack](http://cloudstack.apache.org) need to include complex
21+
logic to keep track of the disks they are actively managing
22+
- Because the snapshot is cloned and the original vdi is deleted, VDI
23+
references to the VDI become invalid, like `VDI.snapshot_of`. This means
24+
that the database has to be combed through to change these references.
25+
Because the database doesn't support transactions this operation is not atomic
26+
and can produce inconsistent database states.
2327

24-
1. adding a `VDI.revert` to the SMAPIv2 and calling this from `VM.revert`
25-
2. defining a new SMAPIv1 operation `vdi_revert` and a corresponding capability
26-
`VDI_REVERT`
27-
3. the Xapi implementation of `VDI.revert` will first try the `vdi_revert`,
28-
and fall back to `VDI.clone` if that fails
29-
4. implement `vdi_revert` for common storage types, including File and LVM-based
30-
SRs.
28+
Additionally, some filesystems support snapshots natively, doing the clone
29+
procedure is much costlier than allowing the filesystem to do the revert.
3130

32-
XenAPI changes
33-
--------------
31+
We will fix these problems by:
3432

35-
We will add the function `VDI.revert` with arguments:
33+
- introducing the new feature `VDI_REVERT` in SM interface (`xapi_smint`). This
34+
allows backends to advertise that they support the new functionality
35+
- defining a new storage operation `VDI.revert` in storage_interface, which is
36+
gated by the feature `VDI_REVERT`
37+
- proxying the storage operation to SMAPIv3 and SMAPv1 backends accordingly
38+
- adding `VDI.revert` to xapi_vdi which will call the storage operation if the
39+
backend advertises it, and fallback to the previous method that uses
40+
`VDI.clone` if it doesn't advertise it, or issues are detected at runtime
41+
that prevent it
42+
- changing the Xapi implementation of `VM.revert` to use `VDI.revert`
43+
- implement `vdi_revert` for common storage types, including File and LVM-based
44+
SRs
45+
- adding unit and quick tests to xapi to test that `VM.revert` does not regress
46+
47+
## Current VM.revert behaviour
48+
49+
The code that reverts the state of storage is located in
50+
[update_vifs_vbds_vgpus_and_vusbs](https://github.com/xapi-project/xen-api/blob/bc0ba4e9dc8dc4b85b7cbdbf3e0ba5915b4ad76d/ocaml/xapi/xapi_vm_snapshot.ml#L211).
51+
The steps it does is:
52+
1. destroys the VM's VBDs (both disks and CDs)
53+
2. destroys the VM's VDI (disks only), referenced by the snapshot's VDIs using
54+
`snapshot_of`; as well as the suspend VDI.
55+
3. clones the snapshot's VDIs (disks and CDs), if one clone fails none remain.
56+
4. searches the database for all `snapshot_of` references to the deleted VDIs
57+
and replaces them with the referenced of the newly cloned snapshots.
58+
5. clones the snapshot's resume VDI
59+
6. creates copies of all the cloned VBDs and associates them with the cloned VDIs
60+
7. assigns the new resume VDI to the VM
61+
62+
## XenAPI design
63+
64+
### API
65+
66+
The function `VDI.revert` will be added, with arguments:
3667

3768
- in: `snapshot: Ref(VDI)`: the snapshot to which we want to revert
3869
- in: `driver_params: Map(String,String)`: optional extra parameters
39-
- out: `Ref(VDI)` the new VDI
70+
- out: `Ref(VDI)` reference to the new VDI with the reverted contents
71+
72+
The function will extract the reference of VDI whose contents need to be
73+
replaced. This is the snapshot's `snapshot_of` field, then it will call the
74+
storage function function `VDI.revert` to have its contents replaced with the
75+
snapshot's. The VDI object will not be modified, and the reference returned is
76+
the VDI's original reference.
77+
If anything impedes the successful finish of an in-place revert, like the SM
78+
backend does not advertising the feature `VDI_REVERT`, not implement the
79+
feature, or the `snapshot_of` reference is invalid; an exception will be
80+
raised.
4081

41-
The function will look up the VDI which this is a `snapshot_of`, and change
42-
the VDI to have the same contents as the snapshot. The snapshot will not be
43-
modified. If the implementation is able to revert in-place, then the reference
44-
returned will be the VDI this is a `snapshot_of`; otherwise it is a reference
45-
to a fresh VDI (created by the `VDI.clone` fallback path)
82+
### Xapi Storage
4683

47-
References:
84+
The function `VDI.revert` is added, with the following arguments:
4885

49-
- @johnelse's [pull request](https://github.com/xapi-project/xen-api/pull/1963)
50-
which implements this
86+
- in: `dbg`: the task identifier, useful for tracing
87+
- in: `sr`: SR where the new VDI must be created
88+
- in: `snapshot_info`: metadata of the snapshot, the contents of which must be
89+
made available in the VDI indicated by the `snapshot_of` field
5190

52-
SMAPIv1 changes
53-
---------------
91+
#### SMAPIv1
5492

55-
We will define the function `vdi_revert` with arguments:
93+
The function `vdi_revert` is defined with the following arguments:
5694

5795
- in: `sr_uuid`: the UUID of the SR containing both the VDI and the snapshot
58-
- in: `vdi_uuid`: the UUID of the snapshot whose contents should be duplicated
59-
- in: `target_uuid`: the UUID of the target whose contents should be replaced
96+
- in: `vdi_uuid`: the UUID of the snapshot whose contents must be duplicated
97+
- in: `target_uuid`: the UUID of the target whose contents must be replaced
6098

6199
The function will replace the contents of the `target_uuid` VDI with the
62100
contents of the `vdi_uuid` VDI without changing the identify of the target
63101
(i.e. name-label, uuid and location are guaranteed to remain the same).
64102
The `vdi_uuid` is preserved by this operation. The operation is obvoiusly
65103
idempotent.
66104

67-
Xapi changes
68-
------------
105+
#### SMAPIv3
69106

70-
Xapi will
107+
In an analogous way to SMAPIv1, the function `Volume.revert` is defined with the
108+
following arguments:
71109

72-
- use `VDI.revert` in the `VM.revert` code-path
73-
- expose a new `xe vdi-revert` CLI command
74-
- implement the `VDI.revert` by calling the SMAPIv1 function and falling back
75-
to `VDI.clone` if a `Not_implemented` exception is thrown
110+
- in: `dbg`: the task identifier, useful for tracing
111+
- in: `sr`: the UUID of the SR containing both the VDI and the snapshot
112+
- in: `snapshot`: the UUID of the snapshot whose contents must be duplicated
113+
- in: `vdi`: the UUID of the VDI whose contents must be replaced
76114

77-
References:
115+
### Xapi
78116

79-
- @johnelse's [pull request](https://github.com/xapi-project/xen-api/pull/1963)
117+
- add the capability `VDI_REVERT` so backends can advertise it
118+
- use `VDI.revert` in the `VM.revert` after the VDIs have been destroyed, and
119+
before the snapshot's VDIs have been cloned. If any of the reverts fail
120+
because a `Not_implemented` exception is thrown, or the `snapshot_of`
121+
contains an invalid reference, add the affected VDIs to the list to be cloned
122+
and recovered, using the existing method
123+
- expose a new `xe vdi-revert` CLI command
80124

81-
SM changes
82-
----------
125+
## SM changes
83126

84127
We will modify
85128

@@ -92,8 +135,7 @@ We will modify
92135
snapshot/clone machinery
93136
- LVHDoISCSISR.py and LVHDoHBASR.py to advertise the `VDI_REVERT` capability
94137

95-
Prototype code
96-
==============
138+
# Prototype code from the previous proposal
97139

98140
Prototype code exists here:
99141

‎doc/content/xapi/storage/_index.md‎

Lines changed: 58 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -26,57 +26,60 @@ D --> F[SMAPIv3 plugins]
2626

2727
## SMAPIv1
2828

29-
These are the files related to SMAPIv1 in `xen-api/ocaml/xapi/`:
30-
31-
- sm.ml: OCaml "bindings" for the SMAPIv1 Python "drivers" (SM)
32-
- sm_exec.ml:
33-
support for implementing the above "bindings". The
34-
parameters are converted to XML-RPC, passed to the relevant python
35-
script ("driver"), and then the standard output of the program is
36-
parsed as an XML-RPC response (we use
37-
`xen-api-libs-transitional/http-svr/xMLRPC.ml` for parsing XML-RPC).
29+
These are the files related to SMAPIv1 in [`/ocaml/xapi/`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi):
30+
31+
- [`sm.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/sm.ml):
32+
OCaml "bindings" for the SMAPIv1 Python "drivers" (SM)
33+
- [`sm_exec.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/sm_exec.ml):
34+
support for implementing the above "bindings".
35+
The parameters are converted to XML-RPC, passed to the relevant python script ("driver"),
36+
and then the standard output of the program is parsed as an XML-RPC response (we use
37+
[`ocaml/libs/http-lib/xMLRPC.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/libs/http-lib/xMLRPC.ml)
38+
for parsing XML-RPC).
3839
When adding new functionality, we can modify `type call` to add parameters,
3940
but when we don't add any common ones, we should just pass the new
4041
parameters in the args record.
41-
- `smint.ml`: Contains types, exceptions, ... for the SMAPIv1 OCaml
42-
interface
43-
- `storage_smapiv1_wrapper.ml`: A state machine for SMAPIv1 operations. It computes
44-
the required actions to reach the desired state from the current state.
45-
- `storage_smapiv1.ml`: Contains the actual translation of SMAPIv2 calls to SMAPIv1
46-
calls, by calling the bindings provided in sm.ml.
42+
- [`smint.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/smint.ml):
43+
Contains types, exceptions, ... for the SMAPIv1 OCaml interface.
44+
- [`storage_smapiv1_wrapper.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1_wrapper.ml):
45+
The [`Wrapper`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1_wrapper.ml#L360)
46+
module wraps a SMAPIv2 server (`Server_impl`) and takes care of
47+
locking and datapaths (in case of multiple connections (=datapaths)
48+
from VMs to the same VDI, using a state machine for SMAPIv1 operations.
49+
It will use the superstate computed by the
50+
[`vdi_automaton.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/vdi_automaton.ml)
51+
in xapi-idl) to compute the required actions to reach the desired state from the current one.
52+
It also implements some functionality, like the `DP` module, that is not implemented in lower layers.
53+
- [`storage_smapiv1.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1.ml):
54+
a SMAPIv2 server that translates SMAPIv2 calls to SMAPIv1 ones, by calling
55+
[`ocaml/xapi/sm.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/sm.ml).
56+
It calls passes the XML-RPC requests as the first command-line argument to the
57+
corresponding Python script, which returns an XML-RPC response on standard
58+
output.
4759

4860
## SMAPIv2
4961

5062
These are the files related to SMAPIv2, which need to be modified to
5163
implement new calls:
5264

53-
- [xcp-idl/storage/storage\_interface.ml](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/storage_interface.ml):
65+
- [`ocaml/xapi-idl/storage/storage_interface.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/storage_interface.ml):
5466
Contains the SMAPIv2 interface
55-
- [xcp-idl/storage/storage\_skeleton.ml](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/storage_skeleton.ml):
67+
- [`ocaml/xapi-idl/storage/storage_skeleton.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/storage_skeleton.ml):
5668
A stub SMAPIv2 storage server implementation that matches the
5769
SMAPIv2 storage server interface (this is verified by
58-
[storage\_skeleton\_test.ml](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/storage_skeleton_test.ml)),
70+
[`storage_skeleton_test.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/storage_skeleton_test.ml)),
5971
each of its function just raise a `Storage_interface.Unimplemented`
6072
error. This skeleton is used to automatically fill the unimplemented
6173
methods of the below storage servers to satisfy the interface.
62-
- [xen-api/ocaml/xapi/storage\_smapiv1.ml](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1.ml):
63-
a SMAPIv2 server that does SMAPIv2 -> SMAPIv1 translation.
64-
It passes the XML-RPC requests as the first command-line argument to the
65-
corresponding Python script, which returns an XML-RPC response on standard
66-
output.
67-
- [xen-api/ocaml/xapi/storage_smapiv1_wrapper.ml](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1_wrapper.ml):
68-
The [Wrapper](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1_wrapper.ml#L360)
69-
module wraps a SMAPIv2 server (Server\_impl) and takes care of
70-
locking and datapaths (in case of multiple connections (=datapaths)
71-
from VMs to the same VDI, it will use the superstate computed by the
72-
[Vdi_automaton](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-idl/storage/vdi_automaton.ml)
73-
in xcp-idl). It also implements some functionality, like the `DP`
74-
module, that is not implemented in lower layers.
75-
- [xen-api/ocaml/xapi/storage\_mux.ml](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_mux.ml):
74+
- [`ocaml/xapi/storage_mux.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_mux.ml):
7675
A SMAPIv2 server, which multiplexes between other servers. A
7776
different SMAPIv2 server can be registered for each SR. Then it
7877
forwards the calls for each SR to the "storage plugin" registered
7978
for that SR.
79+
- [`ocaml/xapi/storage_smapiv1_wrapper.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1_wrapper.ml):
80+
Implements a state machine to compute SMAPIv1 actions needed to reach the desired state, see [SMAPIv1](#smapiv1).
81+
- [`ocaml/xapi/storage_smapiv1.ml`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi/storage_smapiv1.ml):
82+
Translates the SMAPIv2 calls to SMAPIv1, see [SMAPIv1](#smapiv1).
8083

8184
### How SMAPIv2 works:
8285

@@ -211,31 +214,28 @@ It also implements the `Policy` module from the SMAPIv2 interface.
211214

212215
## SMAPIv3
213216

214-
[SMAPIv3](https://xapi-project.github.io/xapi-storage/) has a slightly
215-
different interface from SMAPIv2.The
216-
[xapi-storage-script](https://github.com/xapi-project/xen-api/tree/v25.11.0/ocaml/xapi-storage-script)
217-
daemon is a SMAPIv2 plugin separate from xapi that is doing the SMAPIv2
218-
↔ SMAPIv3 translation. It keeps the plugins registered with xapi-idl
219-
(their message-switch queues) up to date as their files appear or
220-
disappear from the relevant directory.
217+
[SMAPIv3](https://xapi-project.github.io/xapi-storage/) has a slightly different interface from SMAPIv2.
218+
The
219+
[`xapi-storage-script`](https://github.com/xapi-project/xen-api/tree/v25.11.0/ocaml/xapi-storage-script)
220+
daemon is a SMAPIv2 plugin separate from xapi that is doing the SMAPIv2 ↔ SMAPIv3 translation.
221+
It keeps the plugins registered with xapi-idl (their message-switch queues)
222+
up to date as their files appear or disappear from the relevant directory.
221223

222224
### SMAPIv3 Interface
223225

224226
The SMAPIv3 interface is defined using an OCaml-based IDL from the
225-
[ocaml-rpc](https://github.com/mirage/ocaml-rpc) library, and is in this
226-
repo: <https://github.com/xapi-project/xapi-storage>
227+
[`ocaml-rpc`](https://github.com/mirage/ocaml-rpc) library, and is located at
228+
[`xen-api/ocaml/xapi-storage`](https://github.com/xapi-project/xen-api/tree/v25.11.0/ocaml/xapi-storage)
227229

228230
From this interface we generate
229231

230-
- OCaml RPC client bindings used in
231-
[xapi-storage-script](https://github.com/xapi-project/xapi-storage-script)
232-
- The [SMAPIv3 API
233-
reference](https://xapi-project.github.io/xapi-storage)
232+
- OCaml RPC client bindings used in `xapi-storage-script`
233+
- The
234+
[SMAPIv3 API reference](https://xapi-project.github.io/xapi-storage)
234235
- Python bindings, used by the SM scripts that implement the SMAPIv3
235236
interface.
236-
- These bindings are built by running "`make`" in the root
237-
[xapi-storage](https://github.com/xapi-project/xapi-storage),
238-
and appear in the` _build/default/python/xapi/storage/api/v5`
237+
- These bindings are built by running `make` at the root level,
238+
and appear in the` _build/default/ocaml/xapi-storage/python/xapi/storage/api/v5/`
239239
directory.
240240
- On a XenServer host, they are stored in the
241241
`/usr/lib/python3.6/site-packages/xapi/storage/api/v5/`
@@ -258,20 +258,20 @@ stored in subdirectories of the
258258
`/usr/libexec/xapi-storage-script/volume/` and
259259
`/usr/libexec/xapi-storage-script/datapath/` directories, respectively.
260260
When it finds a new datapath plugin, it adds the plugin to a lookup table and
261-
uses it the next time that datapath is required. When it finds a new volume
262-
plugin, it binds a new [message-switch](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-storage-script/main.ml#L2023) queue named after the plugin's
263-
subdirectory to a new server instance that uses these volume scripts.
261+
uses it the next time that datapath is required.
262+
When it finds a new volume plugin, it binds a new
263+
[`message-switch`](https://github.com/xapi-project/xen-api/blob/v25.11.0/ocaml/xapi-storage-script/main.ml#L2023)
264+
queue named after the plugin's subdirectory to a new server instance that uses these volume scripts.
264265

265266
To invoke a SMAPIv3 method, it executes a program named
266-
`<Interface name>.<function name>` in the plugin's directory, for
267-
example
267+
`<Interface name>.<function name>` in the plugin's directory,
268+
for example
268269
`/usr/libexec/xapi-storage-script/volume/org.xen.xapi.storage.gfs2/SR.ls`.
269270
The inputs to each script can be passed as command-line arguments and
270-
are type-checked using the generated Python bindings, and so are the
271-
outputs. The URIs of the SRs that xapi-storage-script knows about are
272-
stored in the `/var/run/nonpersistent/xapi-storage-script/state.db`
273-
file, these URIs can be used on the command line when an sr argument is
274-
expected.
271+
are type-checked using the generated Python bindings, and so are the outputs.
272+
The URIs of the SRs that xapi-storage-script knows about are stored in the
273+
 `/var/run/nonpersistent/xapi-storage-script/state.db` file,
274+
these URIs can be used on the command line when an sr argument is expected.
275275

276276
#### Registration of the various SMAPIv3 plugins
277277

0 commit comments

Comments
 (0)