Add a defer flag to iscsi target/extent/targetextent create/update/delete

[jbendtsen]

Add a defer flag to iscsi target/extent/targetextent create/update/delete. This results in a measureable speedup in iscsi performance: roughly 3.75x faster per create/delete cycle. This commit is part of the effort to add a TrueNAS driver to Incus.

This is BEFORE

jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --debug --config fangtooth share iscsi create --target-prefix=incus dozer/jacks-incus/iscsi-vol2
[[["alias","in",["dozer/jacks-incus/iscsi-vol2"]]],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
tncdaemon connection time: 886.063µs
iscsi.target.query: 185.321513ms
resultsList was empty
iscsi.portal.query: 40.013578ms
iscsi.initiator.query: 33.608441ms
iscsi.target.create: 43.041363ms
defer not supported - was not aware
[{"alias":"dozer/jacks-incus/iscsi-vol2","groups":[{"initiator":1,"portal":1}],"name":"incus:dozer:jacks-incus:iscsi-vol2"}]
iscsi.target.create: 2.169660167s
[[["disk","in",["zvol/dozer/jacks-incus/iscsi-vol2"]]],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
iscsi.extent.query: 166.283708ms
resultsList was empty
defer not supported - aware
[{"disk":"zvol/dozer/jacks-incus/iscsi-vol2","name":"incus:dozer:jacks-incus:iscsi-vol2","ro":false}]
iscsi.extent.create: 390.085498ms
[[],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
iscsi.targetextent.query: 40.775749ms
defer not supported - aware
[{"extent":12233,"lunid":0,"target":12255}]
iscsi.targetextent.create: 2.187420955s
created	dozer/jacks-incus/iscsi-vol2
undoIscsiCreateList

real	0m5.304s
user	0m0.001s
sys	0m0.010s
jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --debug --config fangtooth share iscsi delete --target-prefix=incus dozer/jacks-incus/iscsi-vol2
[[["alias","in",["dozer/jacks-incus/iscsi-vol2"]]],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
tncdaemon connection time: 888.107µs
iscsi.target.query: 160.478358ms
[[["name","in",["dozer/jacks-incus/iscsi-vol2"]]],{"extra":{"flat":false,"properties":[],"retrieve_children":true,"user_properties":false}}]
pool.dataset.query: 49.809661ms
iscsi.target.delete: 40.862491ms
defer not supported - was not aware
[12255,true,true]
iscsi.target.delete: 14.918416722s
deleted	dozer/jacks-incus/iscsi-vol2
undoIscsiDeleteList

real	0m15.220s
user	0m0.002s
sys	0m0.009s

So, 5.3s to create a share. 15.22s to delete it.
And then the same tool, when running against a “defer” enhanced middleware.

jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --debug --config testing share iscsi create puddle/defer-testing/vol
[[["alias","in",["puddle/defer-testing/vol"]]],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
tncdaemon connection time: 368.361µs
iscsi.target.query: 87.970319ms
resultsList was empty
iscsi.portal.query: 33.123852ms
iscsi.initiator.query: 30.907891ms
iscsi.target.create: 209.844556ms
[[["disk","in",["zvol/puddle/defer-testing/vol"]]],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
iscsi.extent.query: 74.20844ms
resultsList was empty
iscsi.extent.create: 210.260233ms
[[],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
iscsi.targetextent.query: 51.338144ms
iscsi.targetextent.create: 139.465783ms
created	puddle/defer-testing/vol
undoIscsiCreateList
reloading iscsitarget
["iscsitarget",{}]
service.reload: 1.079414432s
{"jsonrpc": "2.0", "result": true, "id": 24}

real	0m1.934s
user	0m0.003s
sys	0m0.008s
jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --debug --config testing share iscsi delete puddle/defer-testing/vol
[[["alias","in",["puddle/defer-testing/vol"]]],{"extra":{"flat":false,"properties":null,"retrieve_children":false,"user_properties":false}}]
tncdaemon connection time: 440.288µs
iscsi.target.query: 119.316894ms
[[["name","in",["puddle/defer-testing/vol"]]],{"extra":{"flat":false,"properties":[],"retrieve_children":true,"user_properties":false}}]
pool.dataset.query: 49.232517ms
iscsi.target.delete: 2.788363735s
deleted	puddle/defer-testing/vol
undoIscsiDeleteList
reloading iscsitarget
["iscsitarget",{}]
service.reload: 577.556624ms
{"jsonrpc": "2.0", "result": true, "id": 28}

real	0m3.552s
user	0m0.002s
sys	0m0.009s
jack@jack-desktop:~/truenas/truenas_incus_ctl$ 

1.934s to create, and 3.5s to delete.

That makes deleting 4.3x faster, and creating 2.74x faster.
Here’s a clearer run without all the debug:

jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --config fangtooth share iscsi create --target-prefix=incus dozer/jacks-incus/iscsi-vol2
created	dozer/jacks-incus/iscsi-vol2

real	0m6.210s
user	0m0.003s
sys	0m0.008s
jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --config fangtooth share iscsi delete --target-prefix=incus dozer/jacks-incus/iscsi-vol2
deleted	dozer/jacks-incus/iscsi-vol2

real	0m16.167s
user	0m0.002s
sys	0m0.008s
jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --config testing share iscsi create puddle/defer-testing/vol
created	puddle/defer-testing/vol

real	0m3.403s
user	0m0.003s
sys	0m0.008s
jack@jack-desktop:~/truenas/truenas_incus_ctl$ time sudo ./truenas_incus_ctl --config testing share iscsi delete puddle/defer-testing/vol
deleted	puddle/defer-testing/vol

real	0m3.233s
user	0m0.002s
sys	0m0.008s
jack@jack-desktop:~/truenas/truenas_incus_ctl$ 

Showing 5x faster deletes.

Here’s where the rubber hits the road.
Previously, when running Incus’s basic_usage integration suite against a TrueNAS host... they’d take about 30 minutes.

==> TEST DONE: test_basic_usage (1752s)
+ TEST_RESULT=success

But with the changes to defer reload until the end of the procedure its under 10 minutes.

==> TEST DONE: test_basic_usage (563s)
+ TEST_RESULT=success

That is a 3.1x improvement in the test run time… and not all of the run time is due to create/delete overhead, but it is very significant.
The full standalone Incus test suite takes about 5hrs to run against a TrueNAS. We’re hoping they reduce significantly with these changes.

@kmoore134
@william-gr