Skip to content

xenopsd: set xen-platform-pci-bar-uc key in xenstore #6566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 15, 2025
Merged

xenopsd: set xen-platform-pci-bar-uc key in xenstore #6566

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions ocaml/xenopsd/lib/xenopsd.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ let default_vbd_backend_kind = ref "vbd"

let ca_140252_workaround = ref false

let xen_platform_pci_bar_uc = ref true

let action_after_qemu_crash = ref None

let additional_ballooning_timeout = ref 120.
Expand Down Expand Up @@ -207,6 +209,14 @@ let options =
, (fun () -> string_of_bool !ca_140252_workaround)
, "Workaround for evtchn misalignment for legacy PV tools"
)
; ( "xen-platform-pci-bar-uc"
, Arg.Bool (fun x -> xen_platform_pci_bar_uc := x)
, (fun () -> string_of_bool !xen_platform_pci_bar_uc)
, "Controls whether, when the VM starts in HVM mode, the Xen PCI MMIO used \
by grant tables is mapped as Uncached (UC, the default) or WriteBack \
(WB, the workaround). WB mapping could improve performance of devices \
using grant tables. This is useful on AMD platform only."
Comment on lines +217 to +218
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understood Roger's explanation, the current behaviour behaviour of both hvmloader and guest kernels is that they set the uncacheable bit for for the foreign mapping for AMD systems.
In the future, the hvmloader will add a shortcut to allow guest kernels to use write-back to the foreign mappings, (this is xen_platform_pci_bar_uc=0) but only some of the guest kernels will benefit from it (linux kernel when using memremap).

What I don't quite grasp is:

  • Does setting this to 1 have negative effects? This includes
    • Intel systems where this setting, according to the description, shouldn't change behaviour.
    • AMD systems with unpatched linux kernels in guests; or Windows guests

I'm asking this because it looks to me this is a per-guest setting, and logically it should be overrdable by the platform data coming from the guest template. Having this setting be per-host can be dangerous if the setting has negative side-effects since not all guest have the same kernel.

Alternatively, if this workaround doesn't have negative side-effects, it might be worthwhile to simply set it to 0 and have the free perf boost on some of the VMs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your understanding is good.

I didn't see any regression or negative effect in my tests (Intel or AMD with windows guests or linux guest without memremap patch). There is not effect on those configurations. But, of course, there are configurations I didn't test.

I agree with you. I'll work on the platform data to set this parameter.

Copy link
Contributor

@edwintorok edwintorok Jul 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my testing there is a massive performance improvement on AMD when this is set to 0.
Setting it to 1 is a noop, because that is what Xen already sets it to.

On Intel this is mostly a noop because Xen already sets it to WB (UC=0) via another mechanism.
I've tested Intel and UC=0 has no measurable effect, so it is better to just set it to UC=0 to eliminate differences between how different CPU vendors are handled in Xen (Intel has the iPAT mechanism, but then that won't be needed anymore).

)
; ( "additional-ballooning-timeout"
, Arg.Set_float additional_ballooning_timeout
, (fun () -> string_of_float !additional_ballooning_timeout)
Expand Down
3 changes: 3 additions & 0 deletions ocaml/xenopsd/xc/domain.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,9 @@ let make ~xc ~xs vm_info vcpus domain_config uuid final_uuid no_sharept =
xs.Xs.writev (dom_path ^ "/bios-strings") vm_info.bios_strings ;
if vm_info.is_uefi then
xs.Xs.write (dom_path ^ "/hvmloader/bios") "ovmf" ;
xs.Xs.write
(dom_path ^ "/hvmloader/pci/xen-platform-pci-bar-uc")
(if !Xenopsd.xen_platform_pci_bar_uc then "1" else "0") ;
(* If a toolstack sees a domain which it should own in this state then the
domain is not completely setup and should be shutdown. *)
xs.Xs.write (dom_path ^ "/action-request") "poweroff" ;
Expand Down
6 changes: 6 additions & 0 deletions ocaml/xenopsd/xenopsd.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,9 @@ disable-logging-for=http tracing tracing_export
# time to wait for in-guest PV drivers to acknowledge a shutdown request
# before we conclude that the drivers have failed
# domain_shutdown_ack_timeout = 60

# Controls whether, when the VM starts in HVM mode, the Xen PCI MMIO used
# by grant tables is mapped as Uncached (UC, the default) or WriteBack
# (WB, the workaround). WB mapping could improve performance of devices
# using grant tables. This is useful on AMD platform only.
# xen-platform-pci-bar-uc=true
Loading