HostProcess containers with Hypervisor Isolation #2531

rawahars · 2025-10-07T20:58:24Z

Summary

This PR adds support for running HostProcess containers with hypervisor isolation. These HPCs will run as privileged processes within the UVM. Few key aspects of this feature are-

This feature will be applicable for WS2025 and later.
A pod needs to be marked privileged by passing in annotation 'microsoft.com/hostprocess-container'. This ensures that the UVM can support HPCs alongside normal containers.
Any container which has 'microsoft.com/hostprocess-container' annotation will be launched as HostProcess Container within the UVM.
Other containers will be launched as normal process-isolated containers within the UVM.
GCS will create the HPCs based on the specs. The behavior is similar to existing HPCs for process-isolated cases.
Annotations for Inherit user will lead to the HPC running as System user within UVM.
Annotation for custom rootfs will lead to the container filesystem being virtualized in a directory with that custom name. If not provided, C:\hpc is used by default.

Primary changes

Refactored pod sandbox logic to use a centralized updateConfigForHostProcessContainer method for handling privileged container configuration and annotation passthrough. This method enforces correct sandbox/container combinations and properly sets user and annotation values for HostProcess containers.
Added new fields to the container and storage schema (ContainerType and ContainerRootPath) to support HostProcess container identification and custom root filesystem locations. These are set during container creation when an isolated job container is detected.
Introduced handleProcessArgsForIsolatedJobContainer to ensure that isolated job containers have their process arguments correctly prefixed with cmd /c when necessary, supporting both CommandLine and Args fields. This logic is invoked during container and exec creation.
Improved container creation flow to distinguish between job containers running on the host and those inside a UVM, ensuring the right creation path is used.

Testing Enhancements

Added comprehensive unit tests for both HostProcess container configuration updates and process argument handling, covering a wide range of scenarios and edge cases to ensure robust behavior.
Tests have been added to cover most of the newly added logic.

internal/hcsoci/hcsdoc_wcow.go

jterry75 · 2025-10-31T16:59:12Z

internal/oci/util.go

+
+// IsIsolatedJobContainer checks if `s` is asking for a Windows job container.
+// This request is for running HPC within guest.
+func IsIsolatedJobContainer(s *specs.Spec) bool {


I'd prefer we stick with the above. IsJobContainer returns true when Linux == nil && Windows != nil && annotation[HPC] == true

IsIsolated is already true above based on setting HyperV.

So why can't you just do if IsJobContainer(s) && IsIsolated(s) of you want both?

It's just for being explicitness. It just increases the length of each if condition.
If you don't have major concerns, we can leave this be. It makes the code readability better IMO.

jterry75 · 2025-10-31T17:00:12Z

internal/hcsoci/hcsdoc_wcow.go

+
+		// If the customer specified a custom rootfs path then use that instead of default c:\hpc.
+		if customRootFsPath, ok := coi.Spec.Annotations[annotations.HostProcessRootfsLocation]; ok {
+			v2Container.Storage.PrivilegedContainerRootPath = customRootFsPath


This path needs to be sanitized right?

Added a method to sanitize as well protect against mount attacks.

internal/hcs/schema2/storage.go

jterry75 · 2025-10-31T17:03:49Z

cmd/containerd-shim-runhcs-v1/task_hcs_test.go

+			taskSpec := &specs.Spec{Annotations: tt.taskAnnotations}
+			// Ensure initial Username is set if provided
+			if tt.initialUsername != "" {
+				tt.specs.User.Username = tt.initialUsername


You already set this in the process spec above for each test case. Why override it? I dont get it

Nice catch! I had initially planned on test cases to change the usernames via this parameter. I will remove this in the revision.

jterry75 · 2025-10-31T17:05:58Z

cmd/containerd-shim-runhcs-v1/task_hcs.go

+	// For Isolated Job containers, we need special handling wherein if the command line
+	// is not specified, then we add 'cmd /C' by default.
+	if oci.IsIsolatedJobContainer(s) {
+		handleProcessArgsForIsolatedJobContainer(s, s.Process)


Why is this part of it special for HPC? I dont get it, this is true for all Windows containers right? At this point its just another container

For some reason, if we do not pass the command line to HCS, the call fails.
HPCs don't follow the exact same path as normal Argon containers. Argons are created as full Silos and might have a default shell but HPCs are partial Silos and seemingly don't have one.

jterry75 · 2025-10-31T17:07:51Z

cmd/containerd-shim-runhcs-v1/pod.go

 	var parent *uvm.UtilityVM
 	var lopts *uvm.OptionsLCOW
-	if oci.IsIsolated(s) {
+	if oci.IsIsolated(s) || oci.IsIsolatedJobContainer(s) {


You dont need to add this. We always make a UVM for any "IsIsolated" so you dont need to also handle the case for IsIsolatedJobContainer. That also "IsIsolated"

That's right. This was not needed per se as we will always go through the if condition. However, I added this for being explicit. I can remove it.

This commit adds support for running HostProcess containers with hypervisor isolated workloads. Therefore, these HPCs will run as privileged processes within the UVM. Some of the key aspects of this feature are- - A pod needs to be marked privileged by passing in annotation 'microsoft.com/hostprocess-container' - Any container which has 'microsoft.com/hostprocess-container' annotation will be launched as HostProcess Container. - Other containers will be launched as normal process-isolated containers within the UVM - HPC within UVM will not have access to any network since the sandbox (UVM) does not have any network. - GCS will create the HPCs based on the specs. The behaviour is similar to existing HPCs for process-isolated case. - Annotations for Inherit user will lead to the HPC running as System user within UVM. - Annotation for custom rootfs will lead to the container filesystem being virtualized in a directory with that custom name. If not provided, `C:\hpc` is used by default. Signed-off-by: Harsh Rawat <[email protected]>

This commit adds thorough unit tests for: - IsJobContainer - IsIsolatedJobContainer These tests explicitly verify the expected behavior for Windows HostProcess containers across isolation modes and OS targets. Signed-off-by: Harsh Rawat <[email protected]>

This commit adds thorough unit tests for the updateConfigForHostProcessContainer method. We test the following functionality- - Reject privileged container in unprivileged sandbox (isolated HPC) — return error when container requests microsoft.com/hostprocess-container=true but the pod lacks it. - Allow privileged container in privileged sandbox (isolated HPC) — ensure no unexpected mutations when no passthrough annotations are present. - Block normal process-isolated container in process-isolated HPC pod — enforce constraint that all containers in a process HPC pod must also be job containers (error on mixing). - Allow normal hypervisor-isolated container in hypervisor-isolated HPC pod. - Passthrough annotations to privileged containers — propagate HostProcessInheritUser and HostProcessRootfsLocation from pod → container for: hypervisor-isolated HPC pods with privileged containers, and process-isolated HPC pods with privileged containers. - No passthrough for normal containers — verify annotations aren’t propagated when the container is non-privileged. - Set SYSTEM user for hypervisor-isolated privileged containers — when HostProcessInheritUser=true, set Process.User.Username to NT AUTHORITY\SYSTEM. - Do not change user for normal containers — ensure container user remains unchanged even if the pod has HostProcessInheritUser=true. - Force HostProcessInheritUser to "false" for normal containers — in hypervisor-isolated privileged pods, if a non-privileged container sets the inherit annotation, flip it to "false". Signed-off-by: Harsh Rawat <[email protected]>

Signed-off-by: Harsh Rawat <[email protected]>

rawahars requested a review from a team as a code owner October 7, 2025 20:58

rawahars force-pushed the hpc_inboc_hcs_for_hypervisor_isolation branch from 76a7277 to b56ed8c Compare October 7, 2025 21:02

rawahars changed the title ~~Hpc inboc hcs for hypervisor isolation~~ HostProcess containers with Hypervisor Isolation Oct 7, 2025

jterry75 reviewed Oct 7, 2025

View reviewed changes

internal/hcsoci/hcsdoc_wcow.go Outdated Show resolved Hide resolved

rawahars force-pushed the hpc_inboc_hcs_for_hypervisor_isolation branch from b56ed8c to e1e1b82 Compare October 8, 2025 07:40

anmaxvl self-assigned this Oct 9, 2025

jterry75 reviewed Oct 31, 2025

View reviewed changes

internal/hcs/schema2/storage.go Show resolved Hide resolved

jterry75 reviewed Oct 31, 2025

View reviewed changes

rawahars force-pushed the hpc_inboc_hcs_for_hypervisor_isolation branch 2 times, most recently from 4eba3a9 to 6f6cae7 Compare October 31, 2025 21:55

rawahars added 8 commits November 23, 2025 00:12

Added unit tests for handleProcessArgsForIsolatedJobContainer

726bb24

Signed-off-by: Harsh Rawat <[email protected]>

Rectified the schema for custom rootfs for HCS HPCs

4da7f19

Signed-off-by: Harsh Rawat <[email protected]>

Rectified the logic for user inherit functionality for HPCs

507f9a7

Signed-off-by: Harsh Rawat <[email protected]>

Addressed PR feedback

5c861ad

Signed-off-by: Harsh Rawat <[email protected]>

Schema changes inline with HCS

056de18

Signed-off-by: Harsh Rawat <[email protected]>

rawahars force-pushed the hpc_inboc_hcs_for_hypervisor_isolation branch from 36a6efa to 056de18 Compare November 22, 2025 18:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HostProcess containers with Hypervisor Isolation #2531

HostProcess containers with Hypervisor Isolation #2531

Uh oh!

rawahars commented Oct 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

jterry75 Oct 31, 2025

Uh oh!

rawahars Oct 31, 2025

Uh oh!

jterry75 Oct 31, 2025

Uh oh!

rawahars Oct 31, 2025

Uh oh!

Uh oh!

jterry75 Oct 31, 2025

Uh oh!

rawahars Oct 31, 2025

Uh oh!

jterry75 Oct 31, 2025

Uh oh!

rawahars Oct 31, 2025

Uh oh!

jterry75 Oct 31, 2025

Uh oh!

rawahars Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HostProcess containers with Hypervisor Isolation #2531

Are you sure you want to change the base?

HostProcess containers with Hypervisor Isolation #2531

Uh oh!

Conversation

rawahars commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Primary changes

Testing Enhancements

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rawahars commented Oct 7, 2025 •

edited

Loading