Avoid duplicate 'WindowsAppRuntimeAutoInitializer.cpp' compilation

[mschofie]

Fixes #5395.

build/NuSpecs/WindowsAppSDK-Nuget-Native.AutoInitializer.targets attempts to add a ClCompile Item during the WindowsAppRuntimeAutoInitializer Target. In doing so it attempts to use the existing metadata by uttering %(PreprocessorDefinitions). This syntax is fine for ItemGroups outside of a Target, but within a Target, it causes the (implicit) task that creates the Item to be batched (documentation) - causing an Item with the Identity of WindowsAppRuntimeAutoInitializer.cpp to be added for every unique set of ClCompile.PreprocessorDefinitions. The duplicate ClCompile Items cause a cascade of subsequent errors.

The fix is to add the ClCompile in two steps:

1. Create the ClCompile without specifying any metadata.
2. Modify the metadata for the ClCompile, using a condition for '%(Identity)' == '$(WindowsAppRuntimeAutoInitializerPath)' to force a single task batch (for the Item we're looking for).

Thanks to @chwarr for help with this.

[chwarr]

LGTM

[Scottj1s]

@mschofie thanks for taking a stab at this. I don't think it's a complete fix, since we had to scrap a couple earlier attempts (#5685, #5782).

My concern is that any batching at all (whether PreprocessorDefinitions, AdditionalOptions, etc) just sets a trap for subsequent dynamic source additions.

Can we avoid batching entirely by applying macros to all sources after they've been dynamically added, and that supports Xaml generated sources, etc?
#5685 (comment)

[mschofie]

I don't think it's a complete fix, since we had to scrap a couple earlier attempts (#5685, #5782).

Thanks for linking to the earlier attempts. TBH, both of those attempts appear to have the same problem as the original bug, that my approach doesn't. For example, #5782 says:

Root Cause: When two targets in BeforeClCompileTargets both modify the same parameter (e.g., PreprocessorDefinitions), the second target to execute may cause duplicate entries to be added to the cl compile command due to parameter modification conflicts.

And that's not right - it's not that they're modifying the same parameter Item metadata, it's because they're creating multiple sets of Items when batched by %(PreprocessorDefinitions). Even if those fixes worked, they'd break if a consumer had ClCompile Items with different %(PreprocessorDefinitions). This PR uses the '%(Identity)' == '$(WindowsAppRuntimeAutoInitializerPath)' Condition to force a single Item batch.

Can we avoid batching entirely by applying macros to all sources after they've been dynamically added, and that supports Xaml generated sources, etc?
#5685 (comment)

I don't understand how that mitigates the batching. If there's two ClCompile Items with unique PreprocessorDefinitions metadata created during the evaluation phase, then you'd get two batches (each with one ClCompile Item in) if you were to access %(PreprocessorDefinitions) during an ItemGroup within a Target (without forcing a smaller batch first). Here's a trivial example that shows batching of 'static' Items:

<Project>
  <ItemGroup>
    <ClCompile Include="foo" PreprocessorDefinitions="FOO" />
    <ClCompile Include="bar" PreprocessorDefinitions="BAR" />
  </ItemGroup>

  <Target Name="Build">
    <Message Text="Batch: %(ClCompile.PreprocessorDefinitions) Items: %(ClCompile.Identity)" />
  </Target>
</Project>

If you're OK applying the preprocessor defines globally, then there's a much easier fix - simply decouple the ClCompile Item creation from setting the PreprocessorDefinitions metadata; i.e.

1. If any of the necessary properties are set; add the 'ClCompile' Item
2. Based on which are set, update the PreprocessorDefinitions in an ItemDefinitionGroup.

Property evaluation and ItemDefinitionGroup updates would have to happen outside of the 'Target', but I'm not sure that's a problem...? This is more 'order dependent' than this current PR, but we're in a .targets file, so I think it's OK. Thoughts? So, something like (I'm coding in an edit box; apologies for typos)

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <WindowsAppRuntimeAutoInitializerPath>$(MSBuildThisFileDirectory)..\..\include\WindowsAppRuntimeAutoInitializer.cpp</WindowsAppRuntimeAutoInitializerPath>
    <WindowsAppRuntimeAutoInitializerEnabled>false</WindowsAppRuntimeAutoInitializerEnabled>
    <WindowsAppRuntimeAutoInitializerEnabled Condition="'$(WindowsAppSdkBootstrapInitialize)'=='true'">true</WindowsAppRuntimeAutoInitializerEnabled>
    <WindowsAppRuntimeAutoInitializerEnabled Condition="'$(WindowsAppSdkDeploymentManagerInitialize)'=='true'">true</WindowsAppRuntimeAutoInitializerEnabled>
    <WindowsAppRuntimeAutoInitializerEnabled Condition="'$(WindowsAppSdkUndockedRegFreeWinRTInitialize)'=='true'">true</WindowsAppRuntimeAutoInitializerEnabled>
    <WindowsAppRuntimeAutoInitializerEnabled Condition="'$(WindowsAppSdkCompatibilityInitialize)'=='true'">true</WindowsAppRuntimeAutoInitializerEnabled>
  </PropertyGroup>

  <Target Name="WindowsAppRuntimeAutoInitializer" Condition=" '$(WindowsAppRuntimeAutoInitializerEnabled)'=='true' ">
    <ItemGroup>
      <ClCompile Include="$(WindowsAppRuntimeAutoInitializerPath)" />
    </ItemGroup>
  </Target>

  <ItemDefinitionGroup>
    <ClCompile>
      <PreprocessorDefinitions Condition="'$(WindowsAppSdkBootstrapInitialize)'=='true'">MICROSOFT_WINDOWSAPPSDK_AUTOINITIALIZE_BOOTSTRAP;%(PreprocessorDefinitions)</PreprocessorDefinitions>
      <PreprocessorDefinitions Condition="'$(WindowsAppSdkDeploymentManagerInitialize)'=='true'">MICROSOFT_WINDOWSAPPSDK_AUTOINITIALIZE_DEPLOYMENTMANAGER;%(PreprocessorDefinitions)</PreprocessorDefinitions>
      <PreprocessorDefinitions Condition="'$(WindowsAppSdkUndockedRegFreeWinRTInitialize)'=='true'">MICROSOFT_WINDOWSAPPSDK_AUTOINITIALIZE_UNDOCKEDREGFREEWINRT;%(PreprocessorDefinitions)</PreprocessorDefinitions>
      <PreprocessorDefinitions Condition="'$(WindowsAppSdkCompatibilityInitialize)'=='true'">MICROSOFT_WINDOWSAPPSDK_AUTOINITIALIZE_COMPATIBILITY;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
  </ItemDefinitionGroup>
</Project>

[Scottj1s]

And that's not right - it's not that they're modifying the same parameter Item metadata,

Agreed - that's not a correct RCA

I don't understand how that mitigates the batching. If there's two ClCompile Items with unique PreprocessorDefinitions metadata

Right - I'm suggesting we eliminate batching via speciation of PreprocessorDefinitions (or any other %BatchingMetadata) and just apply metadata globally to all sources - pch, module, whatever. I had suggested doing that during execution after all sources were sure to have been ClCompile included. But I'm fine with doing it during evaluation phase, via ItemDefinitionGroup - that's very appealing.

[mschofie]

Sorry, I don't understand your suggestion. How would you add the PreprocessorDefines? You'd still need to utter %(PreprocessorDefines) in an ItemGroup in a Target, wouldn't you?

At the minute, the only two paths I see are:

1. Force a single item batch. What this PR does.
2. Use an ItemDefinitionGroup. Which becomes a little order dependent (i.e. the WindowsAppSdkBootstrapInitialize (and other) properties would have to be set before build/NuSpecs/WindowsAppSDK-Nuget-Native.AutoInitializer.targets was evaluated).

I'll update this PR with the ItemDefinitionGroup after I've buddy tested it, and we can see which approach folks like the best.

[Scottj1s]

You'd still need to utter %(PreprocessorDefines) in an ItemGroup in a Target, wouldn't you?

Yes, you would - but that's not the issue per se. Having different definitions of PreprocessorDefines is what causes the batching. If we apply the same value to all sources, we get no batching and consequently no duplicate source compilation.

<Target Name="AddFoo"  BeforeTargets="ClCompile" AfterTargets="$(BeforeClCompileTargets);$(ComputeCompileInputsTargets);MakeDirsForCl" > 
  <ItemGroup>
    <ClCompile Include="Foo.Cpp">
        <AnyMetadataExceptPreprocessorDefinitions>this_is_safe</>
    </ClCompile>
    <ClCompile>
        <!-- Now, add macros to all sources, immediately before ClCompile, so no batching concerns -->
        <PreprocessorDefinitions>%PreprocessorDefinitions);FOO_DEFINES</PreprocessorDefinitions>
    </ClCompile>
  </ItemGroup>
</Target> 

[mschofie]

Having different definitions of PreprocessorDefines is what causes the batching.

Indeed, and I don't think we can prevent different definitions of PreprocessorDefines, because customers can easily write valid vcxproj files with different definitions of PreprocessorDefines. A customer can write the following in their .vcxproj:

<Project>
<ItemGroup>
    <ClCompile Include="CustomerSource.cpp" />
    <ClCompile Include="MoreCustomerSource.cpp" PreprocessorDefines="%(PreprocessorDefines);ENABLE_FANCY_MODE"/>
</ItemGroup>
</Project>

And then batching will occur in an ItemGroup (in a Target) that uses %(PreprocessorDefines).

[Scottj1s]

Having different definitions of PreprocessorDefines is what causes the batching.

Indeed, and I don't think we can prevent different definitions of PreprocessorDefines, because customers can easily write valid vcxproj files with different definitions of PreprocessorDefines. A customer can write the following in their .vcxproj:

<Project>
<ItemGroup>
    <ClCompile Include="CustomerSource.cpp" />
    <ClCompile Include="MoreCustomerSource.cpp" PreprocessorDefines="%(PreprocessorDefines);ENABLE_FANCY_MODE"/>
</ItemGroup>
</Project>

And then batching will occur in an ItemGroup (in a Target) that uses %(PreprocessorDefines).

That's true - we can't avoid that. But we can at least avoid self-inflicted issues like this one, by applying PreprocessorDefinitions globally. The interesting thing about this issue is that it's the result of our two targets both including ClCompile items with specific PreprocessorDefinitions: AddUndockedRegFreeWinRTCppDefines and WindowsAppRuntimeAutoInitializer. It's actually the second target that loses, by including its source file into separate batches that the first target has already created. So I'm fine with any solution that avoids our own (re)batching of existing batches.

[mschofie]

I've updated this PR with an implementation that takes the ItemDefinitionGroup-approach. This is - IMHO - a tidier approach. The only downside is that if customers are setting any of

• WindowsAppSdkBootstrapInitialize
• WindowsAppSdkDeploymentManagerInitialize
• WindowsAppSdkUndockedRegFreeWinRTInitialize
• WindowsAppSdkCompatibilityInitialize

within a 'Target', then it won't be honored, they need to be set during evaluation. This seems like a reasonable expectation to me.

Thoughts?

[mschofie]

I've addressed Christopher's comments - thanks! - and pushed a final version. TBH, I hadn't realized that the same pattern is used in multiple places across the build infrastructure - are folks with this PR going in without canonicalizing across the build?

Where should I document - release notes? - the shift that the listed properties have to be set at evaluation time now?

• WindowsAppSdkBootstrapInitialize
• WindowsAppSdkDeploymentManagerInitialize
• WindowsAppSdkUndockedRegFreeWinRTInitialize
• WindowsAppSdkCompatibilityInitialize

[Scottj1s]

I've addressed Christopher's comments - thanks! - and pushed a final version. TBH, I hadn't realized that the same pattern is used in multiple places across the build infrastructure - are folks with this PR going in without canonicalizing across the build?

Where should I document - release notes? - the shift that the listed properties have to be set at evaluation time now?

• WindowsAppSdkBootstrapInitialize
• WindowsAppSdkDeploymentManagerInitialize
• WindowsAppSdkUndockedRegFreeWinRTInitialize
• WindowsAppSdkCompatibilityInitialize

We've already documented that these properties should be set in the project file (that language is imprecise, but implies outside a target):
https://learn.microsoft.com/en-us/windows/apps/package-and-deploy/project-properties#project-properties
We just happened to evaluate them late, in a target.

[Scottj1s]

@mschofie Is this ready to complete? Have we confirmed it won't regress WinUI builds?

[qiutongMS]

@mschofie Is this ready to complete? Have we confirmed it won't regress WinUI builds?

After adding <PrecompiledHeader>NotUsing</PrecompiledHeader>, I successfully built the Foundation package and tested it on WinUI. Unfortunately, the same issue still occurs on arm64ec build. (The arm64 error seems unrelated to this.)
https://microsoft.visualstudio.com/DefaultCollection/WinUI/_build/results?buildId=132757478&view=results

[mschofie]

@mschofie Is this ready to complete? Have we confirmed it won't regress WinUI builds?

After adding <PrecompiledHeader>NotUsing</PrecompiledHeader>, I successfully built the Foundation package and tested it on WinUI. Unfortunately, the same issue still occurs on arm64ec build. (The arm64 error seems unrelated to this.) https://microsoft.visualstudio.com/DefaultCollection/WinUI/_build/results?buildId=132757478&view=results

Thanks for finding the 'PrecompiledHeader' 'NotUsing' regression - the WindowsML samples that I'm using to validate locally don't use precompiled headers, so I missed that - so sorry. Your questions:

1. Is it ready to complete? I think so.
2. Have we confirmed it won't regress WinUI builds? I'm not sure I really understand the question. The implementation that was settled on will change behavior, but it's moving to behavior that - as per Scott's comment - is documented as the expectation. Beyond that, I've been building the WindowsML samples locally, and assuming that the CI build is covering other bases. Is there more due diligence needed? A pointer to the details would be great.

Regarding "Unfortunately, the same issue still occurs on arm64ec build." - can you share a link to the logs that show this? I'm looking through the linked build, on the "BuildMUX Release_arm64ec" job, there's five stages that error:

1. Build WinUICppDesktopSampleApp (windows_build_container) - This has warnings that appear to be unrelated (link), and an error that is an Internal Compiler Error (link).

2. Build WinUIGallery (windows_build_container) - This has warnings that are unrelated (link), and an error (link) that's similar but - from the looks of things, unrelated:

   "C:\__w\1\s\Samples\WinUIGallery\WinUIGallery\WinUIGallery.sln" (Publish target) (1:2) ->
   "C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj" (Publish target) (3:6) ->
   (_ComputeAppxPackagePayload target) ->
       C:\__w\1\s\packages\microsoft.windows.sdk.buildtools.msix\1.5.20250417.1\build\Microsoft.Windows.SDK.BuildTools.MSIX.Packaging.targets(1364,5): error APPX1101: Payload contains two or more files with the same destination path 'WinUIGallery.pdb'. Source files:  [C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj]
       C:\__w\1\s\packages\microsoft.windows.sdk.buildtools.msix\1.5.20250417.1\build\Microsoft.Windows.SDK.BuildTools.MSIX.Packaging.targets(1364,5): error APPX1101:         C:\__w\1\s\BuildOutput\obj\arm64ecfre\Samples\WinUIGallery\WinUIGalleryUnitTests\net8.0-windows10.0.22621.0\win-x64\linked\WinUIGallery.pdb [C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj]
       C:\__w\1\s\packages\microsoft.windows.sdk.buildtools.msix\1.5.20250417.1\build\Microsoft.Windows.SDK.BuildTools.MSIX.Packaging.targets(1364,5): error APPX1101:         C:\__w\1\s\BuildOutput\obj\arm64ecfre\Samples\WinUIGallery\WinUIGallery\WinUIGallery\net8.0-windows10.0.22621.0\win-x64\linked\WinUIGallery.pdb [C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj]
   

3. Build NativeDX12DesktopSample (windows_build_container) - Unrelated warnings (link) and another Internal Compiler Error (link).

4. Build WinUICppIsland2SampleApp (windows_build_container) - Unrelated warnings (link) and another Internal Compiler Error (link).

5. Build WinUICppIslandsSampleApp (windows_build_container) - Unrelated warnings (link) and another Internal Compiler Error (link).

I'm going to go through the passing stages, to look for warnings, but a link would be super handy. Thanks!

I've rebased onto the latest main to see if that helps with the Internal Compiler Errors.

[Scottj1s]

2. Have we confirmed it won't regress WinUI builds? I'm not sure I really understand the question. The implementation that was settled on will change behavior, but it's moving to behavior that - as per Scott's comment - is documented as the expectation. Beyond that, I've been building the WindowsML samples locally, and assuming that the CI build is covering other bases. Is there more due diligence needed? A pointer to the details would be great.

This refers to the comment I made earlier about the Xaml compiler also dynamically adding generated sources. I expect your PR should be compatible, but want to confirm.

[qiutongMS]

@mschofie Is this ready to complete? Have we confirmed it won't regress WinUI builds?

After adding <PrecompiledHeader>NotUsing</PrecompiledHeader>, I successfully built the Foundation package and tested it on WinUI. Unfortunately, the same issue still occurs on arm64ec build. (The arm64 error seems unrelated to this.) https://microsoft.visualstudio.com/DefaultCollection/WinUI/_build/results?buildId=132757478&view=results

Thanks for finding the 'PrecompiledHeader' 'NotUsing' regression - the WindowsML samples that I'm using to validate locally don't use precompiled headers, so I missed that - so sorry. Your questions:

1. Is it ready to complete? I think so.
2. Have we confirmed it won't regress WinUI builds? I'm not sure I really understand the question. The implementation that was settled on will change behavior, but it's moving to behavior that - as per Scott's comment - is documented as the expectation. Beyond that, I've been building the WindowsML samples locally, and assuming that the CI build is covering other bases. Is there more due diligence needed? A pointer to the details would be great.

Regarding "Unfortunately, the same issue still occurs on arm64ec build." - can you share a link to the logs that show this? I'm looking through the linked build, on the "BuildMUX Release_arm64ec" job, there's five stages that error:

1. Build WinUICppDesktopSampleApp (windows_build_container) - This has warnings that appear to be unrelated (link), and an error that is an Internal Compiler Error (link).
2. Build WinUIGallery (windows_build_container) - This has warnings that are unrelated (link), and an error (link) that's similar but - from the looks of things, unrelated:

   "C:\__w\1\s\Samples\WinUIGallery\WinUIGallery\WinUIGallery.sln" (Publish target) (1:2) ->
   "C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj" (Publish target) (3:6) ->
   (_ComputeAppxPackagePayload target) ->
       C:\__w\1\s\packages\microsoft.windows.sdk.buildtools.msix\1.5.20250417.1\build\Microsoft.Windows.SDK.BuildTools.MSIX.Packaging.targets(1364,5): error APPX1101: Payload contains two or more files with the same destination path 'WinUIGallery.pdb'. Source files:  [C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj]
       C:\__w\1\s\packages\microsoft.windows.sdk.buildtools.msix\1.5.20250417.1\build\Microsoft.Windows.SDK.BuildTools.MSIX.Packaging.targets(1364,5): error APPX1101:         C:\__w\1\s\BuildOutput\obj\arm64ecfre\Samples\WinUIGallery\WinUIGalleryUnitTests\net8.0-windows10.0.22621.0\win-x64\linked\WinUIGallery.pdb [C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj]
       C:\__w\1\s\packages\microsoft.windows.sdk.buildtools.msix\1.5.20250417.1\build\Microsoft.Windows.SDK.BuildTools.MSIX.Packaging.targets(1364,5): error APPX1101:         C:\__w\1\s\BuildOutput\obj\arm64ecfre\Samples\WinUIGallery\WinUIGallery\WinUIGallery\net8.0-windows10.0.22621.0\win-x64\linked\WinUIGallery.pdb [C:\__w\1\s\Samples\WinUIGallery\WinUIGalleryUnitTests\WinUIGalleryUnitTests.csproj]
   

3. Build NativeDX12DesktopSample (windows_build_container) - Unrelated warnings (link) and another Internal Compiler Error (link).
4. Build WinUICppIsland2SampleApp (windows_build_container) - Unrelated warnings (link) and another Internal Compiler Error (link).
5. Build WinUICppIslandsSampleApp (windows_build_container) - Unrelated warnings (link) and another Internal Compiler Error (link).

I'm going to go through the passing stages, to look for warnings, but a link would be super handy. Thanks!

I've rebased onto the latest main to see if that helps with the Internal Compiler Errors.

Actually, the errors observed on WinUI are:
Error C1001: Internal compiler error.
Error LNK1000: Internal error during IMAGE::BuildImage.
I understand they may seem unrelated to the fix, but both the PRs Scott mentioned and other fixes I’ve tried (without submitting a PR) for this issue have triggered the same error during check-in or testing — specifically on these WinUI arm64ec sample apps: WinUICppDesktopSampleApp/NativeDX12DesktopSample/WinUICppIsland2SampleApp/WinUICppIslandsSampleApp. These errors haven’t appeared on other samples or platforms.
I haven’t yet identified the connection between these changes and the errors. I’m currently trying to build the affected WinUI samples locally to investigate what might be unique about it.