Fix issue #382 (multiple publish output files with same relative path)

[LucaMaccarini]

Fixes #382: On Windows, runtime files from backend packages are currently flattened during copy. This means that all files from subfolders end up in the same output directory, causing conflicts when multiple files share the same name.

Proposed fix

1. Correct the file LLamaSharpBackend.props as shown in the changes included in this PR.

2. When building for Windows, and after installing the Windows-specific backend packages, make sure to add ExcludeAssets="Native" to the PackageReference in the .csproj, as shown below:

<PackageReference Include="LLamaSharp.Backend.Cpu" Version="0.24.0" ExcludeAssets="Native"/>

Although ExcludeAssets="Native" is strictly necessary only for Windows backends when building on Windows, it shouldn’t cause issues for other backends. To be safe, it can be applied to all imported backend packages. I have tested this with both LLamaSharp.Backend.Cpu and LLamaSharp.Backend.Android installed in the same MAUI project. This attribute tells MSBuild not to copy the runtime files from the NuGet package to the output directory, because copying is already handled correctly in the LLamaSharpBackend.props file updated in step 1.

How to Apply this fix before it is merged

If you urgently need to fix this issue, you can temporarily apply this change by copying the corrected LLamaSharpBackend.props content (from step 1) into each backend package directory, e.g.:

C:\Users\<your user>\.nuget\packages\<llamasharp backend>\<version>\build\netstandard2.0\<llamasharp backend>.props

and then import the package as indicated in step 2.

Note: this modification will be lost if the package is updated.

Future improvements:

Avoid using the default Runtimes folder, since MSBuild always flattens its contents building for windows causing conflicts. Using a custom folder for runtime files would eliminate the need to set ExcludeAssets="Native" in the package reference, and then change the LLamaSharpBackend.props file to correctly fetch runtime files from the custom folder.

Tests

I tested this fix on a limited scenario with a MAUI project having both LLamaSharp.Backend.Cpu and LLamaSharp.Backend.Android installed. The goal was to ensure the app works on both types of devices. Given the large heterogeneity of the backends, I hope this fix will work for other backends as well.

[martindevans]

Thanks for looking into this. #382 has been a long standing issue that a few people (including me) have had a look at but failed to solve, I'm happy to see the fix turns out to be so simple!

I'll find some time to test this out soon, I'd really like to get it into the next update if possible.

Would you be interested in developing a followup PR with your suggested improvements? Removing the extra installation step and making it just work would be fantastic.

[LucaMaccarini]

Glad I could help, this issue had been around for a while, let’s see if the tests confirm the fix.
At the moment, with my fix, the runtimes are packaged into the NuGet packages only under the LLamaSharpRuntimes directory, but then imported into the project under the runtimes folder. This way, nothing should change on the code side and the runtimes end up where they’re expected to be.

One note

I’m not sure why, but the last time I worked on this bug I was able to pack the NuGets with the original paths. This time, however, I had to adjust them. In all the .nuspec files the src attribute of the <file> elements no longer starts with runtimes/ but with ../.

For example:
<file src="../deps/noavx/ggml.dll" target="LLamaSharpRuntimes\win-x64\native\noavx\ggml.dll" />

I hope this doesn’t break the way you build the packages, if it does, please let me know and I’ll restore the src paths to their previous state.

Testing

As before, I tested with a MAUI project using LLamaSharp.Backend.Cpu and LLamaSharp.Backend.Android. This time, I added the packages using the usual NuGet Package Manager GUI, and it was not necessary to use ExcludeAssets="Native".

[m0nsky]

Hi @LucaMaccarini , does this PR need any additional changes? I'm not sure what you mean by that last note (have you found out if we need to start with runtimes/ or ../ ?). I'm currently running into the same issue on linux (but have not yet tested this PR).

Edit
Ok, I had a chance to test this PR. I built the nuget packages locally and added the local source to my project. The PR it's current state, it will result in a bunch of errors like this while packing the nuget files:

Could not find a part of the path 'C:\Users\m0nsky\Desktop\nuget_publish_binaries\LLamaSharp\deps\cu12.4.0'.
"Packing LLamaSharp.Backend.Vulkan.Linux.nuspec"
Attempting to build package from 'LLamaSharp.Backend.Vulkan.Linux.nuspec'.
Could not find a part of the path 'C:\Users\m0nsky\Desktop\nuget_publish_binaries\LLamaSharp\deps\vulkan'.

After replacing all ../deps/ with runtimes/deps/ in the .nuspec files, the .nupkg files build correctly. I installed the packages in my project, and published a linux x64 release. The initial issue is fixed, however, the runtimes folder only seems to contain CPU backend folders (even though I have the CUDA12 backend installed). I think we're close, any ideas?

[LucaMaccarini]

Hi @m0nsky, Thanks a lot for testing and for the detailed feedback

I actually suspected that the path part should have remained runtimes/deps/.
In my local setup, I was using nuget.exe directly, passing the .nuspec file to create the package. Since the .nuspec files are located one level deeper respect to deps folder, I assumed I needed to prepend ../deps.

Example of the command I used locally:

.\nuget.exe pack "C:\Users\luca.maccarini\Desktop\luca\myProjects\LLamaSharp\LLamaSharp\LLama\runtimes\build\LLamaSharp.Backend.Cuda12.Linux.nuspec" -Version 0.25.0

I’ll fix the initial path in this PR so that it uses runtimes/deps/.

Could you kindly share how you usually generate the NuGet packages on your side? (what script or command do you run). That would help me align with your workflow and avoid mismatches like this.

Regarding the point you mentioned about the LLamaSharp.Backend.Cuda12.Linux package also including the CPU backend: this is expected, because in its .nuspec there is an explicit dependency on LLamaSharp.Backend.Cpu (introduced by @martindevans in commit 02eedd94).
Here’s the relevant metadata:

<dependencies>
    <dependency id="LLamaSharp.Backend.Cpu" version="$version$" />
</dependencies>

That dependency is the reason why the CPU backend is included when packing the CUDA12 backend.

[m0nsky]

Yes, absolutely, here are my steps:

• Download nuget.exe and add to PATH (environment variable, Windows 11)
• Clone your WIP branch, build the project
• Pack the nuget packages, using my method described here

Basically, I put that nuget_pack.bat inside the LLamaSharp folder, and run it. After packing the nuget packages, the .nupkg files are located in LLamaSharp/temp.

What I meant by the CPU/CUDA12 issue, is that the CUDA12 runtimes seem to be completely missing when building my application.

When I extract LLamaSharp.Backend.Cuda12.Linux.3.0.1.nupkg using 7zip, I can see that LLamaSharpRuntimes\linux-x64\native\cuda12 exists in the nuget package. However, when building my application which has the LLamaSharp and LLamaSharp.Backend.Cuda12 nuget packages installed (from the local source, which we just built), the runtimes folder only contains the CPU backends.

I'm using the following command to build/publish my application:
dotnet publish -c Release -r linux-x64 --self-contained

[LucaMaccarini]

Thanks for pointing me to the script for packing all the backends, I tried it and it works. I missunderstood your previous comment: I thought the CUDA files were there and I was asking why the CPU ones were included as well, but as you said the CUDA files are actually missing. I’ll investigate further...
if we can fix this, we should finally be able to close this annoying issue

[LSXAxeller]

I am attempting to use a locally built version of the LLamaSharp CPU and Vulkan backends to use this fix, but I'm encountering a build error during the publish step.

Here is my process:

1. I cloned the fix-issue-#382 branch from your fork: https://github.com/LucaMaccarini/LLamaSharp/tree/fix-issue-%23382.
2. I used the nuget_pack.bat script to build and pack the local packages.
3. I copied the generated packages to a local-packages directory in my repository's root.
4. I added the following nuget.config file to my repository root to define the local package source for the CPU and Vulkan backends:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
        <clear />
        <!-- Define local source using a relative path -->
        <add key="local-packages" value="./local-packages" />
        <!-- Add the official nuget.org source -->
        <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    </packageSources>
    <packageSourceMapping>
        <packageSource key="nuget.org">
            <package pattern="*" />
        </packageSource>
        <packageSource key="local-packages">
            <package pattern="LLamaSharp.Backend.Cpu" />
            <package pattern="LLamaSharp.Backend.Vulkan" />
            <package pattern="LLamaSharp.Backend.Vulkan.Linux" />
            <package pattern="LLamaSharp.Backend.Vulkan.Windows" />
        </packageSource>
    </packageSourceMapping>
</configuration>

In my IDE's NuGet Package Manager, I selected the local-packages source and installed the new CPU and Vulkan backend packages. The base LlamaSharp package is still from the official nuget.org feed.

When I try to publish my application with the following command, the build fails:

dotnet publish -c Release -r win-x64 --self-contained -f net8.0-windows

I receive the NETSDK1152 error, which a conflict due to multiple publish output files having the same relative path.

Here is the error output:

E:\Others\Native\Workspace\ProseFlow\ProseFlow.UI>dotnet publish -c Release -r win-x64 --self-contained -f net8.0-windows
Determining projects to restore...
All projects are up-to-date for restore.
ProseFlow.Core -> E:\Others\Native\Workspace\ProseFlow\ProseFlow.Core\bin\Release\net8.0\ProseFlow.Core.dll
ProseFlow.Application -> E:\Others\Native\Workspace\ProseFlow\ProseFlow.Application\bin\Release\net8.0\ProseFlow.Application.dll
ProseFlow.Infrastructure -> E:\Others\Native\Workspace\ProseFlow\ProseFlow.Infrastructure\bin\Release\net8.0\ProseFlow.Infrastructure.dll
ProseFlow.UI -> E:\Others\Native\Workspace\ProseFlow\ProseFlow.UI\bin\Release\net8.0-windows\win-x64\ProseFlow.dll

C:\Program Files\dotnet\sdk\8.0.414\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targets(112,5): error NETSDK1152: Found multiple publish output files with the same relative path:

C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx\ggml-base.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx2\ggml-base.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx512\ggml-base.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\noavx\ggml-base.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx\ggml-cpu.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx2\ggml-cpu.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx512\ggml-cpu.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\noavx\ggml-cpu.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx\ggml.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx2\ggml.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx512\ggml.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\noavx\ggml.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx\llama.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx2\llama.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx512\llama.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\noavx\llama.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx\mtmd.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx2\mtmd.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\avx512\mtmd.dll,
C:\Users\PRIME\.nuget\packages\llamasharp.backend.cpu\0.25.0\runtimes\win-x64\native\noavx\mtmd.dll.

[E:\Others\Native\Workspace\ProseFlow\ProseFlow.UI\ProseFlow.UI.csproj::TargetFramework=net8.0-windows]

Any guidance would be greatly appreciated.

---

Edit: Actually never mind, It was a version conflict. When I set the version to 0.25.0, it used cached backends from the main feed. Changing it to 0.25.1 resolved the issue, and it built successfully.

However, it is still packing the Linux and macOS binaries into the runtimes folder, can we avoid that? Also, is there a way to specify the AVX version to use (e.g., by setting DefineConstants in the *.csproj file) to help MSBuild flatten the runtimes folder content directly into the application directory?

I can also confirm that adding any backend (Vulkan in my case) only adds the CPU backend binaries to the build, not the specific backend itself (e.g., Vulkan, CUDA).