Permission Issue on TrueNas

[Nevah5]

General information and introduction

Hey there! I appreciate that you are reading my question/help request. I'm currently completely stuck with trying to run a fabric server on my TrueNas Scale server via Apps. Apps are basically docker, but have a scale option. In the backend it runs kubernetes (k3s), but I do everything via the frontend, because this is apparently the intended way and I should touch the system itself.

The issue

I think it has something to do with the Linux permissions I have set, but it's still weird, because I can run spigot and vanilla servers fine. (I haven't tried others). On my system, the root user is id 0. So I have created a system user for applications that run on 1000, so that the permissions from the host system can be mounted via volumes and so that no permission issues occur.
(I'm not that advanced in permissions stuff on Linux, so I'm sorry in advance.)

2023-05-04 21:01:20.140637+00:00chmod: changing permissions of '/data': Operation not permitted
2023-05-04 21:01:20.154087+00:00[init] Running as uid=1000 gid=1000 with /data as 'drwxrwx--- 2 1000 1000 2 May  4 20:57 /data'
2023-05-04 21:01:20.160198+00:00[init] Resolved version given 1.19.4 into 1.19.4 and major version 1.19
2023-05-04 21:01:20.941565+00:00[init] Resolving type given FABRIC
2023-05-04 21:01:22.667303+00:00[mc-image-helper] 21:01:22.664 INFO  : Installing Fabric Versions(game=1.19.4, loader=0.14.19, installer=0.11.2)
2023-05-04 21:01:23.174410+00:00[init] Creating server properties in /data/server.properties
2023-05-04 21:01:23.174984+00:00[init] Disabling whitelist functionality
2023-05-04 21:01:25.378967+00:00[init] Setting initial memory to 5G and max to 5G
2023-05-04 21:01:25.379758+00:00[init] Starting the Minecraft server...
2023-05-04 21:01:25.575135+00:00Downloading Minecraft server
2023-05-04 21:01:27.387826+00:00Installing Fabric Loader 0.14.19(1.19.4) on the server
2023-05-04 21:01:27.387967+00:00Downloading required files
2023-05-04 21:01:27.485275+00:00Downloading library net.fabricmc:tiny-mappings-parser:0.3.0+build.17
2023-05-04 21:01:27.523680+00:00Downloading library net.fabricmc:sponge-mixin:0.12.4+mixin.0.8.5
2023-05-04 21:01:27.598538+00:00Downloading library net.fabricmc:tiny-remapper:0.8.2
2023-05-04 21:01:27.622906+00:00Downloading library net.fabricmc:access-widener:2.1.0
2023-05-04 21:01:27.644922+00:00Downloading library org.ow2.asm:asm:9.4
2023-05-04 21:01:27.679724+00:00Downloading library org.ow2.asm:asm-analysis:9.4
2023-05-04 21:01:27.702920+00:00Downloading library org.ow2.asm:asm-commons:9.4
2023-05-04 21:01:27.732630+00:00Downloading library org.ow2.asm:asm-tree:9.4
2023-05-04 21:01:27.755285+00:00Downloading library org.ow2.asm:asm-util:9.4
2023-05-04 21:01:27.777241+00:00Downloading library net.fabricmc:intermediary:1.19.4
2023-05-04 21:01:27.806283+00:00Downloading library net.fabricmc:fabric-loader:0.14.19
2023-05-04 21:01:27.846404+00:00Generating server launch JAR
2023-05-04 21:01:27.923557+00:00Unpacking 1.19.4/server-1.19.4.jar (versions:1.19.4) to versions/1.19.4/server-1.19.4.jar
2023-05-04 21:01:27.989019+00:00Unpacking com/github/oshi/oshi-core/6.2.2/oshi-core-6.2.2.jar (libraries:com.github.oshi:oshi-core:6.2.2) to libraries/com/github/oshi/oshi-core/6.2.2/oshi-core-6.2.2.jar
2023-05-04 21:01:27.994206+00:00Unpacking com/google/code/gson/gson/2.10/gson-2.10.jar (libraries:com.google.code.gson:gson:2.10) to libraries/com/google/code/gson/gson/2.10/gson-2.10.jar
2023-05-04 21:01:27.996638+00:00Unpacking com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar (libraries:com.google.guava:failureaccess:1.0.1) to libraries/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar
2023-05-04 21:01:27.997925+00:00Unpacking com/google/guava/guava/31.1-jre/guava-31.1-jre.jar (libraries:com.google.guava:guava:31.1-jre) to libraries/com/google/guava/guava/31.1-jre/guava-31.1-jre.jar
2023-05-04 21:01:28.010265+00:00Unpacking com/mojang/authlib/3.18.38/authlib-3.18.38.jar (libraries:com.mojang:authlib:3.18.38) to libraries/com/mojang/authlib/3.18.38/authlib-3.18.38.jar
2023-05-04 21:01:28.011797+00:00Unpacking com/mojang/brigadier/1.0.18/brigadier-1.0.18.jar (libraries:com.mojang:brigadier:1.0.18) to libraries/com/mojang/brigadier/1.0.18/brigadier-1.0.18.jar
2023-05-04 21:01:28.013213+00:00Unpacking com/mojang/datafixerupper/6.0.6/datafixerupper-6.0.6.jar (libraries:com.mojang:datafixerupper:6.0.6) to libraries/com/mojang/datafixerupper/6.0.6/datafixerupper-6.0.6.jar
2023-05-04 21:01:28.017236+00:00Unpacking com/mojang/logging/1.1.1/logging-1.1.1.jar (libraries:com.mojang:logging:1.1.1) to libraries/com/mojang/logging/1.1.1/logging-1.1.1.jar
2023-05-04 21:01:28.018225+00:00Unpacking commons-io/commons-io/2.11.0/commons-io-2.11.0.jar (libraries:commons-io:commons-io:2.11.0) to libraries/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar
2023-05-04 21:01:28.020668+00:00Unpacking io/netty/netty-buffer/4.1.82.Final/netty-buffer-4.1.82.Final.jar (libraries:io.netty:netty-buffer:4.1.82.Final) to libraries/io/netty/netty-buffer/4.1.82.Final/netty-buffer-4.1.82.Final.jar
2023-05-04 21:01:28.023248+00:00Unpacking io/netty/netty-codec/4.1.82.Final/netty-codec-4.1.82.Final.jar (libraries:io.netty:netty-codec:4.1.82.Final) to libraries/io/netty/netty-codec/4.1.82.Final/netty-codec-4.1.82.Final.jar
2023-05-04 21:01:28.025862+00:00Unpacking io/netty/netty-common/4.1.82.Final/netty-common-4.1.82.Final.jar (libraries:io.netty:netty-common:4.1.82.Final) to libraries/io/netty/netty-common/4.1.82.Final/netty-common-4.1.82.Final.jar
2023-05-04 21:01:28.029649+00:00Unpacking io/netty/netty-handler/4.1.82.Final/netty-handler-4.1.82.Final.jar (libraries:io.netty:netty-handler:4.1.82.Final) to libraries/io/netty/netty-handler/4.1.82.Final/netty-handler-4.1.82.Final.jar
2023-05-04 21:01:28.033635+00:00Unpacking io/netty/netty-resolver/4.1.82.Final/netty-resolver-4.1.82.Final.jar (libraries:io.netty:netty-resolver:4.1.82.Final) to libraries/io/netty/netty-resolver/4.1.82.Final/netty-resolver-4.1.82.Final.jar
2023-05-04 21:01:28.034824+00:00Unpacking io/netty/netty-transport/4.1.82.Final/netty-transport-4.1.82.Final.jar (libraries:io.netty:netty-transport:4.1.82.Final) to libraries/io/netty/netty-transport/4.1.82.Final/netty-transport-4.1.82.Final.jar
2023-05-04 21:01:28.038445+00:00Unpacking io/netty/netty-transport-classes-epoll/4.1.82.Final/netty-transport-classes-epoll-4.1.82.Final.jar (libraries:io.netty:netty-transport-classes-epoll:4.1.82.Final) to libraries/io/netty/netty-transport-classes-epoll/4.1.82.Final/netty-transport-classes-epoll-4.1.82.Final.jar
2023-05-04 21:01:28.040315+00:00Unpacking io/netty/netty-transport-native-epoll/4.1.82.Final/netty-transport-native-epoll-4.1.82.Final-linux-x86_64.jar (libraries:io.netty:netty-transport-native-epoll:4.1.82.Final:linux-x86_64) to libraries/io/netty/netty-transport-native-epoll/4.1.82.Final/netty-transport-native-epoll-4.1.82.Final-linux-x86_64.jar
2023-05-04 21:01:28.041541+00:00Unpacking io/netty/netty-transport-native-epoll/4.1.82.Final/netty-transport-native-epoll-4.1.82.Final-linux-aarch_64.jar (libraries:io.netty:netty-transport-native-epoll:4.1.82.Final:linux-aarch_64) to libraries/io/netty/netty-transport-native-epoll/4.1.82.Final/netty-transport-native-epoll-4.1.82.Final-linux-aarch_64.jar
2023-05-04 21:01:28.042496+00:00Unpacking io/netty/netty-transport-native-unix-common/4.1.82.Final/netty-transport-native-unix-common-4.1.82.Final.jar (libraries:io.netty:netty-transport-native-unix-common:4.1.82.Final) to libraries/io/netty/netty-transport-native-unix-common/4.1.82.Final/netty-transport-native-unix-common-4.1.82.Final.jar
2023-05-04 21:01:28.043687+00:00Unpacking it/unimi/dsi/fastutil/8.5.9/fastutil-8.5.9.jar (libraries:it.unimi.dsi:fastutil:8.5.9) to libraries/it/unimi/dsi/fastutil/8.5.9/fastutil-8.5.9.jar
2023-05-04 21:01:28.142057+00:00Unpacking net/java/dev/jna/jna/5.12.1/jna-5.12.1.jar (libraries:net.java.dev.jna:jna:5.12.1) to libraries/net/java/dev/jna/jna/5.12.1/jna-5.12.1.jar
2023-05-04 21:01:28.149316+00:00Unpacking net/java/dev/jna/jna-platform/5.12.1/jna-platform-5.12.1.jar (libraries:net.java.dev.jna:jna-platform:5.12.1) to libraries/net/java/dev/jna/jna-platform/5.12.1/jna-platform-5.12.1.jar
2023-05-04 21:01:28.155548+00:00Unpacking net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar (libraries:net.sf.jopt-simple:jopt-simple:5.0.4) to libraries/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar
2023-05-04 21:01:28.157109+00:00Unpacking org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar (libraries:org.apache.commons:commons-lang3:3.12.0) to libraries/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
2023-05-04 21:01:28.160934+00:00Unpacking org/apache/logging/log4j/log4j-api/2.19.0/log4j-api-2.19.0.jar (libraries:org.apache.logging.log4j:log4j-api:2.19.0) to libraries/org/apache/logging/log4j/log4j-api/2.19.0/log4j-api-2.19.0.jar
2023-05-04 21:01:28.163408+00:00Unpacking org/apache/logging/log4j/log4j-core/2.19.0/log4j-core-2.19.0.jar (libraries:org.apache.logging.log4j:log4j-core:2.19.0) to libraries/org/apache/logging/log4j/log4j-core/2.19.0/log4j-core-2.19.0.jar
2023-05-04 21:01:28.172694+00:00Unpacking org/apache/logging/log4j/log4j-slf4j2-impl/2.19.0/log4j-slf4j2-impl-2.19.0.jar (libraries:org.apache.logging.log4j:log4j-slf4j2-impl:2.19.0) to libraries/org/apache/logging/log4j/log4j-slf4j2-impl/2.19.0/log4j-slf4j2-impl-2.19.0.jar
2023-05-04 21:01:28.174054+00:00Unpacking org/joml/joml/1.10.5/joml-1.10.5.jar (libraries:org.joml:joml:1.10.5) to libraries/org/joml/joml/1.10.5/joml-1.10.5.jar
2023-05-04 21:01:28.178608+00:00Unpacking org/slf4j/slf4j-api/2.0.1/slf4j-api-2.0.1.jar (libraries:org.slf4j:slf4j-api:2.0.1) to libraries/org/slf4j/slf4j-api/2.0.1/slf4j-api-2.0.1.jar
2023-05-04 21:01:28.179994+00:00Starting net.fabricmc.loader.impl.game.minecraft.BundlerClassPathCapture
2023-05-04 21:01:30.592925+00:00[21:01:28] [INFO] [FabricLoader/GameProvider]: Loading Minecraft 1.19.4 with Fabric Loader 0.14.19
2023-05-04 21:01:30.593151+00:00[21:01:28] [INFO] [FabricLoader/GameRemap]: Fabric is preparing JARs on first launch, this may take a few seconds...
2023-05-04 21:01:30.594239+00:00[21:01:30] [ERROR] [FabricLoader/]: Uncaught exception in thread "main"
2023-05-04 21:01:30.594302+00:00java.lang.RuntimeException: An exception occurred when launching the server!
2023-05-04 21:01:30.594322+00:00at net.fabricmc.loader.impl.launch.server.FabricServerLauncher.main(FabricServerLauncher.java:71)
2023-05-04 21:01:30.594358+00:00at net.fabricmc.installer.ServerLauncher.main(ServerLauncher.java:69)
2023-05-04 21:01:30.594365+00:00Caused by: java.lang.RuntimeException: error remapping game jars [/data/versions/1.19.4/server-1.19.4.jar]
2023-05-04 21:01:30.594374+00:00at net.fabricmc.loader.impl.game.GameProviderHelper.deobfuscate(GameProviderHelper.java:240)
2023-05-04 21:01:30.594386+00:00at net.fabricmc.loader.impl.game.minecraft.MinecraftGameProvider.initialize(MinecraftGameProvider.java:313)
2023-05-04 21:01:30.594392+00:00at net.fabricmc.loader.impl.launch.knot.Knot.init(Knot.java:142)
2023-05-04 21:01:30.594399+00:00at net.fabricmc.loader.impl.launch.knot.Knot.launch(Knot.java:68)
2023-05-04 21:01:30.594413+00:00at net.fabricmc.loader.impl.launch.knot.KnotServer.main(KnotServer.java:23)
2023-05-04 21:01:30.594421+00:00at net.fabricmc.loader.impl.launch.server.FabricServerLauncher.main(FabricServerLauncher.java:69)
2023-05-04 21:01:30.594428+00:00... 1 more
2023-05-04 21:01:30.594434+00:00Caused by: java.nio.file.FileSystemException: /data/./.fabric/remappedJars/minecraft-1.19.4-0.14.19/zipfstmp11551843125674825625.tmp: Operation not permitted
2023-05-04 21:01:30.594446+00:00at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
2023-05-04 21:01:30.594453+00:00at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
2023-05-04 21:01:30.594460+00:00at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
2023-05-04 21:01:30.594469+00:00at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(Unknown Source)
2023-05-04 21:01:30.594476+00:00at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(Unknown Source)
2023-05-04 21:01:30.594483+00:00at java.base/java.nio.file.Files.setPosixFilePermissions(Unknown Source)
2023-05-04 21:01:30.594494+00:00at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.sync(Unknown Source)
2023-05-04 21:01:30.594501+00:00at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.lambda$close$9(Unknown Source)
2023-05-04 21:01:30.594510+00:00at java.base/java.security.AccessController.doPrivileged(Unknown Source)
2023-05-04 21:01:30.594521+00:00at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.close(Unknown Source)
2023-05-04 21:01:30.594530+00:00at net.fabricmc.tinyremapper.OutputConsumerPath.close(OutputConsumerPath.java:226)
2023-05-04 21:01:30.594537+00:00at net.fabricmc.loader.impl.game.GameProviderHelper.deobfuscate0(GameProviderHelper.java:308)
2023-05-04 21:01:30.594547+00:00at net.fabricmc.loader.impl.game.GameProviderHelper.deobfuscate(GameProviderHelper.java:238)
2023-05-04 21:01:30.594557+00:00... 6 more

Only here (Fabric) the server won't start, but on vanilla or spigot, it works fine, even though I can find same logs with Operation not permitted

Configuration

I'm using the latest image.
I'm always running any container as UID 1000 and GID 1000 (I know this doesn't do anything in this case, I should use the env variables, but I've also tried that already.)

Environment Variables

• MEMORY=5G
• VERSION=1.19.4
• TYPE=FABRIC
• EULA=TRUE

Storage

/mnt/applications/minecraft-fabric:/data

Volume Permissions

root@nas[/mnt/applications/minecraft-fabric]# ls -lah
total 209K
drwxrwx---  5 system system   10 May  4 23:01 .
drwxr-xr-x 18 root   root     18 May  3 22:38 ..
drwxrwx---  4 system system    4 May  4 23:01 .fabric
-rwxrwx---  1 system system  329 May  4 23:01 .fabric-manifest.json
-rwxrwx---  1 system system   86 May  4 23:01 .install-fabric.env
-rwxrwx---  1 system system   68 May  4 23:01 eula.txt
-rwxrwx---  1 system system 152K May  4 23:01 fabric-server-mc.1.19.4-loader.0.14.19-launcher.0.11.2.jar
drwxrwx---  8 system system    8 May  4 23:01 libraries
-rwxrwx---  1 system system  123 May  4 23:01 server.properties
drwxrwx---  3 system system    3 May  4 23:01 versions

I've also tried setting the permissions to 777 with ACL, without success to fix the issue.

[itzg]

I'm wondering if that specific file or it's immediate parent got a different ownership at some point. Can you do these:

ls -ld .fabric/remappedJars/

ls -ld .fabric/remappedJars/minecraft-1.19.4-0.14.19/

ls -l .fabric/remappedJars/minecraft-1.19.4-0.14.19

You might just need to run this to fix it up

sudo chown -R system:system .fabric

[Nevah5]

Seems like this didn't help. Same logs.
I couldn't find any files or folders with different ownership. I also made sure, I forgot to mention this, that I've deleted everything before starting the container, including hidden files/folders. So all the files should have been created by the minecraft (UID 1000) user in the container, making the owner on the nas to system, right?

[Nevah5]

root@nas[/mnt/applications/minecraft-fabric]# ls -ld .fabric/remappedJars
drwxrwx--- 3 system system 3 May  4 23:01 .fabric/remappedJars

root@nas[/mnt/applications/minecraft-fabric]# ls -ld .fabric/remappedJars/minecraft-1.19.4-0.14.19
drwxrwx--- 2 system system 4 May  5 19:23 .fabric/remappedJars/minecraft-1.19.4-0.14.19

root@nas[/mnt/applications/minecraft-fabric]# ls -l .fabric/remappedJars/minecraft-1.19.4-0.14.19
total 27021
-rwxrwx--- 1 system system 15189595 May  4 23:01 zipfstmp11551843125674825625.tmp
-rwxrwx--- 1 system system 15189595 May  5 19:23 zipfstmp16725093547262309557.tmp

I forgot to supply these.

[itzg]

Are you using SElinux or AppArmor on your system. When I read the error again, it sounds like it's disallowing the file mode change rather than reporting a permission issue:

2023-05-04 21:01:30.594434+00:00Caused by: java.nio.file.FileSystemException: /data/./.fabric/remappedJars/minecraft-1.19.4-0.14.19/zipfstmp11551843125674825625.tmp: Operation not permitted
2023-05-04 21:01:30.594446+00:00at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
2023-05-04 21:01:30.594453+00:00at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
2023-05-04 21:01:30.594460+00:00at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
2023-05-04 21:01:30.594469+00:00at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(Unknown Source)
2023-05-04 21:01:30.594476+00:00at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(Unknown Source)

[Nevah5]

Yeah, it looks like that TrueNAS Scale uses AppArmor.

[itzg]

Aha. You'll need to research how to identify and configure the appropriate allowance for the chmod type operations.

[Nevah5]

I fixed it by creating my own Dockerfile. I also tried to fix AppArmor, but I don't think it's fixed because of it.
Well, I will try some time later again, when I have more time. The drawback I have now, is that I'm missing console access over the RCON script in this repo.
Thanks @itzg