Error when starting the containers: "can't get final child's PID from pipe"

[rg-gapa-ecom]

Description

After some days of system uptime and multiple deployments using docker swarm some containers all of a sudden stopped working and can't be started again without full system reboot.

Docker logs have this information:

Oct 01 16:15:52 abs-test-docker01 dockerd[3191195]: time="2025-10-01T16:15:52.661198441+02:00" level=error msg="fatal task error" error="starting container failed: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown" module=node/agent/taskmanager node.id=yopdrf3rc0qny07s1buppka5l service.id=v2hhygvl1ablwrl2v73fnlw84 task.id=bewx7fhndpsb36cr00ulc29ol

As I mentioned system reboot fixes this problem but only temporarily - after some days (and deployments) error comes back. I have this issue at multiple virtual machines.

containerd --version
containerd containerd.io 1.7.27 05044ec0a9a75232cad458027ca83437aae3f4da

docker version
Client: Docker Engine - Community
 Version:           28.4.0
 API version:       1.51
 Go version:        go1.24.7
 Git commit:        d8eb465
 Built:             Wed Sep  3 20:57:05 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.4.0
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.7
  Git commit:       249d679
  Built:            Wed Sep  3 20:57:05 2025
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Steps to reproduce the issue

I can't provide any instructions as the reasons behind the error are unknown for me.

Describe the results you received and expected

Simplest test to reproduce the error:

docker run hello-world

Output:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown
 
Run 'docker run --help' for more information

What version of runc are you using?

runc version 1.2.5
commit: v1.2.5-0-g59923ef
spec: 1.2.0
go: go1.23.7
libseccomp: 2.5.3

Host OS information

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Host kernel information

Linux abs-test-docker01 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[rg-gapa-ecom]

Is there any log file I can provide?
I assume this might be not a runc problem, but rather some linux limits, but I'll be grateful for any help.

[kolyshkin]

Most probably you are hitting some system limit. Have you looked into system logs?

[rg-gapa-ecom]

I'm also pretty sure that's some limits issue, but don't know where to look. Didn't find anything useful so far.
I've tried use runc alternative (runsc) to get different runtime implementation exception, and get this:

docker run --runtime=runsc --rm -d alpine sh -c "while true; do echo running; sleep 1; done"

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: creating container: cannot create gofer process: gofer: fork/exec /proc/self/exe: no space left on device: unknown

There's no problem with disk usage, every partition is almost free.

Edit: Another logs

runc run cont1
FATA[0000] nsexec-1[3247792]: failed to unshare remaining namespaces: No space left on device
FATA[0000] nsexec-0[3247790]: failed to sync with stage-1: next state: Success
ERRO[0000] runc run failed: unable to start container process: can't get final child's PID from pipe: EOF

[cyphar]

failed to unshare remaining namespaces: No space left on device

This error probably means you have hit the per-user namespace limit for some namespace. These are configured via /proc/sys/user/max_*_namespaces.

[rg-gapa-ecom]

Yes, found it, fixed with:

sysctl -w user.max_pid_namespaces=1048576

[kolyshkin]

So, do we have a bug in runc or not?

For one thing, it seems that higher level runtimes only show the last line of runc output, but the issue is best described by earlier log lines. We can probably modify runc to report FATAL logs in the last line.

We can also make nsexec.c print a hint when unshare(2) returns ENOSPC. Not sure we should though.

[cyphar]

I think the issue here is that the can't get final child's PID from pipe: EOF error is not helpful at all here -- if runc init dies early, you would always expect to get this error but it doesn't really tell you anything.

Maybe fatal errors from runc init should also be treated as fatal errors in runc? But we can't just kill the process because libcontainer users will be screwed by that. Maybe we should have the log forwarder track if we saw a fatal error and have that string be errors.New-ified and be returned from libcontainer?

We could make this only happen for getChildPid but that seems fragile (if we ever add another blocking stage we would end up with the same situation). We could do it with a defer statement to modify the returned Err error, but that would also cause us to mask actual errors. Maybe only change it for error.Is(Err, io.EOF)?

[kolyshkin]

OK this is what I ended up with: #4928