pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}

[buckets-dev]

I've pushed a fix to the backend that should resolve this loop.

The issue was that the server wasn't clearing the "Update Container" command from its active queue even after the agent successfully completed it. This caused the server to re-send the same command (targeting the now-old container ID) on the next agent heartbeat, which the agent (correctly) rejected because that container had been renamed to a backup.

Fix: Completed commands are now properly removed from the active queue immediately.

This will be included in the next release.

---

Fixed by automated triage.

Originally posted by @pulse-triage in #1010

@pulse-triage, after updating to v5.0.9, Updating the container went successful however when the container restarted, the machine being monitored 'docker-host' disappears from pulse dashboard. The only remedy is to uninstall the pulse-agent, generate a new API token and reinstall the pulse-agent to bring back docker-host onto pulse dashboard. (I tried re-installing pulse-agent using previous API token but it didn't work).

The machine that disappeared from the docker dashboard still shows up on host dashboard but inactive

root@docker-host:~# curl -fsSL http://172.16.10.10:7655/install.sh | bash -s -- --uninstall --url http://172.16.10.10:7655 --token <old-token-api-9ada6f>
[INFO] Detecting available platforms...
[INFO] Docker/Podman detected - enabling container monitoring
[INFO]   (use --disable-docker to skip)
[INFO] Monitoring configuration:
[INFO]   Host metrics: true
[INFO]   Docker/Podman: true
[INFO]   Kubernetes: false
[INFO]   Proxmox: false
[INFO] Uninstalling pulse-agent and cleaning up legacy agents...
[INFO] Uninstallation complete.
root@docker-host:~# curl -fsSL http://172.16.10.10:7655/install.sh | bash -s -- --url http://172.16.10.10:7655 --token <new-token-api-378f55d> --interval 30s --enable-commands
[INFO] Detecting available platforms...
[INFO] Docker/Podman detected - enabling container monitoring
[INFO]   (use --disable-docker to skip)
[INFO] Monitoring configuration:
[INFO]   Host metrics: true
[INFO]   Docker/Podman: true
[INFO]   Kubernetes: false
[INFO]   Proxmox: false
[INFO] Downloading agent from http://172.16.10.10:7655/download/pulse-agent?arch=linux-amd64...
[INFO] Installing binary to /usr/local/bin/pulse-agent...
[INFO] Checking for legacy agents...
[INFO] Configuring Systemd service at /etc/systemd/system/pulse-agent.service...
[INFO] Token stored securely at /var/lib/pulse-agent/token (mode 600)
Created symlink /etc/systemd/system/multi-user.target.wants/pulse-agent.service → /etc/systemd/system/pulse-agent.service.
[INFO] Installation complete! Agent service started.
[INFO] Token file: /var/lib/pulse-agent/token (mode 600, root only)
root@docker-host:~#  journalctl -u pulse-agent -f
Jan 03 12:51:03 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"ghcr.io/hay-kot/homebox:latest","currentDigest":"sha256:b1fbca6fc1ad2bd3634d13056d1bc0242cf92e9bad4ca21ce57589be3fe4987e","latestDigest":"sha256:935c647cc27070deb2295bd6c414b0234bd9a1fd5d8ec40bf24930f2256f66b7","headDigest":"sha256:b1fbca6fc1ad2bd3634d13056d1bc0242cf92e9bad4ca21ce57589be3fe4987e","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:03+08:00","message":"Checked image update"}
Jan 03 12:51:04 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"ghcr.io/paperless-ngx/paperless-ngx:2.14.7","currentDigest":"sha256:2a6d9f6461ad7e8335f5b2123a173b9e6002fda209af8a66483b0c00629569ab","latestDigest":"sha256:4c640183df3a9975255ff9e29eeab3613208b320b69e8081e019f5973bfa8b95","headDigest":"sha256:2a6d9f6461ad7e8335f5b2123a173b9e6002fda209af8a66483b0c00629569ab","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:04+08:00","message":"Checked image update"}
Jan 03 12:51:06 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"ghcr.io/ssddanbrown/rss:latest","currentDigest":"sha256:f11e89f40a02c7670f46d3da9334c1cbf46b7e1ffe866c7be6fc586fcd96e04e","latestDigest":"sha256:3ffb8d2d806922d9051d11306c01e67fb87c8098e1a6b5a9d8acb3ffb59de400","headDigest":"sha256:f11e89f40a02c7670f46d3da9334c1cbf46b7e1ffe866c7be6fc586fcd96e04e","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:06+08:00","message":"Checked image update"}
Jan 03 12:51:10 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"gcr.io/cadvisor/cadvisor:latest","currentDigest":"sha256:3de2bd5203120b866d74a9b283b2ffb8ec382fbf9dc321814700c6ea6f44ec57","latestDigest":"sha256:9cc0da1192a4d0cd3597d16cdc3c625b8cc5636cee201f3d122c8331e1a01069","headDigest":"sha256:3de2bd5203120b866d74a9b283b2ffb8ec382fbf9dc321814700c6ea6f44ec57","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:10+08:00","message":"Checked image update"}
Jan 03 12:51:14 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"ghcr.io/akpw/mktxp:latest","currentDigest":"sha256:5cb7d6cdf6bf15bae21bc147eab3cb989d7bcdc400c8df0f3d37717661096b14","latestDigest":"sha256:ab9eb57d3b5739402624d796157055e909c5f456e9fda934d60b04fcf67cfd8e","headDigest":"sha256:5cb7d6cdf6bf15bae21bc147eab3cb989d7bcdc400c8df0f3d37717661096b14","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:14+08:00","message":"Checked image update"}
Jan 03 12:51:18 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"ghcr.io/tecnativa/docker-socket-proxy:latest","currentDigest":"sha256:1f3a6f303320723d199d2316a3e82b2e2685d86c275d5e3deeaf182573b47476","latestDigest":"sha256:bd2241b3bec83abcff25927a0a7ae518e0c5bef624b3cc247dcb31e68b53f417","headDigest":"sha256:1f3a6f303320723d199d2316a3e82b2e2685d86c275d5e3deeaf182573b47476","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:18+08:00","message":"Checked image update"}
Jan 03 12:51:20 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"amir20/dozzle:latest","currentDigest":"sha256:0df89c904da71e94a0c9ed3c89a890f01488321b5f10ac1e0c0bedcead9af6e4","latestDigest":"sha256:fdf81daad620469b9c91aabfedc41ae3c11da380325356d10ca1e07871a37a77","headDigest":"sha256:0df89c904da71e94a0c9ed3c89a890f01488321b5f10ac1e0c0bedcead9af6e4","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:20+08:00","message":"Checked image update"}
Jan 03 12:51:21 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"privatebin/nginx-fpm-alpine:stable","currentDigest":"sha256:67653ed3823795a76acb5fb5b8d961c95a7948c05277512617d6397fa4d57ec7","latestDigest":"sha256:fc302a7c72c0bc253d9c2ceb3545969bbd8f59c3790803596303d35eb57e007b","headDigest":"sha256:67653ed3823795a76acb5fb5b8d961c95a7948c05277512617d6397fa4d57ec7","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:21+08:00","message":"Checked image update"}
Jan 03 12:51:27 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"grafana/promtail:3.0.0","currentDigest":"sha256:d3de3da9431cfbe74a6a94555050df5257f357e827be8e63f8998d509c37af8b","latestDigest":"sha256:43c497a102e333c30b7c0f9a45d9107151f6424bd40c424d96791e65b6f2aeb0","headDigest":"sha256:d3de3da9431cfbe74a6a94555050df5257f357e827be8e63f8998d509c37af8b","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:27+08:00","message":"Checked image update"}
Jan 03 12:51:28 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"grafana/loki:3.0.0","currentDigest":"sha256:757b5fadf816a1396f1fea598152947421fa49cb8b2db1ddd2a6e30fae003253","latestDigest":"sha256:451563d761403fd66fdf7abef934f8712e864d47ef88f7b64e8ca52852fdaf28","headDigest":"sha256:757b5fadf816a1396f1fea598152947421fa49cb8b2db1ddd2a6e30fae003253","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:28+08:00","message":"Checked image update"}
Jan 03 12:51:32 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"portainer/agent:2.33.2","currentDigest":"sha256:e87f8097495576bf0d0057bdee3aedae3f7ac3bacba190f707bed35a77d71624","latestDigest":"sha256:e390f69be55bf2c42c85f7dfa7d1ebacb1e5244be0e8b8b64fc439d486016541","headDigest":"sha256:e87f8097495576bf0d0057bdee3aedae3f7ac3bacba190f707bed35a77d71624","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:32+08:00","message":"Checked image update"}
Jan 03 12:51:36 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"grafana/grafana:10.4.2","currentDigest":"sha256:7d5faae481a4c6f436c99e98af11534f7fd5e8d3e35213552dd1dd02bc393d2e","latestDigest":"sha256:3aa2a03d88ad9628485f78e89adcd5b18bcde1aced3e6146bbbd7a718b82b797","headDigest":"sha256:7d5faae481a4c6f436c99e98af11534f7fd5e8d3e35213552dd1dd02bc393d2e","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:36+08:00","message":"Checked image update"}
Jan 03 12:51:36 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"henrywhitaker3/speedtest-tracker","currentDigest":"sha256:47159a9402291275bd9f8a8ed4f5c1ce8502de216465ed29da8a3c995b807df1","latestDigest":"sha256:47159a9402291275bd9f8a8ed4f5c1ce8502de216465ed29da8a3c995b807df1","headDigest":"sha256:47159a9402291275bd9f8a8ed4f5c1ce8502de216465ed29da8a3c995b807df1","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:36+08:00","message":"Checked image update"}
Jan 03 12:51:49 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"flowko1/website-shot","currentDigest":"sha256:816d2e0d66a83e0befbd63b3bd174cb239e3d2bd15d469cec72d1bcb6eca26cd","latestDigest":"sha256:816d2e0d66a83e0befbd63b3bd174cb239e3d2bd15d469cec72d1bcb6eca26cd","headDigest":"sha256:816d2e0d66a83e0befbd63b3bd174cb239e3d2bd15d469cec72d1bcb6eca26cd","arch":"amd64","os":"linux","variant":"","updateAvailable":false,"time":"2026-01-03T12:51:49+08:00","message":"Checked image update"}
Jan 03 12:54:27 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","commandID":"057022da-c869-4add-ad15-31b8a4fe78c7:1a045ef20a79c6a5cb12c57c4c107a425dca6b6f9c21bec865c519ae6c2c01e8","containerId":"1a045ef20a79c6a5cb12c57c4c107a425dca6b6f9c21bec865c519ae6c2c01e8","time":"2026-01-03T12:54:27+08:00","message":"Received update_container command from Pulse"}
Jan 03 12:54:27 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","container":"caddy","image":"caddy:latest","time":"2026-01-03T12:54:27+08:00","message":"Starting container update"}
Jan 03 12:54:27 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"caddy:latest","time":"2026-01-03T12:54:27+08:00","message":"Pulling latest image"}
Jan 03 12:54:30 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","image":"caddy:latest","time":"2026-01-03T12:54:30+08:00","message":"Successfully pulled latest image"}
Jan 03 12:54:32 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","container":"caddy","time":"2026-01-03T12:54:32+08:00","message":"Container stopped"}
Jan 03 12:54:32 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","backup":"caddy_pulse_backup_20260103_125432","time":"2026-01-03T12:54:32+08:00","message":"Container renamed for backup"}
Jan 03 12:54:33 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","newContainerId":"01ab3f3dc165c51fec835e838f7d62a4ab66f9b8156636a6310daa34e2b39e70","time":"2026-01-03T12:54:33+08:00","message":"New container created"}
Jan 03 12:54:34 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","container":"caddy","time":"2026-01-03T12:54:34+08:00","message":"New container started, verifying stability..."}
Jan 03 12:54:39 docker-host pulse-agent[1074973]: {"level":"info","component":"pulse-docker-agent","container":"caddy","oldDigest":"sha256:e4bd5","newDigest":"sha256:e4bd5","time":"2026-01-03T12:54:39+08:00","message":"Container update completed successfully"}
Jan 03 12:54:57 docker-host pulse-agent[1074973]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T12:54:57+08:00","message":"Failed to send report, buffering"}
Jan 03 12:54:57 docker-host pulse-agent[1074973]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T12:54:57+08:00","message":"Failed to send docker report, buffering"}
Jan 03 12:55:27 docker-host pulse-agent[1074973]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T12:55:27+08:00","message":"Failed to send report, buffering"}
Jan 03 12:55:27 docker-host pulse-agent[1074973]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T12:55:27+08:00","message":"Failed to send docker report, buffering"}
Jan 03 12:55:57 docker-host pulse-agent[1074973]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T12:55:57+08:00","message":"Failed to send report, buffering"}
Jan 03 12:55:57 docker-host pulse-agent[1074973]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T12:55:57+08:00","message":"Failed to send docker report, buffering"}
Jan 03 12:56:27 docker-host pulse-agent[1074973]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T12:56:27+08:00","message":"Failed to send report, buffering"}
Jan 03 12:56:27 docker-host pulse-agent[1074973]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T12:56:27+08:00","message":"Failed to send docker report, buffering"}

[buckets-dev]

After pulse-agent reinstall, the dashboard indicated the container still not updated....

[rcourtman]

The 401 errors suggest the token binding was invalidated. Token binding maps your API token to a specific agent ID to prevent token reuse. If the agent ID changed after the container update, the server would reject subsequent reports.

Agent ID is derived from (in order): configured agent ID → machine ID → hostname. If any of these changed during the container restart, it would cause this.

To investigate, can you check:

1. Before reinstall, what was the agent ID being reported?

   journalctl -u pulse-agent -n 100 | grep -i "agent.*id\|reporting"

2. Is your machine ID stable across restarts?

   cat /etc/machine-id

3. Does your hostname change during docker operations?

The fact that generating a new token worked confirms the old token was bound to a different agent ID.

---

Automated triage. Waiting for diagnostic info.

[buckets-dev]

**Before conatainer image update

root@docker-host:~# journalctl -u pulse-agent -n 100 | grep -i "agent.*id\|reporting"
Jan 03 12:58:09 docker-host pulse-agent[1080665]: {"level":"warn","component":"command-client","error":"registration failed: registration rejected: Invalid token","failures":10,"time":"2026-01-03T12:58:09+08:00","message":"WebSocket connection failed repeatedly, reconnecting in 10s"}
Jan 03 12:58:19 docker-host pulse-agent[1080665]: {"level":"warn","component":"command-client","error":"registration failed: registration rejected: Invalid token","failures":11,"time":"2026-01-03T12:58:19+08:00","message":"WebSocket connection failed repeatedly, reconnecting in 10s"}

root@docker-host:~# cat /etc/machine-id
320a3d21a28846b39358949c0ca143e9
root@docker-host:~#

**** After container image update******

Jan 03 21:14:26 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","container":"/caddy_pulse_backup_20260103_125432","backupTime":"2026-01-03T12:54:32Z","time":"2026-01-03T21:14:26+08:00","message":"Removing orphaned backup container"}
Jan 03 21:34:26 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","commandID":"aaaefe9b-3699-4307-9b07-4c68bead4f70:01ab3f3dc165c51fec835e838f7d62a4ab66f9b8156636a6310daa34e2b39e70","containerId":"01ab3f3dc165c51fec835e838f7d62a4ab66f9b8156636a6310daa34e2b39e70","time":"2026-01-03T21:34:26+08:00","message":"Received update_container command from Pulse"}
Jan 03 21:34:26 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","container":"caddy","image":"caddy:latest","time":"2026-01-03T21:34:26+08:00","message":"Starting container update"}
Jan 03 21:34:26 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","image":"caddy:latest","time":"2026-01-03T21:34:26+08:00","message":"Pulling latest image"}
Jan 03 21:34:29 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","image":"caddy:latest","time":"2026-01-03T21:34:29+08:00","message":"Successfully pulled latest image"}
Jan 03 21:34:31 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","container":"caddy","time":"2026-01-03T21:34:31+08:00","message":"Container stopped"}
Jan 03 21:34:32 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","backup":"caddy_pulse_backup_20260103_213431","time":"2026-01-03T21:34:32+08:00","message":"Container renamed for backup"}
Jan 03 21:34:32 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","newContainerId":"f6d85402923b69e59f7d3ded2195854677c893f87e7cb3fbfc229ff42f09567c","time":"2026-01-03T21:34:32+08:00","message":"New container created"}
Jan 03 21:34:33 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","container":"caddy","time":"2026-01-03T21:34:33+08:00","message":"New container started, verifying stability..."}
Jan 03 21:34:38 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","container":"caddy","oldDigest":"sha256:e4bd5","newDigest":"sha256:e4bd5","time":"2026-01-03T21:34:38+08:00","message":"Container update completed successfully"}
Jan 03 21:34:56 docker-host pulse-agent[1083383]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T21:34:56+08:00","message":"Failed to send report, buffering"}
Jan 03 21:34:56 docker-host pulse-agent[1083383]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T21:34:56+08:00","message":"Failed to send docker report, buffering"}
Jan 03 21:35:26 docker-host pulse-agent[1083383]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T21:35:26+08:00","message":"Failed to send report, buffering"}
Jan 03 21:35:26 docker-host pulse-agent[1083383]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T21:35:26+08:00","message":"Failed to send docker report, buffering"}
Jan 03 21:35:56 docker-host pulse-agent[1083383]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T21:35:56+08:00","message":"Failed to send report, buffering"}
Jan 03 21:35:56 docker-host pulse-agent[1083383]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T21:35:56+08:00","message":"Failed to send docker report, buffering"}
Jan 03 21:36:26 docker-host pulse-agent[1083383]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T21:36:26+08:00","message":"Failed to send report, buffering"}
Jan 03 21:36:26 docker-host pulse-agent[1083383]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T21:36:26+08:00","message":"Failed to send docker report, buffering"}
Jan 03 21:36:56 docker-host pulse-agent[1083383]: {"level":"warn","component":"host-agent","error":"pulse responded with status 401 Unauthorized","time":"2026-01-03T21:36:56+08:00","message":"Failed to send report, buffering"}
Jan 03 21:36:56 docker-host pulse-agent[1083383]: {"level":"warn","component":"pulse-docker-agent","error":"target http://172.16.10.10:7655: pulse responded 401 Unauthorized: {\"error\":\"Authentication required\"}","time":"2026-01-03T21:36:56+08:00","message":"Failed to send docker report, buffering"}

root@docker-host:~# journalctl -u pulse-agent -n 100 | grep -i "agent.*id\|reporting"
Jan 03 21:34:26 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","commandID":"aaaefe9b-3699-4307-9b07-4c68bead4f70:01ab3f3dc165c51fec835e838f7d62a4ab66f9b8156636a6310daa34e2b39e70","containerId":"01ab3f3dc165c51fec835e838f7d62a4ab66f9b8156636a6310daa34e2b39e70","time":"2026-01-03T21:34:26+08:00","message":"Received update_container command from Pulse"}
Jan 03 21:34:32 docker-host pulse-agent[1083383]: {"level":"info","component":"pulse-docker-agent","newContainerId":"f6d85402923b69e59f7d3ded2195854677c893f87e7cb3fbfc229ff42f09567c","time":"2026-01-03T21:34:32+08:00","message":"New container created"}

root@docker-host:~# cat /etc/machine-id
320a3d21a28846b39358949c0ca143e9
root@docker-host:~#

[pulse-triage]

Found and fixed the root cause.

Bug: When any Docker command completed (including update_container), the server incorrectly set shouldRemove=true. This caused:

1. Container update completes successfully
2. Agent sends "completed" acknowledgment
3. Server removes the Docker host and revokes the API token
4. Agent's next report gets 401 Unauthorized

Fix: shouldRemove is now only true for stop commands, not for update_container or check_updates.

Pushed: d518760

This should be fixed in the next release. Would appreciate confirmation it works for you once available.

---

Automated triage.

[buckets-dev]

Latest Log

Jan 05 06:10:13 docker-host pulse-agent[1571852]: {"level":"warn","component":"host-age                                                                                                       nt","error":"send request: Post \"http://172.16.10.10:7655/api/agents/host/report\": di                                                                                                       al tcp 172.16.10.10:7655: connect: connection refused","time":"2026-01-05T06:10:13+08:0                                                                                                       0","message":"Failed to send report, buffering"}
Jan 05 06:10:14 docker-host pulse-agent[1571852]: {"level":"warn","component":"pulse-do                                                                                                       cker-agent","error":"target http://172.16.10.10:7655: send report: Post \"http://172.16                                                                                                       .10.10:7655/api/agents/docker/report\": dial tcp 172.16.10.10:7655: connect: connection                                                                                                        refused","time":"2026-01-05T06:10:14+08:00","message":"Failed to send docker report, b                                                                                                       uffering"}
Jan 05 06:10:23 docker-host pulse-agent[1571852]: {"level":"warn","component":"command-                                                                                                       client","error":"dial websocket: dial tcp 172.16.10.10:7655: connect: connection refuse                                                                                                       d","failures":3,"time":"2026-01-05T06:10:23+08:00","message":"WebSocket connection fail                                                                                                       ed repeatedly, reconnecting in 10s"}
Jan 05 06:10:33 docker-host pulse-agent[1571852]: {"level":"info","component":"command-                                                                                                       client","time":"2026-01-05T06:10:33+08:00","message":"Connected and registered with Pul                                                                                                       se command server"}
Jan 05 06:10:43 docker-host pulse-agent[1571852]: {"level":"info","component":"host-age                                                                                                       nt","count":1,"time":"2026-01-05T06:10:43+08:00","message":"Flushing buffered reports"}
Jan 05 06:10:44 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","count":1,"time":"2026-01-05T06:10:44+08:00","message":"Flushing buffered d                                                                                                       ocker reports"}
Jan 05 06:11:14 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","commandID":"21ed5a22-be6a-4284-8839-342232aabfc1:f6d85402923b69e59f7d3ded2                                                                                                       195854677c893f87e7cb3fbfc229ff42f09567c","containerId":"f6d85402923b69e59f7d3ded2195854                                                                                                       677c893f87e7cb3fbfc229ff42f09567c","time":"2026-01-05T06:11:14+08:00","message":"Receiv                                                                                                       ed update_container command from Pulse"}
Jan 05 06:11:14 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","container":"caddy","image":"caddy:latest","time":"2026-01-05T06:11:14+08:0                                                                                                       0","message":"Starting container update"}
Jan 05 06:11:14 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","image":"caddy:latest","time":"2026-01-05T06:11:14+08:00","message":"Pullin                                                                                                       g latest image"}
Jan 05 06:11:17 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","image":"caddy:latest","time":"2026-01-05T06:11:17+08:00","message":"Succes                                                                                                       sfully pulled latest image"}
Jan 05 06:11:19 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","container":"caddy","time":"2026-01-05T06:11:19+08:00","message":"Container                                                                                                        stopped"}
Jan 05 06:11:20 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","backup":"caddy_pulse_backup_20260105_061119","time":"2026-01-05T06:11:20+0                                                                                                       8:00","message":"Container renamed for backup"}
Jan 05 06:11:20 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","newContainerId":"32c9f37a8839339444211c12883aa1035f2739655243ae796af71b540                                                                                                       b05910d","time":"2026-01-05T06:11:20+08:00","message":"New container created"}
Jan 05 06:11:21 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","container":"caddy","time":"2026-01-05T06:11:21+08:00","message":"New conta                                                                                                       iner started, verifying stability..."}
Jan 05 06:11:26 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-do                                                                                                       cker-agent","container":"caddy","oldDigest":"sha256:e4bd5","newDigest":"sha256:e4bd5","                                                                                                       time":"2026-01-05T06:11:26+08:00","message":"Container update completed successfully"}
Jan 05 06:11:44 docker-host pulse-agent[1571852]: {"level":"info","component":"pulse-docker-agent","commandID":"21ed5a22-be6a-4284-8839                                                       -342232aabfc1:f6d85402923b69e59f7d3ded2195854677c893f87e7cb3fbfc229ff42f09567c","containerId":"f6d85402923b69e59f7d3ded2195854677c893f8                                                       7e7cb3fbfc229ff42f09567c","time":"2026-01-05T06:11:44+08:00","message":"Received update_container command from Pulse"}
Jan 05 06:11:44 docker-host pulse-agent[1571852]: {"level":"warn","component":"pulse-docker-agent","container":"caddy_pulse_backup_2026                                                       0105_061119","time":"2026-01-05T06:11:44+08:00","message":"Rejecting update request for backup container"}

[pulse-triage]

The latest logs you shared show transient connection refused errors (likely a brief Pulse server restart or network blip), followed by successful reconnection:

Connected and registered with Pulse command server
Flushing buffered reports

This is the expected behavior - the agent buffers reports during outages and flushes them on reconnection.

Are you still experiencing the 401 Unauthorized errors after the fix in v5.0.9? Or has this resolved?

[buckets-dev]

@pulse-triage. Yes, the docker-host is now not getting removed from the dashboard as previously. The container restarts but the image is not updated accordingly, still showing outstanding update on the dashboard.

Having said that, all the other images has updated successfully with no issue at all. I reckon the "caddy" image is the issue, in the log file I could see the new image getting downloaded successful and that is where it stops and the container doesn't apply the new image.

The ticket can be closed as I believe the image is faulty.

[pulse-triage]

Closing as requested. The 401 Unauthorized issue was fixed in v5.0.9, and the remaining container update issue appears to be specific to the caddy image.

---

Closed by automated triage. Feel free to reopen if this needs further attention.