Failed to fetch any free public IP address on K8S cluster creation over Basic Network #11252
Description
problem
We have 3 KVM hosts in a CloudStack cluster. Once we try to start a K8S cluster, we see the error Failed to fetch any free public IP address in the management server logs. At this moment, the first node, the control plane, or the virtual router can't take the IP address from the Guest Network.
If we check the database, we can see that one IP is in state="Allocated" but "allocated" column is NULL.
> select id,public_ip_address,state,allocated from user_ip_address where state!='Free' and allocated is null and removed is null;
+----+-------------------+-----------+-----------+
| id | public_ip_address | state | allocated |
+----+-------------------+-----------+-----------+
| 1 | 172.24.35.182 | Allocated | NULL |
+----+-------------------+-----------+-----------+
We've tried to remove the K8S cluster and to decrease and increase the Guest Network range with no success. Also to create the cluster through UI and Terraform module.
Also, we have tried to setup a new CloudStack cluster using another 3 different servers and we get the same error, but we get in the last k8s cluster node. It had only 2 remaining IPs and increasing by one the CIDR range it worked. However, if we check the database, the k8s control plane node IP (161) it’s not “allocated”:
mysql> select public_ip_address,allocated,state from user_ip_address where data_center_id=2;
+-------------------+---------------------+-----------+
| public_ip_address | allocated | state |
+-------------------+---------------------+-----------+
| 172.24.77.161 | NULL | Allocated |
| 172.24.77.162 | NULL | Free |
| 172.24.77.163 | 2025-07-16 12:23:16 | Allocated |
| 172.24.77.164 | 2025-07-16 12:23:04 | Allocated |
| 172.24.77.165 | 2025-07-16 12:34:28 | Allocated |
| 172.24.77.166 | 2025-07-16 12:22:52 | Allocated |
| 172.24.77.167 | 2025-07-15 10:27:41 | Allocated |
| 172.24.77.168 | 2025-07-15 09:43:15 | Allocated |
| 172.24.77.169 | 2025-07-15 09:43:15 | Allocated |
| 172.24.77.170 | NULL | Free |
| 172.24.77.171 | 2025-07-16 12:22:26 | Allocated |
| 172.24.77.172 | 2025-07-16 12:21:58 | Allocated |
| 172.24.77.173 | 2025-07-16 12:22:12 | Allocated |
| 172.24.77.174 | NULL | Free |
| 172.24.77.175 | NULL | Free |
+-------------------+---------------------+-----------+
15 rows in set (0.00 sec)
Thanks
versions
Oracle Linux 9.6
Kernel 6.12.0-1.23.3.2.el9uek.x86_64
CloudStack management and agent version is 4.20.1
Kubernetes ISO 1.30.3
KVM
mysql Ver 8.0.41
NFS v4
libvirtd (libvirt) 10.10.
The steps to reproduce the bug
- Start a clean CloudStack setup
- Create a zone with his cluster with Basic Network configuration
- Start a K8S cluster
- K8S cluster tries to start but it can't.
What to do about it?
No response