minikube: Improve cluster creation UX on Linux

[illume]

Currently without Docker running the UX on Linux for cluster creation is not great.

• it shows the error message with a link to the documentation
• it allows selecting a different driver

Better would be if it "just works" for more people.

[rata]

Driver to use: kvm2 

Driver docs: https://minikube.sigs.k8s.io/docs/drivers/kvm2/. You can see the instructions for other distros there. 

For example, in Debian/Ubuntu it’s basically (there are more options): 

$ curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-linux-amd64 
$ sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64 
$ sudo apt install --no-install-recommends qemu-system libvirt-clients libvirt-daemon-system 
$ adduser <youruser> libvirt 

Note the last step is “tricky”, because for the new group memership to show you need to start a new session. You can logout and login again from X/Wayland, but that is painful. What I did is run “su - $USER” in the terminal, that opened a new bash and the group membership is effective then. 

Please note this was tested with minikube 1.36.0. 

You can start a cluster like this afterwards: 

$ minikube start --driver=kvm2 -p headlamp-desktop-1 --container-runtime=containerd --kubernetes-version=1.33.1 --cni=calico --memory 3072 --addons="storage-provisioner,default-storageclass" 
$ minikube config set WantBetaUpdateNotification false 
$ minikube config set WantUpdateNotification false 

The cluster connect to the host over a virt netdev, that is not routable from outside the host.

Given that the VM uses 3GB top, my recommendation is a host with at least 8GB of RAM as min requirement. Of course, virtualization by hardware is required too.
 

[rata]

That creates a cluster with persistent storage. Here is a smoke test to verify that:

# Let's create a simple example to test this.
$ cat pod.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
---
apiVersion: v1
kind: Pod
metadata:
  name: storage-test
spec:
  containers:
  - name: writer
    image: busybox
    command: [ "sh", "-c", "echo 'Hello from kartik for persistence check!' > /data/hello.txt && sleep 3600" ]
    volumeMounts:
    - mountPath: /data
      name: test-storage
  volumes:
  - name: test-storage
    persistentVolumeClaim:
      claimName: myclaim
$ kubectl apply -f pod.yaml
$ kubectl exec -ti storage-test -- sh
# Note, this is inside the interactive shell
/ # ls /data
hello.txt
/ # touch /data/rata
/ # exit

$ minikube stop -p headlamp-desktop-1
$ minikube start -p headlamp-desktop-1
$ kubectl exec -ti storage-test -- ls /data
hello.txt  rata

[rata]

Metallb using the minikube addon

The addon is using an old version of metallb. However, some issues to be debugged with newer versions are still not solved. This is the path forward, at least for now.

Let us know if you want to use metallb, though. As you can see in the comment about ingress, it's nos strictly needed (one option is to use metallb, another one is without). If you decide to not use metallb for ingress, maybe plain services can be accessed via nodePorts and that is enough.

# Check the minikube and host subnet for the VM.
$ minikube ip -p headlamp-desktop-1
192.168.39.136
# We need to configure metallb to assign IPs from this subnet
# It is a /24 subnet
$ ip route | grep 192.168.39
192.168.39.0/24 dev virbr1 proto kernel scope link src 192.168.39.1
$ ip a | grep 39
    inet 192.168.39.1/24 brd 192.168.39.255 scope global virbr1

# The .1 is used by ourselves, and .136 by the VM
# Let's configure metallb to use IPs in this subnet
$ minikube addons configure metallb -p headlamp-desktop-1
-- Enter Load Balancer Start IP: 192.168.39.10
-- Enter Load Balancer End IP: 192.168.39.20
...

# Let's test this
$ kubectl create deployment nginx --image=nginx
$ kubectl expose deployment nginx --port=80 --type=LoadBalancer
$ kubectl get svc nginx
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
nginx   LoadBalancer   10.97.148.176   192.168.39.11   80:32144/TCP   8s
$ curl --head  http://192.168.39.11
HTTP/1.1 200 OK
Server: nginx/1.27.5
Date: Tue, 24 Jun 2025 16:33:28 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 16 Apr 2025 12:01:11 GMT
Connection: keep-alive
ETag: "67ff9c07-267"
Accept-Ranges: bytes

# We can see the IP is assigned and announced to this service if we describe it.
$ kubectl describe svc nginx 
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.97.148.176
IPs:                      10.97.148.176
LoadBalancer Ingress:     192.168.39.11 (VIP)
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  32144/TCP
Endpoints:                10.244.56.78:80
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:
  Type    Reason        Age   From                Message
  ----    ------        ----  ----                -------
  Normal  IPAllocated   113s  metallb-controller  Assigned IP "192.168.39.11"
  Normal  nodeAssigned  113s  metallb-speaker     announcing from node "headlamp-desktop-1"

Also, the VM IP seems to be fixed (defined in ~/.minikube/profiles/headlamp-desktop-1/config.json in this case), so the start/stop range doesn't need to be adjusted over time.

[rata]

Ingress

Please note this step assumes you haven't deleted the things from the previous step with metallb (the nginx deployment and service in particular).

$ minikube addons enable ingress -p headlamp-desktop-1
$ kubectl patch configmap ingress-nginx-controller  -n ingress-nginx  --type merge  -p '{"data":{"allow-snippet-annotations":"false"}}'
$ kubectl rollout restart deployment ingress-nginx-controller -n ingress-nginx

Now there are two options to use ingress:
1. Rely on metallb
2. Use the ingress-dns addon

Each one has pros/cons, and you can choose whatever fits better in headlamp.

In the future, if no option is good, we can explore a third one: use the in-cluster coredns server to resolve these IPs + forward to an upstream server (like 1.1.1.1, or 8.8.8.8) for other DNS names. It might be more tricky than it sounds but maybe if we expose coreDNS server to the host, we forward requests to ingresss resources to the ingress-dns server and other things for an upstream recursive server (so github.com and internet in general works), it might be possible.

Ingress with metallb

Make the ingress-controller service type LoadBalancer, so metallb assigns and announces the IP

$ kubectl patch service ingress-nginx-controller  -n ingress-nginx  --type merge  -p '{"spec":{"type":"LoadBalancer"}}'

Check the IP assigned:

$ kubectl get service ingress-nginx-controller  -n ingress-nginx  
NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.104.102.59   192.168.39.12   80:30562/TCP,443:31902/TCP   18h

Now let's create an ingress resource to test this is working:

$ cat ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rata-example
spec:
  rules:
    - host: "rata.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: nginx
              port:
                number: 80
$ kubectl apply -f ingress.yaml 
ingress.networking.k8s.io/rata-example configured

We can add an entry in /etc/hosts so rata.com points to 192.168.39.12.
Please note that all ingress resources created will need to point to the external-IP of the ingress service.

Once you added the entry, you can just do:

$ curl http://rata.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

You can also do it from the browser.

Ingress with ingess-dns

Ingress-dns is a minikube addon to avoid adding entries to /etc/hosts. More
info on the problem here:
https://minikube.sigs.k8s.io/docs/handbook/addons/ingress-dns/

But a big downside that if the user is not using systemd-resolved, then they will lose internet connection when they switch the DNS server. I didn't find a way around that, this DNS server only answers for the resources on the ingress. It doesn't forward to another upstream the requests for github.com, etc. Checking the code, there doesn't seem to be any configuration possible to set an upstream (https://gitlab.com/cryptexlabs/public/development/minikube-ingress-dns/-/blob/master/app/dns-server/nodejs/src/index.js?ref_type=heads). Therefore, the systemd option, where some requests are sent to this DNS server seems to be the best path (as other requests are sent to the currently configured dns servers). But systemd-resolved is not the default in many popular distros :-/

You enable the addon:

$ minikube addons enable ingress-dns -p headlamp-desktop-1

Now let's create an ingress resource to test this is working:

$ cat ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rata-example
spec:
  rules:
    - host: "rata.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: nginx
              port:
                number: 80
$ kubectl apply -f ingress.yaml 
ingress.networking.k8s.io/rata-example configured

Then you need to update the DNS servers for the host according to the
documentation: https://minikube.sigs.k8s.io/docs/handbook/addons/ingress-dns/.
This is the tricky part.

Note that if systemd-resolved is used, it's way simpler. But, of course, not
everyone will be using that.

After that, you can connect to the ingress resource just as before. This time,
we are not using the external-IP for the ingress service (in fact, it doesn't
even have one :)), but we are connecting to the minikube ip directly.

$ curl http://rata.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

[illume]

It looks like there are cases where we need to ask to reboot.

On some distributions, libvirt bridge networking may fail until the host reboots
-- https://minikube.sigs.k8s.io/docs/drivers/kvm2/

I see in the docs there's some checks to query if the system is capable:

virt-host-validate
grep -E -q 'vmx|svm' /proc/cpuinfo && echo yes || echo no

Note: In the docs for each distro I see some other commands being used.

Seems we may need to ask permission to restart libvirt/libvirtd sometimes if networking is not working. See docs.

We can detect Virtualbox (because kvm can't run at the same time) with one of these commands:

VBoxManage list runningvms
lsmod | grep vbox

Supporting each distro will require custom scripts for each distro for libvirt/kvm installation. See docs.

[illume]

• I'm interested to know what testing you've done? Especially other things you've tried, and the machine configurations you tested on.
• Did you try podman/docker? Did you find them lacking in some regard? (both are popular with many linux users I know, and podman is installed by default on Fedora).

Headlamp has an appimage, a flatpack, a tar ball, and a self-made debian package. It's only packaged externally for Arch linux, and someone tried to package it for Fedora but stalled. We only make builds for x64, armv7l, and arm64(aarch64) architectures.

[joaquimrocha]

Let's focus on Ubuntu for now. We can add support for other distros later.
Rebooting is not great, but if necessary, we should be able to ask for that from the user. The next time the user tries to create minikube, we should detect that no reboot is needed and proceed.

[rata]

On Ubuntu 22.04 I haven't seen the reboot problem (nor in Debian testing).

The comment about the reboot issues was added in 2019, but it doesn't point to any specific issue. Checking the first pages of github issues (close and open), I didn't find anything interesting/worrying. The best thing I found is that someone was using the dashboard and the port was being used by another application, they rebooted and that solve the issue (presumably the application taking that port was not auto-started): kubernetes/minikube#13361 (comment).

There are a few more issues, but either not for Linux (kubernetes/minikube#13734) about not using systemd in devuan (kubernetes/minikube#13628). I couldn't find any more "interesting" issues that might affect users today.

@illume

• I'm interested to know what testing you've done? Especially other things you've tried, and the machine configurations you tested on.

What I've done is mostly shared in the comments above. I tested in a Lenovo Carbon X1 laptop from 2019, it has 16GB of RAM, an nvme disk and is running ubuntu 22.04.

Feel free to check what minikube upstream is testing on, though. Maybe that is more interesting for you?

• Did you try podman/docker? Did you find them lacking in some regard? (both are popular with many linux users I know, and podman is installed by default on Fedora).

No. We focused on VM drivers due to the requirement to run Inspektor Gadget that Chris said. IG has several kernel requirements, having control over that to provide a good user-experience was a requirement.

[illume]

Thanks @rata

On Ubuntu 22.04 I haven't seen the reboot problem (nor in Debian testing).

I also could not reproduce, and asked one other person and they couldn't. Made a PR to remove it from minikube docs:

• site: kvm2: Remove old part about needing to reboot kubernetes/minikube#21020

We focused on VM drivers due to the requirement to run Inspektor Gadget

Inspektor gadget works well in my experience with the docker driver. It's the driver the Inspektor gadget docs recommend works best, and I confirmed with someone from the Inspektor gadget team and they said "it works fine".

I'm not sure about podman driver and Inspektor gadget, I don't know of anyone who uses that.