Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 4 additions & 28 deletions docs/k8s-kubespray-upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,38 +28,14 @@ Once the group variables are set, you can proceed with the upgrade execution.

Running an upgrade with Kubespray is fairly simple and executed via `ansible-playbook`.

Before running the playbook be sure to source your environment variables.
* We recoommend running the upgrade from a control node that is not part of the cluster being upgraded. This helps to ensure that the upgrade process is not competing for resources with the cluster being upgraded.

``` shell
source /opt/genestack/scripts/genestack.rc
```

Change to the `kubespary` directory.

``` shell
cd /opt/genestack/submodules/kubespray
```

!!! note

When running an upgrade be sure to set the `kube_version` variable to the desired version number.

Now run the upgrade.

``` shell
ansible-playbook upgrade-cluster.yml -e kube_version=${VERSION_NUMBER}
```

!!! note

While the basic command could work, be sure to include any and all flags needed for your environment before running the upgrade.

### Running an unsafe upgrade
* We also recommend running the upgrade within a `screen` or `tmux` session to help ensure that the upgrade process is not interrupted by network issues.

When running an upgrade, it is possible to force the upgrade by running the cluster playbook with the `upgrade_cluster_setup` flag set to **true**. This option is a lot faster, though does introduce the possibility of service disruption during the upgrade operation.
The script to run the upgrade can be found within the `scripts/kubespray-major-upgrade.sh` file; however, the process is fairly straightforward.

``` shell
ansible-playbook cluster.yml -e upgrade_cluster_setup=true -e kube_version=${VERSION_NUMBER}
--8<-- "scripts/kubespray-major-upgrade.sh"
```

### Upgrade Hangs
Expand Down
60 changes: 60 additions & 0 deletions scripts/kubespray-major-upgrade.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env bash

set -e

function gitRepoVersion() {
# Returns the current git branch name, tag, or commit hash
echo "$(git symbolic-ref -q --short HEAD || git describe --tags --exact-match 2>/dev/null || git rev-parse HEAD)"
}

echo "This script will help you upgrade your Kubernetes cluster managed by Kubespray."

# Ask the user for the target Kubernetes version
read -p "Enter the target Kubernetes version number (e.g., 1.34.0): " VERSION_NUMBER

# Confirm the version number with the user
echo "You have entered Kubernetes version number: ${VERSION_NUMBER}"

echo "Your current setup is as follows:"
# Show checkout version of Genestack and the branch being used
pushd /opt/genestack &>/dev/null
echo "[+] Current Genestack version: $(gitRepoVersion) (SHA:$(git rev-parse HEAD))"
popd &>/dev/null

pushd /opt/genestack/submodules/kubespray &>/dev/null
echo "[+] Current Kubespray version: $(gitRepoVersion) (SHA:$(git rev-parse HEAD))"
popd &>/dev/null

read -p "Is all of this correct? If yes type \`DOTHETHINGNOW\`: " CONFIRMATION

if [[ "$CONFIRMATION" != "DOTHETHINGNOW" ]]; then
echo "Aborting. Please run the script again and enter the correct version number and confirmation."
exit 1
fi

set -v

# Load Genestack environment variables
. /opt/genestack/scripts/genestack.rc

# Navigate to the Kubespray directory and perform the upgrade
pushd /opt/genestack/submodules/kubespray &>/dev/null
echo "Gathering cluster facts"
ansible-playbook playbooks/facts.yml --become

echo "Upgrading cluster to Kubernetes version ${VERSION_NUMBER}"
ansible-playbook upgrade-cluster.yml --become -e kube_version${VERSION_NUMBER} --limit "kube_control_plane:etcd"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we pass the k8s desired version here, will the playbook ignore whats in group_vars/k8s_cluster?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. but the script updates the group vars as soon as the upgrade is complete.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed upgrade-cluster.yml will attempt to drain the node(s) vs cluster.yml with the -e var will not. What's the preference?

The documentation mentions that "components will be upgraded in the same order they were installed", which does present an issue with kubelet being upgraded prior to kube-apiserver. Kubelet does still run on these control plane nodes, so I wonder if there is some logic that could be added to the playbooks to ensure ordering or a validation that kube-apiserver is >= new kubelet prior to upgrade.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should look into that.


echo "Upgrading worker nodes to Kubernetes version ${VERSION_NUMBER}"
ansible-playbook upgrade-cluster.yml --become -e kube_version${VERSION_NUMBER} --limit "!kube_control_plane:!etcd"
popd &>/dev/null

echo "Kubernetes cluster upgrade to version ${VERSION_NUMBER} completed successfully."

if command -v yq &>/dev/null; then
echo "Updating Kubernetes version in inventory files"
yq -i ".kube_version = \"${VERSION_NUMBER}\"" /etc/genestack/inventory/group_vars/k8s_cluster/k8s-cluster.yml
else
echo "yq command not found. Please install yq to update inventory files."
echo "update the Kubernetes version in /etc/genestack/inventory/group_vars/k8s_cluster/k8s-cluster.yml manually to complete the upgrade."
fi