Skip to content

Note regarding mega nodes #3823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Note regarding mega nodes #3823

Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f674d89
Note regarding mega nodes
Tejeev Jan 25, 2022
3c78d32
Edited note
Jan 25, 2022
b2695da
Fixed extra line spacing
Jan 25, 2022
d1c364b
Edited spacing
Jan 25, 2022
a0a20a2
Removed unnecessary space
Jan 25, 2022
def366e
Removed unnecessary space
Jan 25, 2022
e190af3
Added link
Jan 25, 2022
f78fcf2
Updated verbiage per feedback
Feb 10, 2022
8a99c3d
Update content/rke/latest/en/os/_index.md
Tejeev Mar 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion content/rke/latest/en/os/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,7 +297,17 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne

Copy link
Contributor

@cbron cbron Jul 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we put this in the rke docs ? Wouldn't this apply to any large cluster, including rke2.
We need to make a scaling section, but for now most of that data lives over here: https://rancher.com/docs/rancher/v2.5/en/installation/requirements/

### Large Kubernetes Clusters

For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters](https://kubernetes.io/docs/setup/best-practices/cluster-large/).
Kubernetes is engineered around the concept of horizontal scaling for redundancy, so scaling vertically with large servers can be problematic if the proper minimum/maximum requirements aren’t followed. The following are tips and recommendations for large Kubernetes clusters:

- If you must use servers larger than 24 CPU, use virtualization tooling, such as what [Harvester](https://docs.harvesterhci.io/v1.0/rancher/virtualization-management/) provides, to subdivide the servers.
Copy link
Contributor

@cbron cbron Jul 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you must use servers larger than 24 CPU

Where did that 24 CPU number come from ?


- Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a minimum of roughly 24 cores per server and a maximum of the recommended 110 pods per server.
Copy link
Contributor

@jtravee jtravee Mar 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a minimum of roughly 24 cores per server and a maximum of the recommended 110 pods per server.
- Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a maximum of roughly 24 cores per server and a maximum of the recommended 110 pods per server.

Copy link
Contributor

@jtravee jtravee Mar 8, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Tejeev, checking that minimum should be maximum here?

Copy link
Contributor Author

@Tejeev Tejeev Mar 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, though these are somewhat softer than I think I make it sound. Folks that know what they are doing may up the pod count to the real limit of 251 (IIRC. we should double check that) and there are use cases that might use more cores, per pod, though at the moment they are rare. Not sure how to convey this a bit better or if we leave it as is to prevent over committing by folks that do not have real need of increased limits

Copy link
Contributor Author

@Tejeev Tejeev Mar 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps:
"It is recommended to maintain a maximum of roughly 24 cores and 110 pods per server"


- If you are deploying an application or system that specifically requires a large number of pods, you may increase the pod limit. Please note, however, that going above 254 pods per server is not supported by default pod CIDR settings unless the pods are using host networking.
Copy link
Contributor

@cbron cbron Jul 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This goes against upstream docs best practice: "No more than 110 pods per node": https://kubernetes.io/docs/setup/best-practices/cluster-large/

Also saying "you may increase the pod limi" implies that this is supported, and we have not tested this in our scaling tests, we do not want to give this impression.


- Even when deploying small workloads, it is recommended that you use a virtualization layer to facilitate less downtime and shorter reboots during upgrades and failures.

For additional hardware recommendations for large Kubernetes clusters and nodes, refer to the official Kubernetes documentation on [building large clusters](https://kubernetes.io/docs/setup/best-practices/cluster-large/).

### Etcd Clusters

Expand Down