-
Notifications
You must be signed in to change notification settings - Fork 5
docs: Move memory tuning to an advanced topic #970
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
7327627
5119c3c
668ec22
f0825a6
455954f
707930c
5a8f1f0
12d9d53
11f0625
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| --- | ||
| title: "JVM memory tuning" | ||
| description: Configure JVM memory parameters for Seqera Platform Enterprise deployments | ||
| date created: "2025-12-17" | ||
| tags: [configuration, jvm, memory, tuning] | ||
| --- | ||
|
|
||
| # JVM memory tuning | ||
|
|
||
| :::warning | ||
| JVM memory tuning is an advanced topic that may cause instability and performance issues. | ||
| ::: | ||
|
|
||
| Seqera Platform scales memory allocation based on resources allocated to the application. To best inform available memory, set memory requests and limits on your deployments. We recommend increasing memory allocation before manually configuring JVM settings. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "increasing memory allocation" -- I assume this means requests / limits in the K8s manifests? Vertical scaling on a docker compose node?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This applies to docker compose as-well. We should be setting these values
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not currently defined in the docker-compose template (maybe we should add it so things are aligned?) # https://docs.seqera.io/assets/files/docker-compose-0655848af8f21b6e6211d1a9c8ebc702.yml
backend:
image: cr.seqera.io/private/nf-tower-enterprise/backend:v25.3.0
platform: linux/amd64
command: -c '/wait-for-it.sh db:3306 -t 60; /tower.sh'
networks:
- frontend
- backend
expose:
- 8080
volumes:
- $PWD/tower.yml:/tower.yml
# Data studios RSA key is required for the data studios functionality. Uncomment the line below to mount the key.
#- $PWD/data-studios-rsa.pem:/data-studios-rsa.pem
env_file:
# Seqera environment variables — see https://docs.seqera.io/platform-enterprise/enterprise/configuration/overview for details
- tower.env
environment:
# Micronaut environments are required. Do not edit these values
- MICRONAUT_ENVIRONMENTS=prod,redis,ha
restart: always
depends_on:
- db
- redis
- cron
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a scenario when we expect a client would need to start tinkering with the JVM settings? When is it? How would it be identified?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Some of this is answered by #954 where both of these will need revisions. JVM monitoring links back to overall system monitoring. |
||
|
|
||
| If you wish to manually configure JVM memory, use the following baseline recommendations. | ||
|
|
||
| ## Memory parameters | ||
|
|
||
| Set JVM memory parameters using the `JAVA_OPTS` environment variable. The following parameters control memory allocation: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where is this being set? Given what's been pulled out, looks like as an environment variable in tower.env / an env in the K8s manifests.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Something like JVM memory parameters can be configured using the |
||
|
|
||
| | Parameter | Description | | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there an external source we can direct folks to to see more fulsome examples? Or is the idea "you should NOT be messing with this if you don't understand what you are doing"?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I could not find an easily accessible external source which did not go deep into compiler and runtime behaviors |
||
| | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | ||
| | `-Xms` / `-Xmx` | Set the initial (`Xms`) and maximum (`Xmx`) heap size. The heap stores Java objects and should be 50-70% of total allocated memory. | | ||
| | `-XX:MaxDirectMemorySize` | Set the maximum direct (off-heap) memory. Used for NIO operations, network buffers, and file I/O. | | ||
| | `-XX:ActiveProcessorCount` | Set the number of CPUs available to the JVM. Should match the number of vCPUs allocated to the container. | | ||
|
|
||
| ## Resource allocation guidelines | ||
|
|
||
| - **Heap (`-Xmx`)**: 50-70% of total allocated memory | ||
| - **Direct memory**: 10-20% of total allocated memory | ||
| - **Overhead** (metaspace, thread stacks, native memory): ~10% of total allocated memory | ||
|
|
||
| Ensure total JVM memory (heap + direct memory + overhead) does not exceed container memory limits. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I understand what this means for K8s. Less sure on first glance how I'd do it with docker compose.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Answered above we can configure limits on compose it is a recommended pattern especially when running other services on the cluster As a general compose best practice we should be allocating memory limits to all pods and ensuring that total limit gives enough headroom for the host OS for example 0.5 -2Gb |
||
|
|
||
| ## Example configurations | ||
|
|
||
| The following table provides example configurations for common deployment sizes. These are starting points and may need to be tuned based on your specific usage patterns. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See earlier question about "What scenario would trigger a need to tinker?" |
||
|
|
||
| | vCPU | RAM | Heap (`-Xmx`) | Direct Memory | `JAVA_OPTS` | | ||
| | :--: | :---: | :-----------: | :-----------: | ------------------------------------------------------------------------------- | | ||
| | 1 | 2 GB | 1 GB | 512 MB | `-XX:ActiveProcessorCount=1 -Xms500M -Xmx1000M -XX:MaxDirectMemorySize=512m` | | ||
| | 1 | 4 GB | 2.5 GB | 800 MB | `-XX:ActiveProcessorCount=1 -Xms1000M -Xmx2500M -XX:MaxDirectMemorySize=800m` | | ||
| | 2 | 2 GB | 1 GB | 512 MB | `-XX:ActiveProcessorCount=2 -Xms500M -Xmx1000M -XX:MaxDirectMemorySize=512m` | | ||
| | 2 | 4 GB | 2 GB | 800 MB | `-XX:ActiveProcessorCount=2 -Xms1000M -Xmx2000M -XX:MaxDirectMemorySize=800m` | | ||
| | 2 | 8 GB | 5 GB | 1.5 GB | `-XX:ActiveProcessorCount=2 -Xms2000M -Xmx5000M -XX:MaxDirectMemorySize=1500m` | | ||
| | 3 | 2 GB | 1 GB | 512 MB | `-XX:ActiveProcessorCount=3 -Xms500M -Xmx1000M -XX:MaxDirectMemorySize=512m` | | ||
| | 3 | 4 GB | 2 GB | 800 MB | `-XX:ActiveProcessorCount=3 -Xms1000M -Xmx2000M -XX:MaxDirectMemorySize=800m` | | ||
| | 3 | 8 GB | 5 GB | 1.5 GB | `-XX:ActiveProcessorCount=3 -Xms2000M -Xmx5000M -XX:MaxDirectMemorySize=1500m` | | ||
| | 3 | 16 GB | 11 GB | 2.5 GB | `-XX:ActiveProcessorCount=3 -Xms4000M -Xmx11000M -XX:MaxDirectMemorySize=2500m` | | ||
|
|
||
| ## When to adjust memory settings | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, it's at the bottom. Based on my questions, I think this would be more useful nearer to the top. |
||
|
|
||
| Adjust your JVM memory settings if you observe the following issues in your deployment: | ||
|
|
||
| **Increase heap memory (`-Xmx`)** if you see: | ||
|
|
||
| - `OutOfMemoryError: Java heap space` errors in logs | ||
| - Garbage collection pauses affecting performance | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How are we expecting these metrics to be visible. IIRC you dont get memory metrics on the standards EC2 monitoring package. Do we expect the client to upgrade their monitoring system / be using an aggregating agent like Datadog?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As above they would need to be monitoring via prometheus / or another agent and monitoring JVM stats. |
||
| - Steadily growing memory usage under sustained load | ||
|
|
||
| **Increase direct memory (`MaxDirectMemorySize`)** if you see: | ||
|
|
||
| - `OutOfMemoryError: Direct buffer memory` errors in logs | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Increase relative to what?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It should not be at the expense of the other. There is a typical expected ratio of heap vs direct memory. If the heap is hitting 100% useage that can be scaled on it's own you can then review your direct memory usage and opt to reduce if you have overhead or increase memory allocated to the pod. |
||
| - High concurrent workflow launch rates (more than 100 simultaneous workflows) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is 100 a known pain point when using the default options or was this just chosen because it's a nice number?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes workflows is bad here as it's not nextflow it's Java task allocation related. |
||
| - Large configuration payloads or extensive API usage | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Large" ==?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Happy to drop these |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| --- | ||
| title: "JVM memory tuning" | ||
| description: Configure JVM memory parameters for Seqera Platform Enterprise deployments | ||
| date created: "2025-12-17" | ||
| tags: [configuration, jvm, memory, tuning] | ||
| --- | ||
|
|
||
| # JVM memory tuning | ||
|
|
||
| :::warning | ||
| JVM memory tuning is an advanced topic that may cause instability and performance issues. | ||
| ::: | ||
|
|
||
| Seqera Platform scales memory allocation based on resources allocated to the application. To best inform available memory, set memory requests and limits on your deployments. We recommend increasing memory allocation before manually configuring JVM settings. | ||
|
|
||
| If you wish to manually configure JVM memory, use the following baseline recommendations. | ||
|
|
||
| ## Memory parameters | ||
|
|
||
| Set JVM memory parameters using the `JAVA_OPTS` environment variable. The following parameters control memory allocation: | ||
|
|
||
| | Parameter | Description | | ||
| | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | ||
| | `-Xms` / `-Xmx` | Set the initial (`Xms`) and maximum (`Xmx`) heap size. The heap stores Java objects and should be 50-70% of total allocated memory. | | ||
| | `-XX:MaxDirectMemorySize` | Set the maximum direct (off-heap) memory. Used for NIO operations, network buffers, and file I/O. | | ||
| | `-XX:ActiveProcessorCount` | Set the number of CPUs available to the JVM. Should match the number of vCPUs allocated to the container. | | ||
|
|
||
| ## Resource allocation guidelines | ||
|
|
||
| - **Heap (`-Xmx`)**: 50-70% of total allocated memory | ||
| - **Direct memory**: 10-20% of total allocated memory | ||
| - **Overhead** (metaspace, thread stacks, native memory): ~10% of total allocated memory | ||
|
|
||
| Ensure total JVM memory (heap + direct memory + overhead) does not exceed container memory limits. | ||
|
|
||
| ## Example configurations | ||
|
|
||
| The following table provides example configurations for common deployment sizes. These are starting points and may need to be tuned based on your specific usage patterns. | ||
|
|
||
| | vCPU | RAM | Heap (`-Xmx`) | Direct Memory | `JAVA_OPTS` | | ||
| | :--: | :---: | :-----------: | :-----------: | ------------------------------------------------------------------------------- | | ||
| | 1 | 2 GB | 1 GB | 512 MB | `-XX:ActiveProcessorCount=1 -Xms500M -Xmx1000M -XX:MaxDirectMemorySize=512m` | | ||
| | 1 | 4 GB | 2.5 GB | 800 MB | `-XX:ActiveProcessorCount=1 -Xms1000M -Xmx2500M -XX:MaxDirectMemorySize=800m` | | ||
| | 2 | 2 GB | 1 GB | 512 MB | `-XX:ActiveProcessorCount=2 -Xms500M -Xmx1000M -XX:MaxDirectMemorySize=512m` | | ||
| | 2 | 4 GB | 2 GB | 800 MB | `-XX:ActiveProcessorCount=2 -Xms1000M -Xmx2000M -XX:MaxDirectMemorySize=800m` | | ||
| | 2 | 8 GB | 5 GB | 1.5 GB | `-XX:ActiveProcessorCount=2 -Xms2000M -Xmx5000M -XX:MaxDirectMemorySize=1500m` | | ||
| | 3 | 2 GB | 1 GB | 512 MB | `-XX:ActiveProcessorCount=3 -Xms500M -Xmx1000M -XX:MaxDirectMemorySize=512m` | | ||
| | 3 | 4 GB | 2 GB | 800 MB | `-XX:ActiveProcessorCount=3 -Xms1000M -Xmx2000M -XX:MaxDirectMemorySize=800m` | | ||
| | 3 | 8 GB | 5 GB | 1.5 GB | `-XX:ActiveProcessorCount=3 -Xms2000M -Xmx5000M -XX:MaxDirectMemorySize=1500m` | | ||
| | 3 | 16 GB | 11 GB | 2.5 GB | `-XX:ActiveProcessorCount=3 -Xms4000M -Xmx11000M -XX:MaxDirectMemorySize=2500m` | | ||
|
|
||
| ## When to adjust memory settings | ||
|
|
||
| Adjust your JVM memory settings if you observe the following issues in your deployment: | ||
|
|
||
| **Increase heap memory (`-Xmx`)** if you see: | ||
|
|
||
| - `OutOfMemoryError: Java heap space` errors in logs | ||
| - Garbage collection pauses affecting performance | ||
| - Steadily growing memory usage under sustained load | ||
|
|
||
| **Increase direct memory (`MaxDirectMemorySize`)** if you see: | ||
|
|
||
| - `OutOfMemoryError: Direct buffer memory` errors in logs | ||
| - High concurrent workflow launch rates (more than 100 simultaneous workflows) | ||
| - Large configuration payloads or extensive API usage |
Uh oh!
There was an error while loading. Please reload this page.