Skip to content

Conversation

@klutchell
Copy link
Contributor

@klutchell klutchell commented Sep 5, 2025

This PR introduces runtime kernel memory management tweaks to address frequent "out of memory error 12" issues during firecracker VM startup by optimizing memory allocation behavior and cache reclamation.

  • Adds a new memory-tweaks service that applies kernel memory tuning parameters
  • Configures aggressive memory compaction and higher minimum free memory thresholds
  • Disables transparent hugepages to reduce memory fragmentation

Change-type: minor
See: https://balena.fibery.io/Security/Information_Security_and_Reliability_Incident/GitHub-Runners-Failure-during-vcpu-run-Out-of-memory-(os-error-12)-130

@aethernet
Copy link

/rebase

@git-basejump git-basejump bot force-pushed the kyle/memory-tweaks branch from d993257 to 7afe809 Compare September 8, 2025 13:12
@klutchell klutchell marked this pull request as ready for review September 19, 2025 20:20
@klutchell
Copy link
Contributor Author

/rebase

@klutchell klutchell marked this pull request as draft September 19, 2025 20:20
@klutchell klutchell requested a review from Copilot October 14, 2025 14:28
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces runtime kernel memory management tweaks to address frequent "out of memory error 12" issues during firecracker VM startup by optimizing memory allocation behavior and cache reclamation.

  • Adds a new memory-tweaks service that applies kernel memory tuning parameters
  • Configures aggressive memory compaction and higher minimum free memory thresholds
  • Disables transparent hugepages to reduce memory fragmentation

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

These adjustments prefer writing to disk and reclaiming cache
earlier in order to avoid the frequent "out of memory error 12" we
have been experiencing on firecracker VM startup.

We observed very large (50%) memory caches that were not freed
up in time to start the next VM, so these settings will optimize
for freeing the memory sooner.

Change-type: minor
Signed-off-by: Kyle Harding <[email protected]>
Change-type: patch
Signed-off-by: Kyle Harding <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants