[Draft]: Add policy entropy metric #214
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SamComber
force-pushed
the
add-policy-entropy
branch
from
d18f7ec to
08eab04
Compare
SamComber
changed the title
SamComber
changed the title
SamComber
changed the title
SamComber
force-pushed
the
add-policy-entropy
branch
3 times, most recently
from
bec0673 to
16a14d4
Compare
qgallouedec
reviewed
Aug 19, 2025
verifiers/trainers/grpo_trainer.py
Outdated
| logits = logits / self.temperature | ||
|
|
||
| if compute_entropy: | ||
| entropy = entropy_from_logits_memory_efficient(logits, chunk_size=32) |
There was a problem hiding this comment.
hey! if you want something aligned with trl, we have entropy_from_logits that you can directly import
it does the same as here
There was a problem hiding this comment.
Oh this is perfect! Thanks for the suggestion
There was a problem hiding this comment.
Still getting OOMs with chunk size 1 here in places where we definitely should not be hmm @qgallouedec , does entropy_from_logits internally need to torch.no_grad() perhaps?
SamComber
force-pushed
the
add-policy-entropy
branch
from
16a14d4 to
8f4d018
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Would be good to have an option to understand potential policy entropy collapse. Exhaustion is typically indicative of growing determinism in token decoding and diminishes exploration behaviour (see https://arxiv.org/pdf/2505.22617 paper)
WIP
Need to optimise so we don't create
B*L*Vintermediate tensors inside the entropy calc (i.e. basically 3x the memory required right now)... will likely batch it and put a warning flag when calculate entropy is enabled ??