Skip to content

Commit e736878

Browse files
authored
added note for sampling parameter with log probs (#798)
1 parent 423bfc7 commit e736878

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

docs/model_config.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -244,7 +244,7 @@ Below is the lists of input and output tensors for the `tensorrt_llm` and
244244
| `presence_penalty` | [1] | `float` | Sampling Config param: `presencePenalty` |
245245
| `frequency_penalty` | [1] | `float` | Sampling Config param: `frequencyPenalty` |
246246
| `seed` | [1] | `uint64_t` | Sampling Config param: `seed` |
247-
| `return_log_probs` | [1] | `bool` | When `true`, include log probs in the output |
247+
| `return_log_probs` | [1] | `bool` | When `true`, include log probs in the output. Note: This requires at least one sampling parameter to be set (e.g., `runtime_top_k`, `runtime_top_p` for `tensorrt_llm` model, or `top_k`, `top_p` for `tensorrt_llm_bls` model). |
248248
| `return_context_logits` | [1] | `bool` | When `true`, include context logits in the output |
249249
| `return_generation_logits` | [1] | `bool` | When `true`, include generation logits in the output |
250250
| `num_return_sequences` | [1] | `int32_t` | Number of generated sequences per request. (Default=1) |
@@ -272,7 +272,7 @@ Note: the timing metrics oputputs are represented as the number of nanoseconds s
272272
| Name | Shape | Type | Description |
273273
| :------------: | :---------------: | :-----------: | :--------: |
274274
| `cum_log_probs` | [-1] | `float` | Cumulative probabilities for each output |
275-
| `output_log_probs` | [beam_width, -1] | `float` | Log probabilities for each output |
275+
| `output_log_probs` | [beam_width, -1] | `float` | Per-token log probabilities for each output. Only returned when `return_log_probs` is `true` and sampling parameters are set. |
276276
| `context_logits` | [-1, vocab_size] | `float` | Context logits for input |
277277
| `generation_logits` | [beam_width, seq_len, vocab_size] | `float` | Generation logits for each output |
278278
| `batch_index` | [1] | `int32` | Batch index |

0 commit comments

Comments
 (0)