[TRTLLM-5930][doc] 1.0 Documentation. (#6696)

Signed-off-by: nv-guomingz <[email protected]>
Signed-off-by: Wangshanshan <[email protected]>

Author: nv-guomingz

docs/source/advanced/lora.md

@@ -133,9 +133,9 @@ Next, consider this linear layer is a `RowLinear` layer. When we partition the w
 
 #### DoRA
 
-TRTLLM supports DoRA as described in https://arxiv.org/abs/2402.09353 . To enable DoRA, you must add the additional `--dora_plugin enable` flag to the `trtllm-build` command.
+TensorRT-LLM supports DoRA as described in https://arxiv.org/abs/2402.09353 . To enable DoRA, you must add the additional `--dora_plugin enable` flag to the `trtllm-build` command.
 
-The DoRA scales must be normalized before they are submitted to TRTLLM in an inference request. The normalization requires the base model weights. To normalize your adapter you may use the script provided in `tensorrt_llm/examples/dora/normalize_weights.py`.
+The DoRA scales must be normalized before they are submitted to TensorRT-LLM in an inference request. The normalization requires the base model weights. To normalize your adapter you may use the script provided in `tensorrt_llm/examples/dora/normalize_weights.py`.
 
 When using DoRA, the format of `LoraWeights` and `LoraConfig` changes slightly.
 The shape of `LoraConfig` becomes `[module_id, layer_idx, adapter_size D (i.e. R value), is_dora]`, with `is_dora` a boolean flag that determines whether the supplied adapter contains DoRA scales or not. If the old config shape is used, it is assumed the adapter does not have DoRA scales.

docs/source/advanced/speculative-decoding.md

@@ -173,7 +173,7 @@ Similarly to ReDrafter, TensorRT-LLM implements the EAGLE model such that logits
 
 ### Disaggregated Serving
 
-[Disaggregated Serving](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/advanced/disaggregated-service.md) with EAGLE-3 using the two-model approach is supported in the PyTorch backend.
+[Disaggregated Serving](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disaggregated-service.md) with EAGLE3 using the two model approach is supported in the Pytorch backend. Please refer to the following [Dynamo example](https://github.com/ai-dynamo/dynamo/blob/main/examples/tensorrt_llm/llama4_plus_eagle.md) on how to run EAGLE3 with Disaggregated Serving for Llama 4 Maverick.
 
 ## Lookahead Decoding
 

docs/source/architecture/overview.md

@@ -1,18 +1,75 @@
-(architecture-overview)=
+# Architecture Overview
 
-# TensorRT-LLM Architecture
+The `LLM` class is a core entry point for the TensorRT-LLM, providing a simplified `generate()` API for efficient large language model inference. This abstraction aims to streamline the user experience, as demonstrated with TinyLlama:
 
-TensorRT-LLM is a toolkit to assemble optimized solutions to perform Large Language Model (LLM) inference. It offers a Model Definition API to define models and compile efficient [TensorRT](https://developer.nvidia.com/tensorrt) engines for NVIDIA GPUs. It also contains Python and C++ components to build runtimes to execute those engines as well as backends for the [Triton Inference
-Server](https://developer.nvidia.com/nvidia-triton-inference-server) to easily create web-based services for LLMs. TensorRT-LLM supports multi-GPU and multi-node configurations (through MPI).
+```python
+from tensorrt_llm import LLM
 
-As a user, the very first step to create an inference solution is to either define your own model or select a pre-defined network architecture (refer to {ref}`models` for the list of models supported by TensorRT-LLM). Once defined, that model must be trained using a training framework (training is outside of the scope of TensorRT-LLM). For pre-defined models, checkpoints can be downloaded from various providers. To illustrate that point, a lot of examples in TensorRT-LLM use model weights obtained from the [Hugging Face](https://huggingface.co) hub and trained using [NVIDIA Nemo](https://developer.nvidia.com/nemo) or [PyTorch](https://pytorch.org).
+# Initialize the LLM with a specified model
+llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
 
-Equipped with the model definition and the weights, a user must use TensorRT-LLM's Model Definition API to recreate the model in a way that can be compiled by TensorRT into an efficient engine. For ease of use, TensorRT-LLM already supports a handful of standard models.
+# Generate text using the model
+output = llm.generate("Hello, my name is")
+```
 
-Together with the Model Definition API to describe models, TensorRT-LLM provides users with components to create a runtime that executes the efficient TensorRT engine. Runtime components offer beam-search, along with extensive sampling functionalities such as top-K and top-P sampling. The exhaustive list can be found in the documentation of the {ref}`gpt-runtime`. The C++ runtime is the recommended runtime.
+The `LLM` class automatically manages essential pre and post-processing steps, including tokenization (encoding input prompts into numerical representations) and detokenization (decoding model outputs back into human-readable text).
 
-TensorRT-LLM also includes Python and C++ backends for NVIDIA Triton Inference Server to assemble solutions for LLM online serving. The C++ backend implements in-flight batching as explained in the {ref}`executor` documentation and is the recommended backend.
+Internally, the `LLM` class orchestrates the creation of a dedicated `PyExecutor(Worker)` process on each rank.
 
-## Model Weights
+![TRT-LLM Architecture Overview](../media/TRTLLM_Architecture_Overview.png)
 
-TensorRT-LLM is a library for LLM inference, and so to use it, you need to supply a set of trained weights. You can either use your own model weights trained in a framework like [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework/) or pull a set of pretrained weights from repositories like the Hugging Face Hub.
+This `PyExecutor` operates in a continuous background loop, designed for the efficient, asynchronous processing of inference requests.
+
+The `PyExecutor`'s functionality is built upon several key components:
+
+- `Scheduler`: Responsible for determining which active requests are ready for execution at each processing step.
+
+- `KVCacheManager`: Manages the allocation, deallocation, and maintenance of the Key-Value (KV) Cache. This is a critical optimization for Transformer models, significantly enhancing performance during autoregressive text generation by storing previously computed attention keys and values.
+
+- `ModelEngine`: Handles the loading and highly efficient execution of the language model on the GPU hardware.
+
+- `Sampler`: Takes the raw outputs (logits) from the ModelEngine and applies appropriate sampling strategies (e.g., greedy, top-k, top-p, beam search) to generate the final output tokens.
+
+During each iteration of its background loop, the `PyExecutor` performs the following sequence of operations:
+
+- Request Fetching: Retrieves new inference requests from an internal request queue, if available.
+
+- Scheduling: Interacts with the `Scheduler` to identify and prioritize requests that are ready to be processed in the current step.
+
+- Resource Preparation: Coordinates with the `KVCacheManager` to ensure that the necessary Key-Value (KV) Cache resources are allocated for the selected requests.
+
+- Model Execution: Invokes the `ModelEngine` to perform a forward pass on the scheduled requests, predicting the next output tokens.
+
+- Output Handling: Updates the partial outputs for ongoing requests and finalizes the results for any requests that have reached completion, returning them to the user.
+
+
+## Runtime Optimizations
+
+TensorRT-LLM enhances inference throughput and reduces latency by integrating a suite of runtime optimizations, including CUDA Graph, [Overlap Scheduler](../features/overlap-scheduler.md), [Speculative decoding](../features/speculative-decoding.md), etc.
+
+### CUDA Graph
+
+CUDA Graphs drastically reduce the CPU-side overhead associated with launching GPU kernels, which is particularly impactful in PyTorch-based inference where Python's host-side code can be a bottleneck. By capturing a sequence of CUDA operations as a single graph, the entire sequence can be launched with one API call, minimizing CPU-GPU synchronization and driver overhead.
+
+To maximize the "hit rate" of these cached graphs, TensorRT-LLM employs CUDA Graph padding. If an incoming batch's size doesn't match a captured graph, it's padded to the nearest larger, supported size for which a graph exists. While this incurs minor overhead from computing "wasted" tokens, it's often a better trade-off than falling back to slower eager mode execution. This optimization has a significant impact, demonstrating up to a 22% end-to-end throughput increase on certain models and hardware.
+
+### Overlap Scheduler
+
+The Overlap Scheduler maximizes GPU utilization by hiding CPU-bound latency behind GPU computation.
+
+The key strategy is to launch the GPU's work for the next step (n+1) immediately, without waiting for the CPU to finish processing the results of the current step (n). This allows the CPU to handle tasks like checking stop criteria or updating responses for one batch while the GPU is already executing the model for the subsequent batch.
+
+This concurrent execution pipeline is illustrated in the `PyExecutor`'s logic:
+
+```python
+# Schedule and launch GPU work for the current step (n)
+scheduled_batch, _, _ = self._schedule()
+batch_outputs = self._forward_step(scheduled_batch, previous_tensors_device)
+sample_state = self._sample_async(scheduled_batch, batch_outputs)
+
+# While the GPU is busy, process the CPU-bound results from the previous step (n-1)
+if self.previous_batch is not None:
+    self._process_previous_batch()
+```
+
+This approach effectively reduces GPU idle time and improves overall hardware occupancy. While it introduces one extra decoding step into the pipeline, the resulting throughput gain is a significant trade-off. For this reason, the Overlap Scheduler is enabled by default in TensorRT-LLM.

docs/source/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.md

@@ -172,7 +172,7 @@ To minimize KV cache transmission latency, TensorRT-LLM currently uses direct tr
 </div>
 <p align="center"><sub><em>Figure 8. KV cache layout conversion</em></sub></p>
 
-The optimizations required for KV cache transmission vary depending on whether it's single-node multi-GPU, multi-node multi-GPU, or different GPU models. To accommodate this, TensorRT-LLM provides a set of environment variables for selection in different environments. Please refer to [this document](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/advanced/disaggregated-service.md) for details.
+The optimizations required for KV cache transmission vary depending on whether it's single-node multi-GPU, multi-node multi-GPU, or different GPU models. To accommodate this, TensorRT-LLM provides a set of environment variables for selection in different environments. Please refer to [this document](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disagg-serving.md) for details.
 
 ## Performance Studies
 

docs/source/commands/trtllm-eval.rst

@@ -0,0 +1,89 @@
+trtllm-eval
+===========
+
+About
+-----
+
+The ``trtllm-eval`` command provides developers with a unified entry point for accuracy evaluation. It shares the core evaluation logic with the `accuracy test suite <https://github.com/NVIDIA/TensorRT-LLM/tree/main/tests/integration/defs/accuracy>`_ of TensorRT-LLM.
+
+``trtllm-eval`` is built on the offline API -- LLM API. Compared to the online ``trtllm-serve``, the offline API provides clearer error messages and simplifies the debugging workflow.
+
+The following tasks are currently supported:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 25 15 15 15
+
+   * - Dataset
+     - Task
+     - Metric
+     - Default ISL
+     - Default OSL
+   * - CNN Dailymail
+     - summarization
+     - rouge
+     - 924
+     - 100
+   * - MMLU
+     - QA; multiple choice
+     - accuracy
+     - 4,094
+     - 2
+   * - GSM8K
+     - QA; regex matching
+     - accuracy
+     - 4,096
+     - 256
+   * - GPQA
+     - QA; multiple choice
+     - accuracy
+     - 32,768
+     - 4,096
+   * - JSON mode eval
+     - structured generation
+     - accuracy
+     - 1,024
+     - 512
+
+.. note::
+
+    ``trtllm-eval`` originates from the TensorRT-LLM accuracy test suite and serves as a lightweight utility for verifying and debugging accuracy. At this time, ``trtllm-eval`` is intended solely for development and is not recommended for production use.
+
+Usage and Examples
+------------------
+
+Some evaluation tasks (e.g., GSM8K and GPQA) depend on the ``lm_eval`` package. To run these tasks, you need to install ``lm_eval`` with:
+
+.. code-block:: bash
+
+   pip install -r requirements-dev.txt
+
+Alternatively, you can install the ``lm_eval`` version specified in ``requirements-dev.txt``.
+
+Here are some examples:
+
+.. code-block:: bash
+
+   # Evaluate Llama-3.1-8B-Instruct on MMLU
+   trtllm-eval --model meta-llama/Llama-3.1-8B-Instruct mmlu
+
+   # Evaluate Llama-3.1-8B-Instruct on GSM8K
+   trtllm-eval --model meta-llama/Llama-3.1-8B-Instruct gsm8k
+
+   # Evaluate Llama-3.3-70B-Instruct on GPQA Diamond
+   trtllm-eval --model meta-llama/Llama-3.3-70B-Instruct gpqa_diamond
+
+The ``--model`` argument accepts either a Hugging Face model ID or a local checkpoint path. By default, ``trtllm-eval`` runs the model with the PyTorch backend; you can pass ``--backend tensorrt`` to switch to the TensorRT backend.
+
+Alternatively, the ``--model`` argument also accepts a local path to pre-built TensorRT engines. In this case, you should pass the Hugging Face tokenizer path to the ``--tokenizer`` argument.
+
+For more details, see ``trtllm-eval --help`` and ``trtllm-eval <task> --help``.
+
+
+
+Syntax
+------
+
+.. click:: tensorrt_llm.commands.eval:main
+   :prog: trtllm-eval
+   :nested: full

docs/source/conf.py

@@ -106,7 +106,7 @@
 [GitHub pre-release or release](https://github.com/NVIDIA/TensorRT-LLM/releases)
 (see also [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags)).
 ```
-    """,
+    """
 }
 
 autosummary_generate = True

docs/source/deployment-guide/index.rst

@@ -0,0 +1,11 @@
+Model Recipes
+================
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Model Recipes
+   :name: Model Recipes
+
+   quick-start-recipe-for-deepseek-r1-on-trtllm.md
+   quick-start-recipe-for-llama3.3-70b-on-trtllm.md
+   quick-start-recipe-for-llama4-scout-on-trtllm.md

docs/source/deployment-guide/quick-start-recipe-for-deepseek-r1-on-trtllm.md

@@ -275,7 +275,7 @@ To run the evaluation harness exec into the running TensorRT-LLM container and i
 ```shell
 docker exec -it tensorrt_llm /bin/bash
 
-pip install lm_eval
+pip install -U lm-eval
 ```
 
 FP8 command for GSM8K: