feat: configure nixl agents with custom backend parameters

[cheese-head]

Overview:

This PR allows users to specify NIXL Backend parameters

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• relates to GitHub issue: #900

Summary by CodeRabbit

• New Features
  • Backend configuration now supports environment variables with automatic fallback to defaults
  • Improved error reporting with detailed feedback on backend initialization failures and consolidated error messaging

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi cheese-head! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Walkthrough

Enhanced backend initialization in the NIXL agent module with environment-driven parameter loading. Three new public functions enable flexible backend configuration: parameter creation from environment variables, explicit parameter-based initialization, and required-backends setup. Existing constructor functions now prioritize environment variables over defaults with consolidated error reporting.

Changes

Cohort / File(s)

Change Summary

Backend parameter loading and initialization enhancement lib/llm/src/block_manager/v2/physical/transfer/nixl_agent/mod.rs

Added create_backend_params_from_env() helper to load backend parameters from environment variables (pattern: DYN_KVBM_NIXL_BACKEND_{BACKEND}_{PARAM}). Introduced new_with_backends_and_params() and require_backends_with_params() public APIs for explicit per-backend parameter configuration. Enhanced existing new_with_backends() and require_backends() to attempt environment-derived parameters first, then fall back to defaults, with expanded error handling and per-backend logging.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• Environment variable parsing logic: Verify correctness of the DYN_KVBM_NIXL_BACKEND_* pattern matching and parameter extraction
• Error aggregation and reporting: Ensure consolidated error reporting across multiple backends captures all failure modes and provides clear diagnostics
• Fallback logic sequence: Confirm that environment-first → plugin-defaults flow correctly prioritizes and logs outcomes for each initialization path
• Function interaction: Review how the three new functions interact with existing constructors and ensure consistent behavior

Poem

🐰 Hark! Environment whispers secrets true,
Parameters dance from variables anew,
With fallback grace when defaults align,
Backend agents now configure divine!
Errors consolidated, clear as starlight,
NIXL flows smoothly through the night!

Pre-merge checks

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is incomplete. It includes an Overview and Related Issues, but is missing the Details and Where should the reviewer start sections specified in the template.	Add the missing 'Details' and 'Where should the reviewer start' sections to provide more comprehensive information about the changes and guide reviewers.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title check	✅ Passed	The title accurately summarizes the main change: adding support for configuring NIXL agents with custom backend parameters through new public APIs and environment-driven parameter loading.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

lib/llm/src/block_manager/v2/physical/transfer/nixl_agent/mod.rs (1)

343-372: Inconsistent error message formatting.

Lines 343 and 357 prefix error messages with "✗", but lines 363 and 370 do not. For consistency and readability, either use the symbol for all error logs or none.

Apply this diff to make formatting consistent:

                         Err(e) => {
-                            tracing::error!("✗ Failed to create {} backend with custom params: {}", backend_upper, e);
+                            tracing::error!("Failed to create {} backend with custom params: {}", backend_upper, e);
                             failed_backends
                                 .push((backend_upper.clone(), format!("create with custom params failed: {}", e)));
                         }
                     }
                 }
                 Ok(None) => {
                     // No custom parameters - fall back to default plugin parameters
                     match agent.get_plugin_params(&backend_upper) {
                         Ok((_, params)) => match agent.create_backend(&backend_upper, &params) {
                             Ok(_) => {
                                 available_backends.insert(backend_upper);
                             }
                             Err(e) => {
-                                tracing::error!("✗ Failed to create {} backend: {}", backend_upper, e);
+                                tracing::error!("Failed to create {} backend: {}", backend_upper, e);
                                 failed_backends
                                     .push((backend_upper.clone(), format!("create failed: {}", e)));
                             }

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6f8fd86 and bef8aaa.

📒 Files selected for processing (1)

• lib/llm/src/block_manager/v2/physical/transfer/nixl_agent/mod.rs (6 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

lib/llm/src/block_manager/v2/physical/transfer/nixl_agent/mod.rs (3)

lib/llm/benches/transfer_context_v2.rs (1)

• new (24-38)

lib/llm/src/block_manager/block/transfer/context.rs (5)

• new (180-292)
• new (424-434)
• std (203-203)
• std (350-350)
• std (387-387)

lib/llm/src/block_manager/v2/physical/transfer/nixl_agent/config.rs (1)

• backends (101-103)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)

• GitHub Check: clippy (.)
• GitHub Check: clippy (lib/bindings/python)
• GitHub Check: clippy (launch/dynamo-run)
• GitHub Check: Build and Test - dynamo

🔇 Additional comments (2)

lib/llm/src/block_manager/v2/physical/transfer/nixl_agent/mod.rs (2)

114-144: LGTM! Clean implementation of explicit parameter initialization.

The function correctly handles partial failures, provides clear error messages, and ensures at least one backend succeeds.

---

260-296: LGTM! Proper strict validation with consolidated error reporting.

The function correctly requires all backends to succeed and provides detailed error messages listing all failures.