Skip to content

Conversation

@indrajit96
Copy link
Contributor

Overview:

Add support for full EPD for Image url in dynamo trtllm workers

Details:

  • Encode Worker: Uses MultimodalEncoder to process images and generate disaggregated params with embedding handles
  • Prefill Worker: Receives disaggregated params from Encode worker, handles prompts with tokens
  • Decode Worker: Restores context including token IDs and max_tokens from Prefill worker
  • Enhanced DisaggregatedParamsCodec to support multimodal embedding handles
  • Updated engine initialization to conditionally use MultimodalEncoder vs LLM based on mode

Where should the reviewer start?

  • request_handlers/handler_base.py - Core EPD pipeline flow logic
  • request_handlers/handlers.py - EncodeHandler and PrefillHandler EPD support
  • encode_helper.py - New process_encode_request for image URLs
  • engine.py - MultimodalEncoder initialization

Signed-off-by: Indrajit Bhosale <[email protected]>
Signed-off-by: Indrajit Bhosale <[email protected]>
Signed-off-by: Indrajit Bhosale <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants