Skip to content

feat: concurrent multi stream executor for rec_model. #548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zhang-minchao wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: concurrent multi stream executor for rec_model. #548

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions xllm/core/common/global_flags.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,6 +478,17 @@ DEFINE_bool(enable_constrained_decoding,
"that the output meets specific format or structural requirements "
"through pre-defined rules.");

// --- concurrent llm worker config ---
DEFINE_uint32(llm_worker_max_concurrency,
1,
"Concurrency for llm worker parallel execution. Less than or "
"equal to 1 means disable concurrent llm worker.");

// --- fixedsteps scheduler config ---
DEFINE_bool(enable_fixedsteps_scheduler,
false,
"Whether to use fixsteps scheduler.");

#if defined(USE_NPU)
DEFINE_string(
npu_kernel_backend,
Expand Down
3 changes: 3 additions & 0 deletions xllm/core/common/global_flags.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,9 @@ DECLARE_double(dit_cache_residual_diff_threshold);

DECLARE_bool(enable_constrained_decoding);

DECLARE_uint32(llm_worker_max_concurrency);

DECLARE_bool(enable_fixedsteps_scheduler);
#if defined(USE_NPU)
DECLARE_string(npu_kernel_backend);
#endif
2 changes: 2 additions & 0 deletions xllm/core/runtime/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ cc_library(
worker.h
worker_impl.h
llm_worker_impl.h
concurrent_llm_worker_impl.h
vlm_worker_impl.h
dit_worker.h
embed_worker_impl.h
Expand All @@ -34,6 +35,7 @@ cc_library(
worker.cpp
worker_impl.cpp
llm_worker_impl.cpp
concurrent_llm_worker_impl.cpp
vlm_worker_impl.cpp
dit_worker.cpp
embed_worker_impl.cpp
Expand Down
Loading
Loading