Skip to content

Conversation

@qw86972190
Copy link
Contributor

@qw86972190 qw86972190 commented Nov 27, 2025

Motivation

This PR primarily adds Logprobs support for XPU (Kunlun Chip) on the FastDeploy LLM inference engine.

Previously, Logprobs functionality was restricted only to CUDA platforms, which prevented users from leveraging advanced sampling features on XPU devices.

Modifications

This PR involves changes across configuration, worker logic, and the custom XPU operators

Usage or Command

python -m fastdeploy.entrypoints.openai.api_server
--model /work/PaddlePaddle/ERNIE-4.5-0.3B-Paddle
--port 8188
--tensor-parallel-size 1
--max-model-len 32768
--max-num-seqs 128
--quantization "wint8"
--gpu-memory-utilization 0.9
--enable-logprob

Accuracy Tests

This change affects the Logprobs output structure and platform support, not the core inference results.

Checklist

  • Add at least a tag in the PR title.
    • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
    • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@paddle-bot
Copy link

paddle-bot bot commented Nov 27, 2025

Thanks for your contribution!

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ qw86972190
❌ root


root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codecov-commenter
Copy link

codecov-commenter commented Nov 27, 2025

Codecov Report

❌ Patch coverage is 21.42857% with 11 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@68533eb). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...tdeploy/model_executor/xpu_pre_and_post_process.py 0.00% 7 Missing ⚠️
fastdeploy/output/token_processor.py 40.00% 2 Missing and 1 partial ⚠️
fastdeploy/engine/args_utils.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             develop    #5279   +/-   ##
==========================================
  Coverage           ?   59.60%           
==========================================
  Files              ?      324           
  Lines              ?    39711           
  Branches           ?     5976           
==========================================
  Hits               ?    23669           
  Misses             ?    14158           
  Partials           ?     1884           
Flag Coverage Δ
GPU 59.60% <21.42%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

DDDivano
DDDivano previously approved these changes Nov 28, 2025
gongshaotian
gongshaotian previously approved these changes Nov 28, 2025
Copy link
Collaborator

@gongshaotian gongshaotian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


PD_BUILD_STATIC_OP(get_output_topk)
.Inputs({"x", "scores", "ranks"})
.Attrs({"k: int", "rank_id: int64_t", "wait_flag: bool"})
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是最大不超过上面定义的 K 5个嘛?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XPU之前内部版的算子默认K=5 BS=128, 这里直接把算子迁移过来了,所以配置上没完全和GPU对齐。我们可以改改自定义算子,把这些配置和GPU对齐看看有没有问题

@qw86972190 qw86972190 dismissed stale reviews from gongshaotian and DDDivano via 5628413 December 1, 2025 06:16
Copy link

@jeff41404 jeff41404 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XPU存量自定义算子迁移,短期先豁免

Copy link
Collaborator

@hong19860320 hong19860320 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@qingqing01 qingqing01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

后续需要支持 zmq

@EmmonsCurse EmmonsCurse merged commit 6048ea3 into PaddlePaddle:develop Dec 2, 2025
21 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants