-
Notifications
You must be signed in to change notification settings - Fork 310
[moe training] add bench script for fp8 rowwise kernels and update autotune configs #2697
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: danielvegamyhre/stack/30
Are you sure you want to change the base?
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2697
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
59e34b1
to
362cfb2
Compare
af159db
to
f6688be
Compare
362cfb2
to
14acee2
Compare
14acee2
to
e3c97ec
Compare
e3c97ec
to
5118bcf
Compare
5118bcf
to
0305192
Compare
0305192
to
c621ce5
Compare
c621ce5
to
ab43973
Compare
ab43973
to
1fb9ee1
Compare
print(tabulate(rows, headers=headers)) | ||
|
||
|
||
def benchmark_cuda_function_in_microseconds(f, *args): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have so many of these, maybe reuse in a separate PR?
6704fd3
to
c789281
Compare
…totune configs stack-info: PR: #2697, branch: danielvegamyhre/stack/31
1fb9ee1
to
f124235
Compare
…totune configs stack-info: PR: #2697, branch: danielvegamyhre/stack/31
f124235
to
73b26f1
Compare
Stacked PRs:
[moe training] add bench script for fp8 rowwise kernels and update autotune configs
Performance vs torch.compile
It's faster for llama4 shape (16, 5120, 4*5120), but slower for skinny shapes.
There is more we can do, for example writing row major outputs was roughly 2x faster, but we need the outputs in col-major. I can probably look with NCU and figure out what's going on but for now this is a start.