Make metadata a workproduct and reuse it #114669

cjgillot · 2023-08-09T19:57:37Z

This PR aims to skip the generation of metadata by reusing the infrastructure that already exists for compiled codegen-units, namely "workproducts".

This can yield substantial gains (~10%) when we can demonstrate that metadata does not change between an incremental session and the next. This is the case if the crate is unchanged, or if all the changes are in upstream crates and have no effect on it. This latter case is most interesting, as it arises regularly for users with several crates in their workspace.

TODO:

Materialize the fact that metadata encoding relies on the relative order of definitions;
Refactor the handling of doc links.

cjgillot · 2023-08-09T20:01:11Z

@bors try @rust-timer queue

bors · 2023-08-09T20:01:21Z

⌛ Trying commit 3895d701e4a4c854cc003e599ac6efd49435aaef with merge 6a3ac25372eb1f930be018a4061b9b281bb3f42f...

bors · 2023-08-09T22:11:00Z

☀️ Try build successful - checks-actions
Build commit: 6a3ac25372eb1f930be018a4061b9b281bb3f42f (6a3ac25372eb1f930be018a4061b9b281bb3f42f)

rust-timer · 2023-08-09T23:14:48Z

Finished benchmarking commit (6a3ac25372eb1f930be018a4061b9b281bb3f42f): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Warning ⚠: The following benchmark(s) failed to build:

hyper-0.14.18
exa-0.10.1
cranelift-codegen-0.82.1
bitmaps-3.1.0
stm32f4-0.14.0
ctfe-stress-5
helloworld
unused-warnings
cargo-0.60.0
image-0.24.1
libc-0.2.124
tt-muncher
issue-88862
issue-46449
ucd
regression-31157
unify-linearly
webrender-2022
deep-vector
unicode-normalization-0.1.19
await-call-tree
clap-3.1.6
many-assoc-items
serde-1.0.136
wg-grammar
html5ever-0.26.0
externs
issue-58319
diesel-1.4.8
regex-1.5.5
ripgrep-13.0.0
wf-projection-stress-65510
match-stress
token-stream-stress
serde_derive-1.0.136
coercions
projection-caching
deeply-nested-multi
syn-1.0.89
tuple-stress
derive

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.1%	[1.3%, 3.2%]	7
Regressions ❌ (secondary)	1.1%	[0.9%, 1.7%]	3
Improvements ✅ (primary)	-4.6%	[-17.9%, -2.4%]	30
Improvements ✅ (secondary)	-5.5%	[-10.8%, -1.9%]	19
All ❌✅ (primary)	-3.4%	[-17.9%, 3.2%]	37

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.4%	[0.8%, 2.3%]	11
Regressions ❌ (secondary)	2.1%	[1.3%, 2.9%]	2
Improvements ✅ (primary)	-1.6%	[-2.1%, -1.0%]	8
Improvements ✅ (secondary)	-2.8%	[-4.1%, -1.4%]	5
All ❌✅ (primary)	0.1%	[-2.1%, 2.3%]	19

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.6%	[1.6%, 3.3%]	10
Regressions ❌ (secondary)	2.3%	[2.3%, 2.3%]	1
Improvements ✅ (primary)	-7.2%	[-18.2%, -2.1%]	32
Improvements ✅ (secondary)	-8.1%	[-14.7%, -2.1%]	21
All ❌✅ (primary)	-4.9%	[-18.2%, 3.3%]	42

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.3%	[-2.3%, -0.0%]	60
Improvements ✅ (secondary)	-0.8%	[-2.2%, -0.1%]	6
All ❌✅ (primary)	-0.3%	[-2.3%, -0.0%]	60

Bootstrap: 632.289s -> 632.478s (0.03%)

compiler/rustc_metadata/src/rmeta/encoder.rs

cjgillot · 2023-08-12T08:24:28Z

@bors try @rust-timer queue

bors · 2023-08-12T08:24:36Z

⌛ Trying commit 5f13bb1 with merge 2fce67ca52380d54b739a68cac1efc7c468cbaa6...

bors · 2023-08-12T10:35:13Z

☀️ Try build successful - checks-actions
Build commit: 2fce67ca52380d54b739a68cac1efc7c468cbaa6 (2fce67ca52380d54b739a68cac1efc7c468cbaa6)

rust-timer · 2023-08-12T12:52:07Z

Finished benchmarking commit (2fce67ca52380d54b739a68cac1efc7c468cbaa6): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.4%	[0.6%, 3.2%]	52
Regressions ❌ (secondary)	1.0%	[0.5%, 1.8%]	12
Improvements ✅ (primary)	-4.6%	[-17.9%, -1.0%]	48
Improvements ✅ (secondary)	-5.0%	[-10.8%, -0.5%]	33
All ❌✅ (primary)	-1.5%	[-17.9%, 3.2%]	100

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.4%	[0.6%, 2.1%]	14
Regressions ❌ (secondary)	1.7%	[1.2%, 2.8%]	3
Improvements ✅ (primary)	-1.8%	[-3.9%, -0.7%]	20
Improvements ✅ (secondary)	-2.1%	[-4.1%, -1.0%]	8
All ❌✅ (primary)	-0.5%	[-3.9%, 2.1%]	34

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.4%	[0.7%, 3.9%]	20
Regressions ❌ (secondary)	3.2%	[3.2%, 3.2%]	1
Improvements ✅ (primary)	-7.0%	[-18.3%, -2.1%]	48
Improvements ✅ (secondary)	-8.3%	[-13.8%, -2.5%]	29
All ❌✅ (primary)	-4.2%	[-18.3%, 3.9%]	68

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.0%, 2.6%]	72
Regressions ❌ (secondary)	1.7%	[0.0%, 3.8%]	13
Improvements ✅ (primary)	-0.3%	[-0.9%, -0.0%]	21
Improvements ✅ (secondary)	-0.9%	[-2.2%, -0.1%]	10
All ❌✅ (primary)	0.3%	[-0.9%, 2.6%]	93

Bootstrap: 632.009s -> 632.421s (0.07%)

compiler/rustc_codegen_ssa/src/back/write.rs

Avoid depending on forever-red DepNode when encoding metadata. Split from #114669 for perf r? `@petrochenkov`

petrochenkov · 2025-06-30T16:54:14Z

Blocked on #143247.
@rustbot blocked

Avoid depending on forever-red DepNode when encoding metadata. Split from #114669 for perf r? `@petrochenkov`

petrochenkov · 2025-07-04T14:17:04Z

@bors r+

bors · 2025-07-04T14:17:07Z

📌 Commit 17ce06a has been approved by petrochenkov

It is now in the queue for this repository.

bors · 2025-07-04T18:06:23Z

⌛ Testing commit 17ce06a with merge e384365...

bors · 2025-07-04T21:04:59Z

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing e384365 to master...

github-actions · 2025-07-04T21:08:17Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 0c4fa26 (parent) -> e384365 (this PR)

Test differences

Show 5 test diffs

5 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard e3843659e9f65f589d184d1221ac6149d5fa07b5 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

dist-x86_64-apple: 11672.3s -> 8033.9s (-31.2%)
dist-apple-various: 8451.4s -> 5954.3s (-29.5%)
x86_64-apple-1: 8177.7s -> 6533.0s (-20.1%)
x86_64-apple-2: 5507.1s -> 4657.6s (-15.4%)
pr-check-1: 1858.9s -> 1674.5s (-9.9%)
aarch64-apple: 4207.3s -> 4600.8s (9.4%)
tidy: 67.5s -> 73.1s (8.3%)
dist-aarch64-apple: 5433.7s -> 5041.6s (-7.2%)
pr-check-2: 2135.3s -> 1994.3s (-6.6%)
dist-x86_64-netbsd: 4332.2s -> 4556.1s (5.2%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2025-07-05T00:42:21Z

Finished benchmarking commit (e384365): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

If the regression was expected or you think it can be justified,
please write a comment with sufficient written justification, and add
@rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
If you think that you know of a way to resolve the regression, try to create
a new PR with a fix for the regression.
If you do not understand the regression or you think that it is just noise,
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	1.0%	[0.2%, 2.2%]	76
Regressions ❌ (secondary)	0.8%	[0.3%, 1.7%]	23
Improvements ✅ (primary)	-4.3%	[-7.5%, -1.1%]	45
Improvements ✅ (secondary)	-4.9%	[-11.6%, -0.5%]	36
All ❌✅ (primary)	-1.0%	[-7.5%, 2.2%]	121

Max RSS (memory usage)

Results (primary -0.6%, secondary -0.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.6%	[0.7%, 3.2%]	17
Regressions ❌ (secondary)	2.6%	[2.4%, 2.7%]	2
Improvements ✅ (primary)	-2.3%	[-6.4%, -0.8%]	23
Improvements ✅ (secondary)	-2.1%	[-2.6%, -1.0%]	5
All ❌✅ (primary)	-0.6%	[-6.4%, 3.2%]	40

Cycles

Results (primary -5.8%, secondary -7.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.1%	[2.2%, 4.2%]	8
Regressions ❌ (secondary)	1.7%	[0.7%, 2.7%]	2
Improvements ✅ (primary)	-7.4%	[-11.7%, -2.7%]	43
Improvements ✅ (secondary)	-7.8%	[-13.6%, -1.6%]	32
All ❌✅ (primary)	-5.8%	[-11.7%, 4.2%]	51

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 460.42s -> 460.645s (0.05%)
Artifact size: 372.16 MiB -> 372.20 MiB (0.01%)

Avoid depending on forever-red DepNode when encoding metadata. Split from rust-lang/rust#114669 for perf r? `@petrochenkov`

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 9, 2023

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Aug 9, 2023

cjgillot commented Aug 10, 2023

View reviewed changes

compiler/rustc_metadata/src/rmeta/encoder.rs Outdated Show resolved Hide resolved

cjgillot force-pushed the metadata-wp branch from 3895d70 to 5f13bb1 Compare August 12, 2023 08:13

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 12, 2023

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 12, 2023

petrochenkov self-assigned this Aug 13, 2023

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 13, 2023

bjorn3 reviewed Aug 16, 2023

View reviewed changes

compiler/rustc_codegen_ssa/src/back/write.rs Outdated Show resolved Hide resolved

cjgillot force-pushed the metadata-wp branch from 5f13bb1 to 9742c49 Compare October 1, 2023 14:33

cjgillot mentioned this pull request Jun 30, 2025

Avoid depending on forever-red DepNode when encoding metadata. #143247

Merged

bors added a commit that referenced this pull request Jun 30, 2025

Auto merge of #143247 - cjgillot:metadata-no-red, r=<try>

22d2c51

Avoid depending on forever-red DepNode when encoding metadata. Split from #114669 for perf r? `@petrochenkov`

rustbot added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 30, 2025

bors added a commit that referenced this pull request Jul 1, 2025

Auto merge of #143247 - cjgillot:metadata-no-red, r=<try>

3797379

Avoid depending on forever-red DepNode when encoding metadata. Split from #114669 for perf r? `@petrochenkov`

bors added a commit that referenced this pull request Jul 2, 2025

Auto merge of #143247 - cjgillot:metadata-no-red, r=<try>

a46eba4

Avoid depending on forever-red DepNode when encoding metadata. Split from #114669 for perf r? `@petrochenkov`

cjgillot force-pushed the metadata-wp branch 2 times, most recently from 3391579 to 490e6ad Compare July 3, 2025 14:15

bors added a commit that referenced this pull request Jul 4, 2025

Auto merge of #143247 - cjgillot:metadata-no-red, r=petrochenkov

556d20a

Avoid depending on forever-red DepNode when encoding metadata. Split from #114669 for perf r? `@petrochenkov`

cjgillot added 4 commits July 4, 2025 14:02

Save metadata among work products.

c7ee3a5

Reuse metadata file from work products.

761d366

Lighten formatting.

904bc29

Add comment and move assertion.

17ce06a

cjgillot force-pushed the metadata-wp branch from 490e6ad to 17ce06a Compare July 4, 2025 14:02

cjgillot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Jul 4, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 4, 2025

bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 4, 2025

bors merged commit e384365 into rust-lang:master Jul 4, 2025
12 checks passed

rustbot added this to the 1.90.0 milestone Jul 4, 2025

cjgillot deleted the metadata-wp branch July 4, 2025 21:17

Make metadata a workproduct and reuse it #114669

Make metadata a workproduct and reuse it #114669

Conversation

cjgillot commented Aug 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjgillot commented Aug 9, 2023

Uh oh!

This comment has been minimized.

bors commented Aug 9, 2023

Uh oh!

This comment has been minimized.

bors commented Aug 9, 2023

Uh oh!

This comment has been minimized.

rust-timer commented Aug 9, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Uh oh!

Uh oh!

cjgillot commented Aug 12, 2023

Uh oh!

This comment has been minimized.

bors commented Aug 12, 2023

Uh oh!

bors commented Aug 12, 2023

Uh oh!

This comment has been minimized.

rust-timer commented Aug 12, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Uh oh!

Uh oh!

petrochenkov commented Jun 30, 2025

Uh oh!

petrochenkov commented Jul 4, 2025

Uh oh!

bors commented Jul 4, 2025

Uh oh!

bors commented Jul 4, 2025

Uh oh!

bors commented Jul 4, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 4, 2025

Test differences

Job duration changes

Uh oh!

rust-timer commented Jul 5, 2025

Overall result: ❌✅ regressions and improvements - please read the text below

Uh oh!

Uh oh!

cjgillot commented Aug 9, 2023 •

edited

Loading