diff --git a/docs/.custom_wordlist.txt b/docs/.custom_wordlist.txt index 4ebf78b7..91f8527b 100644 --- a/docs/.custom_wordlist.txt +++ b/docs/.custom_wordlist.txt @@ -5,6 +5,7 @@ apport auditable autobuilders autofill +autogenerated autopkgtests autosync backgrounding @@ -188,6 +189,7 @@ unapplied unapply uncompress unpatched +unpruned unshareable unstaged untracked diff --git a/docs/maintainers/niche-package-maintenance/rustc/common/uscan.md b/docs/maintainers/niche-package-maintenance/rustc/common/uscan.md index 1ec05ef0..47f352a9 100644 --- a/docs/maintainers/niche-package-maintenance/rustc/common/uscan.md +++ b/docs/maintainers/niche-package-maintenance/rustc/common/uscan.md @@ -10,4 +10,4 @@ $ uscan --download-version -v 2>&1 | tee This process can take a while. Once it is complete, you will find a file with an `.orig.tar.xz` suffix in your parent `rustc` directory. That is your orig tarball. It contains the new upstream source code for the new Rust version. -You must then rename the orig tarball to match the first part of your package version number, i.e., `rustc-_+dfsg0ubuntu0`. +You must then rename the orig tarball to match the first part of your package version number. diff --git a/docs/maintainers/niche-package-maintenance/rustc/update-rust.md b/docs/maintainers/niche-package-maintenance/rustc/update-rust.md index ce7b57d1..7eb9cd81 100644 --- a/docs/maintainers/niche-package-maintenance/rustc/update-rust.md +++ b/docs/maintainers/niche-package-maintenance/rustc/update-rust.md @@ -1,5 +1,4 @@ (how-to-update-rust)= - # How to update Rust This guide details the process of creating a new versioned `rustc` Ubuntu package for a new upstream Rust release. @@ -80,53 +79,64 @@ $ git push merge- In this step, you get the source code of the new Rust version. The {term}`watch file`, `debian/watch`, automates this process. -(updating-rust-removing-the-old-vendor-prune-list)= - -#### Removing the old vendor prune list - -First, understand that a great deal of unnecessary vendored crate dependencies must be pruned from the `vendor` directory. +#### Updating the changelog and package name -We tell the Debian packaging tools which files from the Rust team's source code it can ignore in the {term}`copyright file`, `debian/copyright`. The list of ignored vendored dependencies is huge — you'll automatically generate that list later. For now, we will include _all_ vendored dependencies from the new Rust version to ensure we don't miss anything. +:::{important} +The changelog version string is complicated. It's _strongly recommended_ to consult the {ref}`rust-version-strings` article before making this change to ensure you understand the version string. +::: -To do this, remove the `DO NOT EDIT (...) AUTOGENERATED` chunk from `debian/copyright`: +First, update the {term}`changelog` at `debian/changelog`, manually setting the version number: -```diff -- # DO NOT EDIT below, AUTOGENERATED -- vendor/addr2line-0.17.0 -- vendor/aes-0.8.4 -- vendor/ahash-0.8.10 -- [...] -- vendor/zerocopy-derive-0.8.14 -- vendor/zeroize_derive-1.4.2 -- # DO NOT EDIT above, AUTOGENERATED +```none +$ dch -v +dfsg-0ubuntu1 ``` -#### Updating the changelog and package name +You must also update the versioned package name in the changelog, i.e., `rustc-` -> `rustc-` + +Finally, within the changelog entry, you must declare the new Rust version you're packaging, including the Launchpad bug number. -You can also update the {term}`changelog` at `debian/changelog`, manually setting the version number: +Your new changelog entry should look similar to this: ```none -$ dch -v +dfsg0ubuntu0-0ubuntu0 +rustc- (+dfsg-0ubuntu1) ; urgency=medium + + * New upstream version (LP: #) + + -- Jane Doe Thu, 01 Jan 1970 00:00:00 -0000 ``` :::{important} -The changelog version string is complicated. It's _strongly recommended_ to consult the {ref}`rust-version-strings` article before making this change to ensure you understand the version string. - -Even though the `ubuntu` suffix in version strings starts at 1, the fact that both suffixes are `ubuntu0` here is not a typo. This version string will _not_ be added to the archive — it is simply an interim number to be used temporarily until we're finished repacking the tarball. +Make sure `` matches the bug number you created earlier! ::: -Don't forget to manually change the versioned package name in the changelog too! (i.e., `rustc-` -> `rustc-`) +(updating-rust-including-all-vendored-dependencies)= -You can also create your first changelog bullet point — the "New upstream version" point. It should look something like the following; consult previous changelog entries for examples: +#### Temporarily including all vendored dependencies -```none -* New upstream version (LP: #) +Later on in the upgrade process, unwanted {term}`vendored dependencies ` will be pruned from the upstream source. Right now, however, we don't know _which_ dependencies must be pruned, so we must temporarily include _all_ vendored dependencies. + +_Temporarily_ modify `debian/copyright`, commenting out the line in `Files-Excluded` which excludes the `vendor/` directory from the {term}`orig tarball`: + +```diff +--- a/debian/copyright ++++ b/debian/copyright +@@ -10,7 +10,7 @@ Files-Excluded: + # Pre-generated docs + src/tools/rustfmt/docs + # Exclusions on the vendor/ dir should be in Files-Excluded-vendor +- vendor ++# vendor + # DOCX versions of TRPL book prepared for No Starch Press + src/doc/book/nostarch/docx + # Exclude submodules https://github.com/rust-lang/rust/tree/master/src/tools ``` :::{important} -Make sure `` matches the bug number you created earlier! +Remember, this change shouldn't be committed to version control. It's just a temporary measure for the next step. ::: +This means that in the next step, _all_ vendored dependencies will be included in the tarball. + (updating-rust-getting-the-new-source-and-orig-tarball-with-uscan)= #### Getting the new source and orig tarball with uscan @@ -135,17 +145,25 @@ Make sure `` matches the bug number you created earlier! ``` -:::{note} -This `0ubuntu0` is semantically _and_ functionally important. It not only says "I am not finished repacking the orig tarball yet", but if you were to start with `0ubuntu1` instead, you'd run into problems in later steps when trying to overlay an updated tarball using the same version number as before. -::: +You are now free to restore `debian/copyright` to its original state, un-commenting the `vendor` line in `Files-Excluded`: + +```none +$ git restore debian/copyright +``` + +Since this particular tarball contains all vendored dependencies and will therefore be different from the final tarball, it should be renamed with an `~old` suffix, like so: + +```none +$ mv ../rustc-_+dfsg{1,~old}.orig.tar.xz +``` (updating-rust-updating-the-source-code-in-your-repository)= #### Updating the source code in your repository -`uscan` just downloads the new Rust source, yanks out the ignored files, and packs the orig tarball. Your actual Git repository hasn't changed at all yet. To do that, we can use `gbp` to import the new Rust version onto your existing repository. +`uscan` just downloads the new Rust source, yanks out the ignored files, and packs the orig tarball. Your actual Git repository hasn't changed at all yet. To do that, we can use `gbp` to import the new Rust source code onto your existing repository. -:::{important} +:::{note} This is point at which your _actual source code_ moves from `` to ``. ::: @@ -164,11 +182,18 @@ $ gbp import-orig \ --no-pristine-tar \ --upstream-branch=experimental \ --debian-branch=merge- \ - ../rustc-_+dfsg0ubuntu0.orig.tar.xz + ../rustc-_+dfsg~old.orig.tar.xz ``` Afterwards, you should now see two commits in your Git log stating that your upstream source has been updated. +It can be useful to be able to return to this point just in case you make a mistake repacking the tarballs. It's recommended to create a branch here for safekeeping: + +```none +git branch import-old- +git push import-old- +``` + (updating-rust-initial-patch-refresh)= ### Initial Patch Refresh @@ -219,196 +244,38 @@ In this case, you must consult the upstream changes to figure out what replaced Remember, your goal here is to preserve the _intent_ of the patch. If, for example, a patch disables certain tests which require an internet connection, and those tests get refactored completely, it's your responsibility to track down the tests which require an internet connection and disable them accordingly. -#### Pruned vendored dependency patch exception - -Finally, `debian/patches/prune/d-0021-vendor-remove-windows-dependencies.patch` is a special case. Many of its changes will fail to apply because their targeted vendored crates have changed versions. - -You will later have to manually update this patch, so you may simply refresh this patch after force-applying it, dropping all the missing vendored crates. +(updating-rust-pruning-unwanted-dependencies)= ### Pruning Unwanted Dependencies As mentioned above, we don't want to include unnecessary dependencies, especially Windows-related crates like `windows-sys`. This pruning ensures adherence to free software principles, reduces the attack surface of the binary packages, and reduces the binary package size on the end user's hard drive. -Since we [removed the auto-generated `debian/copyright` chunk earlier](updating-rust-removing-the-old-vendor-prune-list) before [getting the upstream source with `uscan`](updating-rust-getting-the-new-source-and-orig-tarball-with-uscan), our `vendor` directory will contain _everything_. We must now remove the dependencies on things we don't need. - -You must prune the unwanted dependencies of both the Rust source code itself _and_ its vendored dependencies. - -#### Get a list of things to prune - -To get a list of possibly-unwanted crate dependencies, search for Windows-related words in the `Cargo.toml` files: - -```none -$ quilt push debian/patches/prune/d-0021-vendor-remove-windows-dependencies.patch -$ find vendor -iname Cargo.toml -exec grep -H -n \ - -e 'schannel' \ - -e 'windows-sys' \ - -e 'winapi' \ - -e 'ntapi' \ - -e 'wincon' \ - -e 'winreg' \ - -e 'windows' \ - {} \; -``` - -:::{note} -The majority of this list will be taken up by lines from the `Cargo.toml`s of various versions of the Windows crates like `windows`, `windows-sys`, etc. You don't need to prune these! Basically, if you _know_ that a given crate will be removed, you _don't_ need to prune its `Cargo.toml`. -::: - -#### Prune the Cargo.toml files - -Your top priority will be removing any dependencies which pull in `windows-sys`, `winapi`, `ntapi`, `windows`, etc. Go through the prune list you just generated and inspect the flagged lines to see whether or not it's something that should be pruned. - -Here's an example of something which should definitely be pruned: - -```diff ---- a/vendor/dbus-0.9.7/Cargo.toml -+++ b/vendor/dbus-0.9.7/Cargo.toml -@@ -63,9 +63,5 @@ - stdfd = [] - vendored = ["libdbus-sys/vendored"] - --[target."cfg(windows)".dependencies.winapi] --version = "0.3.0" --features = ["winsock2"] -- - [badges.maintenance] - status = "actively-developed" -``` - -In this example, the vendored `dbus-0.9.7` crate pulls in `winapi` as a dependency on Windows targets. We obviously aren't targeting Windows, so we can delete this whole chunk. - -Here's another example of something which should be deleted: - -```diff ---- a/vendor/opener-0.7.2/Cargo.toml -+++ b/vendor/opener-0.7.2/Cargo.toml -@@ -48,7 +48,6 @@ - reveal = [ - "dep:url", - "dep:dbus", -- "windows-sys/Win32_System_Com", - ] - - [target.'cfg(target_os = "linux")'.dependencies.bstr] -@@ -62,16 +61,5 @@ - version = "2" - optional = true - --[target."cfg(windows)".dependencies.normpath] --version = "1" -- --[target."cfg(windows)".dependencies.windows-sys] --version = "0.59" --features = [ -- "Win32_Foundation", -- "Win32_UI_Shell", -- "Win32_UI_WindowsAndMessaging", --] -- - [badges.maintenance] - status = "passively-maintained" -``` - -In this case, the `reveal` crate feature relies on something from `windows-sys`, so we _also_ remove that line. We know that the `reveal` feature doesn't _actually_ need `windows-sys/Win32_System_Com` on non-Windows targets, because `windows-sys` is a conditional dependency, so it's safe to remove that line. - -Here's an example of something that `win-rustc-vendored-prune-list` picks up that you _don't_ want to remove: - -```toml -[target.'cfg(any(unix, windows, target_os = "wasi"))'.dependencies.getrandom] -version = "0.3.0" -optional = true -default-features = false -``` - -`win-rustc-vendored-prune-list` just sees `windows` and flags it, so it's your job to recognize that it's also required for `unix`. - -Finally, if you're not sure about how/if you should prune something, you can take a look at versions of the patch from earlier Rust updates to see if a different version of the vendored crate has been pruned before. [Here's](https://git.launchpad.net/~canonical-foundations/ubuntu/+source/rustc/tree/debian/patches/prune/d-0021-vendor-remove-windows-dependencies.patch?h=merge-1.85&id=80e81b6f85ef6086177991e34100e520bd142327) an example of a patch with a large list of vendored crates. - -:::{note} -You may notice that `windows-bindgen` and `windows-metadata` aren't included in the exclusion list — they don't pull in `windows-sys` and friends, and (at least in earlier versions) they're necessary for the build process, so it's not the end of the world if they don't get pruned. - -That said, it's been possible to prune `windows-metadata` from more recent `rustc` packages, so we may potentially be able to consistently prune both in the future. More research is needed on this topic. -::: - -#### Final manual crate checks - -While the above content focuses on removing Windows dependencies, there are a few specific libraries we target for pruning. Review the following patches, and make sure that nothing else is trying to use the libraries they target: - -- `prune/d-0005-no-jemalloc.patch`: `tikv-jemalloc-sys` and `jemalloc-ctl` -- `prune/d-0011-cargo-remove-nghttp2.patch`: `libnghttp2-sys` - -### Removing Unused Dependencies - -Once you've removed all `Cargo.toml` lines which pull in unnecessary vendored dependencies, you're ready to remove said dependencies from the orig tarball and `vendor` directory entirely. - -#### prune-unused-deps - -Previous Rust maintainers have been kind enough to create a script for this purpose: `debian/prune-unused-deps`. This script locates the unneeded vendored crates and adds the list to the `Files-Excluded` field of `d/copyright`. - -:::{note} -The auto-generated chunk generated by `prune-unused-deps` is an updated version of the one you deleted at the start of the update process! -::: - -In order to use the script, you need the previous Rust toolchain to bootstrap. It's easiest to get the toolchain using the `rustup` snap: - -```none -$ rustup install -``` - -You're now ready to run the script, pointing it to your Rust toolchain: - -```none -$ RUST_BOOTSTRAP_DIR=~/.rustup/toolchains/--unknown-linux-gnu/bin/rustc \ - debian/prune-unused-deps -``` - -If you have issues running `prune-unused-deps` due to features requiring "nightly version\[s\] of Cargo", set `RUSTC_BOOTSTRAP=1` at the `cargo update` command within `debian/prune-unused-deps`: +Since we [included all vendored dependencies](updating-rust-including-all-vendored-dependencies) before [getting the upstream source](updating-rust-getting-the-new-source-and-orig-tarball-with-uscan), our `vendor` directory will contain _everything_. We must now remove the dependencies on things we don't need. -```diff ---- a/debian/prune-unused-deps -+++ b/debian/prune-unused-deps -@@ -24,7 +24,7 @@ done - find vendor -name .cargo-checksum.json -execdir "$scriptdir/debian/prune-checksums" "{}" + - - for ws in $workspaces; do -- (cd "$ws" && cargo update --offline) -+ (cd "$ws" && RUSTC_BOOTSTRAP=1 cargo update --offline) - done - - needed_crates() { -``` +#### Generating the pruned vendor tarball component -:::{note} -It's possible the above change can be made permanent, but such a change should be discussed as a team before making that decision. -::: +In addition to the standard orig tarball containing all the upstream code, the Rust toolchain Ubuntu package also comes with an additional {term}`orig tarball` component called `vendor`. This tarball component contains _just_ the pruned vendored dependencies of the Rust toolchain. -#### Committing the right changes - -After running `prune-unused-deps`, there will be many changes, almost none of which you actually need. - -First, if you needed to edit `prune-unused-deps` earlier, restore it: +[`cargo-vendor-filterer`](https://crates.io/crates/cargo-vendor-filterer/) is used to generate a pruned vendor directory. It's recommended to download the `rustup` snap, use `rustup` to download the toolchain matching your target version, then install `cargo-vendor-filterer` for that Rust installation: ```none -$ git restore debian/prune-unused-deps +# snap install rustup +$ rustup install +$ cargo + install cargo-vendor-filterer ``` -Next, commit the changes to {term}`debian/control ` and`debian/source/lintian-overrides`. Their version numbers have been updated. - -#### Double-checking your pruning +Make sure that `~/.cargo/bin` is in your `$PATH`, otherwise you won't be able to run `cargo-vendor-filterer`. -After that, consult the new auto-generated block underneath the `Files-Excluded` field of `debian/copyright`. This lists all the crates within `vendor` that aren't needed for the source package build. +(updating-rust-vendor-tarball-rule)= -Make sure that the Windows crates you pruned earlier are included within that list. If they _aren't_ in that list, you need to go back, check for anything that pulls in those dependencies, and prune them. - -I recommend comparing your new list with the previous version's list as well: +After that, call the `vendor-tarball` rule in `debian/rules`. This will use `cargo-vendor-filterer` to generate a vendor directory which _only_ contains the dependencies required by supported Ubuntu targets. It then repacks this directory into the `vendor` tarball component. Make sure you point it to your installed Rust toolchain via `RUST_BOOTSTRAP_DIR`: ```none -$ git diff merge- -- debian/copyright +$ RUST_BOOTSTRAP_DIR=~/.rustup/toolchains/-x86_64-unknown-linux-gnu/bin/rustc \ + debian/rules vendor-tarball ``` -You shouldn't see many dramatic changes. Remember, anything removed from the diff means that the crate _used_ to be excluded, but isn't anymore. This often just means that the version number changed. In this case, you can usually see a crate with the same name and a different version number that's _new_ to the `Files-Excluded` list. - -Once you've checked over your new list of excluded vendored crates, you can commit your `debian/copyright` changes, restore your Git repository from all the other changes, and continue. +You should now see a new tarball in the parent directory: `../rustc-_+dfsg.orig-vendor.tar.xz`. In later steps, we will use this to replace the existing `vendor/` directory. ### Removing Vendored C Libraries @@ -439,36 +306,45 @@ Build-Depends: These two changes form the basis of removing vendored C dependencies. -#### Find vendored C dependencies +#### Finding vendored C dependencies -Search for C source files within the `vendor/` directory: +Search for C source files within your newly-pruned `vendor` tarball: ```none -$ cd vendor -$ fdfind -e c +tar -tJf ../rustc-_+dfsg.orig-vendor.tar.xz | grep '\.c$' ``` -You can now check this list and figure out which of these are bundled C libraries. However, remember that you just pruned some of these! You can cross-reference this list with your auto-generated `Files-Excluded` field from `d/copyright`. You only have to worry about the vendored C libraries which _aren't_ contained within that list. +:::{note} +You don't want to search your unpacked source directory right now because it contains a bunch of things we just pruned in the [previous step](updating-rust-pruning-unwanted-dependencies). +::: + +Individual C files are likely fine. You're just looking for entire C libraries which have been bundled in with vendored crates. -#### Removing C dependencies from the next orig tarball +#### Removing C dependencies from the vendored orig tarball Naturally, the process of pruning a vendored C library varies from library to library. As an example, we will use a removal of the bundled `oniguruma` library from `rustc-1.86`, which caused some {lpbug}`build failures <2119556>` when it wasn't removed. -Next time we run `uscan`, we want to make sure that the bundled C libraries we want to remove aren't included. To do that, simply add the C library directory to `Files-Excluded` in `debian/copyright`: +To do this, simply add the C library directory to `Files-Excluded-vendor` in `debian/copyright`: ```diff --- a/debian/copyright +++ b/debian/copyright -@@ -50,6 +50,7 @@ Files-Excluded: +@@ -52,6 +52,7 @@ Files-Excluded-vendor: vendor/libsqlite3-sys-*/sqlcipher vendor/libz-sys-*/src/zlib* vendor/lzma-sys*/xz-* + vendor/onig_sys*/oniguruma # Embedded binary blobs - vendor/jsonpath_lib-*/docs - vendor/mdbook-*/src/theme/playground_editor + # vendor/jsonpath_lib-*/docs + vendor/mdbook-*/src/front-end/playground_editor ``` +:::{caution} +Remember, this new exclusion should be under `Files-Excluded-vendor`, _not_ `Files-Excluded`! The primary tarball doesn't contain anything in the `vendor/` directory. +::: + +After that, return to the previous step and [regenerate the vendored tarball component](updating-rust-vendor-tarball-rule). The `vendor-tarball` rule will read your new `debian/copyright` and generate a new vendor tarball without the C library source code you just excluded. + #### Adding the system library as a build dependency We can't remove a C library needed by a vendored dependency without providing a proper equivalent of said library in its place. Instead, we can use the oniguruma Ubuntu package, {lpsrc}`libonig-dev`. We do this by adding the package to `Build-Depends` in `d/control` AND `d/control.in`: @@ -521,21 +397,32 @@ In the case of `onig_sys`, we can simply patch it to use the system library by d ### Updating the Source Tree Again -At this point, you've updated the list of files which must be excluded from the upstream source. This means that you need a new upstream tarball and a new Rust source. +Now that you have a vendored tarball with all unwanted vendored crates and vendored C libraries removed, you are now ready to update your source tree once more. + +To recap, your parent directory should contain the following: + +- `rustc-_+dfsg~old.orig.tar.xz`: The original upstream source code tarball with an unpruned vendor directory. +- `rustc-_+dfsg.orig-vendor.tar.xz`: The pruned vendor directory. + +You must now update the orig tarball and unpacked source tree to match your pruned files. + +#### Updating the orig tarball -We download the upstream Rust source code using `uscan` again, which will use your new `debian/copyright` exclusion list to yank out everything you just pruned: +First, double-check your `debian/copyright` to ensure that `vendor` is listed under `Files-Excluded` (but _NOT_ `Files-Excluded-vendor`). You can now generate a new orig tarball without a `vendor` directory, since the vendor tarball component provides that directory: ```none $ uscan --download-version -v 2>&1 | tee ``` -Since we've now updated the orig tarball repack, we can rename this new orig tarball by making its last number `1`, i.e., `rustc-_+dfsg0ubuntu1`, symbolizing that we've repacked the source once again. +This is the version we will be using for the final package upload. Therefore, we shall rename it to the standard orig tarball format: -We can also update our `debian/changelog` version number to `+dfsg0ubuntu1-0ubuntu1`. Make sure that the portion of the version number before the hyphen matches the new orig tarball. You can use `dch -r` to automatically bump the changelog timestamp. +```none +$ mv ../rustc-_+dfsg{1,}.orig.tar.xz +``` -#### Keeping the tree clean +#### The source tree update -To keep the Git tree clean, we must rebase all our changes on top of the newly-packed orig tarball. +To keep the Git tree clean, we must rebase all our changes on top of the newly-pruned orig tarballs. First, make a backup just to be safe: @@ -543,44 +430,17 @@ First, make a backup just to be safe: $ git branch backup ``` -Next, start an interactive rebase, dropping the `gbp` commit where we [imported the upstream source](updating-rust-updating-the-source-code-in-your-repository): +Next, return to the previous Rust version and create a new branch to import the updated tarballs: ```none -$ git rebase -i merge- +$ git checkout merge- +$ git checkout -b import-new- ``` -Here's an example of what the 1.87 rebase todo looked like. Note that we took the opportunity to clean up the Git tree in general: squashing the large list of vendored pruning commits into one, dropping the initial removal of the old auto-generated `Files-Excluded` chunk, etc.: +The version string in `debian/changelog` must match the names of the tarballs we generated. Consult `git log merge-` and cherry-pick the commit where you added the changelog entry for ``: ```none -drop f24d250953b Remove autogenerated Files-Excluded chunk -pick 222264f2705 Create 1.87.0 changelog entry -drop 93acc98c031 New upstream version 1.87.0+dfsg0ubuntu0 -pick ec0e3f69232 Refresh upstream/u-ignore-ppc-hangs.patch -squash 7c73c0c7640 Refresh upstream/u-hurd-tests.patch -squash a7aecdca8cc Refresh upstream/d-ignore-test_arc_condvar_poison-ppc.patch -[...] -squash 8269adcb6d8 Refresh ubuntu/ubuntu-enzyme-use-system-llvm.patch -pick d045017bed6 d/p/series: Drop u-mdbook-robust-chapter-path.patch applied upstream -pick 960395225de Prune termcolor-1.1.2 -squash 0c8624920e7 Prune ring-0.16.20 -squash b5316157d1f Prune mio-1.0.1 -[...] -squash d0696832bf1 Prune openssl-sys from libssh2-sys-0.2.23 -pick 1239a9b4309 d/control, d/s/lintian-overrides: Update versioned identifiers -pick 713e26dc275 d/copyright: Update Files-Excluded with newly-pruned vendored crates -pick 3a590ea16cd d/copyright: Remove src/tools/rls from Files-Excluded (dropped upstream) -pick 46d9e994fbd Update changelog -``` - -:::{important} -Remember, the only change that's actually necessary for this step is the dropping of the `New upstream version +dfsg0ubuntu0` commit. -::: - -After the interactive rebase, we can go back to the previous Rust version and re-import the new orig tarball: - -```none -$ git checkout merge- -$ git checkout -b import-new- +$ git cherry-pick ``` Recreate the `experimental` branch: @@ -590,7 +450,7 @@ $ git branch -D experimental $ git branch experimental ``` -Then, we can merge our newly-pruned upstream source onto the previous Rust version's source cleanly: +Then, we can merge our newly-pruned tarballs onto the previous Rust version's source cleanly. Note the added `component` argument pointing `gbp` to the vendor tarball component. ```none $ gbp import-orig \ @@ -598,45 +458,76 @@ $ gbp import-orig \ --no-pristine-tar \ --upstream-branch=experimental \ --debian-branch=import-new- \ - ../rustc-_+dfsg0ubuntu1.orig.tar.xz + --component=vendor \ + ../rustc-_+dfsg.orig.tar.xz ``` Finally, we can switch back to our actual branch and rebase: ```none $ git checkout merge- -$ git rebase import-new- +$ git rebase -i import-new- +``` + +When consulting the list of Git rebase commands, don't forget to drop the commit in which you imported the old version of the tarball. Example for `rustc-1.91`: + +```none +drop 0759faf6707 New upstream version 1.91.1+dfsg~old +pick 85d8e6af63d Refresh d-0000-ignore-removed-submodules.patch +pick f31152ca1b9 Refresh d-0010-cargo-remove-vendored-c-crates.patch +[...] ``` -Consulting your `git log`, you should see the two new `gbp` commits immediately after the final commit of the last Rust version. +Consulting your `git log`, you should see the two new `gbp` commits immediately after the creation of the `` changelog entry. -#### Verifying your Git changes +#### Verifying your changes -Before you force-push to your personal Launchpad remote, you must be absolutely sure you didn't miss anything. Compare your branch with your `backup` branch: +To ensure that things were correctly pruned, take a look at the list of vendored Windows crates: ```none -$ git diff backup --name-status +$ ls -1 vendor | grep 'windows' ``` -You should **only** see a huge list of deleted `vendor` files. +You should see a list of windows-related crates. This may seem counter-intuitive — shouldn't these crates have been pruned from the `vendor/` directory? -:::{warning} -If there are _any other changes_ apart from deleted `vendor` files, then you made a mistake during the rebase — reset to your backup branch and try again. -::: +In actuality, these crates _have_ been pruned. `cargo-vendor-filterer` replaces pruned crates with _stubs_ – empty crates with the minimum `Cargo.toml` files required to satisfy `cargo` whilst building the compiler. -Once you're absolutely sure everything is good, you may safely delete your `import-new-` and `backup` branches. This is also a good time to force-push to your `` remote. +To verify that unwanted crates have been pruned properly, spot-check a few of the Windows crates inside `vendor/`. They should have an empty `lib.rs` and have a structure like this: + +```none +$ tree vendor/windows_x86_64_msvc-0.53.0 +vendor/windows_x86_64_msvc-0.53.0 +├── Cargo.toml +└── src + └── lib.rs + +2 directories, 2 files +``` + +### Updating Versioned Package References in Control Files + +Certain {term}`control files `, such as `debian/control` and `debian/source/lintian-overrides`, list versioned package names. These files must be updated to match the new version numbers. + +`update-version`, in `debian/rules`, updates all relevant control files automatically. In order to run it, it must be given an up-to-date Rust toolchain via `RUST_BOOTSTRAP_DIR`: + +```none +$ RUST_BOOTSTRAP_DIR=~/.rustup/toolchains/-x86_64-unknown-linux-gnu/bin/rustc \ + debian/rules update-version +``` + +After running the script, consult `git diff` and verify that, in `debian/control`, the two `Build-Depends` options for a bootstrapping compiler are `rustc-` and `rustc-`. + +After checking that the changes are correct, you may commit these changes and continue. ### After-Repack Patch Refreshes Some of the patches will no longer apply now that more files have been removed. You must refresh all the patches so they once again apply cleanly onto the newly-pruned source. -In general, you will follow the same protocol as the [initial patch refresh](updating-rust-initial-patch-refresh). - -Naturally, many lines will be removed from `debian/patches/prune/d-0021-vendor-remove-windows-dependencies` because many of the vendored crates you pruned were _themselves_ unnecessary. +In general, you will follow the same protocol as the [initial patch refresh](updating-rust-initial-patch-refresh). The most common change is dropping patches of pruned vendored files. ### Updating XS-Vendored-Sources-Rust -Inside of `debian/control` and `debian/control.in`, there's a special field called `XS-Vendored-Sources-Rust` which must be updated. It simply lists all the vendored crate dependencies, along with their version numbers, on a single line. +Inside of `debian/control`, there's a special field called `XS-Vendored-Sources-Rust` which must be updated. It simply lists all the vendored crate dependencies, along with their version numbers, on a single line. Luckily, the {lpsrc}`dh-cargo` package contains a script for automatically generating this line. Push all your patches, then run the script: @@ -645,63 +536,63 @@ $ quilt push -a $ CARGO_VENDOR_DIR=vendor/ /usr/share/cargo/bin/dh-cargo-vendored-sources ``` -Copy-paste the expected value it provides to both `debian/control` AND `debian/control.in`. +Replace the existing `XS-Vendored-Sources-Rust` field in `debian/control` with this new expected value. :::{attention} Make sure there's still an empty line after the end of the field! Mistakenly dropping the empty line will result in a build failure right at the end of the test build. ::: -#### Outdated toolchain issues +This is another opportunity to verify that you pruned unwanted crates properly. You shouldn't see any of the pruned Windows crates in the new `XS-Vendored-Sources-Rust` field. -If you're running a pre-versioned Rust Ubuntu release, then there's a decent chance the `cargo` installation required by `dh-cargo` will be too old. In this case, don't use `dh-cargo`—instead, manually download [`dh-cargo-vendored-sources`](https://git.launchpad.net/ubuntu/+source/dh-cargo/tree/dh-cargo-vendored-sources) (it's just a Perl script) and use it _without_ deb-based installations of Rust, which ensures that the Rustup snap's version will be used instead. +:::{note} +If you're running a pre-versioned Rust Ubuntu release, then there's a decent chance the `cargo` installation required by `dh-cargo` will be too old. In this case, don't use `dh-cargo`—instead, manually download [`dh-cargo-vendored-sources`](https://git.launchpad.net/ubuntu/+source/dh-cargo/tree/dh-cargo-vendored-sources) (it's just a Perl script) and use it _without_ deb-based installations of Rust, which ensures that the `rustup` snap's version will be used instead. +::: -### Updating debian/copyright +### Updating Vendored Copyright Overrides -All the new `vendor` files must be added to `debian/copyright`. Luckily, we can use a script which uses {term}`Lintian` ({manpage}`lintian(1)`) to generate all the missing copyright stanzas. +`debian/copyright` contains copyright stanzas for all the vendored dependencies of `rustc`. However, the crate stubs are "red herrings" for the purposes of `debian/copyright`. They're just empty crates; they don't contain any copyrighted code. -This script requires `pytoml`, so we must create and enter a virtual environment. +To prevent packaging tools from complaining that the stubbed crates are missing copyright stanzas, run `debian/add-vendored-copyright-overrides` to automatically update `debian/source/lintian-overrides`: ```none -# apt install python3-venv +$ debian/add-vendored-copyright-overrides ``` -Create the virtual environment somewhere convenient, such as `~/.venvs/`: +### Updating debian/copyright -```none -$ python3 -m venv rustc-lintian-to-copyright -``` +All the new `vendor` files must be added to `debian/copyright`. Luckily, we can use a script which uses {term}`Lintian` ({manpage}`lintian(1)`) to generate all the missing copyright stanzas. -After that, enter the virtual environment and install `pytoml`: +#### Generate the Lintian report + +[Clean up previous build artifacts](updating-rust-clean-build), build the source package using {manpage}`dpkg-buildpackage(1)`, then run Lintian, redirecting the output to somewhere convenient: + +(updating-rust-lintian-command)= ```none -$ source ~/.venvs/rustc-lintian-to-copyright/bin/activate -$ which python3 -$ python3 -m pip install pytoml +$ dpkg-buildpackage -S -I -i -nc -d -sa +$ lintian -i -I -E --pedantic | tee ``` -Temporarily edit `debian/lintian-to-copyright.sh` to recognize the virtual environment: +#### Clean up lintian-overrides -```diff -@@ -1,5 +1,5 @@ - #!/bin/sh - # Pipe the output of lintian into this. - sed -ne 's/.* file-without-copyright-information //p' | cut -d/ -f1-2 | sort -u | while read x; do -- /usr/share/cargo/scripts/guess-crate-copyright "$x" -+ python3 /home/maxgmr/rustc/rustc/debian/scripts/guess-crate-copyright "$x" - done -``` +Some of the crate stubs just happen to be covered by `debian/copyright` stanzas because of glob matching. For example, if `vendor/foo-1.0.2` was included and `vendor/foo-1.0.3` was stubbed, then a `debian/copyright` stanza matching `vendor/foo-*` will cover both. In this case, the autogenerated Lintian override for `vendor/foo-1.0.3` will trigger a Lintian warning for being mismatched. -[Clean up previous build artifacts](updating-rust-clean-build), build the source package using {manpage}`dpkg-buildpackage(1)`, then run Lintian and pipe the output to the script: +Get a list of these unnecessary overrides: ```none -$ dpkg-buildpackage -S -I -i -nc -d -sa -$ lintian -i -I -E --pedantic | debian/lintian-to-copyright.sh +$ grep 'mismatched-override file-without-copyright-information' ``` -Leave the virtual environment afterwards: +All of these overrides can be removed from `debian/source/lintian-overrides`. After removing the overrides, clean up old build artifacts, rebuild the source package, and re-run Lintian, just like you did in the [last step](updating-rust-lintian-command). + +#### Get missing copyright stanzas + +Getting the missing copyright stanzas is a tedious process. Luckily, `debian/lintian-to-copyright.sh` automates this. + +Pipe the output from your call to Lintian to the script: ```none -$ deactivate +$ cat | debian/lintian-to-copyright.sh ``` You may need to fill in some fields manually. [This](https://stackoverflow.com/questions/23611669/how-to-find-the-created-date-of-a-repository-project-on-github) is an easy way to find the start date of a GitHub repo. @@ -712,6 +603,36 @@ Keep things clean by adding the new `d/copyright` stanzas alphabetically. It mak ``` +## Common Update Build Issues + +### Unresolved import + +When building the compiler, you may see an error like the following: + +```none +error[E0432]: unresolved import `foo::Bar` + --> /<>/vendor/baz-1.0.4/src/quux.rs:16:5 + | +16 | use foo::Bar; + | ^^^^^^^^ no `Bar` in the root +``` + +This most commonly happens when there are two different versions of the same vendored crate, and the newer version gets stubbed; e.g.: + +- `vendor/foo-1.0.20` (STUB) +- `vendor/foo-1.0.19` (NOT A STUB) + +In this case, create (or modify) `debian/patches/ubuntu/ubuntu-pin-dep-ver.patch`, add the `Cargo.toml` of the offending file's crate (in this case `vendor/baz-1.0.4/Cargo.toml`) to the patch, and pin the version to the older one: + +```diff +--- a/vendor/baz-1.0.4/Cargo.toml ++++ b/vendor/baz-1.0.4/Cargo.toml +@@ -136,3 +136,3 @@ + [dependencies.foo] +-version = "1.0.17" ++version = "= 1.0.19" +``` + (updating-rust-lintian-checks)= ```{include} common/lintian-checks.md