Complete `target-release` test simulating an entire update #9059

jgallagher · 2025-09-22T22:08:53Z

(This is staged on top of #8936.)

Finishes the target-release reconfigurator-test, showing the simulate update walking through the process of starting new Nexus zones, waiting for handoff, then expunging the old Nexus zones.

Has two tweaks:

Fixes a planning report off-by-one bug where we'd claim a zone was both out of date and expunged (or updated) within the same plan.
Adds a set active-nexus-gen N command to reconfigurator-cli to control Nexus handoff instead of assuming it completes instantly.

Closes #8478

…ing new nexus

…pdated

dev-tools/reconfigurator-cli/tests/input/cmds-target-release.txt

smklein · 2025-09-22T22:17:19Z

nexus/types/src/deployment/planning_report.rs

            .and_modify(|zones| zones.push(zone_config.to_owned()))
            .or_insert_with(|| vec![zone_config.to_owned()]);
+
+        // We check for out-of-date zones before expunging zones. If we just


I do want this change, but should it be in this PR?

I think so? The test changing in this PR is the only place (at least in terms of tests / CI) where this change is visible.

smklein · 2025-09-22T22:31:39Z

dev-tools/reconfigurator-cli/tests/input/cmds-target-release.txt


-# We ought to update the inventory for the final sled and then step through
-# the Nexus handoff process, but that work is still in progress.
+# Start the Nexus handoff process: A planning step here should place three new


FYI, I did add this test to #8936 in a recent update, but I'm totally fine with deleting my changes and using what you have here.

(I think your addition of set active-nexus-gen 2 will become necessary anyway)

davepacheco

This looks pretty good to me. I haven't had a chance to look at the test output yet.

davepacheco · 2025-09-22T23:51:27Z

dev-tools/reconfigurator-cli/src/lib.rs

    Seed { seed: String },
    /// target number of Nexus instances (for planning)
    NumNexus { num_nexus: u16 },
+    /// generation of the active Nexus zones (controlling handoff)


I wonder if it's worth calling out explicitly somewhere around here that this is controlling the simulated state about which (simulated) Nexus instances think they're active, as opposed to changing the generation in the blueprint. ("active generation" shouldn't imply a blueprint change, and obviously they're not doing blueprint-edit here, but I think it's still easy for someone to forget this and get confused.)

Yeah, fair. I was trying to keep this short for --help but it's intricate enough that's probably the wrong thing to target. How about

/// specify the generation of Nexus zones that are considered /// active when running the blueprint planner

?

david-crespo · 2025-09-23T15:47:35Z

Nexus runs as part of this test, right? Is there a way I could hook into this (or just do my own similar thing) and hit the update status endpoint at various points throughout the process? See my test so far, very meh.

jgallagher · 2025-09-23T16:09:00Z

Nexus runs as part of this test, right? Is there a way I could hook into this (or just do my own similar thing) and hit the update status endpoint at various points throughout the process? See my test so far, very meh.

Nexus is not running in this test, no. :(

…est-nexus-handoff

nexus/reconfigurator/planning/src/blueprint_builder/builder.rs

davepacheco

This looks good. One thing I'm worried is missing is serializing this state (in the save command), exporting it from a live system (i.e., putting it into UnstableReconfiguratorState), and loading that back in. I'm not sure how much of a problem that is? It does feel like kind of a problem when using either of those workflows (save + load or omdb reconfigurator export + load) once the active generation is not 1.

jgallagher · 2025-09-23T18:19:52Z

This looks good. One thing I'm worried is missing is serializing this state (in the save command), exporting it from a live system (i.e., putting it into UnstableReconfiguratorState), and loading that back in. I'm not sure how much of a problem that is? It does feel like kind of a problem when using either of those workflows (save + load or omdb reconfigurator export + load) once the active generation is not 1.

Hmm, great point. Is "the set of active Nexuses" an inherent problem with saved state? I.e., there isn't anything in that state that tells us which set of Nexuses is active, if there are multiple generations.

If there's only one generation, we could assume that's the active set. Maybe load should check for this, and (a) set the active generation if there's only one, or (b) require the user to specify the active generation if there are two?

davepacheco · 2025-09-23T18:27:17Z

This looks good. One thing I'm worried is missing is serializing this state (in the save command), exporting it from a live system (i.e., putting it into UnstableReconfiguratorState), and loading that back in. I'm not sure how much of a problem that is? It does feel like kind of a problem when using either of those workflows (save + load or omdb reconfigurator export + load) once the active generation is not 1.

Hmm, great point. Is "the set of active Nexuses" an inherent problem with saved state? I.e., there isn't anything in that state that tells us which set of Nexuses is active, if there are multiple generations.

I think there's probably nothing in the UnstableReconfiguratorState today that tells us with certainty what it is, but I was assuming we'd just add it.

If there's only one generation, we could assume that's the active set. Maybe load should check for this, and (a) set the active generation if there's only one, or (b) require the user to specify the active generation if there are two?

This is also an option.

jgallagher · 2025-09-23T18:37:52Z

I think there's probably nothing in the UnstableReconfiguratorState today that tells us with certainty what it is, but I was assuming we'd just add it.

Oh! I'm wrong; I think we do have this. UnstableReconfiguratorState includes the planning input, which includes the set of active_nexus_zones. We should be able to infer the generation from that.

jgallagher · 2025-09-23T20:55:31Z

Oh! I'm wrong; I think we do have this. UnstableReconfiguratorState includes the planning input, which includes the set of active_nexus_zones. We should be able to infer the generation from that.

This is implemented in ddf1afb. I tested it by hand, although if it seems important I might be able to finagle an automated test for it too? At least confirming that it can load a generation greater than 1.

davepacheco · 2025-09-23T22:00:32Z

dev-tools/reconfigurator-cli/tests/output/cmds-stdout

+warnings:
+  could not determine active Nexus generation from serialized state: no target blueprint set (using default of 1)


Why's this? Aren't we loaded something we just saved, and wouldn't the saved version have what we need?

This test doesn't use the example system; it's just showing a handful of commands. Earlier in this file we see no blueprints at all:

> blueprint-list T ENA ID PARENT TIME_CREATED

which is consistent with the warning here that there's no target blueprint.

davepacheco · 2025-09-23T22:02:07Z

nexus/reconfigurator/simulation/src/config.rs

+                res.warnings.push(format!(
+                    "could not determine active Nexus \
+                     generation from serialized state: \
+                     {message} (using default of 1)"
+                ));


I'm basically wondering if this should be an error (similar to what happens if you try to load it without specifying an inventory collection and there's more than one of them). But it depends on why this could happen.

This should never happen in a state blob pulled from a real system. But it can happen in some stripped down reconfigurator-cli contexts (like the one from your other comment).

davepacheco

Marking approved because I'd like to figure that out (the thing I commented on), but I don't want to block this too much longer than that concern.

Finishes the `target-release` `reconfigurator-test`, showing the simulate update walking through the process of starting new Nexus zones, waiting for handoff, then expunging the old Nexus zones. Has two tweaks: * Fixes a planning report off-by-one bug where we'd claim a zone was both out of date and expunged (or updated) within the same plan. * Adds a `set active-nexus-gen N` command to `reconfigurator-cli` to control Nexus handoff instead of assuming it completes instantly. Closes #8478 --------- Co-authored-by: Sean Klein <[email protected]>

smklein added 30 commits September 2, 2025 11:46

(4/N) db_metadata_nexus database queries (handoff)

38a265b

(5/N) Read database access records on boot

3e61a97

(6/N) Add image to planning report for zones

e1b43c3

(7/N) Use nexus_generation, update it

6b7e9ed

Merge with main

db263c1

drop keys for InlineErrorChain

bbb204c

Merge

67505ec

Merge

e1f2fe1

update nexus internal API

529874d

Actually create NotYet records when appropriate

961338c

Merge with main

fed7aa0

Merge

bf79e74

Merge

51b3103

Fix database_nexus_access_create to use active Nexus

9514deb

Make clippy happy

ce31570

Simplify database_nexus_access_create, move tests

70d4ab9

Unify queries for all db_metadata_nexus states

690ea16

comments, enums, set_..._nexuses

f15fa4b

determine_nexus_generation no longer returns an option

5525529

maybe when we say the gen bumps, we should do that

c8126af

merge

056829f

merge

e3615a9

move some generation checks out of planner, into blippy

a38a811

get_zones_not_yet_propagated_to_inventory comments

b4bcca7

clarifying planner comments

a6548ca

proposed

448a39d

Test cleanup, make lack of nexus zones more of an error, delay deploy…

07d79f3

…ing new nexus

Ensure nexus can always be 'shut down' safely, refactor can_zone_be_u…

730a3b5

…pdated

Merge

f34422f

clippy

4939771

jgallagher requested a review from smklein September 22, 2025 22:08

jgallagher changed the title ~~wip~~ Complete target-release test simulating an entire update Sep 22, 2025

cargo fmt

d491246

smklein reviewed Sep 22, 2025

View reviewed changes

smklein mentioned this pull request Sep 22, 2025

(7/N) Use nexus_generation, update it #8936

Merged

smklein reviewed Sep 22, 2025

View reviewed changes

davepacheco mentioned this pull request Sep 22, 2025

internal/external DNS should only contain records for active Nexus instances #9060

Merged

5 tasks

davepacheco reviewed Sep 22, 2025

View reviewed changes

Base automatically changed from nexus_gen_usage to main September 23, 2025 02:12

jgallagher added 3 commits September 23, 2025 11:21

Merge branch 'main' into john/target-release-test-nexus-handoff

5392671

fix incorrect comments referring to the install dataset

8118180

more specific description of set active-nexus-gen

5fd5517

Merge remote-tracking branch 'origin/main' into john/target-release-t…

19a7276

…est-nexus-handoff

smklein reviewed Sep 23, 2025

View reviewed changes

nexus/reconfigurator/planning/src/blueprint_builder/builder.rs Outdated Show resolved Hide resolved

remove test pruned from main

0ecb80c

smklein approved these changes Sep 23, 2025

View reviewed changes

davepacheco reviewed Sep 23, 2025

View reviewed changes

determine active Nexus generation from UnstableReconfiguratorState

ddf1afb

davepacheco reviewed Sep 23, 2025

View reviewed changes

davepacheco approved these changes Sep 23, 2025

View reviewed changes

jgallagher merged commit 2648592 into main Sep 24, 2025
16 checks passed

jgallagher deleted the john/target-release-test-nexus-handoff branch September 24, 2025 14:13

iliana mentioned this pull request Sep 24, 2025

fix mis-merged expectorate test #9073

Closed

		warnings:
		could not determine active Nexus generation from serialized state: no target blueprint set (using default of 1)

Complete target-release test simulating an entire update #9059

Complete target-release test simulating an entire update #9059

Uh oh!

Conversation

jgallagher commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davepacheco left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-crespo commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgallagher commented Sep 23, 2025

Uh oh!

Uh oh!

davepacheco left a comment

Choose a reason for hiding this comment

Uh oh!

jgallagher commented Sep 23, 2025

Uh oh!

davepacheco commented Sep 23, 2025

Uh oh!

jgallagher commented Sep 23, 2025

Uh oh!

jgallagher commented Sep 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davepacheco left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Complete `target-release` test simulating an entire update #9059

Complete `target-release` test simulating an entire update #9059

jgallagher commented Sep 22, 2025 •

edited

Loading

david-crespo commented Sep 23, 2025 •

edited

Loading