add live test for Nexus handoff #9024

davepacheco · 2025-09-16T03:27:50Z

This PR adds a new live test for Nexus handoff.

Depends on ~~#9022 and~~ #9023.

TODO:

Make sure the test turns off the autoplanner and/or verifies that it is off

…off-live-test

davepacheco · 2025-09-16T17:33:29Z

With a4x2 built from cb83530 and live test built from 8032377, it takes a while, but it works:

#      TMPDIR=/var/tmp ./cargo-nextest nextest run --profile=live-tests          --archive-file live-tests-archive/omicron-live-tests.tar.zst          --workspace-remap live-tests-archive
  Extracting 2 binaries, 1 build script output directory, and 5 linked paths to /var/tmp/nextest-archive-Fer3fa
   Extracted 79 files to /var/tmp/nextest-archive-Fer3fa in 3.66s
info: experimental features enabled: setup-scripts
------------
 Nextest run ID c4a011b3-5629-4c0a-8a84-e186530b0122 with nextest profile: live-tests
    Starting 2 tests across 2 binaries
        SLOW [> 60.000s] omicron-live-tests::test_nexus_add_remove test_nexus_add_remove
        SLOW [>120.000s] omicron-live-tests::test_nexus_add_remove test_nexus_add_remove
        SLOW [>180.000s] omicron-live-tests::test_nexus_add_remove test_nexus_add_remove
        SLOW [>240.000s] omicron-live-tests::test_nexus_add_remove test_nexus_add_remove
        PASS [ 252.738s] omicron-live-tests::test_nexus_add_remove test_nexus_add_remove
        SLOW [> 60.000s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
        SLOW [>120.000s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
        SLOW [>180.000s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
        SLOW [>240.000s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
        SLOW [>300.000s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
        SLOW [>360.000s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
        SLOW [>420.000s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
        PASS [ 433.442s] omicron-live-tests::test_nexus_handoff test_nexus_handoff
------------
     Summary [ 686.212s] 2 tests run: 2 passed (2 slow), 0 skipped

…e-notyet-records

…off-live-test

smklein · 2025-09-17T18:35:21Z

live-tests/tests/common/reconfigurator.rs

+/// Returns whether the given blueprint's sled configurations appear to be
+/// propagated to all sleds.
+///
+/// Returns the inventory collection so that the caller can check additional
+/// details if wanted.


Suggested change

/// Returns whether the given blueprint's sled configurations appear to be

/// propagated to all sleds.

///

/// Returns the inventory collection so that the caller can check additional

/// details if wanted.

/// Verifies that the given blueprint's sled configurations appear to be

/// propagated to all active sleds.

///

/// Returns the inventory collection so that the caller can check additional

/// details if wanted.

Good catch -- updated the comment in 34aa89a.

smklein · 2025-09-17T18:42:46Z

live-tests/tests/test_nexus_handoff.rs

+    // Wait for the zones to be running.
+    // (This does not mean that their Nexus instances are running.)


Suggested change

// Wait for the zones to be running.

// (This does not mean that their Nexus instances are running.)

// Wait for the zones to be running.

//

// We expect that the new Nexus zones will be blocked on the "not yet"

// DbMetadataNexusState, waiting for handoff.

Updated the comment in 34aa89a.

smklein · 2025-09-17T18:46:41Z

live-tests/tests/test_nexus_handoff.rs

+
+    // Make sure we're starting from a known-normal state.
+    // First, we have an enabled target blueprint.
+    let blueprint1 = blueprint_load_target_enabled(log, nexus)


Nitpick; WDYT about giving these more descriptive names than the enumerated "blueprint1"? I just suffered through this in #8936 , where there was a test that went up to like "blueprint12" in a multi-hundred line long test. Refactoring it to add a blueprint in the middle sucks.

Ah, sorry about that. Sometimes I think the numbers are really helpful for keeping track of the order. In this case, at least, the names "initial", "new_nexus", "handoff", and "cleanup" are clear enough about the order. Done in 34aa89a.

smklein · 2025-09-17T18:47:47Z

live-tests/tests/test_nexus_handoff.rs

+    // Wait for this to get propagated everywhere.
+    let _latest_collection = blueprint_wait_sled_configs_propagated(
+        opctx,
+        datastore,
+        &blueprint4,
+        new_nexus,
+        Duration::from_secs(120),
+    )
+    .await
+    .expect("waiting for blueprint4 sled configs");


What are we waiting for here? Should we be verifying that we can't access the "old Nexus quiesce" interface after this or something?

(If we have nothing else to verify, why wait?)

This is just general test hygiene. I don't like to have tests kick off work that might not be finished when they finish running. The main reason is that this can make things flaky when you run multiple tests in a row because a second test winds up confused by the changing state left by the first test.

davepacheco added 25 commits September 5, 2025 16:22

WIP: pretty good changes so far

6545d73

WIP: cont

cba808d

pass OpContext through usefully

8dce7c1

start fixing/writing tests

7ad5d55

fix test

40f75ac

WIP: live test for handoff

b57236b

Merge branch 'main' into dap/quiesce-with-db

ee285d7

final timeout was too short due to transient Cockroach issue on a4x2

f7c92b3

remove stuff that will go into separate PRs

d45a995

clean up / refactor

bdf36a7

fix wrong comment

0830d57

callers of sled_add_zone_nexus() should have to provide Nexus generation

cb98214

fix live test

537fb45

fix clippy

454aef5

write db_metadata_nexus records with the correct state

4d99381

Merge branch 'dap/nexus-gen-required' into dap/handoff-live-test

35d8eb8

add live test for Nexus handoff

c8475e9

fix clippy

cb83530

Merge remote-tracking branch 'origin/main' into dap/quiesce-with-db

fa6d78f

Merge branch 'dap/quiesce-with-db' into dap/write-notyet-records

0f15899

review feedback (update comment)

f01ffe9

Merge commit 'd3ad55bf8bd4f54dcf1c915ebddba30db5b6d3dc' into dap/hand…

d945eaf

…off-live-test

Merge commit '3c10adbe14669455a7670c01125c0de187e8ed72' into dap/hand…

e751a8f

…off-live-test

Merge branch 'dap/write-notyet-records' into dap/handoff-live-test

89c89ee

live test should check planner config

8032377

davepacheco requested review from jgallagher and smklein September 16, 2025 17:33

davepacheco self-assigned this Sep 16, 2025

davepacheco marked this pull request as ready for review September 16, 2025 17:34

davepacheco added 4 commits September 16, 2025 11:01

review feedback

e8eb204

Merge commit 'e8eb204' into dap/write-notyet-records

88d0668

Merge commit 'a96a4e64c1f756cc58340ccb8a1082a6aab90706' into dap/writ…

04cdeb7

…e-notyet-records

Merge commit '2163cfa43af7d5558021946bd642f78e6104ab06' into dap/writ…

08b8580

…e-notyet-records

Base automatically changed from dap/write-notyet-records to main September 17, 2025 13:21

davepacheco added 3 commits September 17, 2025 06:22

Merge commit '08b8580' into dap/handoff-live-test

09a30ac

Merge commit 'f1a21fbecbb677d8600ecccb43cd4ed73077abd1' into dap/hand…

cc86882

…off-live-test

Merge remote-tracking branch 'origin/main' into dap/handoff-live-test

c0b5e3b

smklein approved these changes Sep 17, 2025

View reviewed changes

review feedback

34aa89a

davepacheco enabled auto-merge (squash) September 17, 2025 19:57

davepacheco merged commit 0d82762 into main Sep 17, 2025
16 checks passed

davepacheco deleted the dap/handoff-live-test branch September 17, 2025 21:10

charliepark pushed a commit that referenced this pull request Sep 19, 2025

add live test for Nexus handoff (#9024)

b8bd54c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add live test for Nexus handoff #9024

add live test for Nexus handoff #9024

Uh oh!

davepacheco commented Sep 16, 2025 •

edited

Loading

Uh oh!

davepacheco commented Sep 16, 2025 •

edited

Loading

Uh oh!

smklein Sep 17, 2025

Uh oh!

davepacheco Sep 17, 2025

Uh oh!

smklein Sep 17, 2025

Uh oh!

davepacheco Sep 17, 2025

Uh oh!

smklein Sep 17, 2025

Uh oh!

davepacheco Sep 17, 2025

Uh oh!

smklein Sep 17, 2025

Uh oh!

davepacheco Sep 17, 2025

Uh oh!

Uh oh!

Uh oh!

		// Wait for the zones to be running.
		// (This does not mean that their Nexus instances are running.)

add live test for Nexus handoff #9024

add live test for Nexus handoff #9024

Uh oh!

Conversation

davepacheco commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davepacheco commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davepacheco commented Sep 16, 2025 •

edited

Loading

davepacheco commented Sep 16, 2025 •

edited

Loading