Skip to content

Conversation

@lourens-octopus
Copy link
Contributor

@lourens-octopus lourens-octopus commented Nov 13, 2025

Background

[sc-123004]

The merge of #558 has introduced some flakiness into the deploy_test.go test file.

Investigation

The failures in our build pipeline seems to be very intermittent and would easily resolve be rerunning the action. Running the test locally to reproduce the action requires rerunning the test over and over until a failure is raised.
Go allows us to do this by setting the count property when running the tests. The following command was used to raise the error: go test ./pkg/cmd/release/deploy/deploy_test.go -count=200 -failfast. The failfast property was used to stop the iteration when a failure occured. The result of this command generally looks as follows:

image

In general the failures where always related to one of the following tests:

  1. prompted variable
  2. tenant and tags in a definitely tenanted project
  3. tenant and tags in a mabye tenanted project (choosing tenanted)
  4. prompt if feature toggle is on and a release has missing packages

From the test failure output it was displaying both the general deploy information as well as the additional options, where the test was only expecting the general information. This is surprising for two reasons:

  1. The additional options are contained in a function called PrintAdvancedSummary, that can be found here. This function is always called during the deployment process, so its information should always be displayed to the user.
  2. The test itself does check for the existence of the additional options, for example here.

Findings

The fact that the additional options is always called leads me to believe that what is displayed to the screen is actually correct and should not result in a failure of the test. I believe that the addition of ephemeral environments to the deployment command and its subsequent refactor didn't introduce flakiness but merely surfaced a brittleness in the way the test was structured.

The comparison of the text between the general deploy information and the additional options is done differently in the tests. The general information checks for an exact match (using assert.Equal), where the additional options only checks for a regular expression match (using assert.Regexp). The test thus required the exact text to be displayed to the buffer at a specific time, it would then clear it before looking for more text. With the addition of the new code this meant that in some of the runs the timing would be off and more information would be displayed than was expected.

In order to solve this, the PR changes the exact match to a contains, to check that the information it expects appears somewhere in the output. We are still checking for the same information to be displayed, but we do not care if there is additional options also contained in the buffer. I've also removed the clearing of the buffer further down in the test to account for the timing of additional checks to not be affected by a clearing of the display buffer.

Result

Running the command to reproduce the error locally now results in no error being displayed.

image

@lourens-octopus lourens-octopus force-pushed the lourens/fix-flaky-deploy-test branch from 9a10938 to 4ded43e Compare November 13, 2025 21:35
@lourens-octopus lourens-octopus requested a review from a team November 13, 2025 21:37
@lourens-octopus lourens-octopus marked this pull request as ready for review November 13, 2025 21:37
@lourens-octopus lourens-octopus force-pushed the lourens/fix-flaky-deploy-test branch from 4ded43e to 3ceae00 Compare November 13, 2025 21:41
@lourens-octopus lourens-octopus merged commit 82b8981 into main Nov 13, 2025
4 checks passed
@lourens-octopus lourens-octopus deleted the lourens/fix-flaky-deploy-test branch November 13, 2025 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants