Introduce `RetryOnFailure` lifecycle management strategy

[stefanprodan]

This is a proposal for introducing a new way of performing lifecycle management akin to Flux kustomize-controller operational mode.

The RetryOnFailure strategy is suitable for statefulsets and other workloads that cannot tolerate rollbacks and have a high rollout duration susceptible to health check timeouts and transient capacity errors.

The RetryOnFailure strategy will ensure that:

• An installation failure will not be retried immediately, instead, the controller will retry with an upgrade at a fixed interval defined in the HelmRelease.
• An upgrade failure will leave the Helm release in a failed state without performing any remediations.
• An upgrade failure will never result in a rollback or uninstall.
• Upgrade failures are always retried at a fixed interval defined in the HelmRelease.

API Changes

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
spec:
  install:
    strategy:
      name: RetryOnFailure # defaults to RemediateOnFailure
      retryInterval: 5m # only used for RetryOnFailure (defaults to 5m)
    remediation: # ignored for RetryOnFailure
      retries: 2
  upgrade:
    strategy:
      name: RetryOnFailure # defaults to RemediateOnFailure
      retryInterval: 5m # only used for RetryOnFailure (defaults to 5m)
    remediation: # ignored for RetryOnFailure
      retries: 2
      strategy: rollback

When not specified, or when the strategy is set to RemediateOnFailure, the lifecycle management works like before.

For installations, the RetryOnFailure strategy will perform an uninstall on failure, then will rerun the installation after the specified retry interval.

For upgrades, the RetryOnFailure strategy will behave like flux reconcile hr --force when the remediation retries are set to 0.

[matheuscscp]

the controller will uninstall the release

Is an uninstall strictly required? I was able to trigger an upgrade after a failed install. But not an install, as the Helm SDK errors out with cannot re-use a name that is still in use. If switching to upgrades after a failed install is an acceptable behavior, we can avoid uninstalls. The install strategy name could be RetryWithUpgradeOnFailure.

[matheuscscp]

After running several tests I concluded that both install strategies can be useful, and allowing users to choose the one that better suits their use cases is probably the best choice here. I have implemented both RetryOnFailure and RetryWithUpgradeOnFailure for install, and RetryOnFailure for upgrade, here.

Here is a preview image:

ghcr.io/fluxcd/helm-controller:preview-d7540ecb

This preview image can be used together with the HelmRelease CRD from the same commit, here:

https://github.com/fluxcd/helm-controller/blob/d7540ecb1f440b68cb7682aff43d1a81d5f3f56b/config/crd/bases/helm.toolkit.fluxcd.io_helmreleases.yaml

[tmmorin]

First: this evolution is very relevant 👍

My only comment on the initial strategy proposal would have been the same as the one you made @matheuscscp : on a failed installation, it is useful to have the possibility of retrying without an uninstallation.

As an illustration, in Sylva project today we compensate the absence of "retry without uninstall" with a Kyverno policy that annotates failed HelmReleases with reconcile.fluxcd.io/forceAt and reconcile.fluxcd.io/requestedAt to force a retry. This works well for us for both "install" and "upgrade" situation, the goal for us simply being to retrigger the Helm "wait" processing without uninstalling.

[tmmorin]

After running several tests I concluded that both install strategies can be useful, and allowing users to choose the one that better suits their use cases is probably the best choice here. I have implemented both RetryOnFailure and RetryWithUpgradeOnFailure for install, and RetryOnFailure for upgrade, #1281.

I'm wondering if this couldn't be simplified: maybe you could have a single strategy for the installation step, RetryOnFailure, which would do "retry with a Helm upgrade" if install.remediation.retries is zero ?

[tmmorin]

The example states that upgrade.remediation is purely ignored when strategy is RetryOnFailure:

  upgrade:
    strategy:
      name: RetryOnFailure # defaults to RemediateOnFailure
      retryInterval: 5m # only used for RetryOnFailure (defaults to 5m)
    remediation: # ignored for RetryOnFailure    <<<<<<<<<<<<<<<<<<<<<<<
      retries: 2
      strategy: rollback

This seems to me as the right thing to do.

But then the proposal also says...

For upgrades, the RetryOnFailure strategy will behave like flux reconcile hr --force when the remediation retries are set to 0.

(emphasis mine)

... which seems contradictory with the example saying that upgrade.remediation is purely ignored when strategy is RetryOnFailure.

I think that if the last sentence was just this, we would have something just fine:

... the RetryOnFailure strategy will behave like flux reconcile hr --force when the remediation retries are set to 0.

[matheuscscp]

when the remediation retries are set to 0.

This part is written in the proposal only to further explain the behavior, it doesn't mean that there is any connection with the .remediation.retries field. It's just to explain the new behavior in terms of an existing one

[matheuscscp]

After running several tests I concluded that both install strategies can be useful, and allowing users to choose the one that better suits their use cases is probably the best choice here. I have implemented both RetryOnFailure and RetryWithUpgradeOnFailure for install, and RetryOnFailure for upgrade, #1281.

I'm wondering if this couldn't be simplified: maybe you could have a single strategy for the installation step, RetryOnFailure, which would do "retry with a Helm upgrade" if install.remediation.retries is zero ?

I think a spec like this is a bit hard to read/understand, it's not clear what's gonna happen on an install failure (and you would still need the upgrade strategy to achieve automatic retries)

spec:
  install:
    strategy:
      name: RetryOnFailure
    remediation:
      retries: 0
  upgrade:
    strategy:
      name: RetryOnFailure

So I think the following is simpler and easier to understand:

spec:
  install:
    strategy:
      name: RetryWithUpgradeOnFailure
  upgrade:
    strategy:
      name: RetryOnFailure

[tmmorin]

when the remediation retries are set to 0.

This part is written in the proposal only to further explain the behavior, it doesn't mean that there is any connection with the .remediation.retries field. It's just to explain the new behavior in terms of an existing one

Oh, yes, of course.

I had mistakenly understood "For upgrades, the RetryOnFailure strategy will behave like flux reconcile hr --force when the remediation retries are set to 0." as meaning "For upgrades, when the remediation retries are set to 0, the RetryOnFailure strategy will behave like flux reconcile hr --force".

[tmmorin]

So I think the following is simpler and easier to understand:

spec:
  install:
    strategy:
      name: RetryWithUpgradeOnFailure
  upgrade:
    strategy:
      name: RetryOnFailure

Fair enough, it's indeed simpler.

[matheuscscp]

So I think the following is simpler and easier to understand:

spec:
  install:
    strategy:
      name: RetryWithUpgradeOnFailure
  upgrade:
    strategy:
      name: RetryOnFailure

Fair enough, it's indeed simpler.

The preview I posted above is already matching this API, feel free to take it for a ride 👍

[tmmorin]

Two comments regarding the strategy names:

• RetryOnFailure may be misunderstood as meaning that "spec.(install|upgrade).remediation.retries" is used -- maybe using "NoUninstall" in the strategy names would avoid this possible confusion between the existing "retry with remediation" and the new "retry == just wait again" introduced by this new feature; or perhaps "KeepWaiting", or "NoRemediation"
• RetryWithUpgradeOnFailure may be misunderstood as meaning that, on a failed install, settings from "spec.upgrade" would be used

[matheuscscp]

• RetryWithUpgradeOnFailure may be misunderstood as meaning that, on a failed install, settings from "spec.upgrade" would be used

This is not a misunderstanding, it's the correct interpretation. This setting is telling helm-controller to perform an upgrade when a install fails, this involves using everything under .spec.upgrade, and nothing under .spec.install anymore, except for the .spec.install.strategy.retryInterval when the decision of retrying later with an upgrade takes place.

That's exactly why setting .spec.upgrade.strategy.name is important to achieve automatic retries without remediation, like in the YAML you and I just pasted above.