kops_cluster_updater is broken on new cluster creation in 1.31.0 (but worked in 1.30.1)

[shapirus]

Situation: creation of a new kops cluster in AWS from scratch using the terraform kops provider.

Resources defined:

• kops_cluster
• a master kops_instance_group (min_size=1 max_size=1)
• a non-master kops_instance_group (min_size=1 max_size=1)

terraform plan runs fine. The cluster revision and revisions of both instance groups are present in the keepers attribute. The plan output looks good.

Behavior of kops_cluster_updater in 1.30.1:

• applies the generated cluster manifest
• initializes the launch templates, ASGs and everything else in the cloud
• waits for instances in both IGs to start, validates the cluster, job done

Behavior of kops_cluster_updater in 1.31.0:

• applies a cluster manifest without the non-master instance group, only the master IG is applied
• only the master instance group's cloud resources (LT, ASG) are created
• kops_cluster_updater is stuck in the "still creating state", never being able to validate the cluster, because the desired instance group is never created, and eventually fails on timeout
• if kops update cluster is run at this point manually, it displays a pending diff with all the stuff belonging to the non-master instance group, and if it is run again with --yes, then it will create the respective resources in the cloud.

Since it worked fine in 1.30.1 and is broken in 1.31.0, it should probably be easy enough to track down the change that introduced this bug.

[argoyle]

Aaah, my change broke new clusters of course 🤦🏻 Will take a look shortly on how to handle that. Must be possible to know if it's a new cluster or an update of an existing somehow of course.

[shapirus]

I believe it's caused by this:

* feat: update to kOps 1.31

Make cluster updating follow the new reconcile format where control-planes
are upgraded first followed by remaining nodes and then completed with a prune.

10afb1b#diff-cb8864ab2ac6ea24f1dfce535d85948d1770a0b8bed338c109a6a5fd0c337566

This is a good idea for rolling update, but apparently it doesn't take the new cluster creation situation into account.

@argoyle does it make sense?

(update: your lightning fast reply didn't give me a chance to read it before posting this comment lol)

[argoyle]

Yes, that's the one that caused that behaviour I would say.

[argoyle]

Didn't have the possibility right now to test #218 but maybe you can build and try it and report back @shapirus?

[shapirus]

Didn't have the possibility right now to test #218 but maybe you can build and try it and report back @shapirus?

I can test it right away, but I have no idea how to build, let alone reference this build in my terraform configs.

Any chance you could build a patch revision and push it to the TF registry?

...or maybe some side-project development registry, that'll also be fine.

[argoyle]

I usually do make all and then override the provider in ~/.terraformrc like this:

provider_installation {
  dev_overrides {
    "terraform-kops/kops" = "/Users/argoyle/.terraform.d/plugins/github/terraform-kops/kops/0.0.1/darwin_arm64"
  }

# Install all other providers directly from their origin provider registry as normal.
#   # If this is omittet, terraform will only use the dev_overrides block.
  direct {}
}

[argoyle]

Otherwise I'll test it myself later today and see if it seems to behave correctly.

[shapirus]

We use TF cloud here, so that'll probably not work for my case, as the remote side won't have access to this.

...update: yes, a quick search suggests that there is some way of creating a private registry that can be used with cloud TF, but I think it'll take longer for me to read about it and set it all up than for you to test the fix on your side when you have the time.

[argoyle]

Aaah, then I'll test in a couple of hours.

[argoyle]

I usually do make all and then override the provider in ~/.terraformrc like this:

Correcting myself here. It should be make install.

[argoyle]

Seems to work fine. I'll make the PR ready for review.

[shapirus]

@sl1pm4t @eddycharly any chance you could review, merge the PR and release a fixed version?