[Bug] 'eksctl.cluster.k8s.io/v1alpha1/cluster-name' is not a valid tag key

[rhino5oh]

When creating an EKS cluster, the nodegroup's CloudFormation stack fails to to create. Looking at the AWS GUI, I see the following error message:

This AWS::AutoScaling::AutoScalingGroup resource is in a CREATE_FAILED state.

Resource handler returned message: "Group did not stabilize. Last scaling activity: 'eksctl.cluster.k8s.io/v1alpha1/cluster-name' is not a valid tag key. Tag keys must match pattern ([0-9a-zA-Z\\-_+=,.@:]{1,255}), and must not be a reserved name ('.', '..', '_index'). Launching EC2 instance failed." (RequestToken: eeb4351e-96d0-fef8-a86b-8d70fd3b4f41, HandlerErrorCode: NotStabilized)

Again, this is for the nodegroup CloudFormation template, NOT the cluster CloudFormation template - that gets created successfully

eksctl version: 0.208.0

I attempted to create a cluster via the following command:

eksctl create cluster -f /tmp/eksctl_cluster.yml

Contents of /tmp/eksctl_cluster.yml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: bax-cluster
  region: us-east-1
  version: "1.32"

vpc:
  subnets:
    public:
      us-east-1a: { id: OMITTED }
      us-east-1b: { id: OMITTED }
      us-east-1c: { id: OMITTED }

nodeGroups:
  - name: standard
    # Which security group(s) to assign to all worker nodes
    securityGroups:
      withShared: true
      withLocal: true
      attachIDs: ['OMITTED']
    amiFamily: AmazonLinux2023
    instanceType: t3.small
    desiredCapacity: 2
    ssh:
      allow: true
      publicKeyPath: OMITTED
      # This option tells eksctl to use the specified security group(s) for SSH access, rather
      # than auto-generating a security group rule that allows SSH access from anywhere.
      sourceSecurityGroupIds: ['OMITTED']

[rhino5oh]

More info on restrictions from AWS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions

[rhino5oh]

It appears that the solution is to disable the "Allow tags in metadata" option in the launch template that gets created by eksctl. The interesting thing is that this is disabled when I do not specify a nodeGroup information or VPC info.

For example, if I simply say eksctl create cluster --name my-cluster, then a cluster gets created no problem. And the launch template used by the AutoScaling group does have "Allow tags in metadata" set to disabled

However, if I used the config file from my original description, the launch template does NOT have "Allow tags in metadata" disabled, causing this issue.

Any info on what the deal is here? Why is the launch template behaving differently between using a config file vs not using one?

[rhino5oh]

And finally, the example config file from the docs (using nodeGroups) results in the same error: https://eksctl.io/usage/creating-and-managing-clusters/#using-config-files

[naclonts]

Hi @rhino5oh, thanks for the bug report and the detailed repro steps!

Could you confirm what eksctl version you're using, and which example from the Eksctl docs page you used that had the error? I tried creating a cluster with this on eksctl v0.208.0:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: basic-cluster
  region: eu-north-1

nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 3
    volumeSize: 80
    ssh:
      allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key
  - name: ng-2
    instanceType: m5.xlarge
    desiredCapacity: 2
    volumeSize: 100
    ssh:
      allow: true

(Copied from the docs, although I modified the ssh and desiredCapacity fields)

Then ran eksctl create cluster -f config.yaml. The cluster and launch templates created successfully; the "Allow tags in metadata" option was also set to disabled.

I wonder if I have some different config in my AWS account. I do see a reference to the tag you mentioned as OldClusterNameTag in the source code. Since instance tag keys can't contain / characters, we may need to modify something to prevent this (possibly around makeMetadataOptions).

[rhino5oh]

I was using 0.208.0 .... It worked because you had Allow Tags in metadata disabled :) ... In AWS, if you go to EC2 -> Scroll all the way to the bottom on the left nav and click Settings -> IMDS defaults section, you can see what it is set to in your account. If you enable "Access to tags in metadata", it'll break because the launch template tries to use a tag with a / in the key.

AWS makes the tagging requirements more strict if this option is enabled. So perhaps eksctl can make a change to use different key names for tags so that this issue won't occur if someone has this setting enabled in AWS?