providerID not set with karpenter nodes

[Sindvero]

Hello,

I'm seeing issue in aws cluster (not eks) with karpenter deployed where nodes created by Karpenter do not get their providerID set ever by the AWS CCM. The cluster is being created with ASG and when trying similar action on ASG created nodes, the nodes join the cluster without issue and get the providerID set by the CCM. Here's the error from the CCM:

error syncing 'ip-<node-ip>.<region>.compute.internal': failed to get instance metadata for node ip-<node-ip>.<region>.compute.internal: instance not found, requeuing
E0602 19:56:35.311131       1 node_controller.go:244] "Unhandled Error" err="error syncing 'ip-<node-ip>.<region>.compute.internal': failed to get instance metadata for node ip-<node-ip>.<region>.compute.internal: instance not found, requeuing" logger="UnhandledError" 
E0602 19:56:36.976054       1 node_lifecycle_controller.go:156] error checking if node ip-<node-ip>.<region>.compute.internal exists: instance not found
I0602 19:56:39.224230       1 node_controller.go:271] Update 7 nodes status took 1.627947232s.         
E0602 19:56:42.128159       1 node_lifecycle_controller.go:156] error checking if node ip-<node-ip>.<region>.compute.internal exists: instance not found
E0602 19:56:47.302222       1 node_lifecycle_controller.go:156] error checking if node ip-<node-ip>.<region>.compute.internal exists: instance not found

When setting the providerID manually or through another application, The node gets picked up by the CCM and everything's fine.

NodeClass config:

spec:
  amiFamily: Custom
  amiSelectorTerms:
  - tags:
      <custom-name>-karpenter-version: current
  associatePublicIPAddress: false
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      deleteOnTermination: true
      encrypted: true
      kmsKeyID: <kms-key-id>
      volumeSize: 100Gi
      volumeType: gp3
  detailedMonitoring: true
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 3
    httpTokens: optional
  role: <iam-role>
  securityGroupSelectorTerms:
  - id: <sg>
  subnetSelectorTerms:
  - id: <subnet1>
  - id: <subnet2>
  - id: <subnet3>
  tags:
    <some-tag>
    <name>:rancher:managementApi: <api-url>
  userData: |
    #cloud-config
    yum_repos:
      artifactory:
        baseurl: <url>
        enabled: true
        gpgcheck: false
        name: "<name>"
    package_update: true
    package_upgrade: true
    write_files:
      - <file_config>
    runcmd:
      - |
        <Some script and initialization steps with CA and so on>
nodepool:
spec:
  disruption:
    budgets:
    - nodes: "1"
    consolidateAfter: 5m
    consolidationPolicy: WhenEmpty
  limits:
    cpu: 512
  template:
    metadata:
      labels:
        <some-tag>
    spec:
      expireAfter: Never
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: prometheus
      requirements:
      - key: kubernetes.io/arch
        operator: In
        values:
        - amd64
      - key: kubernetes.io/os
        operator: In
        values:
        - linux
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - on-demand
      startupTaints:
      - effect: NoExecute
        key: node.cilium.io/agent-not-ready
        value: "true"
      taints:
      - effect: NoExecute
        key: <prefix>/dedicated
        value: prometheus

The cluster is managed with rke2 and has cloud-provider=external set.

Is there a particular setting that needs to be done on either side that I'm missing?

(cross-posting from kubernetes-sigs/karpenter#2281)

Thanks!

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[cartermckinnon]

Do you mean CCM is never able to determine the providerID, or just that you see it fail early on?

When the providerID isn't set, CCM tries to find the Node's corresponding EC2 instance by doing a reverse-lookup on the private DNS name (the name of the Node object):

cloud-provider-aws/pkg/providers/v1/aws.go

Line 3274 in b47d2cf

func (c *Cloud) findInstanceByNodeName(ctx context.Context, nodeName types.NodeName) (*ec2types.Instance, error) {

If you use a custom domain name in your VPC, this may not work. It can also cause issues in AWS accounts with many EC2 instances, due to the nature of the filter.

What AMI are you using where the providerID isn't set already? It's significantly more efficient to set it in the kubelet config using a couple IMDS calls vs. relying on CCM to fill it in.

[Sindvero]

I mean it never able to set it at all. As said on asg nodes it works fine without any issue. The node name is also the one expected by aws and the ccm as seen in the logs. It had the priper format.
I thought the point of having the ccm working, apart from other features, was to let it set the providerID by https://kubernetes.io/docs/concepts/architecture/cloud-controller/ and not use kubelet to set it anymore like on the older k8s versions?

Thanks

[cartermckinnon]

I'd really need to look at your control plane logs to see what's going wrong. Can you open a case with AWS support?

I thought the point of having the ccm working, apart from other features, was to let it set the providerID

CCM setting the providerID is more of a convenience/backwards-compat than a core function. It's a best guess using heuristics that don't work in every situation, and it's a lot less efficient than passing it to kubelet from the beginning. We do this on Amazon Linux EKS AMIs and on Bottlerocket for that reason.

[Sindvero]

What specific logs would you want to look at on the control plane? I can manage to give them to you.

Thanks

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale