[RFC-0010] Introduce workload identity auth for remote clusters

Signed-off-by: Matheus Pimenta <[email protected]>

Author: matheuscscp

api/go.mod

@@ -4,7 +4,7 @@ go 1.24.0
 
 require (
 	github.com/fluxcd/pkg/apis/kustomize v1.10.0
-	github.com/fluxcd/pkg/apis/meta v1.12.0
+	github.com/fluxcd/pkg/apis/meta v1.13.1-0.20250703201432-fe8e8dd5ce64
 	k8s.io/apiextensions-apiserver v0.33.0
 	k8s.io/apimachinery v0.33.0
 	sigs.k8s.io/controller-runtime v0.21.0

api/go.sum

@@ -4,8 +4,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/fluxcd/pkg/apis/kustomize v1.10.0 h1:47EeSzkQvlQZdH92vHMe2lK2iR8aOSEJq95avw5idts=
 github.com/fluxcd/pkg/apis/kustomize v1.10.0/go.mod h1:UsqMV4sqNa1Yg0pmTsdkHRJr7bafBOENIJoAN+3ezaQ=
-github.com/fluxcd/pkg/apis/meta v1.12.0 h1:XW15TKZieC2b7MN8VS85stqZJOx+/b8jATQ/xTUhVYg=
-github.com/fluxcd/pkg/apis/meta v1.12.0/go.mod h1:+son1Va60x2eiDcTwd7lcctbI6C+K3gM7R+ULmEq1SI=
+github.com/fluxcd/pkg/apis/meta v1.13.1-0.20250703201432-fe8e8dd5ce64 h1:DktET84J5DO8BugAm898A0dK+S4wC/1Rpr1emnCiULU=
+github.com/fluxcd/pkg/apis/meta v1.13.1-0.20250703201432-fe8e8dd5ce64/go.mod h1:+son1Va60x2eiDcTwd7lcctbI6C+K3gM7R+ULmEq1SI=
 github.com/fxamacker/cbor/v2 v2.8.0 h1:fFtUGXUzXPHTIUdne5+zzMPTfffl3RD5qYnkY40vtxU=
 github.com/fxamacker/cbor/v2 v2.8.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
 github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=

api/v1/zz_generated.deepcopy.go

@@ -256,16 +256,47 @@ spec:
                   a controller level fallback for when KustomizationSpec.ServiceAccountName
                   is empty.
                 properties:
+                  address:
+                    description: |-
+                      Address is the optional address of the Kubernetes API server.
+                      Not supported for the generic provider, optional for the
+                      other providers. The address is used to select among a list
+                      of endpoints in the cluster resource. If not set, the first
+                      endpoint on the list is used. If none of the addresses in the
+                      cluster resource match a provided address, the controller will
+                      error out and the reconciliation will fail. Must be a valid
+                      HTTPS endpoint, e.g. "https://api.example.com:6443".
+                    pattern: ^https://.*
+                    type: string
+                  cluster:
+                    description: |-
+                      Cluster is the optional fully qualified resource name of the
+                      Kubernetes cluster in the cloud provider to connect to.
+                      Not supported for the generic provider, required for the
+                      other providers.
+                    type: string
+                  provider:
+                    default: generic
+                    description: |-
+                      Provider is the optional name of the cloud provider that should be used
+                      to authenticate to the Kubernetes API server. Can be one of "aws",
+                      "azure", "gcp", or "generic". Defaults to "generic".
+                    enum:
+                    - aws
+                    - azure
+                    - gcp
+                    - generic
+                    type: string
                   secretRef:
                     description: |-
-                      SecretRef holds the name of a secret that contains a key with
+                      SecretRef holds an optional name of a secret that contains a key with
                       the kubeconfig file as the value. If no key is set, the key will default
                       to 'value'.
                       It is recommended that the kubeconfig is self-contained, and the secret
                       is regularly updated if credentials such as a cloud-access-token expire.
                       Cloud specific `cmd-path` auth helpers will not function without adding
                       binaries and credentials to the Pod that is responsible for reconciling
-                      Kubernetes resources.
+                      Kubernetes resources. Supported only for the generic provider.
                     properties:
                       key:
                         description: Key in the Secret, when not specified an implementation-specific
@@ -277,9 +308,29 @@ spec:
                     required:
                     - name
                     type: object
-                required:
-                - secretRef
+                  serviceAccountName:
+                    description: |-
+                      ServiceAccountName is the optional name of the Kubernetes
+                      ServiceAccount in the same namespace that should be used
+                      to authenticate to the Kubernetes API server. If not set,
+                      the controller ServiceAccount will be used. Not supported
+                      for the generic provider.
+                    type: string
                 type: object
+                x-kubernetes-validations:
+                - message: .secretRef is not supported for the specified .provider
+                  rule: '!has(self.secretRef) || !has(self.provider) || self.provider
+                    == ''generic'''
+                - message: .serviceAccountName is not supported when .provider is
+                    empty or 'generic'
+                  rule: '!has(self.serviceAccountName) || (has(self.provider) && self.provider
+                    != ''generic'')'
+                - message: .cluster is not supported when .provider is empty or 'generic'
+                  rule: '!has(self.cluster) || (has(self.provider) && self.provider
+                    != ''generic'')'
+                - message: .address is not supported when .provider is empty or 'generic'
+                  rule: '!has(self.address) || (has(self.provider) && self.provider
+                    != ''generic'')'
               namePrefix:
                 description: NamePrefix will prefix the names of all managed resources.
                 maxLength: 200
@@ -1347,16 +1398,47 @@ spec:
                   a controller level fallback for when KustomizationSpec.ServiceAccountName
                   is empty.
                 properties:
+                  address:
+                    description: |-
+                      Address is the optional address of the Kubernetes API server.
+                      Not supported for the generic provider, optional for the
+                      other providers. The address is used to select among a list
+                      of endpoints in the cluster resource. If not set, the first
+                      endpoint on the list is used. If none of the addresses in the
+                      cluster resource match a provided address, the controller will
+                      error out and the reconciliation will fail. Must be a valid
+                      HTTPS endpoint, e.g. "https://api.example.com:6443".
+                    pattern: ^https://.*
+                    type: string
+                  cluster:
+                    description: |-
+                      Cluster is the optional fully qualified resource name of the
+                      Kubernetes cluster in the cloud provider to connect to.
+                      Not supported for the generic provider, required for the
+                      other providers.
+                    type: string
+                  provider:
+                    default: generic
+                    description: |-
+                      Provider is the optional name of the cloud provider that should be used
+                      to authenticate to the Kubernetes API server. Can be one of "aws",
+                      "azure", "gcp", or "generic". Defaults to "generic".
+                    enum:
+                    - aws
+                    - azure
+                    - gcp
+                    - generic
+                    type: string
                   secretRef:
                     description: |-
-                      SecretRef holds the name of a secret that contains a key with
+                      SecretRef holds an optional name of a secret that contains a key with
                       the kubeconfig file as the value. If no key is set, the key will default
                       to 'value'.
                       It is recommended that the kubeconfig is self-contained, and the secret
                       is regularly updated if credentials such as a cloud-access-token expire.
                       Cloud specific `cmd-path` auth helpers will not function without adding
                       binaries and credentials to the Pod that is responsible for reconciling
-                      Kubernetes resources.
+                      Kubernetes resources. Supported only for the generic provider.
                     properties:
                       key:
                         description: Key in the Secret, when not specified an implementation-specific
@@ -1368,9 +1450,29 @@ spec:
                     required:
                     - name
                     type: object
-                required:
-                - secretRef
+                  serviceAccountName:
+                    description: |-
+                      ServiceAccountName is the optional name of the Kubernetes
+                      ServiceAccount in the same namespace that should be used
+                      to authenticate to the Kubernetes API server. If not set,
+                      the controller ServiceAccount will be used. Not supported
+                      for the generic provider.
+                    type: string
                 type: object
+                x-kubernetes-validations:
+                - message: .secretRef is not supported for the specified .provider
+                  rule: '!has(self.secretRef) || !has(self.provider) || self.provider
+                    == ''generic'''
+                - message: .serviceAccountName is not supported when .provider is
+                    empty or 'generic'
+                  rule: '!has(self.serviceAccountName) || (has(self.provider) && self.provider
+                    != ''generic'')'
+                - message: .cluster is not supported when .provider is empty or 'generic'
+                  rule: '!has(self.cluster) || (has(self.provider) && self.provider
+                    != ''generic'')'
+                - message: .address is not supported when .provider is empty or 'generic'
+                  rule: '!has(self.address) || (has(self.provider) && self.provider
+                    != ''generic'')'
               patches:
                 description: |-
                   Strategic merge and JSON patches, defined as inline YAML objects,

api/v1beta2/zz_generated.deepcopy.go

@@ -791,14 +791,42 @@ with:
 kustomize.toolkit.fluxcd.io/force: enabled
 ```
 
-### KubeConfig reference
+### KubeConfig (Remote clusters)
+
+With the `.spec.kubeConfig` field a Kustomization
+can apply and manage resources on a remote cluster.
+
+The field `.spec.kubeConfig.provider` determines the type of remote cluster
+and the authentication method used to connect to it. The following providers
+are supported:
+
+- `generic`: The default. Uses a KubeConfig stored in the Kubernetes Secret
+  referenced by `.spec.kubeConfig.secretRef`.
+- `aws`: Secret-less authentication for remote EKS clusters with IAM Roles.
+- `azure`: Secret-less authentication for remote AKS clusters with Managed
+  Identities.
+- `gcp`: Secret-less authentication for remote GKE clusters with GCP Service
+  Accounts or Workload Identity Federation.
+
+When both `.spec.kubeConfig` and `.spec.ServiceAccountName` are specified,
+the controller will impersonate the ServiceAccount on the target cluster,
+i.e. a ServiceAccount with name `.spec.serviceAccountName` must exist in
+the target cluster inside a namespace with the same name as the namespace
+of the Kustomization. For example, if the Kustomization is in the namespace
+`apps` of the cluster where Flux is running from, then the ServiceAccount
+must be in the `apps` namespace of the target remote cluster, and have the
+name `.spec.serviceAccountName`. In other words, the namespace of the
+Kustomization must exist both in the cluster where Flux is running from
+and in the target remote cluster where Flux will apply resources.
+
+#### Secret-based authentication
 
 `.spec.kubeConfig.secretRef.Name` is an optional field to specify the name of
 the secret containing a KubeConfig. If specified, objects will be applied,
 health-checked, pruned, and deleted for the default cluster specified in that
 KubeConfig instead of using the in-cluster ServiceAccount.
 
-The secret defined in the `kubeConfig.SecretRef` must exist in the same
+The secret defined in the `.spec.kubeConfig.secretRef` must exist in the same
 namespace as the Kustomization. On every reconciliation, the KubeConfig bytes
 will be loaded from the `.secretRef.key` key (default: `value` or `value.yaml`)
 of the Secret’s data , and the Secret can thus be regularly updated if
@@ -824,11 +852,61 @@ This matches the constraints of KubeConfigs from current Cluster API providers.
 KubeConfigs with `cmd-path` in them likely won't work without a custom,
 per-provider installation of kustomize-controller.
 
-When both `.spec.kubeConfig` and `.spec.ServiceAccountName` are specified,
-the controller will impersonate the service account on the target cluster.
-
 For more information, see [remote clusters/Cluster-API](#remote-clusterscluster-api).
 
+#### Secret-less authentication
+
+For the `aws`, `azure` and `gcp` providers, the controller supports
+secret-less authentication to remote EKS, AKS and GKE clusters
+respectively.
+
+When `.spec.kubeConfig.provider` is set to `aws`, `azure` or `gcp`,
+the field `.spec.kubeConfig.cluster` becomes required and must be
+set to the fully qualified name of the cluster resource in the
+respective cloud provider:
+
+* `aws`: `arn:<partition>:eks:<region>:<account-id>:cluster/<cluster-name>`
+* `azure`: `/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ContainerService/managedClusters/<cluster-name>`
+* `gcp`: `projects/<project-id>/locations/<location>/clusters/<cluster-name>`
+
+Users can also optionally specify the address of the remote cluster
+API server with the `.spec.kubeConfig.address` field. This field is
+necessary in case the remote cluster has multiple addresses, e.g.
+public and private addresses. If not specified, the controller will
+select the first address available. If specified, the address has
+to match at least one of the addresses of the remote cluster,
+otherwise the controller will return an error.
+
+The optional `.spec.kubeConfig.serviceAccountName` field can be used to
+specify a ServiceAccount in the same namespace as the Kustomization for
+object-level workload identity. Leaving this field empty will cause the
+controller to use its identity for getting access to the remote cluster.
+
+For complete guides on workload identity and setting up permissions for
+this feature, see the following docs:
+
+* [EKS](/flux/integrations/aws/#for-amazon-elastic-kubernetes-service)
+* [AKS](/flux/integrations/azure/#for-azure-kubernetes-service)
+* [GKE](/flux/integrations/gcp/#for-google-kubernetes-engine)
+
+Example for an EKS cluster:
+
+```yaml
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: apps
+  namespace: flux-system
+spec:
+  ... # other fields omitted for brevity
+  kubeConfig:
+    provider: aws
+    cluster: arn:aws:eks:eu-central-1:123456789012:cluster/my-cluster
+    address: https://my-prod-cluster.us-east-1.eks.amazonaws.com # optional
+    serviceAccountName: apps-iam-role # optional. maps to a cloud identity. used for authentication
+  serviceAccountName: apps-sa # optional. must exist in the target cluster. user for impersonation
+```
+
 ### Decryption
 
 Storing Secrets in Git repositories in plain text or base64 is unsafe,
@@ -1355,7 +1433,7 @@ specified will use the service account name provided by
 
 ### Remote clusters/Cluster-API
 
-With the [`.spec.kubeConfig` field](#kubeconfig-reference) a Kustomization can be fully
+With the [`.spec.kubeConfig` field](#kubeconfig-remote-clusters) a Kustomization can be fully
 reconciled on a remote cluster. This composes well with Cluster API bootstrap
 providers such as CAPBK (kubeadm), CAPA (AWS) and others.