Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The [Amazon Elastic Block Store](https://aws.amazon.com/ebs/) Container Storage
* **Volume Snapshots** - Create and restore [snapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) taken from a volume in Kubernetes.
* **Volume Resizing** - Expand the volume by specifying a new size in the [PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims) (PVC).
* **Volume Modification** - Change the properties (type, iops, or throughput) [via a `VolumeAttributesClass`](examples/kubernetes/modify-volume).
* **Node-Local Volumes** - Mount pre-attached, node-specific EBS volumes using a single cluster-wide PV/PVC for node-local caching scenarios.

## Container Images

Expand Down Expand Up @@ -51,6 +52,7 @@ The EBS CSI Driver implements the [Container Storage Interface specification](ht
* [Driver Installation](docs/install.md)
* [Driver Launch Options](docs/options.md)
* [StorageClass Parameters](docs/parameters.md)
* [Node-Local Volumes](docs/node-local-volumes.md)
* [Frequently Asked Questions](docs/faq.md)
* [Volume Tagging](docs/tagging.md)
* [Volume Modification](docs/modify-volume.md)
Expand Down
3 changes: 3 additions & 0 deletions charts/aws-ebs-csi-driver/templates/controller.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,9 @@ spec:
{{- if .Values.controller.batching }}
- --batching=true
{{- end}}
{{- if .Values.controller.enableNodeLocalVolumes }}
- --enable-node-local-volumes=true
{{- end}}
{{- with .Values.controller.loggingFormat }}
- --logging-format={{ . }}
{{- end }}
Expand Down
5 changes: 5 additions & 0 deletions charts/aws-ebs-csi-driver/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,11 @@
"default": false
}
}
},
"enableNodeLocalVolumes": {
"type": "boolean",
"description": "Enable support for node-local volumes that use pre-attached EBS volumes",
"default": false
}
}
},
Expand Down
2 changes: 2 additions & 0 deletions charts/aws-ebs-csi-driver/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,8 @@ controller:
batching: true
volumeModificationFeature:
enabled: false
# Enable support for node-local volumes that use pre-attached EBS volumes
enableNodeLocalVolumes: false
# Additional parameters provided by aws-ebs-csi-driver controller.
additionalArgs: []
sdkDebugLog: false
Expand Down
129 changes: 129 additions & 0 deletions docs/node-local-volumes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Node-Local Volumes

## Overview

Node-local volumes enable a single cluster-wide PersistentVolume (PV) and PersistentVolumeClaim (PVC) to mount pre-attached, node-specific EBS volumes. When pods reference this PVC, each node independently mounts its own local EBS device, and all pods on that node share the mount.

This feature is useful for scenarios where:
- Multiple co-located pods need access to a shared dataset (e.g., cached files from S3).
- You want to avoid using `hostPath` volumes for security reasons.
- You need to scale workloads across nodes while maintaining node-local caching.

## Prerequisites

- EBS volumes must be pre-attached to each node at a consistent device name (e.g., `/dev/xvdbz`).
- The driver must be deployed with `controller.enableNodeLocalVolumes=true`.

## Enabling the Feature

### Helm Installation

```bash
helm upgrade --install aws-ebs-csi-driver \
--namespace kube-system \
./charts/aws-ebs-csi-driver \
--set controller.enableNodeLocalVolumes=true
```

## Usage

### 1. Pre-attach EBS Volumes to Nodes

Each node must have an EBS volume attached at the same device name. For example:

**EC2 Launch Template:**
```json
{
"BlockDeviceMappings": [{
"DeviceName": "/dev/xvdbz",
"Ebs": {
"VolumeSize": 100,
"VolumeType": "gp3",
"DeleteOnTermination": true
}
}]
}
```

### 2. Create cluster-wide PV and PVC

Create a PersistentVolume with `volumeHandle` using the `local-ebs://` prefix:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: node-local-cache-pv
spec:
capacity:
storage: 100Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: ebs.csi.aws.com
volumeHandle: local-ebs://dev/xvdbz
volumeAttributes:
ebs.csi.aws.com/fsType: "xfs"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: node-local-cache-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
volumeName: node-local-cache-pv
```

**Important:** The `volumeHandle` format is `local-ebs://<device-name>` where `<device-name>` is the device path without the leading slash (e.g., `dev/xvdbz` for `/dev/xvdbz`).

### 3. Use in Workloads

Reference the PVC in your pod or deployment:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 10
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: my-app:latest
volumeMounts:
- name: cache
mountPath: /cache
volumes:
- name: cache
persistentVolumeClaim:
claimName: node-local-cache-pvc
```

## Access Mode Requirements

Node-local volumes may use `ReadWriteMany` (RWX) access mode. This tells the Kubernetes scheduler it's safe to place pods on multiple nodes. Each node uses its own physical EBS volume, so there's no actual cross-node sharing.

## Limitations

- Volumes must be statically provisioned and pre-attached at the specified device path.
- Cross-node data sharing is not supported (each node has its own volume).
- Volume snapshots and modifications for local cache volumes are not supported.
- The root device cannot be used as a node-local volume.

## Examples

See [examples/kubernetes/node-local-volumes](../examples/kubernetes/node-local-volumes) for complete working examples.
1 change: 1 addition & 0 deletions docs/options.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ There are a couple of driver options that can be passed as arguments when starti
| reserved-volume-attachments | 2 | -1 | Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes. |
| legacy-xfs | true | false | Warning: This option will be removed in a future release. It is a temporary workaround for users unable to immediately migrate off of older kernel versions. Formats XFS volumes with `bigtime=0,inobtcount=0,reflink=0`, so that they can be mounted onto nodes with linux kernel ≤ v5.4. Volumes formatted with this option may experience issues after 2038, and will be unable to use some XFS features (for example, reflinks). |
| metadata-sources | imds | imds,kubernetes,metadalabeler | Dictates which sources are used to retrieve instance metadata. The driver will attempt to rely on each source in order until one succeeds. Valid options include 'imds', 'kubernetes', and (ALPHA)'metadata-labeler'. |
| enable-node-local-volumes | true | false | If set to true, enables support for node-local volumes that use pre-attached EBS volumes. See [node-local-volumes.md](node-local-volumes.md) for details. |
47 changes: 47 additions & 0 deletions examples/kubernetes/node-local-volumes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Node-Local Volumes Example

This example demonstrates how to use node-local volumes with the EBS CSI Driver.

## Prerequisites

1. Enable the feature in the driver:
```bash
helm upgrade --install aws-ebs-csi-driver \
--namespace kube-system \
./charts/aws-ebs-csi-driver \
--set controller.enableNodeLocalVolumes=true
```

2. Pre-attach EBS volumes to each node using an EC2 Launch Template:
```json
{
"BlockDeviceMappings": [{
"DeviceName": "/dev/xvdbz",
"Ebs": {
"VolumeSize": 100,
"VolumeType": "gp3",
"DeleteOnTermination": true
}
}]
}
```
Apply this launch template to your node group or managed node group.

## Deploy

```bash
kubectl apply -f manifests/
```

## Verify

```bash
# Check pods are running
kubectl get pods -l app=cache-app -o wide
```

## Cleanup

```bash
kubectl delete -f manifests/
```
41 changes: 41 additions & 0 deletions examples/kubernetes/node-local-volumes/manifests/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Copyright 2025 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cache-app
spec:
replicas: 4
selector:
matchLabels:
app: cache-app
template:
metadata:
labels:
app: cache-app
spec:
containers:
- name: app
image: public.ecr.aws/amazonlinux/amazonlinux
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /cache/out.txt; sleep 5; done"]
volumeMounts:
- name: cache
mountPath: /cache
volumes:
- name: cache
persistentVolumeClaim:
claimName: node-local-cache-pvc
43 changes: 43 additions & 0 deletions examples/kubernetes/node-local-volumes/manifests/pv-pvc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Copyright 2025 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: node-local-cache-pv
spec:
capacity:
storage: 100Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: ebs.csi.aws.com
volumeHandle: local-ebs://dev/xvdbz
volumeAttributes:
ebs.csi.aws.com/fsType: "xfs"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: node-local-cache-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
volumeName: node-local-cache-pv
21 changes: 21 additions & 0 deletions pkg/cloud/cloud.go
Original file line number Diff line number Diff line change
Expand Up @@ -1637,6 +1637,27 @@ func (c *cloud) GetDiskByID(ctx context.Context, volumeID string) (*Disk, error)
return disk, nil
}

func (c *cloud) GetVolumeIDByNodeAndDevice(ctx context.Context, nodeID string, deviceName string) (string, error) {
instance, err := c.getInstance(ctx, nodeID)
if err != nil {
return "", fmt.Errorf("failed to get instance %s: %w", nodeID, err)
}

if instance.RootDeviceName != nil && *instance.RootDeviceName == deviceName {
return "", fmt.Errorf("device %s is the root device: %w", deviceName, ErrInvalidRequest)
}

for _, bdm := range instance.BlockDeviceMappings {
if bdm.DeviceName != nil && *bdm.DeviceName == deviceName {
if bdm.Ebs != nil && bdm.Ebs.VolumeId != nil {
return *bdm.Ebs.VolumeId, nil
}
}
}

return "", fmt.Errorf("volume not found at device %s on node %s: %w", deviceName, nodeID, ErrNotFound)
}

func isHyperPodNode(nodeID string) bool {
return strings.HasPrefix(nodeID, "hyperpod-")
}
Expand Down
Loading