ETCD-less Dynamo in K8s #37

mohammedabdulwahhab · 2025-09-04T19:51:15Z

No description provided.

Signed-off-by: mohammedabdulwahhab <[email protected]>

Clarified trade-offs and Kubernetes concepts in etcd-k8s documentation. Signed-off-by: mohammedabdulwahhab <[email protected]>

This update introduces a new approach for managing DynamoEndpoint resources, combining pod lifecycle management with controller oversight. It details the server, controller, and client-side interactions for improved service discovery and resource management. Signed-off-by: mohammedabdulwahhab <[email protected]>

Signed-off-by: mohammedabdulwahhab <[email protected]>

Clarified wording and improved structure in the EndpointSlice based discovery section. Signed-off-by: mohammedabdulwahhab <[email protected]>

deps/etcd-k8s.md

Signed-off-by: mohammedabdulwahhab <[email protected]>

deps/etcd-k8s.md

nnshah1 · 2025-09-09T13:37:50Z

deps/etcd-k8s.md

+    - Lists all instances that match the given namespace and component.
+- `get_metadata`(namespace: str, component: str, instance_id: str) -> Instance
+    - Returns the metadata for an instance.
+- `subscribe`(namespace: str, component: str) -> EventStream


Maybe to distinguish this from event mechanisms - we can call this watch

and instead of eventstream we have InstanceAdded, InstanceRemoved

Yep, makes sense to me.

This will be the hardest one to support. For example NATS doesn't have a watch mechanism, so we'd have to poll. Maybe we should split this out to the notification mechanism.

@nnshah1 Fixed to watch. I do think the return value of this will be an EventStream that can yield an InstanceAddedEvent and InstanceRemovedEvent

@grahamking Potentially, I'm thinking of this as an analog to the kv_get_and_watch_prefix which is implemented with ETCD right now.

nnshah1 · 2025-09-09T13:41:53Z

deps/etcd-k8s.md

+    - Creates an instance and persists associated immutable metadata. Does not mark the instance as ready.
+- `list_instances`(namespace: str, component: str) -> List[Instance]
+    - Lists all instances that match the given namespace and component.
+- `get_metadata`(namespace: str, component: str, instance_id: str) -> Instance


While not critical here - we can make this an object model where the Instance has a metadata property instead of get and set on the ServiceDiscovery interface.

I'm thinking this is all part of the runtime object model

Sure, I think this makes sense.

nnshah1 · 2025-09-09T13:49:00Z

deps/etcd-k8s.md

+    - Returns the metadata for an instance.
+- `subscribe`(namespace: str, component: str) -> EventStream
+    - Subscribes to events for the set of instances that match (namespace, component). Returns a stream of events giving signals for when instances that match the subscription are created, deleted, or readiness status changes.
+- `set_instance_status`(instance: Instance, status: str)


Not clear to me immediately why we need a separate step for setting status - could we have "add" and "remove" - so if an instance is ready it is added, if it is not ready it is 'removed'.

just a thought in comparison to what we have today - we have add and remove but readiness is implied.

Pending discussion

also in which scenario are we going to call this function ?

nvm, it's detailed below in the doc

nnshah1 · 2025-09-09T13:51:03Z

deps/etcd-k8s.md

+#### Frontend Discovers Decode Workers
+
+- Decode calls `create_instance("dynamo", "decode", metadata)` and `set_instance_status(decode_worker, "ready")` to register itself with the service discovery backend. Its metadata includes the model it serves.
+- Frontend calls `list_instances("dynamo", "decode")` and `get_metadata` to get all decode workers and the associated model information to bootstrap its cache.


this implies frontend knows at launch that it is in namespace 'dynamo' and is looking for 'decode' - just wanting to confirm here.

That's right

Does this extrapolate to other workers, like "planner"? Does Frontend know what workers it has and how many of each to expect?

If i understand your question correctly, any component can list and watch any tuple of (namespace, component)

nnshah1 · 2025-09-09T13:53:06Z

deps/etcd-k8s.md

+
+#### Frontend Needs to Know Some Model Specifics (like Tokenizer)
+
+- Decode writes a Model Card using the `write_kv` method. Frontend can simply read this entry.


Could be in the metadata - because if not this also creates a race condition for readiness - i.e. is it ready if the model card is not written?

I'm in favor of putting all required information for interacting with this instance into the metadata - this decouples the instances from each other as well. No shared state is required between instances.

Agreed, that this might be sensible.

nnshah1 · 2025-09-09T13:57:49Z

deps/etcd-k8s.md

+
+```python
+# Context: decode worker is registering the MDC before it marks itself as ready
+service_discovery.write_kv("mdc/Qwen3-32B", mdc)


still unclear to me why the mdc has to live outside the instance metadata - we can discuss

maybe because this mdc would be duplicated in the case of multiple instance of the same worker ?

Yes, the motivation was to not duplicate the MDC, but I believe we reached the consensus that it's okay for it to be duplicated as long as some conditions are checked.

deps/etcd-k8s.md

nnshah1 · 2025-09-09T14:23:58Z

deps/etcd-k8s.md

+
+## ServiceDiscovery Interface
+
+To de-couple Dynamo from ETCD, we define a minimal `ServiceDiscovery` interface that can be satisfied by different backends (etcd, kubernetes, etc). In Kubernetes environments, we will use the Kubernetes APIs to implement this interface.


In a single node, non K8s environment, can this discovery be backed by static configuration? Or local file system with file watchers?

I haven't put too much thought into it, but it should be the case. Could be useful for greg clark's use case and the NIM team.

can we add a section to cover this case - or add a TODO - need to have the local deployment case covered as well / sketeched.

Sure, I think it might be a useful exercise to do to see if this interface generalizes across different backends.

Co-authored-by: Neelay Shah <[email protected]> Signed-off-by: mohammedabdulwahhab <[email protected]>

mohammedabdulwahhab · 2025-09-09T15:27:39Z

deps/etcd-k8s.md

+    - Writes data at a given key. Data not associated with any instance.
+- `read_kv`(key: str)
+    - Reads data at a given key. Data not associated with any instance.
+


TODO: Figure out if there is a compelling reason to keep a separate entity for MDC

if each instance has MDC embedded, there are consistency and race conditions to watch for

mohammedabdulwahhab · 2025-09-09T16:03:28Z

deps/etcd-k8s.md

+
+## ServiceDiscovery Interface
+
+To de-couple Dynamo from ETCD, we define a minimal `ServiceDiscovery` interface that can be satisfied by different backends (etcd, kubernetes, etc). In Kubernetes environments, we will use the Kubernetes APIs to implement this interface.


I haven't put too much thought into it, but it should be the case. Could be useful for greg clark's use case and the NIM team.

mohammedabdulwahhab · 2025-09-09T16:04:22Z

deps/etcd-k8s.md

+    - Creates an instance and persists associated immutable metadata. Does not mark the instance as ready.
+- `list_instances`(namespace: str, component: str) -> List[Instance]
+    - Lists all instances that match the given namespace and component.
+- `get_metadata`(namespace: str, component: str, instance_id: str) -> Instance


Sure, I think this makes sense.

mohammedabdulwahhab · 2025-09-09T16:05:31Z

deps/etcd-k8s.md

+    - Lists all instances that match the given namespace and component.
+- `get_metadata`(namespace: str, component: str, instance_id: str) -> Instance
+    - Returns the metadata for an instance.
+- `subscribe`(namespace: str, component: str) -> EventStream


Yep, makes sense to me.

mohammedabdulwahhab · 2025-09-09T16:06:41Z

deps/etcd-k8s.md

+    - Returns the metadata for an instance.
+- `subscribe`(namespace: str, component: str) -> EventStream
+    - Subscribes to events for the set of instances that match (namespace, component). Returns a stream of events giving signals for when instances that match the subscription are created, deleted, or readiness status changes.
+- `set_instance_status`(instance: Instance, status: str)


Pending discussion

deps/etcd-k8s.md

mohammedabdulwahhab · 2025-09-09T16:07:36Z

deps/etcd-k8s.md

+#### Frontend Discovers Decode Workers
+
+- Decode calls `create_instance("dynamo", "decode", metadata)` and `set_instance_status(decode_worker, "ready")` to register itself with the service discovery backend. Its metadata includes the model it serves.
+- Frontend calls `list_instances("dynamo", "decode")` and `get_metadata` to get all decode workers and the associated model information to bootstrap its cache.


That's right

mohammedabdulwahhab · 2025-09-09T16:07:55Z

deps/etcd-k8s.md

+
+#### Frontend Needs to Know Some Model Specifics (like Tokenizer)
+
+- Decode writes a Model Card using the `write_kv` method. Frontend can simply read this entry.


Agreed, that this might be sensible.

grahamking · 2025-09-09T17:02:08Z

deps/etcd-k8s.md

+
+## ServiceDiscovery Interface
+
+To de-couple Dynamo from ETCD, we define a minimal `ServiceDiscovery` interface that can be satisfied by different backends (etcd, kubernetes, etc). In Kubernetes environments, we will use the Kubernetes APIs to implement this interface.


Who is implementing this interface and who is using it? I'm not clear on where this sits, and everything afterwards depends on that.

Is it provided by the frontend? Is the frontend instead calling into this interface?

By a new discovery component?

Is it provided by dynamo to the worker, so it's the worker calling it?

IIUC - this is provided by the runtime - @mohammedabdulwahhab is that correct?

@grahamking I've added a section about this here: https://github.com/ai-dynamo/enhancements/blob/df9ddfac245e189bb5e01a4d3355b25da166a828/deps/etcd-k8s.md#where-will-these-apis-be-used

grahamking · 2025-09-09T17:03:27Z

deps/etcd-k8s.md

+To de-couple Dynamo from ETCD, we define a minimal `ServiceDiscovery` interface that can be satisfied by different backends (etcd, kubernetes, etc). In Kubernetes environments, we will use the Kubernetes APIs to implement this interface.
+
+### Methods
+- `create_instance`(namespace: str, component: str, metadata: dict) -> Instance


Should this be add_instance? Is this how we tell the frontend that it has a new instance to route to? Or is it how the worker tells the discovery implementation to advertise itself?

I've changed this to be register_instance. It returns an instance identifier but does not mark it as ready until set_ready is called.

grahamking · 2025-09-09T17:07:24Z

deps/etcd-k8s.md

+- `write_kv`(key: str, value: str)
+    - Writes data at a given key. Data not associated with any instance.
+- `read_kv`(key: str)
+    - Reads data at a given key. Data not associated with any instance.


I'm not sure we want to provide generic key/value features. I think this should always be in the domain specific language of dynamo instances.

grahamking · 2025-09-09T17:09:55Z

deps/etcd-k8s.md

+    # Fetch the associated metadata for the instance to register the model in-memory model registry
+    metadata = service_discovery.get_metadata("dynamo", "decode", decode_worker.instance_id)
+    model = metadata["Model"]
+    # map the instance to the model in the in-memory model registry ...


The examples are all in Python, but the implementation will need to be in Rust. That's fine, Python is clearer to illustrate the mechanics.

Yes, this will be written in Rust and consumed in rust

grahamking · 2025-09-09T17:12:29Z

deps/etcd-k8s.md

@@ -0,0 +1,314 @@
+# Service and Model Discovery Interface


Noting what we have right now. This will likely be the integration point.

KeyValueStore interface: https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/storage/key_value_store.rs#L39 which manages keys in KeyValueBucket's.

KeyValueBucket interface https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/storage/key_value_store.rs#L199

hhzhang16 · 2025-09-09T18:15:01Z

deps/etcd-k8s.md

+#### Frontend Discovers Decode Workers
+
+- Decode calls `create_instance("dynamo", "decode", metadata)` and `set_instance_status(decode_worker, "ready")` to register itself with the service discovery backend. Its metadata includes the model it serves.
+- Frontend calls `list_instances("dynamo", "decode")` and `get_metadata` to get all decode workers and the associated model information to bootstrap its cache.


Does this extrapolate to other workers, like "planner"? Does Frontend know what workers it has and how many of each to expect?

hhzhang16 · 2025-09-09T18:17:13Z

deps/etcd-k8s.md

+#### Decode Worker Needs to Know how to Reach Prefill Workers
+
+- Decode worker does the exact same thing as the frontend, instead listing and subscribing to the "prefill" component.


Will prefill workers be doing both frontend + decode worker work? e.g. create_instance("dynamo", "prefill", metadata) and set_instance_status(prefill_worker, "ready") and also list_instances("dynamo", "decode") and get_metadata? Will its instance status be dependent on the decode worker(s)' statuses?

hhzhang16 · 2025-09-09T18:28:33Z

deps/etcd-k8s.md

+
+<!-- ### Simplifications and Assumptions -->


Could you add some diagrams/examples showing some more complex flows? Like Planner + Prometheus disagg?

How is metadata propagated + kept in sync for these components?

@hhzhang16 Added some more details about the way metadata is exposed and read.

hhzhang16 · 2025-09-09T18:32:34Z

deps/etcd-k8s.md

+```
+
+#### Kubernetes Reference Impl
+- The readiness probe of the pod will proxy the result of `get_instance_status`. When the instance will be ready for traffic, the readiness probe will return 200 and EndpointSlices will be updated to include the instance.


There could be a need for customizing readiness probes if instance readiness is more complex than a simple HTTP health check (e.g. waiting for a model to finish loading or for some other aspect to be set up)

hhzhang16 · 2025-09-09T18:33:23Z

deps/etcd-k8s.md

+
+### Discovery Flow
+
+1. **Instance Registration**: When `create_instance()` is called, the pod creates a ConfigMap with instance metadata


Are ConfigMaps immutable per instance or recreated? What happens if metadata changes/gets updated?

hhzhang16 · 2025-09-09T18:34:38Z

deps/etcd-k8s.md

+  name: dynamo-decode-service
+spec:
+  selector:
+    nvidia.com/dynamo-namespace: dynamo


How/when are you avoiding namespace collisions?

hhzhang16 · 2025-09-09T18:35:32Z

deps/etcd-k8s.md

+
+## Kubernetes EndpointSlice Discovery Mechanism
+
+This section provides a detailed explanation of how the ServiceDiscovery interface is implemented using Kubernetes EndpointSlices for service discovery.


Would love to see a diagram for this!! create_instance --> Pod/ConfigMap --> Service --> EndpointSlice --> client subscribe

Yes, a mermaid diagram incoming

Signed-off-by: mohammedabdulwahhab <[email protected]>

Removed outdated implementation details and restructured the breakdown of the implementation section. Signed-off-by: mohammedabdulwahhab <[email protected]>

mohammedabdulwahhab added 4 commits September 4, 2025 12:51

Add document on ETCD alternatives for Dynamo in K8s

eb13577

Signed-off-by: mohammedabdulwahhab <[email protected]>

Revise mermaid diagram for Kubernetes service flow

23fc0d5

Signed-off-by: mohammedabdulwahhab <[email protected]>

Revise trade-offs and notes in etcd-k8s.md

636197a

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update etcd-k8s.md

4d1349c

Signed-off-by: mohammedabdulwahhab <[email protected]>

mohammedabdulwahhab changed the title ~~Add document on ETCD alternatives for Dynamo in K8s~~ ETCD-less Dynamo in K8s Sep 4, 2025

mohammedabdulwahhab added 6 commits September 4, 2025 13:12

Refine etcd-k8s.md with updated notes and trade-offs

2f2d76f

Clarified trade-offs and Kubernetes concepts in etcd-k8s documentation. Signed-off-by: mohammedabdulwahhab <[email protected]>

Update etcd-k8s.md

73902af

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update etcd-k8s.md

372e600

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update title for etcd-k8s documentation

074687b

Signed-off-by: mohammedabdulwahhab <[email protected]>

Refine EndpointSlice discovery documentation

5c79080

Clarified wording and improved structure in the EndpointSlice based discovery section. Signed-off-by: mohammedabdulwahhab <[email protected]>

julienmancuso reviewed Sep 4, 2025

View reviewed changes

deps/etcd-k8s.md Outdated Show resolved Hide resolved

deps/etcd-k8s.md Outdated Show resolved Hide resolved

deps/etcd-k8s.md Outdated Show resolved Hide resolved

deps/etcd-k8s.md Outdated Show resolved Hide resolved

julienmancuso reviewed Sep 4, 2025

View reviewed changes

deps/etcd-k8s.md Show resolved Hide resolved

mohammedabdulwahhab added 3 commits September 5, 2025 14:35

Update etcd-k8s.md

f6ad4f3

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update etcd-k8s.md

4a93fc7

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update etcd-k8s.md

9e1004f

Signed-off-by: mohammedabdulwahhab <[email protected]>

nnshah1 reviewed Sep 9, 2025

View reviewed changes

deps/etcd-k8s.md Outdated Show resolved Hide resolved

nnshah1 reviewed Sep 9, 2025

View reviewed changes

deps/etcd-k8s.md Outdated Show resolved Hide resolved

nnshah1 reviewed Sep 9, 2025

View reviewed changes

Update deps/etcd-k8s.md

63f391a

Co-authored-by: Neelay Shah <[email protected]> Signed-off-by: mohammedabdulwahhab <[email protected]>

mohammedabdulwahhab commented Sep 9, 2025

View reviewed changes

grahamking reviewed Sep 9, 2025

View reviewed changes

hhzhang16 reviewed Sep 9, 2025

View reviewed changes

mohammedabdulwahhab added 5 commits September 10, 2025 08:02

Update etcd-k8s.md

df9ddfa

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update etcd-k8s.md

388dca1

Signed-off-by: mohammedabdulwahhab <[email protected]>

Rename document title to Service and Model Discovery Interface

5aef0b0

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update etcd-k8s.md

bab299a

Signed-off-by: mohammedabdulwahhab <[email protected]>

Update implementation breakdown in etcd-k8s.md

981844c

Removed outdated implementation details and restructured the breakdown of the implementation section. Signed-off-by: mohammedabdulwahhab <[email protected]>


		#### Frontend Needs to Know Some Model Specifics (like Tokenizer)

		- Decode writes a Model Card using the `write_kv` method. Frontend can simply read this entry.


		## ServiceDiscovery Interface

		To de-couple Dynamo from ETCD, we define a minimal `ServiceDiscovery` interface that can be satisfied by different backends (etcd, kubernetes, etc). In Kubernetes environments, we will use the Kubernetes APIs to implement this interface.

		#### Decode Worker Needs to Know how to Reach Prefill Workers

		- Decode worker does the exact same thing as the frontend, instead listing and subscribing to the "prefill" component.


		### Discovery Flow

		1. Instance Registration: When `create_instance()` is called, the pod creates a ConfigMap with instance metadata


		## Kubernetes EndpointSlice Discovery Mechanism

		This section provides a detailed explanation of how the ServiceDiscovery interface is implemented using Kubernetes EndpointSlices for service discovery.

ETCD-less Dynamo in K8s #37

Are you sure you want to change the base?

ETCD-less Dynamo in K8s #37

Conversation

mohammedabdulwahhab commented Sep 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohammedabdulwahhab Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohammedabdulwahhab Sep 10, 2025 •

edited

Loading

grahamking Sep 9, 2025 •

edited

Loading