-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Open
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weekscommunity-backlogcompiled-graphscoreIssues that should be addressed in Ray CoreIssues that should be addressed in Ray CoreenhancementRequest for new feature and/or capabilityRequest for new feature and/or capability
Description
Description
As part of the effort (meta-issue: #47983) to support collective communication ops, we need to support one-to-all (broadcast) patterns. As discussed in the RFC, we will pass the sender worker handle to the collective call as follows:
workers = [Worker.options(num_gpus=1).remote() for _ in range(3)]
nccl_group_handle = ray.collective.NcclGroup(workers)
# One-to-all pattern.
with InputNode() as inp:
result = workers[0].fwd.bind(inp)
results = ray.collective.broadcast.bind(
result, workers,
transport=nccl_group_handle)
dag = MultiOutputNode(results)
# Errors if `broadcast` sender is not part of the group.
dag = dag.experimental_compile()
Use case
No response
Metadata
Metadata
Assignees
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weekscommunity-backlogcompiled-graphscoreIssues that should be addressed in Ray CoreIssues that should be addressed in Ray CoreenhancementRequest for new feature and/or capabilityRequest for new feature and/or capability