Ray Autoscaler for Azure not getting Spot VMs

[omnific9]

I was running a Ray Autoscaler sample script on Azure when I noticed the script was running only on the head node and no worker nodes are launching. The error message was as follows:

  File "/anaconda/lib/python3.8/site-packages/ray/autoscaler/_private/node_launcher.py", line 85, in run

    self._launch_node(config, count, node_type)

  File "/anaconda/lib/python3.8/site-packages/ray/autoscaler/_private/node_launcher.py", line 67, in _launch_node

    self.provider.create_node(node_config, node_tags, count)

  File "/anaconda/lib/python3.8/site-packages/ray/autoscaler/_private/_azure/node_provider.py", line 220, in create_node

    create(

  File "/anaconda/lib/python3.8/site-packages/azure/mgmt/resource/resources/v2021_04_01/operations/_deployments_operations.py", line 3008, in create_or_update

    raw_result = self._create_or_update_initial(

  File "/anaconda/lib/python3.8/site-packages/azure/mgmt/resource/resources/v2021_04_01/operations/_deployments_operations.py", line 2964, in _create_or_update_initial

    raise exp

msrestazure.azure_exceptions.CloudError: Azure Error: MultipleErrorsOccurred

Message: Multiple error occurred: BadRequest,BadRequest. Please see details.

Exception Details:

      Error Code: InvalidTemplateDeployment

      Message: The template deployment failed with error: 'The resource with id: '/subscriptions/42158598-412c-477a-a552-c34f8e8debde/resourceGroups/ray-cluster/providers/Microsoft.Compute/virtualMachines/ray-minimal-worker-c4da9c6a0' failed validation with message: 'The requested size for resource '/subscriptions/42158598-412c-477a-a552-c34f8e8debde/resourceGroups/ray-cluster/providers/Microsoft.Compute/virtualMachines/ray-minimal-worker-c4da9c6a0' is currently not available in location 'westus' zones '' for subscription '42158598-412c-477a-a552-c34f8e8debde'. Please try another size or deploy to a different location or zones. See https://aka.ms/azureskunotavailable for details.'.'.

      Error Code: InvalidTemplateDeployment

      Message: The template deployment failed with error: 'The resource with id: '/subscriptions/42158598-412c-477a-a552-c34f8e8debde/resourceGroups/ray-cluster/providers/Microsoft.Compute/virtualMachines/ray-minimal-worker-c4da9c6a1' failed validation with message: 'The requested size for resource '/subscriptions/42158598-412c-477a-a552-c34f8e8debde/resourceGroups/ray-cluster/providers/Microsoft.Compute/virtualMachines/ray-minimal-worker-c4da9c6a1' is currently not available in location 'westus' zones '' for subscription '42158598-412c-477a-a552-c34f8e8debde'. Please try another size or deploy to a different location or zones. See https://aka.ms/azureskunotavailable for details.'.'.

^C

Shared connection to 168.62.205.103 closed.

I later found out this was caused by the worker node section:

    ray.worker.default:
        # The minimum number of worker nodes of this type to launch.
        # This number should be >= 0.
        min_workers: 0
        # The maximum number of worker nodes of this type to launch.
        # This takes precedence over min_workers.
        max_workers: 2
        # The resources provided by this node type.
        resources: {"CPU": 2}
        # Provider-specific config, e.g. instance type.
        node_config:
            azure_arm_parameters:
                vmSize: Standard_D2s_v3
                # List images https://docs.microsoft.com/en-us/azure/virtual-machines/linux/cli-ps-findimage
                imagePublisher: microsoft-dsvm
                imageOffer: ubuntu-1804
                imageSku: 1804-gen2
                imageVersion: 21.07.12
                # optionally set priority to use Spot instances
                priority: Spot
                # set a maximum price for spot instances if desired
                # billingProfile:
                #     maxPrice: -1

If I remove the "priority: Spot" line, Ray would be able to get and launch worker nodes. So looks like if Ray is not getting Spot instances with this line enabled, it's not getting regular instances either.

CC'ing @DmitriGekhtman

[DmitriGekhtman]

Thanks, @omnific9 .
cc @gramhagen @eisber for your thoughts on this.

[eisber]

Can you check the deployment logs in Azure portal?
Maybe a quota issue?

[gramhagen]

the image versions are being deprecated quicker than anticipated

az vm image list --publisher microsoft-dsvm --offer ubuntu-1804 --sku 1804-gen2 --all --output table
Offer        Publisher       Sku        Urn                                            Version
-----------  --------------  ---------  ---------------------------------------------  ---------
ubuntu-1804  microsoft-dsvm  1804-gen2  microsoft-dsvm:ubuntu-1804:1804-gen2:21.05.14  21.05.14
ubuntu-1804  microsoft-dsvm  1804-gen2  microsoft-dsvm:ubuntu-1804:1804-gen2:21.06.01  21.06.01
ubuntu-1804  microsoft-dsvm  1804-gen2  microsoft-dsvm:ubuntu-1804:1804-gen2:21.07.12  21.07.12

try updating the version to 21.07.02

[omnific9]

@gramhagen thanks for pointing that out. I did use 21.07.12 in my test. I realized I copy-pasted from the example-full script above. I edited my original post.

@eisber my cluster has been deleted but I don't think this is a quota issue. The problem is this:

1. If I don't set "priority: Spot", I can get several regular VMs.
2. If I set "priority: Spot": I can't get any Spot VMs, and I can't get any regular VMs.

The expectation is that in the 2nd case above, if I can't get Spot VMs, I should be able to get regular VMs instead. Does that make sense?

[gramhagen]

got it, i think the "priority" term there is a bit misleading. right now if there is no spot quota available no VMs will be allocated. there is no fallback to using regular VMs. From my perspective this is desired, otherwise it would be worse to hide the issue, allocate regular VMs and then the user is surprised that their cost is higher than expected.

[gramhagen]

note that spot quota is different from normal vm quota, you'll need to look for "spot vCPUs"
https://docs.microsoft.com/en-us/azure/azure-portal/supportability/low-priority-quota

[omnific9]

@gramhagen just checked that I do have a quota of 20 vCPUs on Spot instances (using 0 of 20), and so the script should've allocated the 2 vCPUs that I requested.

[gramhagen]

Ok. Thanks for checking on that. I will try to reproduce the issue.

Could you try using the assure cli to create a spot instance vm just to make sure we're debugging the right issue?

Docs here should be helpful to replicate the same image/sku/priority settings.
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/tutorial-manage-vm
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/spot-cli

[dizhouwu]

Any update?

[gramhagen]

@dizhouwu are you seeing the same issue? I haven't been able to replicate this