[VSphere] 64bitMMIOSizeGB should be configurable

[juulie]

What happened + What you expected to happen

Hi all,

ray/python/ray/autoscaler/_private/vsphere/gpu_utils.py

Line 272 in fc34890

vim.option.OptionValue(key="pciPassthru.64bitMMIOSizeGB", value="64"),

Sets the 64bitMMIOSizeGB paramater to 64, but this is depended on the GPU being passed through:
https://docs.nvidia.com/ai-enterprise/latest/release-notes/index.html#tesla-p40-large-memory-vms
I think we should add a configuration for this in the vsphere gpu_config

Versions / Dependencies

Ray-2.38.0

Reproduction script

Launch cluster on vsphere with a GPU requiring more than 64gb MMIO space

Issue Severity

High: It blocks me from completing my task.

[jcotant1]

Hey @kevin85421 @jjyao - any update on this one? Thx

[rynewang]

@JingChen23 can you take a look at this?

[juulie]

I have a fix for this, just need to do a pull request (which ive never done before on github, so might take a bit to get right)