opencl: add set_rows for f16 and f32

[lhez]

Following a70c8a0, this PR adds set_rows for f16 and f32.

Make sure to read the contributing guidelines before submitting a PR

[ggerganov]

Note that implementing it like this won't be very efficient. This dedicates 256 threads for each row of data. So for small rows with less than 256 elements, there will be wasted resources. For example, when FA is disabled, ggml_set_rows() is used with rows of 1 element (due to the V cache being transposed), so 255 out of the 256 local threads will be idle.

That's why in the Metal implementation I did "threadgroup batching" so that the local threads can work on multiple rows. Might want to consider implementing it here too for improved performance.

[lhez]

Thanks a lot for the suggestion - it makes a good point. Looking into this.

[CISC]

@ggerganov Any chance getting an OpenCL runner on ggml-ci?

[ggerganov]

The easiest way is if there are suitable machines in Azure cloud because we have a grant for these. The other option is someone to donate a dedicated machine (like how the SYCL node is donated by Menlo AI).

[max-krasnyansky]

@ggerganov Any chance getting an OpenCL runner on ggml-ci?

Yeah, I was thinking of adding some X-Elite based runners but didn't get a chance to look into it.
Standard runners do not have GPU -- https://docs.github.com/en/actions/reference/github-hosted-runners-reference#supported-runners-and-hardware-resources
We could probably enable some CPU based OpenCL testing but it won't really test all the features of the backend.

[max-krasnyansky]

The easiest way is if there are suitable machines in Azure cloud because we have a grant for these. The other option is someone to donate a dedicated machine (like how the SYCL node is donated by Menlo AI).

The N-series might work : https://learn.microsoft.com/en-us/azure-stack/user/gpu-vms-about?view=azs-2501
but would still be quite limited due to lack of OpenCL FP16 support.

[CISC]

The easiest way is if there are suitable machines in Azure cloud because we have a grant for these. The other option is someone to donate a dedicated machine (like how the SYCL node is donated by Menlo AI).

The N-series might work : https://learn.microsoft.com/en-us/azure-stack/user/gpu-vms-about?view=azs-2501 but would still be quite limited due to lack of OpenCL FP16 support.

Even the MI25?

[max-krasnyansky]

The easiest way is if there are suitable machines in Azure cloud because we have a grant for these. The other option is someone to donate a dedicated machine (like how the SYCL node is donated by Menlo AI).

The N-series might work : https://learn.microsoft.com/en-us/azure-stack/user/gpu-vms-about?view=azs-2501 but would still be quite limited due to lack of OpenCL FP16 support.

Even the MI25?

Oh, I didn't notice MI25 in there, only saw the NV flavors somehow :).
In the past only the Intel and Qualcomm GPU drivers had decent OpenCL feature-set (for GGML / LLM needs that is).
Not a bad idea to check on the latest stuff.

@ggerganov if you could get one of those VMs hooked up as runner with some new tag we could see which tests we could run on it. It will be useful for the HIP and maybe Vulkan backends as well.

[lhez]

The easiest way is if there are suitable machines in Azure cloud because we have a grant for these. The other option is someone to donate a dedicated machine (like how the SYCL node is donated by Menlo AI).

The N-series might work : https://learn.microsoft.com/en-us/azure-stack/user/gpu-vms-about?view=azs-2501 but would still be quite limited due to lack of OpenCL FP16 support.

Even the MI25?

Oh, I didn't notice MI25 in there, only saw the NV flavors somehow :). In the past only the Intel and Qualcomm GPU drivers had decent OpenCL feature-set (for GGML / LLM needs that is). Not a bad idea to check on the latest stuff.

@ggerganov if you could get one of those VMs hooked up as runner with some new tag we could see which tests we could run on it. It will be useful for the HIP and maybe Vulkan backends as well.

Currently it won't run on AMD. I did try to enable on AMD but have never really finished. I will take a look at it again. Intel should just work if integrated Intel GPUs can be used. Nvidia's OpenCL implementation never gets OpenCL 2.0 features like subgroups (unfortunately subgroups is not mandatory for OpenCL 3.0 so you can claim OpenCL 3.0 without subgroups).

[ggerganov]

Ok, if you confirm support with any of the Azure hosts let me know and I'll add a node.

[max-krasnyansky]

I'm going to merge it now. We can iterate further if needed.