Feat: Add Analyze Image Task Type

[lukasdotcom]

The name for this in my opinion is not great. Some ideas I had: Image Question (Current), or Picture Chat. This should probably also be added as a task type on server.
Relevant server pr: nextcloud/server#53763

[julien-nc]

Nice!
The name could be better indeed. How about "Analyze image"?

[kyteinsky]

Thanks!

[julien-nc]

Optional non-necessary change suggestion.

[kyteinsky]

🚀

[lukasdotcom]

I changed the task type to allow multiple images as an input like recommended by @kyteinsky to allow for comparing images

[kyteinsky]

nice!

from the image requirements https://platform.openai.com/docs/guides/images-vision?api-mode=responses&format=url#image-input-requirements
the max payload size is 50 MB and the max no. of files is 500.
What do you think, should we let the request fail or adjust the request to the limit and make it succeed? It might not be accurate when some images have been dropped, but past 50 images, can the model even work coherently at that point?

[lukasdotcom]

nice!

from the image requirements https://platform.openai.com/docs/guides/images-vision?api-mode=responses&format=url#image-input-requirements the max payload size is 50 MB and the max no. of files is 500. What do you think, should we let the request fail or adjust the request to the limit and make it succeed? It might not be accurate when some images have been dropped, but past 50 images, can the model even work coherently at that point?

I implemented most of the changes given and renamed the file to analyzeimages while updating all that on the server pr to.

About adding a limit. Might also be a good idea to limit the amount of images to prevent the worker from crashing due to running out of memory. Right now all the data is loaded into memory when base64 encoded. Not sure if we should do file size and count though. File size requires a lot more work.

[julien-nc]

A simple approach would be to limit the number of images to a relatively low number like 20 or 30 and also limit sum of image sizes to something lower than 50 MB (or 40 MB seems reasonable to keep a margin for the prompt).
The task would fail if the input is above the limits.
The provider could set an informative message when failing, mentioning if it's because of the number of images or the total payload size.
Wdyt?

[julien-nc]

I'm not in favor of sending a subset of the input images to make the request to the service succeed. There is no way for the user to know this has happened and why the result is no pertinent.

[lukasdotcom]

A simple approach would be to limit the number of images to a relatively low number like 20 or 30 and also limit sum of image sizes to something lower than 50 MB (or 40 MB seems reasonable to keep a margin for the prompt). The task would fail if the input is above the limits. The provider could set an informative message when failing, mentioning if it's because of the number of images or the total payload size. Wdyt?

I think failing is probably the better option definitely after seeing that some models can actually attempt to understand 100s of images.

I did just try out a few questions on about 400 pictures and google's models could answer the questions.
Eg: "There is a picture of a farmers market. What is the name of the market?" or "There is a picture of police officers on horses. How many horses are there?"
So surprisingly many files could be understood in this simple test (might have gotten lucky though).

I'll probably just implement the official openai limits and fail if more than that is given.

[kyteinsky]

looks good!