Map to shared memory

[kodonnell]

Is your feature request related to a problem? Please describe.
I currently have a use case where I'd like multiple (python) processes to access the stream - e.g. one running machine learning, and one providing a live stream. I know I can do this as-is in a single process (main + lores, different encoders etc.) but experience tells me that this means you often end up mixing code (e.g. your machine learning code now has a webserver in it), and mixing patterns (e.g. async and non-async for machine learning / web server has been fun in the past). You also end up managing separate processes with exclusive access to the camera and ensuring they're not running concurrently. And single mega-processes are trickier to dev/debug ... not only are they more complex with lots of moving parts, but things can contend in unusual ways (and you have to be careful not to block the main picamera loop or you'll miss frames) ... but you've got to run on a pi. (I.e. if we're using /dev/shm I could just generate fake frames and all my client code wouldn't know the difference ... which I've done before with mock video on disk, but it's still a bit coupled.)

Describe the solution you'd like
What I think might be 'best' is a single camera service on the host, and all it does is spit the main/lores data to shared memory (and optionally record to disk if you want that). Then any process (in any language) can do what they want with the frames. I've implemented this as a basic memory mapped pseudo ring buffer using posix_ipc - and I can write 1080x1920x3 arrays on a pi zero 2w in about 6ms. So that'll work. I also have a need for buffering, so I keep the last N frames in the buffer ... and this solution gives me that for free. (Put another way, I'm not worried about excess memory as I'm already doing that.) However, I suspect I'm doing unnecessary memory copies, as picamera is also copying memory around (and also using mmaps etc.) - and I suspect picamera and I are both buffering the same thing. (Unlikely to be a major performance/memory hit.)

So, the request: is there currently an easy way to replace the allocators inside of picamera2 to use my own on /dev/shm, in a way I could access nicely from other processes? I can see there are definitely allocators etc. - I'm just not sure if they're as generic as I want, or which one to start with. I current serialize/deserialize the various independent things into a single frame (i.e. main + lores + metadata), but individual buffers would be fine instead as long as they're relatable in some way (e.g. by frame index or capture timestamp).

I am not suggesting building a generic RPC solution in picamera2 etc. - that can happen elsewhere, and is more complicated (what if two processes want cameras that are configured differently? etc.). Just whether I can use my own allocator in /dev/shm that would support that.

Describe alternatives you've considered
I don't think this is too relevant beyond what I've covered. All other IPC solutions will still involve copies, so it comes back to the allocator. Probably the main alternative would be going this in libcamera as the source for getting frames into /dev/shm, and python is just a consumer. But that won't be easy (for me at least). The other alternative is sticking with what I've got ... it should be fine.

Additional context
Not really. The main use will be streaming machine learning sized frames (i.e. low res) around, some going to machine learning (potentially multiple models), some to configuration, and maybe some being recorded. Performance, power consumption, and nice code are high priority, memory utilisation not so much (beyond limits of the Zero 2 W).

[davidplowman]

Hi, thanks for your post.

The question of exporting buffers to other processes is not one I've looked out, and (disclaimer!) I don't know too much about it.

The image buffers that come out of the camera system are dmabufs, so (according to my understanding of all this) it should be possible to share them between processes. I assume there are standard Linux runes for doing this that Python exposes by some means. If anyone were to produce a simple example of this, I'd be delighted to include it among our examples and in our documentation.

Obviously there are risks in all this - so long as another process is using the buffer then you can't recycle it back to the camera, so things my stall or get corrupted. Possibly you can mitigate this by allocating more and more buffers, or perhaps there comes a point where you have to take the hit and copy them.

Picamera2 does have an allocator class (as you've discovered), and in fact our next release (currently on the "next" branch) will have another flavour of allocator too. Whether this is enough to help you already I couldn't say, but we're certainly open to extending what we have if that's going to be useful - but we'd have to know what extensions to make.

Another point of interest is that parts of the Linux world regard cameras as a system resource which needs to be handled by some kind of server, which multiple clients can then talk to. The direction this seems to be heading in is Pipewire, and indeed the expectation is that libcamera will be integrated into other application (web browsers, for example) through Pipewire. One day this may even enable things like Google Meet to work with libcamera-based cameras. So there may be some overlap here too, though I don't know how useful that is in your case.

[kodonnell]

Thanks @davidplowman . I don't know a heap about it either - it's driven by necessity = )

Obviously there are risks in all this - so long as another process is using the buffer then you can't recycle it back to the camera, so things my stall or get corrupted. Possibly you can mitigate this by allocating more and more buffers, or perhaps there comes a point where you have to take the hit and copy them.

I try to have my processes read and then release the shared memory which, for 1920x1080 RGB is about 6ms on the Pi Zero 2, so plenty of wiggle room. But yes, since I want a buffer anyway, the client just reads as fast as it can, or takes the oldest frame if it's that far behind. So it shouldn't ever block the main camera. This design kind of prevents that actually - whereas when you're in a normal python example, you've either got to ensure your processing runs fast, or do the cleverer tricks anyway (separate thread filling up a queue etc.).

Picamera2 does have an allocator class (as you've discovered), and in fact our next release (currently on the "next" branch) will have another flavour of allocator too. Whether this is enough to help you already I couldn't say, but we're certainly open to extending what we have if that's going to be useful - but we'd have to know what extensions to make.

Thanks, I'll try have a quick look. I suspect I'll eventually realise it'll be really easy or not at all.

The direction this seems to be heading in is Pipewire ...

Oooh that looks cool! I'm somewhat dubious as it's hard for a bit of software to do all things and also do everyone's specific use case as fast as possible - but I also suspect it'll be fine here, especially as I'm generally sending around smaller frames. And I'd like not writing my own code.

[kodonnell]

OK, I'm just looking into this again after looking at the change log and reading about the new persistent allocator, and then about the DMA heap etc. This is largely doing what I'm doing with shared memory, so it'd be nice to merge them. Can you check my thinking?

• We use the DMA heap because it's faster (??), so it makes sense vs /dev/shm (??)
• I think I mostly get the libcamera request flow. I'm assuming each buffer contains the pixels etc., but also some metadata like the frame number or timestamp. (Is there a reference for this? I didn't find libcamera python docs.)

So, I'm thinking:

• Use the existing DMA buffers, but with e.g. 100 buffers. Leave the existing libcamera pipeline to fill them.
• Share the buffer somehow - it looks like you can share DMA, though that needs investigating! Let's assume it works for now. As a new request is processed in picamera2, I'll update some shared interprocess state (e.g. "request 57 contains frame 1818").
• In the other process(es), I can somehow enumerate those 100 buffers. Then, if I want frame 1818, I query the aforementioned shared store, and figure out it's request 57, which I can then read from DMA.
• Locking as appropriate. I'm not sure I have any ability to force libcamera to block writing to a DMA while I read it, so it might just have to be best effort (and e.g. check the frame timestamp after accessing). I guess it depends on how the DMA sharing works.

Does that seem reasonable, or have I missed something? The main bit to investigate seems to be how to share DMA from python.

[davidplowman]

OK, I'm just looking into this again after looking at the change log and reading about the new persistent allocator, and then about the DMA heap etc. This is largely doing what I'm doing with shared memory, so it'd be nice to merge them. Can you check my thinking?

• We use the DMA heap because it's faster (??), so it makes sense vs /dev/shm (??)

Can't say I specifically know about that, but in our experience using the DMA heap is best because it can be shared directly between hardware (which needs special contiguous buffers) and the CPU because it can read them through the data cache (though you have to signal when the CPU is going to use the buffer). The default libcamera buffers can't be read through the data cache, so accessing them is much slower.

Even just copying the buffers is faster if they go through the data cache, I think that might have a wider path to memory than the uncached route, even though for a straight copy, the cache in itself isn't useful.

• I think I mostly get the libcamera request flow. I'm assuming each buffer contains the pixels etc., but also some metadata like the frame number or timestamp. (Is there a reference for this? I didn't find libcamera python docs.)

That's correct. Whilst libcamera provides Python bindings, it doesn't actually make an official package for them, and the documentation for all of libcamera (particularly when it comes to "how do I use this?" rather an auto-generated API documentation) is not copious.

Reading the Picamera2 manual in detail might help, perhaps I should consider a "how does libcamera work" section in there.

So, I'm thinking:

• Use the existing DMA buffers, but with e.g. 100 buffers. Leave the existing libcamera pipeline to fill them.

I don't think you're going to be able to allocate 100 buffers (certainly not on a Pi 4 or earlier). As I explained, they're "special" buffers and contiguous memory is in short supply. The number you can allocate obviously depends on how big the images are, but the range is likely to be from one or two (very large images) to a dozen or couple of dozen (much smaller images).

• Share the buffer somehow - it looks like you can share DMA, though that needs investigating! Let's assume it works for now. As a new request is processed in picamera2, I'll update some shared interprocess state (e.g. "request 57 contains frame 1818").
• In the other process(es), I can somehow enumerate those 100 buffers. Then, if I want frame 1818, I query the aforementioned shared store, and figure out it's request 57, which I can then read from DMA.
• Locking as appropriate. I'm not sure I have any ability to force libcamera to block writing to a DMA while I read it, so it might just have to be best effort (and e.g. check the frame timestamp after accessing). I guess it depends on how the DMA sharing works.

So long as the the process running Picamera2 doesn't release the request, then it's safe for you to hang on to those buffers. When you get a request through capture_request, it records a "use" of the buffer, so it can't be re-used by libcamera until you do a release. Of course, as I said previously, 100 buffers sounds unlikely...

Does that seem reasonable, or have I missed something? The main bit to investigate seems to be how to share DMA from python.

On a Pi 4 or earlier, I don't think you'll be able to get that many buffers. However, you probably have the option of creating lots "memfd" buffers which you could share between processes, though I think you'd always have to take the hit of a copy from the libcamera dmabuf to the memfd buffer. (I expect this is a lot like creating a file in /dev/shm, not exactly sure what the differences are.)

Note that on a Pi 5 you probably can have lots more camera dmabufs. That's because a Pi 5 has an IO MMU so it can use "regular" memory, not special contiguous memory, with the hardware.

[kodonnell]

Great, very helpful thanks! The automatic releasing is nice, and confirmation I get can frame timestamp info etc.

Whilst libcamera provides Python bindings ...

Do you generally just look at the libcamera C++ classes and know they're pybind11'd? I.e. how do you develop against libcamera in python?

I don't think you're going to be able to allocate 100 buffers (certainly not on a Pi 4 or earlier).

Ah, right. Is there significantly more data stored in the buffer than the pixel information? E.g. YUV 512x768 is (I think) about 600kb, so 100 images should only be around 60mb. I might need to increase the CMA heap, but that's still well within the realms of a 512mb device. I assume it packs both streams in there (if they're used), but e.g. I'm usually doing main as 512x768 and lores as 192x256, so it won't be a big difference. It currently works in shared memory, but CMA might be different.

FWIW my use case is generally lower power things, so I'm focusing on the Pi Zero 2W. The interest in buffers is because it's not as grunty as the pi 5 so can get a bit behind in processing occasionally, and the shared memory/buffers is just to make code development a lot nicer ... not needing a big monolith etc.

[davidplowman]

Great, very helpful thanks! The automatic releasing is nice, and confirmation I get can frame timestamp info etc.

Whilst libcamera provides Python bindings ...

Do you generally just look at the libcamera C++ classes and know they're pybind11'd? I.e. how do you develop against libcamera in python?

Generally the Python bindings don't change much, so Python code tends to be fairly stable. The only catch is that libcamera beneath is not so stable, so one version of libcamera is generally not compatible with another even though the Python API looks identical. This is why I always recommend updating libcamera and the Python bindings together using apt, we always keep those versions in step.

I don't think you're going to be able to allocate 100 buffers (certainly not on a Pi 4 or earlier).

Ah, right. Is there significantly more data stored in the buffer than the pixel information? E.g. YUV 512x768 is (I think) about 600kb, so 100 images should only be around 60mb. I might need to increase the CMA heap, but that's still well within the realms of a 512mb device. I assume it packs both streams in there (if they're used), but e.g. I'm usually doing main as 512x768 and lores as 192x256, so it won't be a big difference. It currently works in shared memory, but CMA might be different.

I suppose you'll have to try it and see. Your buffers are small, so you may have more luck. There are some other buffers that get allocated too, such as the raw camera buffers. Depending on the sensor and sensor mode, these may or may not be large. But certainly give it a try.

FWIW my use case is generally lower power things, so I'm focusing on the Pi Zero 2W. The interest in buffers is because it's not as grunty as the pi 5 so can get a bit behind in processing occasionally, and the shared memory/buffers is just to make code development a lot nicer ... not needing a big monolith etc.

Pi Zero 2W is of course always a bit short on memory. It's worth checking what your CMA heap is like (grep Cma /proc/meminfo I think). There may be some scope to increase it - again, you'll have to try. On a regular Pi 3 with 1GB, you could probably push it to 512MB. You could try 384MB on a Zero 2W (if it's less that that by default). The system can still use the CMA heap so it might still work. Or it might not boot. Like everything else really - try it and see.

[kodonnell]

OK, while recording at 768x1024 (in binned mode) it's using about 50mb (based on the change in CmaFree from /proc/meminfo). If I record 256x256 it's still 50mb. So it does seem like maybe there are raw frame buffers being allocated, which makes sense if this is about shuffling frames between hardware etc.

Going to 12 buffers uses 60mb. 24 buffers is about 90mb. 100 buffers it can't allocate enough.

So I think this rules out naively increasing the buffer count, because it uses more memory than just rolling my own (and only sharing the streams). I may still use DMA bufs though, as it might be a bunch faster - and maybe there's a quick way to copy out only the stream information from the picamera2 buffers, and into my own. I'll keep this issue open for now as it's potentially easy to do and would make a nice example - even though a full solution is probably outside the scope of picamera2.

[kodonnell]

It seems like using DMA is faster for smaller bits of memory (e.g. 50kb vs 10mb), though it's not an apples-to-apples test so may not be fair.

Any tips on how I'd access an existing dma buffer from a separate process? I was (naively) hoping I could just pass the fd to another process and then mmap it, but no luck. Do you know of any examples that could point me in the right direction? I've not dealt with dma buffers (or ioctls etc.) before, and my googling isn't proving fruitful.

[davidplowman]

Indeed, you can't just pass an fd as a number to another process and use it. This isn't really something I know a great deal about. However...

If you're using Python, I think the multiprocessing module has built-in support for creating shared memory buffers. I expect this is probably the easiest route, though I haven't used it myself so there may be pitfalls I don't know about!

If you're not using Python, I think things get more complicated. The easiest thing is probably to open your fds and then fork the process. Alternatively, it is possible to pass fds from one process to another. There appear to be some unreasonably complicated looking mechanisms (to my eyes, at least!) that involve sending the fd through a socket, and it will get "translated". More recently there's something called pidfd_getfd so perhaps that would be the way to go, though you'd have to read up how to use that.

[kodonnell]

Oooh things are looking promising! https://unix.stackexchange.com/a/773998 I’ve tried a variety of shared memory approaches (posix_ipc and multiprocessing off the top of my head) but hopefully this dma process will be faster … even if it’s a micro-optimisation = ). I’ll see if I can get a prototype running and compare the bandwidth etc. Thanks!

[kodonnell]

Cool, the approach from above definitely works! I'd still need to implement the bit to copy off the parts I need from the picamera2 buffers as requests are processed, but that seems straightforward enough. Main issue I foresee would be managing the buffers (what happens when one process dies and the other still holds them) etc.

[kodonnell]

I'm going to close this, but it's worth noting:

• It works just fine! Didn't have any issues with other processes having things open or whatnot.
• It was a lot faster.
• Not sure if there's a way to copy/write from one mm directly to another, but given we're reading off mmap'd DMA buffers in make_buffer etc, we could possibly skip reading into userspace (if that makes sense) to make it even faster.

I decided to move away from shared memory in the end, mostly because IPC was getting really messy, as was having a single camera process managing various requests and trying to handle them sensible (e.g. when 3 different processes want different resolutions ... and they request them at different times) ... and trying to keep power consumption down. My scenario is inflexible enough to keep it simple = )

[davidplowman]

Thanks for the update. I don't suppose you might have have a short example that shows how you can pass these buffers around that could be added to the examples folder? I think something like that would be helpful to other users (and me!). But no worries if not!