Refactor simulation

[viljarjf]

Simulation is now a single file, and requires no dependencies. It is quite performant, I can run for a million crystals with no stuttering on my desktop.
I tried commenting the maths, but it can be hard to follow some places.
Hopefully you can try this out, @Baharis! I find it strange that it did not work for you before. It should also work with real microscopes, but the way communication is established is not pretty..

Of course, if you still want to keep complete seperation of simulation, then this PR can be ignored. It simply collects the existing simulation into one file, with a slight rewrite of some parts for simplicity.

I have not checked how accurate the spot positions are when accounting for camera length calibration ect, nor did I try to change the unit cell much.

[Baharis]

@viljarjf I would love to be convinced that it is fine to leave, but for the time being the simulation just does not work for me, which has an opposite effect. I get no errors no log messages, but neither do I get an image. Neither brightness nor dynamical range affects it. Maybe there are some special steps to set it up?

Edit: I played with it a bit and I DID see diffraction for a second or so, but it disappeared and I could never get it back. So it definitely does work, but given that I though it doesn't does means user might think so as well. It does not fill me with much confidence for the time being...

[viljarjf]

Strange. Perhaps there are no crystals in that area? Could you try changing to very low magnification and see if any show up?

A more user-friendly experience is necessary regardless, i agree

[Baharis]

I played with it a bit and found a lot of issues. I started writing a structured response, but after a lot of writing and testing I realized that the behavior of Instamatic was changing for me as I kept writing and testing... I think some of the issues are due to some strange interaction with new implementation in #131. If you want to see what I am talking about, try change your display range to the new simulator default of 255 - oddly enough it does not matter whether Auto contrast is on or off. I must look into this code... This is not to say all problems are there – most of the time I still can't see any crystals/diffraction. May be an issue with zoom or the way simulated stage works. Will look later.

Edit: I cannot get an image consistently enough to test for issues. I spent 15 minutes trying to get image to no avail. With that I unfortunately have to pass debugging for the time being.

[viljarjf]

I'll add an option to have crystals in a regular grid on the stage instead of randomly dispersed. Then you should always see one at the origin. I'll also add some noise and fix the value ranges/type. That might hopefully help

Edit: Options can be in the dedicated simulation repo. I'll have one crystal always be at (0, 0), the rest random

[viljarjf]

I added some noise, a 100nm cross at (0, 0), and forced a crystal to be at (0, 0).
This is how it looks after resetting the stage and going to 2500x (I used the control updates from #142):

Are you able to replicate this @Baharis?

I am seeing some strangeness at displayrange 255 when auto contrast is on. It seems like the auto-contrast is off specifically at 255, as images with it off look similar when the displayrange is 254 or 256. I set datatype to int and the max value is 0xf000, if that matters.

[Baharis]

Ok, with your most recent changes I can see things. Apparently also my stage z was off the charts and the hand panel does not allow setting it, so any movements resulted in erratic changes. There are some funny things going on with display range and brightness as well. I think the new method does not work well with the sharp limits of the simulated data. Below is an image with crystal, cross, and background at four brightness settings and DisplayRange 250:

Brightness 4.0	Brightness 4.3	Brightness 4.7	Brightness 5.0

I'm off to test why does that happen, my guess is that some bits overflow or some to-be-variables are hard-coded. Edit: I will edit in my notes below so that I do not have to re-write the entire thought process twice.

Testing notes

Brightness 1.0, DisplayRange 255	Brightness 1.0, DisplayRange 256	Brightness 10, DisplayRange 256	Brightness 200, DisplayRange 256	Brightness -1, DisplayRange 256

Case 1: Brightness 1.0, DisplayRange 255, Auto contrast on.
Simplest case: default settings so VideoStreamProcessor theoretically does not have to scale anything because config.camera.dynamic_range == 255. Image rendered receives frame with floats between -10000 (overlap of crystals) and +70000 (background) while it expects ints between 0 and 255. The rounding function that asserts data is in 0-255 range for plotting uses np.clip(frame.astype(np.int16), 0, 255).astype(np.uint8), so values over 32k overflow ~2 times, causing dark background. Consequently, the crystal is the brightest object on screen, thus white.

• Potential issue: frame provided contains floats, not ints (probably benign)
• Potential issue: frame apparently ignores config.camera.dynamic_range
• Potential issue: overlapping crystals make intensity negative, which oddly enough is fine because the intensity is clipped back to 0, but this makes me question the simulation

Case 2: Brightness 1.0, DisplayRange 256, Auto contrast on.
The image changes significantly because DisplayRange = 256 is not default value, so the scaling actually starts working. Now the data is properly scaled using the following logic:

if self.vsf.display_range != 255.0 or self.vsf.brightness != 1.0:
    if self.vsf.auto_contrast:
        display_range = 1 + np.percentile(frame[::4, ::4], 99.5)
    else:
        display_range = self.vsf.display_range
    frame = (self.vsf.brightness * 255 / display_range) * frame
frame = np.clip(frame.astype(np.int16), 0, 255).astype(np.uint8)

Now the frame is properly scaled to the 255 range before before being displayed, so it it mostly correctly scaled. One remaining issue is that the frame fed to the last line still has scaled negative values for overlapping crystals, but these are small enough for np.int16 to handle and np.clip to round to 0.

Case 3: Brightness 10, DisplayRange 256, Auto contrast on.
This is completely correct and expected: both negative and positive values are scaled by 10 and end up in the limits of np.int16. More image is white because with this high brightness the top crystal * 10 exceeds 255 i.e. white.

Case 4: Brightness 200, DisplayRange 256, Auto contrast on.
Another degenerate case that is the unfortunate side-effect of optimizing the display in #131 , but no when you set brightness at absurdly high levels, you scale entire image after scaling to 0-255, so for brightness above ~128, 255*128+ starts exceeding np.int16 limits again, so due to overflow the dark parts become light again. Can be fixed by hard-limiting brightness to 0 to 100.

Bonus fact: Brightness -1, DisplayRange 256, Auto contrast on.
Setting brightness -1 typically just makes the whole image black because all values end up below 0 and are thus trimmed; however, in this case, they can be used to find negative intensity regions, as they become white!

Notes on notes

I added the following snippet to the image definition shown above to print sparse contents of one every ~500 frames:

if p := (random.random() < 0.002):
    print('Frame:')
    print(frame[::64, ::64].astype(np.int64))

[Baharis]

Based on my notes shown above, I suggest the following:

• Simulation should consider and clip to the dynamic range of the camera set in config from the top and to 0 from the bottom;
• Brightness should be capped to 0 – 128 to protect user from confusing absurd settings; I will make a patch for that;
  • Maybe the check for auto-scaling should still work and force-scale even if scaling should be theoretically unnecessary? This would slow the GUI slightly down, but would avoid odd behavior of this value in the future,
• Hand panel should have an option to set stage Z. This is ugly but straightforward and I will include it in Tighten code and add magnification control to the hand panel #142 .

Edit: I noticed that brightness is already soft-capped at 0 to 10 (other values can still be put in manually if needed, but the buttons do not allow it). I'd call this sufficient for the time being.