Lua draw alpha blending is weird

[warmCabin]

Here are a few example screenshots using Mega Man 2 sprites.
In each image, the left sprite shows the true colors, the middle sprite is FCEUX's attempt at blending it with the background, and the right sprite is a true alpha blend.

• Bubble Lead
• Quick Boomerang

Notice that the gray Bubble Lead color gets blended down to a perfect gray of #808080, instead of something with a bit of blue in it. The soft pink of the Quick Boomerang did not get blended at all.

Not sure if it has to do with the alpha blending, the 8 bit color palette quantizing, or both. Are all 256 colors always taken up by the NES colors and 8 levels of deemphasis?

I'm also not sure how the 256 color palette interacts with the 64 "unvarying colors."

[bbbradsmith]

1. I believe what it's supposed to do is match the intended blended colour to one already in the palette. I don't think it can do actual RGB blending directly.
2. I think when palettes were updated to be bigger than 256 colours, the lua blend result mapping code was left behind only checking the first 256 colours which might leave it getting poorer results than before, especially since the colours have been rearranged. It should search the whole palette table not just 256 colours. (IIRC the "unvarying" colours are there to provide nice bright colours suitable for lua.)
3. It appears to greatly favour matching green over red, and especially over blue. I've seen perceptual justification for that kind of thing but in my experience you better match if you do something like this instead:

test_score =
    abs(r - tr) *  66 +
    abs(g - tg) * 129 +
    abs(b - tb) * 25;

Might do better as:

test+score =
    (r-tr)*(r-tr) +
    (g-tg)*(g-tg) +
    (b-tb)*(b-tb);

i.e. 1. don't let a good green match dominate other colours, and 2. use squared distance instead of abs to favour matching all colours closely instead of the linear trade of error you get with abs.

[warmCabin]

1. For as long as the main window uses a 512-color palette, that seems to be the case. Maybe we could add something about that to the "on colors" section of the documentation.

2. https://github.com/TASVideos/fceux/blob/master/src/lua-engine.cpp#L3611

for (test = 0; test < 0xff; test++)

Yep! I wonder what kind of effect simply changing 0xff to 0x1ff would have on things.

I'm still kind of thrown off by the fact that all of them should be taken up by the 64 main colors * 8 tint modes but somehow aren't. Check out this scene in Woodman's stage with a regular palette and a weirdo sepia palette.

3. Are those ratios based on the biology of the human eye? Maybe I'll open up a PR and see what happens.

[zeromus]

hey @warmCabin will you review the situation if I poke around at the code?

[warmCabin]

Was just thinking about this again. I recently realized that twiddling your palette settings mitigates this issue, which ultimately lead me to our culprit:

fceux/src/lua-engine.cpp

Lines 3748 to 3757 in e550e5c

	static uint8 index_lookup[1 << (3+3+3)];
	int k;
	if (!gui_saw_current_palette)
	{
	memset(index_lookup, GUI_COLOUR_CLEAR, sizeof(index_lookup));
	gui_saw_current_palette = TRUE;
	}
	k = ((r & 0xE0) << 1) | ((g & 0xE0) >> 2) | ((b & 0xE0) >> 5);

It caches color lookups quantized to the upper 3 bits of each channel to keep it nice and lean at 512 bytes. I'd like to increase it to 5 or 6 bits (for 2^15 = 32K or 2^18 = 262K) or just remove it entirely and take the performance hit.

Looks like the palette is still initialized to 256 colors, so the test < 0xFF bound is fine.

Here's a funny little test script you can use to reproduce the issue relatively easily. It draws the full NES palette at 70% alpha and bounces it across the screen. Just play it on top of a random TAS or whatever and see what happens.

gui.register(function()

    local centerX, centerY = 64, 100
    local baseX, baseY = centerX + 60 * math.sin(emu.framecount() / 60), centerY + 80 * math.cos(emu.framecount() / 80)
    local size = 8

    gui.opacity(0.7)
    -- gui.opacity(0.5 + math.sin(emu.framecount() / 300) / 2)
    for i = 0, 0x3F do
        local x, y = baseX + (i % 16) * size, baseY + math.floor(i / 16) * size
        gui.box(x, y, x + size + 1, y + size + 1, string.format("P%02X", i), "clear")
    end

end)

[warmCabin]

#802