fix: type_caster_enum_type for pointer types

[rwgk]

Description

Closes #5694

Add pointer overload to type_caster_enum_type::cast method to handle enum pointer casting. Fixes compilation error when returning pointers to enum types from bound functions.

Experiment for validation:

Temporarily undo the changes in include/pybind11/cast.h:

g++ -o pybind11/tests/test_native_enum.os -c -std=c++20 -fPIC -fvisibility=hidden -O0 -g -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wundef -Wnon-virtual-dtor -Wunused-result -Werror -funsigned-char -Wpedantic -isystem /usr/include/python3.12 -isystem /usr/include/eigen3 -DPYBIND11_SMART_HOLDER_PADDING_ON -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD_IF_AVAILABLE -DPYBIND11_TEST_BOOST -Ipybind11/include -I/home/rgrossekunst/forked/pybind11/include -I/home/rgrossekunst/clone/pybind11/include /home/rgrossekunst/forked/pybind11/tests/test_native_enum.cpp
In file included from /home/rgrossekunst/forked/pybind11/include/pybind11/native_enum.h:10,
                 from /home/rgrossekunst/forked/pybind11/tests/test_native_enum.cpp:1:
/home/rgrossekunst/forked/pybind11/include/pybind11/cast.h: In instantiation of ‘static pybind11::handle pybind11::detail::type_caster_enum_type<EnumType>::cast(SrcType&&, pybind11::return_value_policy, pybind11::handle) [with SrcType = const test_native_enum::color*; EnumType = test_native_enum::color]’:
/home/rgrossekunst/forked/pybind11/include/pybind11/pybind11.h:429:40:   required from ‘void pybind11::cpp_function::initialize(Func&&, Return (*)(Args ...), const Extra& ...) [with Func = test_submodule_native_enum(pybind11::module_&)::<lambda()>; Return = const test_native_enum::color*; Args = {}; Extra = {pybind11::name, pybind11::scope, pybind11::sibling}]’
/home/rgrossekunst/forked/pybind11/include/pybind11/pybind11.h:274:19:   required from ‘pybind11::cpp_function::cpp_function(Func&&, const Extra& ...) [with Func = test_submodule_native_enum(pybind11::module_&)::<lambda()>; Extra = {pybind11::name, pybind11::scope, pybind11::sibling}; <template-parameter-1-3> = void]’
/home/rgrossekunst/forked/pybind11/include/pybind11/pybind11.h:1384:22:   required from ‘pybind11::module_& pybind11::module_::def(const char*, Func&&, const Extra& ...) [with Func = test_submodule_native_enum(pybind11::module_&)::<lambda()>; Extra = {}]’
/home/rgrossekunst/forked/pybind11/tests/test_native_enum.cpp:139:10:   required from here
/home/rgrossekunst/forked/pybind11/include/pybind11/cast.h:70:32: error: invalid ‘static_cast’ from type ‘const test_native_enum::color*’ to type ‘pybind11::detail::type_caster_enum_type<test_native_enum::color>::Underlying’ {aka ‘unsigned int’}
   70 |             return native_enum(static_cast<Underlying>(src)).release();
      |                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/rgrossekunst/forked/pybind11/include/pybind11/cast.h: In instantiation of ‘static pybind11::handle pybind11::detail::type_caster_enum_type<EnumType>::cast(SrcType&&, pybind11::return_value_policy, pybind11::handle) [with SrcType = test_native_enum::color*; EnumType = test_native_enum::color]’:
/home/rgrossekunst/forked/pybind11/include/pybind11/pybind11.h:429:40:   required from ‘void pybind11::cpp_function::initialize(Func&&, Return (*)(Args ...), const Extra& ...) [with Func = test_submodule_native_enum(pybind11::module_&)::<lambda()>; Return = test_native_enum::color*; Args = {}; Extra = {pybind11::name, pybind11::scope, pybind11::sibling}]’
/home/rgrossekunst/forked/pybind11/include/pybind11/pybind11.h:274:19:   required from ‘pybind11::cpp_function::cpp_function(Func&&, const Extra& ...) [with Func = test_submodule_native_enum(pybind11::module_&)::<lambda()>; Extra = {pybind11::name, pybind11::scope, pybind11::sibling}; <template-parameter-1-3> = void]’
/home/rgrossekunst/forked/pybind11/include/pybind11/pybind11.h:1384:22:   required from ‘pybind11::module_& pybind11::module_::def(const char*, Func&&, const Extra& ...) [with Func = test_submodule_native_enum(pybind11::module_&)::<lambda()>; Extra = {}]’
/home/rgrossekunst/forked/pybind11/tests/test_native_enum.cpp:143:10:   required from here
/home/rgrossekunst/forked/pybind11/include/pybind11/cast.h:70:32: error: invalid ‘static_cast’ from type ‘test_native_enum::color*’ to type ‘pybind11::detail::type_caster_enum_type<test_native_enum::color>::Underlying’ {aka ‘unsigned int’}

Suggested changelog entry:

• Fixed compilation error in type_caster_enum_type when casting pointer-to-enum types. Added pointer overload to handle dereferencing before enum conversion.

[rwgk]

Tracking flakes:

🐍 (macos-13, 3.13t, -DCMAKE_CXX_STANDARD=11) / 🧪

test_gil_scoped.py:204: RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

(This is a known instability.)

---

🐍 3.12 • macos-latest • x64 inplace C++14

FAILED: [code=139] tests/test_embed/CMakeFiles/cpptest /Users/runner/work/pybind11/pybind11/tests/test_embed/CMakeFiles/cpptest 
cd /Users/runner/work/pybind11/pybind11/tests/test_embed && /Users/runner/work/pybind11/pybind11/tests/test_embed/test_embed
Exception ignored in: <module 'threading' from '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py'>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1602, in _shutdown
    assert tlock.locked()
           ^^^^^^^^^^^^^^
AssertionError: 
Debug memory block at address p=0x120605040: API ''
    0 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x00 *** OUCH
        at p-6: 0x00 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0x00 *** OUCH
        at p-3: 0x00 *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0x120605040 are not all FORBIDDENBYTE (0xfd):
        at tail+0: 0x2f *** OUCH
        at tail+1: 0x00 *** OUCH
        at tail+2: 0x00 *** OUCH
        at tail+3: 0x00 *** OUCH
        at tail+4: 0x4c *** OUCH
        at tail+5: 0x00 *** OUCH
        at tail+6: 0x00 *** OUCH
        at tail+7: 0x00 *** OUCH

Enable tracemalloc to get the memory block allocation traceback

Fatal Python error: _PyMem_DebugRawFree: bad ID: Allocated using API '', verified using API 'r'
Python runtime state: initialized

Current thread 0x000000016f703000 (most recent call first):
  <no Python frame>
Debug memory block at address p=0x104818150: API '�'
    5437922219062919421 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0xfd
        at p-6: 0xfd
        at p-5: 0xfd
        at p-4: 0xfd
        at p-3: 0xfd
        at p-2: 0xfd
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0x4b7761736bf4824d are /bin/sh: line 1:  5546 Segmentation fault: 11  /Users/runner/work/pybind11/pybind11/tests/test_embed/test_embed
ninja: build stopped: subcommand failed.

@b-pass I know you're working on it already. Tagging you here JIC it helps as a clue.

[rwgk]

@henryiii There seems to be an unrelated problem with 🐍 (macos-13, 3.13t, -DCMAKE_CXX_STANDARD=11) / 🧪 that we need to solve separately. I'm seeing the same failure under #5777. I think this PR is good for merging.

[henryiii]

I don't understand what changed since yesterday, though. It's now reliably falling, it seems. Nothing noticeable has changed in the environment. Same runner version, Python version, etc - if it's logged, it's the same (diff'ed logs).

[rwgk]

Under #5777 it just passed again. Weird.

I cannot imagine how this PR could be related.

I'll try another rerun.

[rwgk]

@henryiii The troublesome test passed now.

[HinTak]

When is this going into a release? 3.0.0 is out and #5694 gotten more serious.

[henryiii]

I'd be in favor of a 3.0.1 soon. :)

[rwgk]

@henryiii wrote:

I'd be in favor of a 3.0.1 soon. :)

+1

I think it's a good time now.

I won't be able to drive this for a while though, but I'll review release PRs asap.