Bug: exception in noexcept what() when Python exception contains a surrogate character

[TheShiftedBit]

By default, Python produces errors when converting encoding strs with utf-8 if the str contains surrogate characters. This can be disabled by passing surrogatepass as a second argument to .encode(). Pybind11 has this same behavior with its str -> std::string conversion. However, the bug is this: if an exception message contains a surrogate character, calling .what() on an error_already_set with such an exception causes another exception to be thrown, but since .what() is noexcept, that exception cannot be caught and the program std::terminates.

I'm not sure what the correct behavior regarding surrogate characters is. Perhaps pybind11 should always use surrogatepass, perhaps not. However, even if that's not the right choice, it should probably use it during exception handling, or Python exceptions like this are extremely difficult to diagnose.

[Skylion007]

@TheShiftedBit Could your provide a minimal example? I think I might have a proposed solution, but I am having trouble replicating this.

[TheShiftedBit]

Here's a reproducer. Tested on Linux.

Extension code:

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <iostream>

void invoke(std::function<void()> cb) {
  try {
    cb();
  } catch (pybind11::error_already_set& ex) {
    std::cout << ex.what() << std::endl;
  }
}

PYBIND11_MODULE(pybind11_repro, m) {
  m.def("invoke", &invoke);
}

Python code:

import pybind11_repro

def foo():
  raise RuntimeError("\ud927")

pybind11_repro.invoke(foo)

This produces the following output:

terminate called after throwing an instance of 'pybind11::cast_error'
  what():  Unable to cast Python instance to C++ type (#define PYBIND11_DETAILED_ERROR_MESSAGES or compile in debug mode for details)
[1]    1227873 IOT instruction  python3 ./repro.py

Defining that macro, we see:

terminate called after throwing an instance of 'pybind11::cast_error'
  what():  Unable to cast Python instance of type <class 'str'> to C++ type 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >'
[1]    1227997 IOT instruction  python3 ./repro.py