Skip to content

Conversation

@murisi
Copy link

@murisi murisi commented Oct 16, 2025

I came across the bug ** (ErlangError) Erlang error: :nif_panicked when trying to do the following NIF call in Elixir: AnomaSDK.Arm.encrypt_cipher(<<96, 109, 146, 170, 18, 27, 248, 118, 203, 31, 65, 72, 51, 153, 179, 35, 168, 62, 88, 150, 244, 119, 198, 51, 20, 106, 74, 8, 143, 32, 45, 168>>). The NIF is defined as follows:

#[nif]
fn encrypt_cipher(SerdeTerm(cipher): SerdeTerm<Vec<u8>>) -> SerdeTerm<Ciphertext> { ... }

Tracking down this error led me to the Deserializer::deserialize_seq implementation. It seems that the function does not have the ability to treat Elixir binary terms as sequences. This PR enables Elixir binaries to be treated as sequences (in addition to lists and empty lists) which fixes the NIF panic described above.

And thanks for maintaining Rustler, this crate is extremely useful!

Copy link
Member

@evnu evnu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting! Can you add test cases as well (especially cases in rustler_tests?

@murisi murisi force-pushed the murisi/fix-binary-deserialization branch from 20b81fd to c64c029 Compare October 21, 2025 03:27
@murisi murisi force-pushed the murisi/fix-binary-deserialization branch from c64c029 to 98abafc Compare October 21, 2025 03:34
@murisi
Copy link
Author

murisi commented Oct 21, 2025

Interesting! Can you add test cases as well (especially cases in rustler_tests?

I have added a test case which checks that <<2, 3, 5, 7, 11, 13>> can deserialise to the Rust Vec<u8>: vec![2, 3, 5, 7, 11, 13]. But vec![2, 3, 5, 7, 11, 13] still serializes to the Elixir list [2, 3, 5, 7, 11, 13] since I'm not sure how to specialize the serialization of Vec<u8>s. So the byte array conversion in https://docs.rs/rustler/latest/rustler/serde/index.html#conversion-table now only half works (when it did not seem to work at all before).

But upon some reflection, this solution is not so nice because it is using Serializer::serialize_seq and Derializer::deserialize_seq to serialize/deserialise binaries when the more appropriate methods are Serializer::serialize_bytes and Deserializer::deserialize_bytes. Using the ByteArray and ByteBuf types from the serde_bytes crate (https://docs.rs/serde_bytes/latest/serde_bytes/) instead of Vec<u8>s causes the latter two methods to be invoked and makes the ** (ErlangError) Erlang error: :nif_panicked error above go away.

So a better/less intrusive solution than this PR might be to just update the documentation at https://docs.rs/rustler/latest/rustler/serde/index.html#conversion-table to either indicate that the Serde (Rust) Value for a byte array is a ByteArray or ByteBuf, or alternatively indicate that Serde (Rust) Value is a Vec<u8> only when serialization/deserialisation happens with serde_bytes (#[serde(with = "serde_bytes")]).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants