Add Qwen3 Moe

[kanpuriyanawab]

Qwen3 Moe backbone output matching with atol 1e-3!

Generate output matching wise We are doing okay here!

keras hub - [ 3838, 374, 730, 9247, 30, 2585, 311, 5443, 730, 9247,
304, 13027, 1939, 42, 9247, 374, 458, 1787, 30774, 5538,
6832, 6733, 5326, 304, 13027, 13, 1084, 374, 6188, 311,
7283, 4937]

huggingface -[ 3838, 374, 730, 9247, 30, 730, 9247, 374, 264, 1550,
11591, 29728, 14155, 5333, 11, 5326, 304, 13027, 323, 12875,
315, 4303, 389, 1909, 315, 94986, 11, 356, 6408, 42,
11, 476]

[mattdangerw]

Thanks! Took an initial pass. Let's try to clean up the config and state passing.

No passing an index down the layer stack, plus data structures that apply to the whole layer stack.

[divyashreepathihalli]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for the Qwen3 MoE model. The implementation looks solid, covering the backbone, attention, decoder, tokenizer, and conversion scripts. I've identified several high-severity issues related to incomplete get_config methods in various new layers, which will prevent model serialization from working correctly. There are also some medium-severity issues like unused parameters and a critical issue in the checkpoint conversion test script where an incorrect preprocessor is used. I've provided suggestions to fix these issues. Once addressed, the PR should be in great shape.