Feature Request: Native Support for Custom Multimodal Models

[DrxcoDev2]

Feature request

I'm currently working on a research project that involves combining textual and audio data in a custom multimodal architecture. While the library already supports several powerful pretrained multimodal models like CLIP and Flamingo, building new custom multimodal models from scratch is still quite manual and repetitive.

I’d love to propose a general framework for defining and training custom multimodal models natively within transformers. I believe this could benefit many researchers and developers looking to explore new combinations of modalities.

Motivation

While working on a project that involves combining textual and audio data, I found it quite cumbersome to build a custom multimodal model using the current transformers library. Although the library supports impressive multimodal architectures like CLIP, Flamingo, and VisionEncoderDecoderModel, these are tied to specific use cases and pretrained models.

I'm often frustrated by the lack of a general, modular interface that allows me to:

Seamlessly combine different pretrained encoders (e.g., BERT + Wav2Vec2).

Handle multimodal inputs using a unified processor.

Train and fine-tune these models using the Trainer API without custom boilerplate code.

This lack of flexibility makes experimenting with new multimodal architectures more difficult and discourages rapid prototyping.

By introducing a native multimodal model base class and processor integration, the transformers library could better support custom research and production use cases involving mixed modalities.

Your contribution

Yes — I’d be happy to contribute to this feature.

If the proposed idea aligns with the maintainers’ vision, I’m willing to:

Collaborate on the design of the API and architecture.

Open a Pull Request implementing a minimal working version of the MultiModalModel base class and example processors.

Write basic documentation and provide a working notebook or demo.