When creating a Trainer object for a MixedModel, the initialization tries to access attribute get_base_model (which does not exist) rather than model

[rluss]

System Info

• transformers version: 4.52.4
• Platform: Linux-5.14.0-503.21.1.el9_5.x86_64-x86_64-with-glibc2.34
• Python version: 3.12.0
• Huggingface_hub version: 0.30.2
• Safetensors version: 0.5.3
• Accelerate version: 1.6.0
• Accelerate config: not found
• DeepSpeed version: 0.16.7
• PyTorch version (GPU?): 2.7.0+cu126 (True)
• Tensorflow version (GPU?): 2.19.0 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: Yes
• GPU type: NVIDIA A100-SXM4-80GB

Who can help?

@zach-huggingface @SunMarc

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from peft import PeftConfig, PeftModel, get_peft_model, LoraConfig, TaskType
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from transformers import Trainer, TrainingArguments, DataCollatorWithPadding

import datasets
import torch
import os

# get base model and tokenizer
model_id = "google/gemma-2-2b-it"
access_token = os.getenv("HF_ACCESS_KEY", None)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2, token=access_token)
tokenizer = AutoTokenizer.from_pretrained(model_id, token=access_token)

peft_config = LoraConfig(
    r=8,
    task_type=TaskType.CAUSAL_LM,
    target_modules=["q_proj", "v_proj"]
)
model = get_peft_model(model, peft_config, adapter_name='mrpc1', mixed=True)
_ = model.add_adapter(adapter_name='mrpc2', peft_config=peft_config)

raw_datasets = datasets.load_dataset("glue", "sst2")
def tokenize_function(example):
    return tokenizer(example["sentence"], truncation=True)
# Tokenize the entire dataset
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

training_args = TrainingArguments("sst2-finetuned-model")
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
)

Expected behavior

I would expect this code to run sucessfully so that I can then run trainer.train(). I pasted the traceback to the error below. I believe it is related to the fact that PeftMixedModel does not provide a get_base_model method. See

transformers/src/transformers/trainer.py

Line 919 in 51f94ea

model_to_inspect = self.model.base_model.model

where the comment "# PeftMixedModel do not provide a get_base_model method" remains in the code. I believe the same fix there was not handled where my code hit an error which is at

transformers/src/transformers/trainer.py

Line 637 in 51f94ea

else unwrapped_model.get_base_model().forward

. I was able to locally fix this for myself at L637 by replacing else unwrapped_model.get_base_model().forward with
else (unwrapped_model.get_base_model().forward if hasattr(unwrapped_model, "get_base_model") else unwrapped_model.base_model.model.forward)

Traceback (most recent call last):
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/peft/mixed_model.py", line 197, in getattr
return super().getattr(name) # defer to nn.Module's logic
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1940, in getattr
raise AttributeError(
AttributeError: 'PeftMixedModel' object has no attribute 'get_base_model'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/peft/tuners/mixed/model.py", line 192, in getattr
return super().getattr(name) # defer to nn.Module's logic
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1940, in getattr
raise AttributeError(
AttributeError: 'MixedModel' object has no attribute 'get_base_model'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/dccstor/rluss1/Research_Projects/PrincipledAI/Orchestrators/lorasub/repos/trainer_expts/temp/HF_peftmixed_model_test.py", line 32, in
trainer = Trainer(
^^^^^^^^
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/transformers/trainer.py", line 636, in init
else unwrapped_model.get_base_model().forward
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/peft/mixed_model.py", line 199, in getattr
return getattr(self.base_model, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/peft/tuners/mixed/model.py", line 194, in getattr
return getattr(self.model, name)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dccstor/rluss1/envs/trainer8/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1940, in getattr
raise AttributeError(
AttributeError: 'Gemma2ForSequenceClassification' object has no attribute 'get_base_model'. Did you mean: 'base_model'?