Inconsistant `input_feature` length and `attention_mask` length in `WhisperFeatureExtractor`

[BakerBunker]

System Info

transformers main branch

Who can help?

@eustlb @zucchini-nlp

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoProcessor
import numpy as np

audios=[np.random.randn(16000*5)]
processor=AutoProcessor.from_pretrained("openai/whisper-large-v3")
print(processor(
    [audios[0][: 160 * 5 - 1]], return_attention_mask=True, sampling_rate=16000, padding=False
)["attention_mask"].shape)
print(processor(
    [audios[0][: 160 * 5]], return_attention_mask=True, sampling_rate=16000, padding=False
)["attention_mask"].shape)
print(processor(
    [audios[0][: 160 * 5 + 1]], return_attention_mask=True, sampling_rate=16000, padding=False
)["attention_mask"].shape)

Expected behavior

The input_feature and attention_mask length should be audio_length // hop_length, but the code here:

transformers/src/transformers/models/whisper/feature_extraction_whisper.py

Lines 327 to 329 in e8e0c76

	if return_attention_mask:
	# rescale from sample (48000) to feature (3000)
	padded_inputs["attention_mask"] = padded_inputs["attention_mask"][:, :: self.hop_length]

makes the attention_mask length equal to (audio_length+hop_length-1) // hop_length, it should be changed to:

diff --git a/src/transformers/models/whisper/feature_extraction_whisper.py b/src/transformers/models/whisper/feature_extraction_whisper.py
index 68c52c6eb3..b9f3b4cb35 100644
--- a/src/transformers/models/whisper/feature_extraction_whisper.py
+++ b/src/transformers/models/whisper/feature_extraction_whisper.py
@@ -326,7 +326,9 @@ class WhisperFeatureExtractor(SequenceFeatureExtractor):
 
         if return_attention_mask:
             # rescale from sample (48000) to feature (3000)
-            padded_inputs["attention_mask"] = padded_inputs["attention_mask"][:, :: self.hop_length]
+            padded_inputs["attention_mask"] = padded_inputs["attention_mask"][
+                :, self.hop_length - 1 :: self.hop_length
+             ]

[eustlb]

Hey @BakerBunker, this looks like a nice catch! It's never been an issue since there is no reason to use Whisper feature extractor with paddind=False since Whisper only accepts 30sec inputs, unless you want to use it for another model of curse (which is likely your use case, right?)

Would you mind opening a fixing PR? happy to review 🤗

[BakerBunker]

In some models using whisper encoders like qwen omni, even with paddind=True, the sum of attention_mask is different with input_feature length, and qwen omni uses attention_mask to calculate the length of input_feature

transformers/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py

Lines 159 to 169 in e6a8063

	if audio is not None:
	output_kwargs["audio_kwargs"]["padding"] = "max_length" # Support "max_length" padding only here
	audio_inputs = self.feature_extractor(audio, **output_kwargs["audio_kwargs"])
	audio_inputs["feature_attention_mask"] = audio_inputs.pop(
	"attention_mask"
	) # rename feature_attention_mask to prevent conflicts later on
	audio_inputs["input_features"] = audio_inputs.pop(
	"input_features"
	) # rename input_features to prevent conflicts later on
	input_lengths = (audio_inputs["feature_attention_mask"].sum(-1) - 1) // 2 + 1
	audio_lengths = iter((input_lengths - 2) // 2 + 1)