Multiprocessing leads to decoding error

[KeKsBoTer]

🐛 Describe the bug

I am trying to assemble batches of images of random frames from multiple videos.
This is my code:

import torch
from torchcodec.decoders import VideoDecoder
from tqdm import tqdm

class Dataset(torch.utils.data.Dataset):

    def __init__(self,video_files:list[str]):
        self.videos = [VideoDecoder(f,device="cpu",num_ffmpeg_threads=1) for f in video_files]
        self.num_frames = len(self.videos[0])

    def __len__(self):
        return self.num_frames*len(self.videos)

    def __getitem__(self, item: int) -> torch.Tensor:
        
        frame_id = item%self.num_frames
        cam_id = item//self.num_frames
        return self.videos[cam_id].get_frame_at(frame_id).data
        
    
ds = Dataset(["video_1.mp4","video_2.mp4","video_3.mp4"])

trainloader = torch.utils.data.DataLoader(
    ds,
    batch_size=4,
    num_workers=1,
)
loader = iter(trainloader)
for l in tqdm(loader):
    pass

As long as num_workers is set to zero or one everything works.
Once I increase the num_workers i get this error for random batches:

RuntimeError: Could not push packet to decoder: Invalid data found when processing input

This seams to be a FFMEG error that only occurs when multiprocessing (by the Dataloader) is used.

Versions

Collecting environment information...
PyTorch version: 2.7.1+cu128
Is debug build: False
CUDA used to build PyTorch: 12.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.2 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version: Could not collect
CMake version: version 3.28.3
Libc version: glibc-2.39

Python version: 3.13.5 | packaged by Anaconda, Inc. | (main, Jun 12 2025, 16:09:02) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.11.0-29-generic-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: 12.8.93
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version: 570.133.20
cuDNN version: Could not collect
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 9950X 16-Core Processor
CPU family: 26
Model: 68
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 0
Frequency boost: enabled
CPU(s) scaling MHz: 84%
CPU max MHz: 5752.0000
CPU min MHz: 600.0000
BogoMIPS: 8583.65
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d amd_lbr_pmc_freeze
Virtualization: AMD-V
L1d cache: 768 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 16 MiB (16 instances)
L3 cache: 64 MiB (2 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] torch==2.7.1+cu128
[pip3] torch-fidelity==0.3.0
[pip3] torch-tb-profiler==0.4.3
[pip3] torchaudio==2.7.1+cu128
[pip3] torchcodec==0.4.0
[pip3] torchmetrics==1.7.3
[pip3] torchvision==0.22.1+cu128
[pip3] triton==3.3.1
[conda] cuda-cudart 12.8.90 0 nvidia/label/cuda-12.8.1
[conda] cuda-cudart-dev 12.8.90 0 nvidia/label/cuda-12.8.1
[conda] cuda-cudart-dev_linux-64 12.8.90 0 nvidia/label/cuda-12.8.1
[conda] cuda-cudart-static 12.8.90 0 nvidia/label/cuda-12.8.1
[conda] cuda-cudart-static_linux-64 12.8.90 0 nvidia/label/cuda-12.8.1
[conda] cuda-cudart_linux-64 12.8.90 0 nvidia/label/cuda-12.8.1
[conda] cuda-cupti 12.8.90 0 nvidia
[conda] cuda-cupti-dev 12.8.90 0 nvidia
[conda] cuda-libraries 12.8.1 0 nvidia
[conda] cuda-libraries-dev 12.8.1 0 nvidia
[conda] cuda-nvrtc 12.8.93 0 nvidia
[conda] cuda-nvrtc-dev 12.8.93 0 nvidia
[conda] cuda-nvtx 12.8.90 0 nvidia
[conda] cuda-opencl 12.8.90 0 nvidia
[conda] cuda-opencl-dev 12.8.90 0 nvidia
[conda] cuda-runtime 12.8.1 0 nvidia
[conda] libcublas 12.8.4.1 0 nvidia
[conda] libcublas-dev 12.8.4.1 0 nvidia
[conda] libcufft 11.3.3.83 0 nvidia
[conda] libcufft-dev 11.3.3.83 0 nvidia
[conda] libcurand 10.3.9.90 0 nvidia
[conda] libcurand-dev 10.3.9.90 0 nvidia
[conda] libcusolver 11.7.3.90 0 nvidia
[conda] libcusolver-dev 11.7.3.90 0 nvidia
[conda] libcusparse 12.5.8.93 0 nvidia
[conda] libcusparse-dev 12.5.8.93 0 nvidia
[conda] libnvjitlink 12.8.93 1 nvidia
[conda] libnvjitlink-dev 12.8.93 1 nvidia
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.8.3.14 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.57 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.61 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.57 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.7.1.26 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.41 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.55 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.2.55 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.7.53 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.3 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.26.2 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.61 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.55 pypi_0 pypi
[conda] torch 2.7.1+cu128 pypi_0 pypi
[conda] torch-fidelity 0.3.0 pypi_0 pypi
[conda] torch-tb-profiler 0.4.3 pypi_0 pypi
[conda] torchaudio 2.7.1+cu128 pypi_0 pypi
[conda] torchcodec 0.4.0 pypi_0 pypi
[conda] torchmetrics 1.7.3 pypi_0 pypi
[conda] torchvision 0.22.1+cu128 pypi_0 pypi
[conda] triton 3.3.1 pypi_0 pypi

[NicolasHug]

Hi @KeKsBoTer , would you be able to cache, for each decoder instance, the sequence of frame indices that are being requested? Then it'd be useful to query a decoder on the same frame index sequence, and see if the error is repeatable.

    self.num_frames = len(self.videos[0])

Seems like you are assuming that all videos have the same number of frames, just making sure this is a correct assumption.

[KeKsBoTer]

Thanks for the fast reply!
Yes, I am making the assumption that all videos have the same number of frames.

I modified the code to only load one video and sync the workers via the file system and make sure they access the frames in ascending order:

import os
import time
import torch
from torchcodec.decoders import VideoDecoder
import fasteners

rw_lock = fasteners.InterProcessLock('lock.file')

with open("current_frame.txt","w") as f:
    f.write(str(0))

with open("history.txt","w") as f:
    pass

class Dataset(torch.utils.data.Dataset):

    def __init__(self,video_file:str):
        self.video = VideoDecoder(video_file,device="cpu",num_ffmpeg_threads=1)
        self.num_frames = len(self.video)

    def __len__(self):
        return self.num_frames

    def __getitem__(self, item: int) -> torch.Tensor:
        with rw_lock:
            with open("current_frame.txt","r+") as f:
                index = int(f.read())
                f.seek(0)
                f.write(str(index+1))
            with open("history.txt","a") as f:
                f.write(f"{os.getpid()}: {index}\n")
            return self.video.get_frame_at(index).data
        
    
ds = Dataset("out.mp4")

trainloader = torch.utils.data.DataLoader(
    ds,
    batch_size=1,
    num_workers=2,
)
try:
    loader = iter(trainloader)
    for l in loader:
        pass
except Exception as e:
    print(open("history.txt","r").read())

This is what the output looks like:

859803: 0
859803: 1
859803: 2
859804: 3
859804: 4
859804: 5
859804: 6
859804: 7
859803: 8
859803: 9

The error remains the same:
RuntimeError: Could not push packet to decoder: Invalid data found when processing input

Edit:
Making sure the two threads perform alternating reads, leads to the same error after 6 reads.