will the output from DummyGPTModel the same regardless of torch.nn version?

[Jessen-Li]

In chapter 4.1, the model output is:
tensor([[[-1.2034, 0.3201, -0.7130, ..., -1.5548, -0.2390, -0.4667],
[-0.1192, 0.4539, -0.4432, ..., 0.2392, 1.3469, 1.2430],
[ 0.5307, 1.6720, -0.4695, ..., 1.1966, 0.0111, 0.5835],
[ 0.0139, 1.6755, -0.3388, ..., 1.1586, -0.0435, -1.0400]],
[[-1.0908, 0.1798, -0.9484, ..., -1.6047, 0.2439, -0.4530],
[-0.7860, 0.5581, -0.0610, ..., 0.4835, -0.0077, 1.6621],
[ 0.3567, 1.2698, -0.6398, ..., -0.0162, -0.1296, 0.3717],
[-0.2407, -0.7349, -0.5102, ..., 2.0057, -0.3694, 0.1814]]], grad_fn=)
I tried in my computer and generates different output. Is it normal because of different version of PyTorch perhaps?

[d-kleine]

Yeah, it seems like the logits will be computed slightly different in newer PyTorch versions. I am using torch 2.5.1 and the output is

Output shape: torch.Size([2, 4, 50257])
tensor([[[-0.9289,  0.2748, -0.7557,  ..., -1.6070,  0.2702, -0.5888],
         [-0.4476,  0.1726,  0.5354,  ..., -0.3932,  1.5285,  0.8557],
         [ 0.5680,  1.6053, -0.2155,  ...,  1.1624,  0.1380,  0.7425],
         [ 0.0447,  2.4787, -0.8843,  ...,  1.3219, -0.0864, -0.5856]],

        [[-1.5474, -0.0542, -1.0571,  ..., -1.8061, -0.4494, -0.6747],
         [-0.8422,  0.8243, -0.1098,  ..., -0.1434,  0.2079,  1.2046],
         [ 0.1355,  1.1858, -0.1453,  ...,  0.0869, -0.1590,  0.1552],
         [ 0.1666, -0.8138,  0.2307,  ...,  2.5035, -0.3055, -0.3083]]],
       grad_fn=<UnsafeViewBackward0>)

[casinca]

@Jessen-Li @d-kleine I am getting the same results with PyTorch 2.1 to 2.6. I think it mainly depends on the OS and how PyTorch was compiled. I am quite sure that it is due to the dropout issue here in PyTorch: pytorch/pytorch#121595

Ye I believe your intuition is right, I just tested on my 2 machines with fresh cloned repos.
I'm getting same result as @d-kleine on Windows and same result as @Jessen-Li on Mac

[rasbt]

Oh nice, thanks for looking into that and confirming!

[Jessen-Li]

@Jessen-Li @d-kleine I am getting the same results with PyTorch 2.1 to 2.6. I think it mainly depends on the OS and how PyTorch was compiled. I am quite sure that it is due to the dropout issue here in PyTorch: pytorch/pytorch#121595

Ye I believe your intuition is right, I just tested on my 2 machines with fresh cloned repos. I'm getting same result as @d-kleine on Windows and same result as @Jessen-Li on Mac

I'm using Ubuntu 22.04

[d-kleine]

I have the same output

tensor([[[-0.9289,  0.2748, -0.7557,  ..., -1.6070,  0.2702, -0.5888],
         [-0.4476,  0.1726,  0.5354,  ..., -0.3932,  1.5285,  0.8557],
         [ 0.5680,  1.6053, -0.2155,  ...,  1.1624,  0.1380,  0.7425],
         [ 0.0447,  2.4787, -0.8843,  ...,  1.3219, -0.0864, -0.5856]],

        [[-1.5474, -0.0542, -1.0571,  ..., -1.8061, -0.4494, -0.6747],
         [-0.8422,  0.8243, -0.1098,  ..., -0.1434,  0.2079,  1.2046],
         [ 0.1355,  1.1858, -0.1453,  ...,  0.0869, -0.1590,  0.1552],
         [ 0.1666, -0.8138,  0.2307,  ...,  2.5035, -0.3055, -0.3083]]],
       grad_fn=<UnsafeViewBackward0>)

on Ubuntu as on Windows as they both use MKL (Intel Math Kernel Library). The output is different on Mac because it uses other libraries.

[Jessen-Li]

Yeah, it seems like the logits will be computed slightly different in newer PyTorch versions. I am using torch 2.5.1 and the output is

Output shape: torch.Size([2, 4, 50257])
tensor([[[-0.9289,  0.2748, -0.7557,  ..., -1.6070,  0.2702, -0.5888],
         [-0.4476,  0.1726,  0.5354,  ..., -0.3932,  1.5285,  0.8557],
         [ 0.5680,  1.6053, -0.2155,  ...,  1.1624,  0.1380,  0.7425],
         [ 0.0447,  2.4787, -0.8843,  ...,  1.3219, -0.0864, -0.5856]],

        [[-1.5474, -0.0542, -1.0571,  ..., -1.8061, -0.4494, -0.6747],
         [-0.8422,  0.8243, -0.1098,  ..., -0.1434,  0.2079,  1.2046],
         [ 0.1355,  1.1858, -0.1453,  ...,  0.0869, -0.1590,  0.1552],
         [ 0.1666, -0.8138,  0.2307,  ...,  2.5035, -0.3055, -0.3083]]],
       grad_fn=<UnsafeViewBackward0>)

After correcting the vocab_size to 50257, my output is the same as yours.

[d-kleine]

Ok cool 👍🏻 🙂

[rasbt]

Oh nice! Glad this resolved itself :)

[Jessen-Li]

Does this difference in the initial stage matter? Or it will diminish with the ongoing training?

[rasbt]

Good question. So we are not using the DummyGPT model further. However, since we are using dropout for the main model as well, it could potentially affect it. Based on my experiments though, running the training on macOS and on a GPU Linux machine, the results were almost identical as far as I remember.

[d-kleine]

Also, the dropout was relatively small ("drop_rate": 0.1)