Toy example: Train on a dataset

[Iosifts]

I do have a simple question:

If we train a llm with 10e100000 repetitions of the sentence: 'there is a cat on the sofa'
like explicitly sentence(x)=there(x1), is(x2),...sofa(x7), feeding the first six and let it predict the 7th
In the end, you have a pre-trained model which all it does is text completion
if you input "there is a cat on the" it will say sofa
but if you just say "there" it will output "is", cause it is the only thing it knows, furthermore if you input "is a cat" it will trigger the problem Y = P(y|cat, a, is) and this is modeled to be Y = "on" cause it is trained implicitly for this data when watching the corpus and changing all the rows of the W_keys, W_q, Wv, not onlly the last one.

Is my thought accurate or not?
Thanks in advance!

[rasbt]

This is actually a really nice description of the next-word prediction task in pretraining! In practice, that's why it's so important to have a large and diverse dataset. But yes, your understanding is spot on there.

[Iosifts]

Thanks a lot for the prompt answer! This is nice to confirm, due to the whole idea of 'language modelling', but as we use only the softmax at the top of the last representation/token/step I don't see explicitly how the previous ones are modelled... As there is no softmax for the previous tokens, just in the last one. Do you get my point? Thanks a lot for the nice book btw!

[rasbt]

Good question, at a given position, it's the last token like you said. But you have all the positions in parallel, i.e.,

• there -> is
• there is -> a
• there is a -> cat
• ...

So you model each position in the text.

[Iosifts]

Oh I see, so the fact that the loss is calculated per batch leads to modeling the distribution of the training dataset! Thanks a lot Sebastian!

[rasbt]

Glad it makes sense now!