Tracking full integration for fill-in-middle (FIM)

[sayakpaul]

Given that most code LLMs are trained using the FIM objective [1], I think it makes a lot of sense to work on:

• A training example just like how we have it here: https://github.com/huggingface/transformers/tree/main/examples/pytorch. This can also be turned into a task guide later (example: https://huggingface.co/docs/transformers/tasks/language_modeling).
• A dedicated pipeline so that users can load FIM-trained models easily with pipeline("fill-in-middle").
• A task page to list all the relevant resources.

Cc @ArthurZucker since we discussed it internally via Slack.

References

[1] Efficient Training of Language Models to Fill in the Middle, https://arxiv.org/abs/2207.14255.

[tanaymeh]

Hi @sayakpaul, I would like to work on this issue!

[sayakpaul]

Awesome!

Pinging @ArthurZucker to check if we should start with pipeline or the example.

[ArthurZucker]

Both are independent, not sure which one will be the easiest

[tanaymeh]

I will start with the example resource!
Do you recommend I check out any existing resources examples/notebooks on FIM?

Thanks!

[sayakpaul]

Yes!

[tanaymeh]

Hi @sayakpaul, I was busy with some work hence starting this now.

Is there a list of models in HF transformers that support FIM? Or perhaps a faster way of identifying which models support that (via a tag or something similar)?

[sayakpaul]

Cc: @pacman100

[tanaymeh]

Hi @sayakpaul, is there a set standard way of naming FIM tokens across tokenizers of all models that support this objective? (for example, is it certain that all prefix tokens will be <fim_prefix>, etc?)

If not, should I add the prefix, suffix and middle FIM token names as data arguments in the example script (and default to <fim_prefix>, <fim_suffix>, and <fim_middle> if not provided)?

EDIT:
In the latter case, there are two possibilities:

1. Either the tokenizer doesn't have those tokens (in which case we add these tokens to the tokenizer vocab and resize model embeddings)
2. The user forgot to provide them (same treatment as the first case along with a warning?).

[tanaymeh]

I have created a draft PR for the example. Also have started the work on the pipeline (will create a PR a little later).

[ArthurZucker]

You can check CodeLlama which sets the precedent for direct integration, but we can also compare starCoder, which does not have this directly in the tokenizers. Fine with setting new standards as the pipeline PR progresses!

[tanaymeh]

@ArthurZucker Cool! CodeLlama seems like a great option to start.