Refactor example prompts fixture

[nwaughachukwuma]

Purpose

I'm taking on the task of improving conftest.py, focusing on type definitions and general code & fixtures improvements, with the objective of creating a better overall testing experience from the perspectives of readability/approachability and test completion time.

This PR focuses on two things:

1. Using list comprehension for example_prompts without list concatenation.

This change is not trivial as it might appear, given that this fixture is heavily used across multiple tests suites

# Before
@pytest.fixture
def example_prompts() -> list[str]:
    prompts = []
    for filename in _TEST_PROMPTS:
        prompts += _read_prompts(filename)
    return prompts

# After
@pytest.fixture
def example_prompts() -> list[str]:
    return [prompt for filename in _TEST_PROMPTS for prompt in _read_prompts(filename)]

2. Improving type definitions and type inference for better IDE type hints

a. tokenizer

from transformers import PreTrainedTokenizer, PreTrainedTokenizerFast

if not skip_tokenizer_init:
    self.tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast = AutoTokenizer.from_pretrained(...)

b. self.model.generate(...)

from transformers.generation.utils import GenerateOutput

for inputs in all_inputs:
    output: GenerateOutput = self.model.generate(..., return_dict_in_generate=True)
    ...

for inputs in all_inputs:
    output_ids: torch.Tensor = self.model.generate(...)

Test Plan

N/A

Test Result

N/A