Skip to content

Conversation

@kudos07
Copy link

@kudos07 kudos07 commented Nov 1, 2025

Summary

This draft PR introduces a parametrized test suite for steerable models
covering multiple Hugging Face models under the Transformers backend.

It focuses on one aspect of issue #1717:

  • Expands initialization and inference coverage to multiple models.
  • Runs each inference several times to surface flaky behavior.
  • Adds a shared Hugging Face cache fixture to prevent rate-limit errors.
  • Includes a backend parameter to allow future vLLM/llama_cpp coverage.

Notes

  • vLLM tests are skipped on Windows; CI can re-enable them.
  • This is a draft PR as discussed with @RobinPicard to align on test organization before expanding further.

@RobinPicard
Copy link
Contributor

Thanks for opening a PR @kudos07! I think maybe it would make sense to keep those test separated by Model as in your commit the code of the test is made for Transformers for instance. What I think we could do is to have in the test file for each Model, a specific model used for all the tests testing the various features and then a parametrized test with many models as the one you have here that would initialize and run a standard constrained generation.

To me one of the main interest of that would be to check that we do run into tokenizer issues for various models as I've noticed it's often where there are problems.

@kudos07
Copy link
Author

kudos07 commented Nov 13, 2025

Update based on review feedback:

  1. Moved the parametrized test into the Transformers backend test suite at tests/models/test_transformers_parametrized.py, since this test targets the Transformers model implementation specifically.

  2. Kept the parametrized set of Hugging Face checkpoints to validate initialization and basic constrained generation across multiple tokenizers.

  3. Added a small tokenizer sanity check (assert tokenizer.vocab_size > 0) to address the note that tokenizer inconsistencies often cause failures.

  4. Left the vLLM entry in the parametrization but skipped on Windows, documenting intended future backend coverage once platform support allows it.

Let me know if you'd prefer this test split differently or want it integrated with any existing Transformers test files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants