Support the free-threaded build

[ngoldbaum]

See #406 (comment).

It may be a little presumptuous to add wheel builds already but it looks like that's how CI is set up in this repo. It may also make sense to add some more tests that use either rayon or the python threading module. That said, IMO you'd be testing guarantees provided by PyO3 more than tiktoken itself.

[ngoldbaum]

From the cibuildwheel logs:

<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'regex._regex', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

https://github.com/ngoldbaum/tiktoken/actions/runs/17308679370/job/49137906087#step:3:1256

See #406 (comment) and replies for more about this.

For now, maybe I should set PYTHON_GIL=0 for the cp313t build?

[ngoldbaum]

Let me know if you'd like me to split off 1ecd757 into its own PR without the cp314t builds.

[hauntsaninja]

Thank you! (We have some tests that do end up lightly using threads internally, so I guess if we switch to a free threaded build internally we'll get a little more coverage)

[ngoldbaum]

so I guess if we switch to a free threaded build internally we'll get a little more coverage

Cool! We have a discord you can join if you want to chat more or ask questions in realtime: https://py-free-threading.github.io/#news-and-getting-help