Skip to content

gh-86353: Deprecate several tokens #138118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

gh-86353: Deprecate several tokens #138118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
48b6cb1
Commit
StanFromIreland Aug 24, 2025
e760fdf
Commit
StanFromIreland Aug 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Doc/deprecations/pending-removal-in-3.17.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,8 @@ Pending removal in Python 3.17
but it has been retained for backward compatibility, with removal scheduled for Python
3.17. Users should use documented introspection helpers like :func:`typing.get_origin`
and :func:`typing.get_args` instead of relying on private implementation details.

* :mod:`token`:

- The :func:`token.ISTERMINAL`, :func:`token.ISNONTERMINAL` and
:func:`token.ISEOF` tokens.
6 changes: 6 additions & 0 deletions Doc/library/token.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,22 @@ a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`.

Return ``True`` for terminal token values.

.. deprecated-removed:: next 3.17


.. function:: ISNONTERMINAL(x)

Return ``True`` for non-terminal token values.

.. deprecated-removed:: next 3.17


.. function:: ISEOF(x)

Return ``True`` if *x* is the marker indicating the end of input.

.. deprecated-removed:: next 3.17


The token constants are:

Expand Down
9 changes: 9 additions & 0 deletions Doc/whatsnew/3.15.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -440,6 +440,15 @@ hashlib
(Contributed by Bénédikt Tran in :gh:`134978`.)


token
-----

* The :func:`token.ISTERMINAL`, :func:`token.ISNONTERMINAL` and
:func:`token.ISEOF` tokens are now deprecated, and will be removed
in Python 3.17.
(Contributed by Stan Ulbrych in :gh:`86353`)


.. Add deprecations above alphabetically, not here at the end.

Removed
Expand Down
6 changes: 3 additions & 3 deletions Include/internal/pycore_token.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,9 @@ extern "C" {

/* Special definitions for cooperation with parser */

#define ISTERMINAL(x) ((x) < NT_OFFSET)
#define ISNONTERMINAL(x) ((x) >= NT_OFFSET)
#define ISEOF(x) ((x) == ENDMARKER)
#define ISTERMINAL(x) ((x) < NT_OFFSET) /* Deprecated and will be removed in 3.17 */
#define ISNONTERMINAL(x) ((x) >= NT_OFFSET) /* Deprecated and will be removed in 3.17 */
#define ISEOF(x) ((x) == ENDMARKER) /* Deprecated and will be removed in 3.17 */
#define ISWHITESPACE(x) ((x) == ENDMARKER || \
(x) == NEWLINE || \
(x) == INDENT || \
Expand Down
9 changes: 9 additions & 0 deletions Lib/token.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2025-08-24-17-54-22.gh-issue-86353.bw0iat.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Deprecate and schedule for removal in 3.17, :func:`token.ISTERMINAL`,
:func:`token.ISNONTERMINAL` and :func:`token.ISEOF`.
15 changes: 12 additions & 3 deletions Tools/build/generate_token.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ def update_file(file, content):
/* Special definitions for cooperation with parser */
#define ISTERMINAL(x) ((x) < NT_OFFSET)
#define ISNONTERMINAL(x) ((x) >= NT_OFFSET)
#define ISEOF(x) ((x) == ENDMARKER)
#define ISTERMINAL(x) ((x) < NT_OFFSET) /* Deprecated and will be removed in 3.17 */
#define ISNONTERMINAL(x) ((x) >= NT_OFFSET) /* Deprecated and will be removed in 3.17 */
#define ISEOF(x) ((x) == ENDMARKER) /* Deprecated and will be removed in 3.17 */
#define ISWHITESPACE(x) ((x) == ENDMARKER || \\
(x) == NEWLINE || \\
(x) == INDENT || \\
Expand Down Expand Up @@ -279,12 +279,21 @@ def make_rst(infile, outfile='Doc/library/token-list.inc',
}
def ISTERMINAL(x: int) -> bool:
import warnings
warnings.warn('token.ISTERMINAL is deprecated and will be removed in 3.17',
DeprecationWarning, stacklevel=2)
return x < NT_OFFSET
def ISNONTERMINAL(x: int) -> bool:
import warnings
warnings.warn('token.ISNONTERMINAL is deprecated and will be removed in 3.17',
DeprecationWarning, stacklevel=2)
return x >= NT_OFFSET
def ISEOF(x: int) -> bool:
import warnings
warnings.warn('token.ISEOF is deprecated and will be removed in 3.17',
DeprecationWarning, stacklevel=2)
return x == ENDMARKER
'''

Expand Down
Loading