Python 3.14 support and dependency pinning and deprecation warning policies

[gotmax23]

Description

Please leave a brief description of the bug or feature request:

Fedora is proud to be an early adopter of Python 3.14 as part of our goal to help the ecosystem by testing and integrating new Python versions, and I worked on getting all dependencies functional and addressing test failures in the scancode stack. I am creating this issue to track supporting Python 3.14 officially upstream. We only run a subset of upstream scancode tests, but this reflects new failures after the Python 3.14 update.

There are also some concerns I have with dependency pinning and deprecation warning practices in the aboutcode projects that I mentioned that created extra friction when updating to Python 3.14. See the Commentary below.

scancode issues

• packagedcode: don't use removed ast module attributes #4539 that I filed earlier today was needed due to ast module removals.

• We ignored the remaining test failures in our scancode build:

  # ERROR: 'sbin/fstrim' is a link to an absolute path
  # These two are due to the tarfile module defaulting to the data filter in py3.14.
  # Let upstream figure out what to do about this issue.
  test_can_get_installed_system_packages_with_license_from_alpine_container_layer
  test_can_scan_installed_system_package_in_alpine_container_layer
  # These test failures have to do with the handling of tempfiles in the CLI
  # and don't seem particularly important. Disable them for now
  test_scan_keep_temp_files_is_false_by_default
  test_scan_keep_temp_files_keeps_files and not

• test_license_reference_to_file_beside_package_manifest test is broken #4540 is not directly caused by Python 3.14 but also impacted us while trying to update scancode to the latest version.

Commentary: Both the ast and tarfile issues were due to known deprecation that upstream seemingly ignored, so perhaps aboutcode should consider adopting a more proactive approach to addressing deprecation warnings (such as running pytest with -W error) and avoid pinning old dependency versions ¹. I'm not sure what's going on with the temp_files tests. I didn't take the time to debug them, and I'm not sure if the issue is with the Fedora build environment.

dependency issues

• I replaced pkginfo2 with pkginfo in scancode with a downstream patch. pkginfo2 seems to be an under-maintained fork of pkginfo and has failures with new setuptools versions that prevented it from being rebuilt for Python 3.14. These were reported in Tests are failing with new setuptools due to new metadata version pkginfo2#5 but never addressed upstream.
• Test failure on Python 3.14 commoncode#88 requires disabling an additional test in our commoncode package. This was also caused by tarfile changes.
• Issues with beartype prevented spdx-tools from being rebuilt. These were fixed in [Feature Request] Official Python 3.14 support! It do be like: ☢ ☣ ⚠ 🦖  beartype/beartype#537, but a new upstream release has not yet been created.
• Once we updated beartype to the latest git snapshot, I backported spdx/tools-python@6817ca7 for beartype compatibility and submitted relationship_writer: properly access __annotations__ dict spdx/tools-python#858 to spdx-tools to fix other issues that came up in the spdx-tools package. The spdx-tools maintainers have not yet applied my patch.

Commentary: The beartype issues were understandable due to major typing changes in Python 3.14. I'm not sure about the state of the pkginfo2 project or spdx-tools.

How To Reproduce

Tell us how to reproduce the issue.

Run scancode tests in an environment with Python 3.14. Make sure to install the latest git snapshots of https://github.com/gotmax23/tools-python/tree/py314-annotations and https://github.com/beartype/beartype that have the patches mentioned.

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? (Windows/MacOS/Linux) Fedora Linux Rawhide (44)
• What version of scancode-toolkit was used to generate the scan file? 32.4.1
• What installation method was used to install/run scancode? (pip/source download/other) source

Thank you for reading my wall of text and for all the work you do on scancode!

Footnotes

1. EDIT: To be clear, I was referring to pinning of test dependencies for CI in requirements.txt, not the dependencies in the packaging metadata (setup.cfg) which are specified properly without excessive pinning for the most part. After looking again, it seems like requirements.txt does get updated on a somewhat regular basis but importlib-metadata was just held back in https://github.com/aboutcode-org/scancode-toolkit/commit/97bc9b5a6e752d06124280c66d5965b6be723293. Using something like dependabot or a Github Actions scheduled pipeline to automatically submit PRs to bump requirements.txt is still a good idea. ↩

[reneleonhardt]

Great to see Python 3.14 support!

I can't even install it on Python 3.13 😅

uvx scancode-toolkit 
      Built ftfy==4.4.3                                                                                                                                                                × Failed to build `bitarray==0.9.3`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)

      [stderr]
      /Users/rene/.cache/uv/builds-v0/.tmp5o48Gd/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!

              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:

              License :: OSI Approved :: Python Software Foundation License

              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************

      !!
        self._finalize_license_expression()
      bitarray/_bitarray.c:131:23: error: expression is not assignable
        131 |         Py_SIZE(self) = newsize;
            |         ~~~~~~~~~~~~~ ^
      bitarray/_bitarray.c:156:19: error: expression is not assignable
        156 |     Py_SIZE(self) = newsize;
            |     ~~~~~~~~~~~~~ ^
      bitarray/_bitarray.c:177:18: error: expression is not assignable
        177 |     Py_SIZE(obj) = nbytes;
            |     ~~~~~~~~~~~~ ^
      bitarray/_bitarray.c:1372:14: error: call to undeclared function 'PyEval_CallObject'; ISO C99 and later do not support implicit function declarations
      [-Wimplicit-function-declaration]
       1372 |     result = PyEval_CallObject(reader, rargs);
            |              ^
      bitarray/_bitarray.c:1372:12: error: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
       1372 |     result = PyEval_CallObject(reader, rargs);
            |            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      bitarray/_bitarray.c:1511:14: error: call to undeclared function 'PyEval_CallObject'; ISO C99 and later do not support implicit function declarations
      [-Wimplicit-function-declaration]
       1511 |     result = PyEval_CallObject(writer, args);
            |              ^
      bitarray/_bitarray.c:1511:12: error: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
       1511 |     result = PyEval_CallObject(writer, args);
            |            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      bitarray/_bitarray.c:3206:28: error: expression is not assignable
       3206 |     Py_TYPE(&Bitarraytype) = &PyType_Type;
            |     ~~~~~~~~~~~~~~~~~~~~~~ ^
      bitarray/_bitarray.c:3207:31: error: expression is not assignable
       3207 |     Py_TYPE(&SearchIter_Type) = &PyType_Type;
            |     ~~~~~~~~~~~~~~~~~~~~~~~~~ ^
      bitarray/_bitarray.c:3208:31: error: expression is not assignable
       3208 |     Py_TYPE(&DecodeIter_Type) = &PyType_Type;
            |     ~~~~~~~~~~~~~~~~~~~~~~~~~ ^
      bitarray/_bitarray.c:3209:33: error: expression is not assignable
       3209 |     Py_TYPE(&BitarrayIter_Type) = &PyType_Type;
            |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
      11 errors generated.
      error: command '/usr/bin/cc' failed with exit code 1

      hint: This usually indicates a problem with the package or the build environment.
  help: `bitarray` (v0.9.3) was included because `scancode-toolkit` (v21.2.9) depends on `bitarray`

bitarray v0.9.3 is 6 years old: https://pypi.org/project/bitarray/#history

I can only find outdated test data:
https://github.com/aboutcode-org/scancode-toolkit/blob/develop/tests/packagedcode/data/pypi/develop/scancode_toolkit.egg-info/requires.txt#L3

bitarray<1.0.0,>=0.8.1

But of course nobody should use requirements pinning anymore, exactly why packages can't be installed on newer Python versions anymore when updates are forbidden.
https://iscinumpy.dev/post/bound-version-constraints/

[gotmax23]

help: bitarray (v0.9.3) was included because scancode-toolkit (v21.2.9) depends on bitarray

The latest version of scancode-toolkit is 32.4.1 which should support Python 3.13. You are trying to install 21.2.9 from over four years ago. It looks like you are having a separate issue and should create a new ticket for that.

But of course nobody should use requirements pinning anymore, exactly why packages can't be installed on newer Python versions anymore when updates are forbidden.

I was referring to the fact that tests are all run in upstream CI using library versions pinned in requirements.txt, which is generally a good practice, provided that they are updated on a regular basis and not pinned to versions that are significantly out of date.

Pinning bitarray<1.0.0,>=0.8.1 in the packaging metadata probably made sense at the time if it was expected that bitarray 1.0.0 would have breaking changes.

[reneleonhardt]

You can see my command line, that's what uvx resolves to when ommitting the version.
But

uvx scancode-toolkit==32.4.1

isn't much better for Python 3.13.

  × No solution found when resolving tool dependencies:
  ╰─▶ Because only the following versions of typecode-libmagic are available:
          typecode-libmagic<=5.39.210223
          typecode-libmagic==5.39.210531
      and typecode-libmagic>=5.39.210223 has no wheels with a matching platform tag (e.g., `macosx_15_0_arm64`), we can conclude that typecode-libmagic>=5.39.210223 cannot be
      used. (1)
      And because typecode[full]>=30.0.1 depends on typecode-libmagic>=5.39.210223 and only the following versions of typecode[full] are available:
          typecode[full]<=30.0.1
          typecode[full]==30.0.2
      we can conclude that typecode[full]>=30.0.1,<30.0.2 cannot be used. (2)

      Because we know from (1) that typecode-libmagic>=5.39.210223 cannot be used and typecode[full]==30.0.2 depends on typecode-libmagic>=5.39.210223, we can conclude that
      typecode[full]==30.0.2 cannot be used.
      And because we know from (2) that typecode[full]>=30.0.1,<30.0.2 cannot be used, we can conclude that typecode[full]>=30.0.1 cannot be used.
      And because scancode-toolkit==32.4.1 depends on typecode[full]>=30.0.1 and you require scancode-toolkit==32.4.1, we can conclude that your requirements are unsatisfiable.

      hint: Wheels are available for `typecode-libmagic` (v5.39.210531) on the following platforms: `manylinux1_x86_64`, `macosx_10_14_x86_64`, `win_amd64`

But I didn't want to steal your thread, didn't know that you don't support arm64 or Apple Silicon yet, typecode-libmagic doesn't seem to be maintained anymore, last release is 4 years old:
https://pypi.org/project/typecode-libmagic/#files.

I can only recommend using the standard cibuildwheel action if possible to build for all platforms and architectures.
https://github.com/pypa/cibuildwheel#example-setup

[stefan6419846]

But I didn't want to steal your thread, didn't know that you don't support arm64 or Apple Silicon yet, typecode-libmagic doesn't seem to be maintained anymore, last release is 4 years old

This is a separate issue and is tracked in #3958.

[pombredanne]

@gotmax23 Thanks for all this. This is awesome! As for pkginfo2, it needs to be updated, but the reason for the fork and patch is security: pkginfo does insecure evaluation of code that is possibly dangerous when processing random code at scale. It may be OK for pkginfor, but is not OK for ScanCode