Skip to content

Integration of a new optional package for Khovanov homology #40081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from

Conversation

soehms
Copy link
Member

@soehms soehms commented May 10, 2025

Sage has a native implementation of Khovanov homology. Unfortunately, it's not very performant. For knots and links with more than six crossings, the calculation takes more than a second on a ThinkPad with an i7 (i7-1165G7 @ 2.80GHz) processor (running Ubuntu under WSL2). For a knot with nine crossings, it takes more than a minute, and for 10 crossings, the calculation breaks (after about an hour) with a Pari stack overflow:

sage: K7 = Knots().from_table(7,1)
....: K8 = Knots().from_table(8,1)
....: K9 = Knots().from_table(9,1)
....: K10 = Knots().from_table(10,1)
sage: %time K7.khovanov_homology()
CPU times: user 679 ms, sys: 0 ns, total: 679 ms
Wall time: 679 ms
{-21: {-7: Z},
 -19: {-7: 0, -6: C2},
 -17: {-7: 0, -6: Z, -5: Z},
 -15: {-7: 0, -6: 0, -5: 0, -4: C2},
 -13: {-7: 0, -6: 0, -5: 0, -4: Z, -3: Z},
 -11: {-7: 0, -6: 0, -5: 0, -4: 0, -3: 0, -2: C2},
 -9: {-7: 0, -6: 0, -5: 0, -4: 0, -3: 0, -2: Z, -1: 0, 0: 0},
 -7: {-7: 0, -6: 0, -5: 0, -4: 0, -3: 0, -2: 0, -1: 0, 0: Z},
 -5: {0: Z}}
sage: %time K8.khovanov_homology()
CPU times: user 22.9 s, sys: 74.1 ms, total: 23 s
Wall time: 23 s
{-13: {-6: Z},
 -11: {-6: 0, -5: C2, -4: 0, -3: 0},
 -9: {-6: 0, -5: Z, -4: Z, -3: 0, -2: 0, -1: 0},
 -7: {-6: 0, -5: 0, -4: 0, -3: Z x C2, -2: 0, -1: 0, 0: 0, 1: 0},
 -5: {-6: 0, -5: 0, -4: 0, -3: Z, -2: Z x C2, -1: 0, 0: 0, 1: 0, 2: 0},
 -3: {-6: 0, -5: 0, -4: 0, -3: 0, -2: Z, -1: Z x C2, 0: 0, 1: 0, 2: 0, 3: 0},
 -1: {-4: 0, -3: 0, -2: 0, -1: Z, 0: Z x C2, 1: 0, 2: 0, 3: 0, 4: 0},
 1: {-3: 0, -2: 0, -1: 0, 0: Z x Z, 1: Z, 2: 0, 3: 0, 4: 0},
 3: {-1: 0, 0: 0, 1: 0, 2: C2, 3: 0, 4: 0},
 5: {1: 0, 2: Z, 3: 0, 4: 0}}
sage: %time K9.khovanov_homology()
CPU times: user 1min 23s, sys: 260 ms, total: 1min 23s
Wall time: 1min 23s
{-27: {-9: Z},
 -25: {-9: 0, -8: C2},
 -23: {-9: 0, -8: Z, -7: Z},
 -21: {-9: 0, -8: 0, -7: 0, -6: C2},
 -19: {-9: 0, -8: 0, -7: 0, -6: Z, -5: Z},
 -17: {-9: 0, -8: 0, -7: 0, -6: 0, -5: 0, -4: C2},
 -15: {-9: 0, -8: 0, -7: 0, -6: 0, -5: 0, -4: Z, -3: Z},
 -13: {-9: 0, -8: 0, -7: 0, -6: 0, -5: 0, -4: 0, -3: 0, -2: C2},
 -11: {-9: 0, -8: 0, -7: 0, -6: 0, -5: 0, -4: 0, -3: 0, -2: Z, -1: 0, 0: 0},
 -9: {-9: 0, -8: 0, -7: 0, -6: 0, -5: 0, -4: 0, -3: 0, -2: 0, -1: 0, 0: Z},
 -7: {0: Z}}
sage: %time K10.khovanov_homology()
Traceback (most recent call last):
...
PariError: the PARI stack overflows (current size: 1073741824; maximum size: 1073741824)
You can use pari.allocatemem() to change the stack size and try again

If we do the same with the optional package Khoca implemented in this PR, we get the following: The knot with 9 crossings takes less than half a second and the example with 10 crossings terminates after less than two seconds:

sage: %time K9.khovanov_homology(implementation='Khoca')
CPU times: user 268 ms, sys: 44.7 ms, total: 313 ms
Wall time: 293 ms
{-27: {-9: Z},
 -25: {-9: 0, -8: C2},
 -23: {-9: 0, -8: Z, -7: Z},
 -21: {-8: 0, -7: 0, -6: C2},
 -19: {-7: 0, -6: Z, -5: Z},
 -17: {-6: 0, -5: 0, -4: C2},
 -15: {-5: 0, -4: Z, -3: Z},
 -13: {-4: 0, -3: 0, -2: C2},
 -11: {-3: 0, -2: Z},
 -9: {0: Z},
 -7: {0: Z}}
sage: %time K10.khovanov_homology(implementation='Khoca')
CPU times: user 1.55 s, sys: 123 ms, total: 1.68 s
Wall time: 1.65 s
{-17: {-8: Z},
 -15: {-8: 0, -7: C2},
 -13: {-8: 0, -7: Z, -6: Z},
 -11: {-7: 0, -6: 0, -5: Z x C2},
 -9: {-6: 0, -5: Z, -4: Z x C2},
 -7: {-5: 0, -4: Z, -3: Z x C2},
 -5: {-4: 0, -3: Z, -2: Z x C2},
 -3: {-3: 0, -2: Z, -1: Z x C2},
 -1: {-2: 0, -1: Z, 0: Z x C2},
 1: {-1: 0, 0: Z x Z, 1: Z},
 3: {0: 0, 1: 0, 2: C2},
 5: {1: 0, 2: Z}}

Even knots with 13 crossing only need a few seconds:

sage: K13 = KnotInfo.K13a_1.link()
sage: %time K13.khovanov_homology(implementation='Khoca')
CPU times: user 2.28 s, sys: 578 ms, total: 2.86 s
Wall time: 2.6 s
{-17: {-8: Z},
 -15: {-8: 0, -7: Z x Z x Z x C2},
 -13: {-8: 0, -7: Z, -6: Z^7 x C2 x C2 x C2},
 -11: {-7: 0, -6: Z x Z x Z, -5: Z^13 x C2^7},
 -9: {-6: 0, -5: Z^7, -4: Z^19 x C2^13},
 -7: {-5: 0, -4: Z^13, -3: Z^24 x C2^19},
 -5: {-4: 0, -3: Z^19, -2: Z^27 x C2^24},
 -3: {-3: 0, -2: Z^24, -1: Z^25 x C2^27},
 -1: {-2: 0, -1: Z^27, 0: Z^22 x C2^25},
 1: {-1: 0, 0: Z^26, 1: Z^16 x C2^21},
 3: {0: 0, 1: Z^21, 2: Z^9 x C2^16},
 5: {1: 0, 2: Z^16, 3: Z x Z x Z x Z x C2^9},
 7: {2: 0, 3: Z^9, 4: Z x C2 x C2 x C2 x C2},
 9: {3: 0, 4: Z x Z x Z x Z, 5: C2},
 11: {4: 0, 5: Z}}

The software behind the optional package is Khoca. It was written in C++ by Lukas Lewark. I packaged it on PyPI for integration into Sage (see LLewark/khoca#3).

Currently, there is still an issue with the PyPI package regarding integration with Sage, as both use the Pari library (see LLewark/khoca#3 (comment)). This occurs sporadically and triggers a system error that occurs at cypari2/stack.pyx. The error disappears when the last action is re-executed. I'm leaving this PR as a draft until the issue is resolved.

📝 Checklist

  • The title is concise and informative.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation and checked the documentation preview.

⌛ Dependencies

Copy link

github-actions bot commented May 10, 2025

Documentation preview for this PR (built with commit 8644000; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

@soehms soehms force-pushed the khoca_integration branch from c56ba18 to 213d7d4 Compare May 11, 2025 06:07
@soehms
Copy link
Member Author

soehms commented May 12, 2025

I'm quite surprised to see another segmentation fault. It occurs in the CI-Linux-incremental-run on all Ubuntu and Debian jobs in Line 1293 of src/sage/knots/link.py:

            sage: K.khovanov_homology(implementation='Khoca')                           # optional khoca, needs sage.modules
            {-9: {-3: Z}, -7: {-3: 0, -2: C2}, -5: {-3: 0, -2: Z}, -3: {0: Z}, -1: {0: Z}}

From the log-file:

2025-05-11T07:22:31.6912136Z sage: K.khovanov_homology(implementation='Khoca')                           # optional khoca, needs sage.modules ## line 1293 ##
2025-05-11T07:22:31.6913383Z Traceback (most recent call last):
...
2025-05-11T07:22:31.6924156Z   File "<doctest sage.knots.link.Link.khovanov_homology[2]>", line 1, in <module>
2025-05-11T07:22:31.6925109Z   File "/sage/src/sage/knots/link.py", line 1376, in khovanov_homology
2025-05-11T07:22:31.6926844Z           
2025-05-11T07:22:31.6926045Z     homologies = {j: dict(self._khovanov_homology_cached(j, implementation, ring, **kwds)) for j in heights}
2025-05-11T07:22:31.6926844Z                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-11T07:22:31.6927635Z   File "sage/misc/cachefunc.pyx", line 1975, in sage.misc.cachefunc.CachedMethodCaller.__call__
2025-05-11T07:22:31.6928381Z     w = self._instance_call(*args, **kwds)
2025-05-11T07:22:31.6929160Z   File "sage/misc/cachefunc.pyx", line 1851, in sage.misc.cachefunc.CachedMethodCaller._instance_call
2025-05-11T07:22:31.6929998Z     return self.f(self._instance, *args, **kwds)
2025-05-11T07:22:31.6930680Z   File "/sage/src/sage/knots/link.py", line 1204, in _khovanov_homology_cached
2025-05-11T07:22:31.6931368Z     res += [(d, HomologyGroup(len(ifac), ring, ifac))]
2025-05-11T07:22:31.6931857Z                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-11T07:22:31.6932500Z   File "/sage/src/sage/homology/homology_group.py", line 187, in HomologyGroup
2025-05-11T07:22:31.6933181Z     return HomologyGroup_class(n, invfac)
2025-05-11T07:22:31.6960252Z   File "/sage/src/sage/modules/free_module.py", line 6685, in __init__
2025-05-11T07:22:31.6960865Z     basis = self._echelonized_basis(ambient, basis)
2025-05-11T07:22:31.6961327Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-11T07:22:31.6961946Z   File "/sage/src/sage/modules/free_module.py", line 6875, in _echelonized_basis
2025-05-11T07:22:31.6962568Z     E = A.echelon_form()
2025-05-11T07:22:31.6962899Z         ^^^^^^^^^^^^^^^^
2025-05-11T07:22:31.6963909Z   File "sage/matrix/matrix_integer_dense.pyx", line 2063, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.echelon_form
2025-05-11T07:22:31.6965204Z     H_m = self._hnf_pari(flag, include_zero_rows=include_zero_rows)
2025-05-11T07:22:31.6966156Z   File "sage/matrix/matrix_integer_dense.pyx", line 5830, in sage.matrix.matrix_integer_dense.Matrix_integer_dense._hnf_pari
2025-05-11T07:22:31.6966995Z     sig_on()
2025-05-11T07:22:31.6967368Z cysignals.signals.SignalError: Segmentation fault

Unfortunately, I can't reproduce this locally, even in a Docker container running the exact same image: ghcr.io/sagemath/sage/sage-ubuntu-focal-standard-with-targets:dev@sha256:e487138be53a1f275bf3cc21d8bf5f352a5709776217ea9996b74fd46ad0fa36. So, it seems to be dependent on the GitHub runner's host system.

@soehms soehms force-pushed the khoca_integration branch from 4bf27cb to 213d7d4 Compare May 13, 2025 09:21
@soehms
Copy link
Member Author

soehms commented May 15, 2025

In the current commit, I switched the installation to binary wheels. This fixes the issue explained in my previous comment. Two of the five CI-Linux-incremental jobs failed for other reasons, while all remaining jobs ran the optional Khoca tests and all tests passed. For example, in the case of Ubuntu jammy:

2025-05-14T16:13:21.7201014Z [sagelib-10.7.beta3] installing. Log file: /sage/logs/pkgs/sagelib-10.7.beta3.log
2025-05-14T16:13:39.7457879Z   [sagelib-10.7.beta3] successfully installed (real 0m18.018s user 0m13.044s sys 0m5.874s).
2025-05-14T16:13:39.9423508Z make --no-print-directory khoca-no-deps
2025-05-14T16:13:39.9425263Z sage-logger -p 'sage --pip install -r "/sage/build/pkgs/khoca/requirements.txt"' '/sage/logs/pkgs/khoca.log'
2025-05-14T16:13:39.9426937Z [khoca] installing. Log file: /sage/logs/pkgs/khoca.log
2025-05-14T16:13:45.6360219Z   [khoca] successfully installed.
...
2025-05-14T16:29:09.2985065Z Running doctests with ID 2025-05-14-16-29-09-d4377f03.
2025-05-14T16:29:09.2985980Z Running with SAGE_LOCAL='/sage/local' and SAGE_VENV='/sage/local/var/lib/sage/venv-python3.12.5'
2025-05-14T16:29:09.2987434Z Using --optional=!sagemath_doc_html,!sagemath_doc_pdf,debian,pip,sage,sage_spkg
2025-05-14T16:29:09.4518558Z Features to be detected: 4ti2,SAGE_SRC,benzene,bliss,buckygen,conway_polynomials,coxeter3,csdp,cvxopt,cvxopt,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_ellcurves,database_graphs,database_jones_numfield,database_knotinfo,dot2tex,dvipng,ecm,fpylll,fricas,gap_package_atlasrep,gap_package_design,gap_package_grape,gap_package_guava,gap_package_hap,gap_package_polenta,gap_package_polycyclic,gap_package_qpa,gap_package_quagroup,gfan,giac,glucose,graphviz,imagemagick,info,ipython,jmol,jupymake,jupyter_sphinx,kenzo,khoca,kissat,latte_int,lrcalc_python,lrslib,mathics,matroid_database,mcqd,meataxe,meson_editable,mpmath,msolve,nauty,networkx,numpy,palp,pandoc,pdf2svg,pdftocairo,pexpect,phitigra,pillow,plantri,polytopes_db,polytopes_db_4d,pplpy,primecountpy,ptyprocess,pycosat,pycryptosat,pynormaliz,pyparsing,python_igraph,requests,rpy2,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.braiding,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.giac,sage.libs.homfly,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.sat,sage.schemes,sage.symbolic,sage_numerical_backends_coin,scipy,singular,sirocco,sloane_database,sphinx,symengine_py,sympy,tdlib,threejs,topcom
2025-05-14T16:29:09.4525357Z Doctesting entire Sage library.
2025-05-14T16:29:09.4525616Z Doctesting all documentation sources.
2025-05-14T16:29:09.7653466Z Sorting sources by runtime so that slower doctests are run first....
2025-05-14T16:29:09.9839651Z Doctesting 4707 files using 5 threads.
...
2025-05-14T16:46:35.7485735Z sage -t --warn-long 5.0 --random-seed=28611653333899964209816938163267021840 src/sage/knots/link.py
2025-05-14T16:46:35.7486685Z     [601 tests, 5.65s wall]
...
..
2025-05-14T17:08:36.7590277Z ----------------------------------------------------------------------
2025-05-14T17:08:36.7591065Z All tests passed!
2025-05-14T17:08:36.7591864Z ----------------------------------------------------------------------
2025-05-14T17:08:36.7592418Z Total time for all tests: 2365.3 seconds
2025-05-14T17:08:36.7592845Z     cpu time: 8047.0 seconds
2025-05-14T17:08:36.7593237Z     cumulative wall time: 9913.3 seconds
2025-05-14T17:08:36.7599974Z Features detected for doctesting: bliss,conway_polynomials,coxeter3,cvxopt,database_cremona_mini_ellcurve,database_ellcurves,database_graphs,fpylll,gap_package_polycyclic,gfan,info,jupyter_sphinx,khoca,lrcalc_python,mcqd,meataxe,mpmath,nauty,networkx,numpy,palp,pexpect,pillow,polytopes_db,pplpy,primecountpy,ptyprocess,pyparsing,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.braiding,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.homfly,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.schemes,sage.symbolic,scipy,singular,sirocco,sphinx,sympy,tdlib,threejs

All other unsuccessful CI jobs appear to have failed for independent reasons (except for meson-tests), too.

@soehms
Copy link
Member Author

soehms commented May 20, 2025

Status after merge with 10.7.beta4

All five tests for CI-Linux-incremental are now green, and all tests with the optional package pass.

Regarding the segmentation fault observed at #40081 (comment), I have made further unsuccessful attempts to reproduce it. Specifically, I ran sage -tp --all and sage -tp --all --long in Docker containers created with tox -e docker-<os>-standard-incremental -- khoca-ensure, where <os> stands for ubuntu-focal and debian-bookworm. I created them relative to 213d7d4, which builds khoca from source. I ran this on two hosts (Linux Mint 21.1 on a ThinkPad with an i5 and Ubuntu 20.04 running under WSL2 on a ThinkPad with an i7). In all these attempts, all tests ran successfully.

But the following was different from the tests with the binary wheel installation. Although the number of tests counted in src/sage/knots/link.py is the same (together with the new optional tests for khoca, it is 601 and 602 in the --long case), khoca is not listed as a feature detected for doctesting when installed from sources:

sage -t --warn-long 5.0 --random-seed=210352780579853333040454205222764212850 src/sage/knots/link.py
    [601 tests, 5.43s wall]
...
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 1316.9 seconds
    cpu time: 8661.2 seconds
    cumulative wall time: 8798.8 seconds
Features detected for doctesting: conway_polynomials,cvxopt,database_cremona_mini_ellcurve,database_ellcurves,database_graphs,fpylll,gap_package_polycyclic,gfan,info,jupyter_sphinx,lrcalc_python,mpmath,nauty,networkx,numpy,palp,pexpect,pillow,polytopes_db,pplpy,primecountpy,ptyprocess,pyparsing,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.braiding,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.homfly,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.schemes,sage.symbolic,sagemath_doc_html,scipy,singular,sphinx,sympy,threejs

This was accordingly in the segfault runs. However, it is likely an independent issue in the doctest framework.

Again, the cause of all failed CI runs seems to have nothing to do with this PR.

@soehms
Copy link
Member Author

soehms commented Jul 22, 2025

Currently, there is still an issue with the PyPI package regarding integration with Sage, as both use the Pari library (see LLewark/khoca#3 (comment)). This occurs sporadically and triggers a system error that occurs at cypari2/stack.pyx. The error disappears when the last action is re-executed. I'm leaving this PR as a draft until the issue is resolved.

As explained at LLewark/khoca#3 (comment), this issue is unrelated to the Khoca package. The sporadic SystemError was caused by a regression in cysignals that arose during the build system migration to Meson. This issue is currently being fixed at sagemath/cysignals#230.

Therefore, we can now proceed with the Khoca integration.

@LLewark, @alexchandler100, FYI.

@soehms soehms marked this pull request as ready for review July 22, 2025 06:56
@soehms soehms requested a review from tscrim July 22, 2025 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant