Explicit control of enabled features

[orlitzky]

Sage supports many optional features by detecting their presence at run-time. This is used to show better errors to end users, and to skip # needs <package> doctests when <package> is not installed. For the sake of the discussion, there are roughly four categories of feature:

1. External programs. The sage library doesn't need these present to build, and then can easily be installed or upgraded later on, after sage is installed. Examples: ffmpeg, imagemagick, mwrank.
2. External python libraries. Additional (not required) python or cython packages that sage knows how to use. These are typically easy to install after-the-fact as well. Examples: pillow, rpy2.
3. Sage add-on packages. Python libraries that depend on sage. These would typically only be installed because you want sage to use them, but it still might be nice (as a bonus) to be able to control whether or not they are tested. Examples: sagemath_giac.
4. Internal cython libraries. These are internal libraries (part of sagelib) that can optionally be disabled. Since they are built (when enabled) as part of sage, we typically know at build time whether or not they will be present, but it's not out of the question that some distribution would want to package the result separately. Note that most of these are good candidates to be moved into the third category. Examples: sage.graphs.bliss, sage.libs.eclib.

The benefit of run-time detection is that you don't have to reinstall (or at least, reconfigure) sage to enable new features. There are however many corresponding downsides. And now that the sage library has its own build system (independent of the sage distribution), we are free to consider alternatives.

Since we will eventually discuss other options, we christen the status quo...

Option 1: leave it alone

Pro:

1. New features can be enabled without rebuilding sage.

Con:

1. Tests can fail when features are detected incorrectly. Fixing this with robust run-time checks both duplicates code, and makes the feature test slow.
2. Running tests for features that are only accidentally enabled (because, say, ffmpeg is installed) wastes time.
3. The sage distribution needs to reimplement all of the run-time checks at build-time (in spkg-configure.ac) for packages that are standard in the distribution, but optional to the library.
4. It's impossible to know the dependencies of sage ahead of time for packaging purposes. For example, sage doesn't depend on imagemagick, because it is optional and detected at run-time. If the user installs sage via a package manager and just happens to have imagemagick installed as a dependency of e.g. inkscape, he may write a sage program that uses sage's imagemagick support. If he later uninstalls inkscape, imagemagick will be removed too, and now his program crashes. There is currently no way to address this without requiring every optional dependency unconditionally.
5. Errors can be hidden when the feature test is "does this work?" For example in import error with bliss #40990 we expect bliss to be importable, but it isn't; the feature test hides this because the ImportError perversely disables the associated tests.

Option 2: detect/configure everything at build-time

The opposite of the status quo would be to detect everything at build time, with the user being able to specify explicitly whether or not he expects a particular feature to be enabled. The pros/cons lists are more or less inverted.

Pro

1. All feature tests are localized in the build system, happen once, and don't slow sage down at run-time.
2. All unwanted features can be forced off; all desired features can be explicitly requested (made an error if they are not found).
3. All dependency information is known ahead-of-time.
4. If you explicitly enable a feature and it doesn't work, the tests will fail.

Con

1. If you want to enable any feature, at the very least, you would have to rebuild sage by invoking meson or make. On a source-based distro, you would have to reinstall sage completely. Binary distros would need to find some other way to support optional features.
2. This is a change of behavior that is not backwards-compatible.

Option 3: build-time checks that can be deferred to run-time

This is a compromise between the first two. Basically, we would allow all features to be controlled at build-time, and the default would be to detect them at build-time automatically (if they're present, enable them; otherwise, disable them). However we would provide an override for people who want runtime detection, something like -Ddefer-feature-checks=true. This could be used by binary distros, and by default in the sage distribution to retain backwards compatibility.

Pro

All of the above.

Con

None of the above, except:

1. This would still require duplication of the build-time feature checks at run-time under sage.features. To give one example of this, sage needs ffmpeg to support several file extensions: .avi, .flv, .gif, .mkv, .mov, .mp4, .ogg, .ogv, .webm, .wmv. The test for this is complicated, and we would have to repeat it. A build-time check is necessary to support -Dffmpeg=enabled, and the run-time check is necessary to support -Ddefer-feature-checks=true. This can lead to a lot of code duplication.

Implementation details

The implementation of any of these would be straightforward. When a feature foo is enabled or disabled at build-time, we would write foo_enabled = <value> into a config file. We would then modify sage.features.foo to return the configured value, rather than try to detect the presence of foo itself. With deferred checks, we would just keep doing what we are doing now (i.e. ignore foo_enabled if defer_feature_checks is True).

The existing features would have to be ported to meson one at a time (and there is no hurry to do this). The internal libraries in meson.options would be a good starting point.

[antonio-rojas]

As I see it the pro/cons lists are kind of unbalanced in the sense that in option 1 the "Pros" are mostly for users, while the "Cons" are mostly for developers/packagers (and conversely for option 2). IMHO we should aim to make the experience as smooth as possible for users, even at the cost of slightly increasing the necessary work for developers/packagers.

On a more specific level, option 2 would be quite a step back from my point of view as an Arch packager. That would mean I would have to choose between packaging every single optional dependency of Sage (even the more niche or obscure ones, and the list is increasing with each release: for 10.8 there are at least two new ones, khoca and regina) or make these optional features completely unaccesible to our users. Currently these features provided by non-packaged optional dependencies can easily be enabled by users by installing eg. user created packages from the AUR, or simply by downloading the files locally in the case of database files.

I have no objections to options 1 or 3.

[orlitzky]

As I see it the pro/cons lists are kind of unbalanced in the sense that in option 1 the "Pros" are mostly for users, while the "Cons" are mostly for developers/packagers (and conversely for option 2). IMHO we should aim to make the experience as smooth as possible for users, even at the cost of slightly increasing the necessary work for developers/packagers.

I agree. In my head, the main "cons" were from the perspective of a user, albeit a Gentoo user. Having features mysteriously disappear because some unrelated program was uninstalled is a constant source of bug reports, and we run the test suite during installation (at least if the user wants to) so testing only what is expected to work is particularly important.

[dimpase]

I don't see how "Con" in option 2 is actually correct - building/installing an extra library and a Cython interface to it does not entail re-building everything, quite the opposite.

I also don't see why you say "everything". Surely something like an external binary can be dealt with by the existing mechanics, whereas an extra Cython module, something which needs building anyway, can be a build-time detected feature.

[orlitzky]

I did Ctrl-F "everything" to be sure, and I only ever said detect/configure everything, not rebuild everything.

For developers and users of the sage distribution, Option 2 would only have to reconfigure, and then build the parts of the sage library that are affected by the new feature. That would be relatively quick.

On Gentoo, you would have to rebuild the whole package though.

[tobiasdiez]

Thank you for opening this discussion!

I think it's a good idea to distinguish each of the 4 different kinds of dependencies. There is not much harm in runtime-detection of external Python/Cython modules of the form:

try:
   import external_lib
catch ImportError:
    disable feature

Also you can easily (un)install external_lib from the venv, so I don't see the need for an build-time detection in this case (of course except if you would actually link against some cython modules in external_lib).

The internal Cython modules (point 4) are in disguise external libs that sage links again - so this has to be already controlled by the build system.

Leaves the external programs and databases. I would tend to make those build-time detection-only, with the option to override the location with specialized env variables (eg sage_ffmpeg) to still give users the option to manually use those features on binary distributions that don't ship them. But perhaps it's also okay to look at least for the static db's at runtime in the usual places. I'm honestly missing the user's perspective here, as I never used any of these db's in sage...

[orlitzky]

There is not much harm in runtime-detection of external Python/Cython modules of the form:

try:
   import external_lib
catch ImportError:
    disable feature

Also you can easily (un)install external_lib from the venv, so I don't see the need for an build-time detection in this case (of course except if you would actually link against some cython modules in external_lib).

This is easy to do if you're developing in a venv, or if the external package is sagemath-whatever, but is less easy if the external package is something popular and generic like pillow, and was installed system-wide by your distro because some other package wanted it.

The internal Cython modules (point 4) are in disguise external libs that sage links again - so this has to be already controlled by the build system.

Yeah, the usefulness of the added control would vary greatly depending on the feature. Run-time detection of build-time deps is not very helpful, nor is build-time control over add-ons that are only useful to sage. I focused more on the feature types when I scribbled down my initial ideas on #41042 a few days ago, but I tried to simplify things here based on the assumption that it's better to be consistent, even if the added flexibility will be of little value in some cases.

I really don't want to have to explain four loosely-defined feature types to end users in the documentation, and expect them to distinguish between the types, and then keep four different methods of detection / feature-control in their heads. It would be much better if we could say, unconditionally, "To enable a feature explicitly, use -Dfeature=enabled, to disable it, use -Dfeature=disabled. Otherwise it will be auto-detected at build time. To delay these checks until run-time, use -Ddefer-feature-checks=true." (Even in cases where that won't do you a lot of good.)

Leaves the external programs and databases. I would tend to make those build-time detection-only, with the option to override the location with specialized env variables (eg sage_ffmpeg) to still give users the option to manually use those features on binary distributions that don't ship them. But perhaps it's also okay to look at least for the static db's at runtime in the usual places. I'm honestly missing the user's perspective here, as I never used any of these db's in sage...

This would be more flexible, but it would also be more work -- not just in sage, but also for the binary package maintainer who wants to detect most things at run-time. Personally I'm in the build-time camp, but don't want to do anything that would make life harder for our binary distro friends. I abstain :)