RFC: Developing package configuration system (package options) outside Base and Pkg?

[tkf]

I realized that package options can be implemented outside Base and Pkg so here is my proof of concept implementation: https://github.com/tkf/PkgOptions.jl. Combined with conditional dependencies #1285, I think the examples in JuliaLang/Juleps#38 and #977 (cc @Roger-luo) can be covered (feature flags are just boolean package options). Quoting the README:

Features

• Project/environment-local per-package configuration options.
• Precompilation is triggered when package options are changed.
• Minimal dependency (PkgOptionsLoader) for a package to support package options.

Installation

pkg> add https://github.com/tkf/PkgOptionsLoader.jl.git

pkg> add https://github.com/tkf/PkgOptions.jl.git

pkg> add https://github.com/tkf/PkgOptionsDemo.jl.git  # a demo

Usage

End-users

Use PkgOptions.set(pkg; ...) to configure package options for package pkg:

julia> using PkgOptions

julia> PkgOptions.set(:PkgOptionsDemo, some_option=true)

julia> using PkgOptionsDemo
       PkgOptionsDemo.pkgoptions
[ Info: Precompiling PkgOptionsDemo [bb3b8621-5970-400b-8acd-051caadabee1]
Dict{String,Any} with 1 entry:
  "some_option" => true

Note that package options are precompile-time options. You need to reload package to re-configure it.

To use the default option, remove the package option by:

julia> PkgOptions.rm(:PkgOptionsDemo)

See more details in the documentation.

Package authors

To support package options, use PkgOptionsLoader.@load to load package options (a Dict{String,Any}). For example, PkgOptionsDemo.jl used above in the demo is defined as

module PkgOptionsDemo
using PkgOptionsLoader
const pkgoptions = PkgOptionsLoader.@load
end

See more details in the PkgOptionsLoader.@load documentation.

How it works

PkgOptions.set, PkgOptionsLoader.@load, etc. read and write the package options in a TOML file at

~/.julia/options/$project_slug/$package_slug/$package_name.toml

where

• $project_slug is a hash determined by the project path (Base.active_project()).
• $package_slug is a hash determined by the UUID of the package whose options are configured.
• $package_name is the name of the package whose options are configured.
• ~/.julia may be different if you configure Base.DEPOT_PATH[1]

Changing the TOML file triggers precompilation using Base.include_dependency mechanism.

See more in the documentation: https://tkf.github.io/PkgOptions.jl/dev

What do you think about the approach? Does it have some limitations? What are the alternatives?

A possible improvement would be to increase reproducibility. I think there are several ways to do it (e.g., reflecting options in Project.toml, maybe using format like #458 (comment)).

I think actually the main observation that this can be developed outside Base and Pkg. It's good because the implementation and protocol can be iterated faster in the wild.

[KristofferC]

Does it have some limitations?

I think the main limitation (as you mention later) is that it stores project state outside the Project.toml file. Whether this is a problem or not depends on what type of options are stored. If the options affect the functionality of the package (fast_math = true) then there is a reproducibility problem. If the options are only such that it helps the package run at all (cuda_path = "...") then that should be fine.

It is very good that you have managed to implement this outside of Pkg so that things can be iterated and usecase identified etc.

[tkf]

Thanks for the comments. Yes, reproducibility is the main issue. I have two ideas:

Relative option directory

Option file can be placed at (say)

$PROJECT_DIR/.juliaoptions/$package_slug/$package_name.toml

where $PROJECT_DIR is the directory where you have Manifest.toml. So, a $PROJECT_DIR may look like

$PROJECT_DIR
├── .juliaoptions/
│   └── GkzkC
│       └── PyCall.toml
├── Project.toml
└── Manifest.toml

Advantage:

• Implementation is straightforward and requires no change in Base or Pkg.

Disadvantage:

• You need several files to reproduce a project.

Reflecting data to Project.toml

Another idea is to save the option data both in Project.toml and ~/.julia/options/**/*.toml. PkgOptionsLoader.@load can look at the Project.toml file and copy it to the option TOML files at the first precompile time. However, this does not work if Project.toml is updated after precompilation. Solving this would require change in Base or Pkg. I think the easiest solution would be to rewrite stale option TOML files after instantiate, resolve, activate, etc.

[tkf]

Closing since we have JuliaLang/julia#37595 and https://github.com/JuliaPackaging/Preferences.jl