Limit libraries to be included in the bundle

[simonbyrne]

If julia_libdir() == "/usr/lib" then it will attempt to bundle the whole directory into the app. Fixes #16.

@jlapeyre, @OliverEvans96, @feanor12, @alexriss: would you please be able to try out this branch (]add PackageCompiler#sb/limit-bundle) and see if it fixes the problem?

[alexriss]

Thank you! This seems to fix the bundle_julia_libraries function. But now I get an error in bundle_cert:

IOError: open("/usr/bin/../share/julia/cert.pem", 0, 0): no such file or directory (ENOENT)

at PackageCompiler.jl:1206

I am somehow not able to find a cert.pem belonging to Julia on my system (Manjaro 22, Julia installed via the package manager).

[simonbyrne]

Thanks @alexriss: I'm not sure how the certificates are used, so don't know what the fix is (can we just drop them?)

FWIW, the recommendation is to use the binaries from https://julialang.org/downloads/, as system packages often don't include patches or other fixes to upstream libraries that Julia requires.

[codecov]

Codecov Report

Attention: Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.88%. Comparing base (376e3bf) to head (59b6528).
Report is 138 commits behind head on master.

Files with missing lines	Patch %	Lines
src/PackageCompiler.jl	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #742      +/-   ##
==========================================
- Coverage   93.46%   92.88%   -0.58%     
==========================================
  Files           2        2              
  Lines         520      520              
==========================================
- Hits          486      483       -3     
- Misses         34       37       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[feanor12]

I tried to compile a small demo project with aur/juliaup, aur/julia-bin and the community/julia package.

With the new branch of PackageCompiler aur/juliaup and aur/julia-bin work.

the community/julia package was throwing an error:

ERROR: IOError: open("/usr/bin/../share/julia/cert.pem", 0, 0): no such file or directory (ENOENT)
Stacktrace:
 [1] cp
   @ ./file.jl:376 [inlined]
 [2] bundle_cert
   @ ~/.julia/packages/PackageCompiler/Lz0Nj/src/PackageCompiler.jl:1211 [inlined]
 [3] create_app(package_dir::String, app_dir::String; executables::Nothing, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, force::Bool, c_driver_program::String, cpu_target::String, include_lazy_artifacts::Bool, sysimage_build_args::Cmd, include_transitive_dependencies::Bool)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/Lz0Nj/src/PackageCompiler.jl:712
 [4] create_app(package_dir::String, app_dir::String)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/Lz0Nj/src/PackageCompiler.jl:685
 [5] top-level scope
   @ REPL[3]:1

The community/julia package, however, does not ship cert.pem at all.

[alexriss]

I have done some more tests.

If I install the official binaries on my Manjaro distro, everything works, even with the previous version without any changes.

With the version from the package manager, everything is more complicated. I have adapted the bundle_cert function as follows:

function bundle_cert(dest_dir)
    cert_path = joinpath(Sys.BINDIR, "..", "share", "julia", "cert.pem")
    if Sys.isunix() && !isfile(cert_path)
        cert_path = "/etc/ssl/cert.pem"
    end
    share_path = joinpath(dest_dir, "share", "julia")
    mkpath(share_path)
    isfile(cert_path) && cp(cert_path, joinpath(share_path, "cert.pem"))
end

I am not quite sure if this is the right cert.pem file. However, I get a different error now for some packages that rely on these certs:

Failed to precompile Blink [ad839575-38b3-5650-b840-f874b8c74a25] to /home/riss/.julia/compiled/v1.8/Blink/jl_z6KvWg.
ERROR: LoadError: InitError: SystemError: opening file "/usr/share/julia/cert.pem": No such file or directory

I have traced the problem down further, it seems to come from the package MozillaCACerts_jll.jl, which seems to be generated by JLLWrappers.jl.

Now somewhere there is a function MozillaCACerts_jll.find_artifact_dir() which does basically this:

global artifact_dir = dirname(Sys.BINDIR)
global cacert = normpath(Sys.BINDIR, Base.DATAROOTDIR, "julia", "cert.pem")

and it ends up with:

julia> Base.DATAROOTDIR
"../share"

julia> Sys.BINDIR
"/usr/bin"

julia> MozillaCACerts_jll.artifact_dir
"/usr"

So this does not seem to be a PackageCompiler problem anymore, since I cannot compile these packages directly under my Julia instance, see also this thread. And yes, this should have been the first thing to try, would have saved quite some time.

tl;dr: Seems like the solution is to not use the community/julia package for now.