Possible typo `LLVM` instead of `LVM` in `lower_convention`?

[ymardoukhi]

Enzyme.jl/src/compiler.jl

Line 4404 in f360f6b

LVM.API.LLVMVerifyFunction(wrapper_f, LLVM.API.LLVMPrintMessageAction),

I hit this error when dealing with @views.

ERROR: UndefVarError: `LVM` not defined in `Enzyme.Compiler`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
  [1] (::Enzyme.Compiler.var"#359#366"{LLVM.Module, LLVM.Function})(io::IOBuffer)
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:4402
  [2] sprint(::Function; context::Nothing, sizehint::Int64)
    @ Base ./strings/io.jl:114
  [3] sprint(::Function)
    @ Base ./strings/io.jl:107
  [4] lower_convention(functy::Type, mod::LLVM.Module, entry_f::LLVM.Function, actualRetType::Type, RetActivity::Type, TT::Union{…}, run_enzyme::Bool)
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:4400
  [5] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob{…})
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:4713
  [6] compile(target::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Gp8bZ/src/driver.jl:67
  [7] compile
    @ ~/.julia/packages/GPUCompiler/Gp8bZ/src/driver.jl:55 [inlined]
  [8] _thunk(job::GPUCompiler.CompilerJob{…}, postopt::Bool)
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:5946
  [9] _thunk
    @ ~/.julia/dev/Enzyme/src/compiler.jl:5944 [inlined]
 [10] cached_compilation
    @ ~/.julia/dev/Enzyme/src/compiler.jl:6001 [inlined]
 [11] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{…}, A::Type{…}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, StrongZero::Bool, edges::Vector{…})
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:6117
 [12] thunk_generator(world::UInt64, source::Union{…}, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, StrongZero::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type, strongzero::Type)
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:6312
 [13] augfwd_with_return(::Val{…}, ::Val{…}, ::Val{…}, ::Val{…}, ::Type{…}, ::Val{…}, ::Type{…}, ::Type{…}, ::typeof(QruiseToolset._func), ::Nothing, ::Active{…}, ::Duplicated{…})
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/rules/jitrules.jl:1284
 [14] macro expansion
    @ ~/.julia/dev/Enzyme/src/rules/jitrules.jl:1383 [inlined]
 [15] runtime_iterate_augfwd(::Type{…}, ::Val{…}, ::Val{…}, ::Val{…}, ::Val{…}, ::Val{…}, ::typeof(QruiseToolset._func), ::Nothing, ::Tuple{…}, ::Base.RefValue{…}, ::Vector{…}, ::Vector{…}, ::Vector{…}, ::Vector{…})
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/rules/jitrules.jl:1423

[vchuravy]

That's hilarious! Could you open a PR to fix that and perhaps add your code as a test?

[ymardoukhi]

That's hilarious! Could you open a PR to fix that and perhaps add your code as a test?

Absolutely! My use case is a bit convoluted. I compile an LLVM IR into a shared object via clang.
I wanted to simplify the code and avoid using clang by relying on Base.llvmcall, but ran into an issue which I don't know how to resolve.

const FUNC_LLVM_IR = """
    declare double @llvm.rint.f64(double) #1

    define i32 @func(double* noalias nocapture writeonly %retptr, { i8*, i32, i8*, i8*, i32 }** noalias nocapture readnone %excinfo, double %arg.t, i8* nocapture readnone %arg.arr.0, i8* nocapture readnone %arg.arr.1, i64 %arg.arr.2, i64 %arg.arr.3, double* %arg.arr.4, i64 %arg.arr.5.0, i64 %arg.arr.6.0) local_unnamed_addr #0 {
    common.ret:
      %.27 = fdiv double %arg.t, 1.000000e-02
      %.28 = tail call double @llvm.rint.f64(double %.27)
      %.29 = fptosi double %.28 to i64
      %.42 = icmp slt i64 %.29, 0
      %.43 = select i1 %.42, i64 %arg.arr.5.0, i64 0
      %.44 = add i64 %.43, %.29
      %.55 = mul i64 %.44, %arg.arr.6.0
      %.56 = ptrtoint double* %arg.arr.4 to i64
      %.57 = add i64 %.55, %.56
      %.58 = inttoptr i64 %.57 to double*
      %.59 = load double, double* %.58, align 8
      store double %.59, double* %retptr, align 8
      ret i32 0
    }


    attributes #0 = { mustprogress nofree nosync nounwind willreturn }
    attributes #1 = { mustprogress nocallback nofree nosync nounwind readnone speculatable willreturn }
    attributes #2 = { noinline }
"""

sopath = "./func.so"
run(
    pipeline(
    `clang -x ir - -Xclang -no-opaque-pointers -O3 -fPIC -fembed-bitcode -shared -o $(sopath)`;
    stdin = IOBuffer(FUNC_LLVM_IR)
)
)

using Libdl
lib = Libdl.dlopen(sopath)
const fptr = Libdl.dlsym(lib, :func)

function func_llvm(t::Float64, arr::AbstractVector{Float64})
    nitems = length(arr)
    bitsize = Base.elsize(arr)
    excinfo = Ref{Ptr{Nothing}}()
    res = Ref{Float64}()

    status = Base.llvmcall((FUNC_LLVM_IR, "func"), Cint,
        Tuple{Ref{Cdouble}, Ref{Ptr{Cvoid}},
            Cdouble, Ptr{Cvoid}, Ptr{Cvoid},
            Clong, Clong, Ptr{Cdouble}, Clong, Clong
        },
        res, excinfo, t, C_NULL, C_NULL, nitems, bitsize,
        Base.unsafe_convert(Ptr{Cdouble}, arr), nitems, nitems * bitsize)

    status == 0 || error("returned non-zero status: $status")
    res[]
end

function func_ccall(t::Float64, arr::AbstractVector{Float64})
    nitems = length(arr)
    bitsize = Base.elsize(arr)
    excinfo = Ref{Ptr{Nothing}}()
    res = Ref{Float64}()

    status = ccall(fptr, Cint,
        (Ref{Cdouble}, Ref{Ptr{Cvoid}},
            Cdouble, Ptr{Cvoid}, Ptr{Cvoid},
            Clong, Clong, Ptr{Cdouble}, Clong, Clong),
        res, excinfo, t, C_NULL, C_NULL, nitems, bitsize,
        Base.unsafe_convert(Ptr{Cdouble}, arr), nitems, nitems * bitsize)

    status == 0 || error("returned non-zero status: $status")
    res[]
end

using Enzyme

a = rand(10)
# throws
# Invalid bitcast
#   %18 = bitcast {} addrspace(10)* %"'mi" to double*, !dbg !139
# Invalid bitcast
#   %19 = bitcast {} addrspace(10)* %11 to { i8*, i32, i8*, i8*, i32 }**, !dbg !139
# Invalid bitcast
#   %20 = bitcast i64 %16 to double*, !dbg !139
# Invalid bitcast
#   %21 = bitcast i64 %"'ipc" to double*, !dbg !139
# Invalid bitcast
# double* bitcast ({} addrspace(10)* @ejl_jl_nothing to double*)
# Invalid bitcast
#   %24 = bitcast {} addrspace(10)* %"'mi" to double*, !dbg !139
# Invalid bitcast
#   %25 = bitcast i64 %16 to double*, !dbg !139
# Invalid bitcast
#   %26 = bitcast i64 %"'ipc" to double*, !dbg !139
# Invalid bitcast
# { i8*, i32, i8*, i8*, i32 }** bitcast ({} addrspace(10)* @ejl_jl_nothing to { i8*, i32, i8*, i8*, i32 }**)
gradient(Reverse, func_llvm, Const(0.0), a)

# returns the result as intended
# (nothing, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
gradient(Reverse, func_ccall, Const(0.0), a)

It seems I am messing up with Julia's address space and C's address space. I tried to use Base.Libc.malloc and GC.@preserve to no avail.
If you could give some directions or hints on how to address this, I could figure out the rest.

I tried to use llvmcall since I am not sure clang is available on the CI runner. I had a look at the repo, and it seems it is not?

[vchuravy]

You have a calling convention mismatch in the LLVM IR for exec info? You should be getting an error/warning if you run that without Enzyme being involved (maybe only with assertions on)

[ymardoukhi]

You have a calling convention mismatch in the LLVM IR for exec info? You should be getting an error/warning if you run that without Enzyme being involved (maybe only with assertions on)

Thanks! I managed to fix that. I added a wrapper function in the LLVM IR to avoid malloc since Enzyme complained about it when I wanted to take the gradient.

Now surprisingly, I don't hit that execution path which caused the LVM using llvmcall

using Libdl
using Enzyme

const FUNC_LLVM_IR = """
    declare double @llvm.rint.f64(double) #1

    define i32 @func(double* noalias nocapture writeonly %retptr, { i8*, i32, i8*, i8*, i32 }** noalias nocapture readnone %excinfo, double %arg.t, i8* nocapture readnone %arg.arr.0, i8* nocapture readnone %arg.arr.1, i64 %arg.arr.2, i64 %arg.arr.3, double* %arg.arr.4, i64 %arg.arr.5.0, i64 %arg.arr.6.0) local_unnamed_addr #0 {
    common.ret:
      %.27 = fdiv double %arg.t, 1.000000e-02
      %.28 = tail call double @llvm.rint.f64(double %.27)
      %.29 = fptosi double %.28 to i64
      %.42 = icmp slt i64 %.29, 0
      %.43 = select i1 %.42, i64 %arg.arr.5.0, i64 0
      %.44 = add i64 %.43, %.29
      %.55 = mul i64 %.44, %arg.arr.6.0
      %.56 = ptrtoint double* %arg.arr.4 to i64
      %.57 = add i64 %.55, %.56
      %.58 = inttoptr i64 %.57 to double*
      %.59 = load double, double* %.58, align 8
      store double %.59, double* %retptr, align 8
      ret i32 0
    }

    define double @func_wrap({ i8*, i32, i8*, i8*, i32 }** %excinfo, double %arg.t, i8* %arg.arr.0, i8* %arg.arr.1, i64 %arg.arr.2, i64 %arg.arr.3, double* %arg.arr.4, i64 %arg.arr.5.0, i64 %arg.arr.6.0) {
    entry:
        %tmp = alloca double, align 8
        %st  = call i32 @func(double* %tmp, { i8*, i32, i8*, i8*, i32 }** %excinfo, double %arg.t, i8* %arg.arr.0, i8* %arg.arr.1, i64 %arg.arr.2, i64 %arg.arr.3, double* %arg.arr.4, i64 %arg.arr.5.0, i64 %arg.arr.6.0)
        %val = load double, double* %tmp, align 8
        ret double %val
    }


    attributes #0 = { mustprogress nofree nosync nounwind willreturn }
    attributes #1 = { mustprogress nocallback nofree nosync nounwind readnone speculatable willreturn }
    attributes #2 = { noinline }
"""

sopath = "./func.so"
run(
    pipeline(
    `clang -x ir - -Xclang -no-opaque-pointers -O3 -fPIC -fembed-bitcode -shared -o $(sopath)`;
    stdin = IOBuffer(FUNC_LLVM_IR)
)
)


lib = Libdl.dlopen(sopath)
const fptr = Libdl.dlsym(lib, :func_wrap)

function func_llvm(t::Float64, arr::AbstractVector{Float64})
    nitems = length(arr)
    bitsize = Base.elsize(arr)
    GC.@preserve arr begin
        excinfo = Ptr{Ptr{Cvoid}}(C_NULL)
        base::Ptr{Cdouble} = pointer(arr)

        Base.llvmcall((FUNC_LLVM_IR, "func_wrap"), Cdouble,
            Tuple{Ptr{Ptr{Cvoid}},
                Cdouble, Ptr{Cvoid}, Ptr{Cvoid},
                Clong, Clong, Ptr{Cdouble}, Clong, Clong
            },
            excinfo, t, C_NULL, C_NULL, nitems, bitsize,
            base, nitems, nitems * bitsize)
    end
end

function func_ccall(t::Float64, arr::AbstractVector{Float64})
    nitems = length(arr)
    bitsize = Base.elsize(arr)
    GC.@preserve arr begin
        excinfo = Ptr{Ptr{Cvoid}}(C_NULL)
        base::Ptr{Cdouble} = pointer(arr)

        ccall(fptr, Cdouble,
            (Ptr{Ptr{Cvoid}}, Cdouble, Ptr{Cvoid}, Ptr{Cvoid},
                Clong, Clong, Ptr{Cdouble}, Clong, Clong),
            excinfo, t, C_NULL, C_NULL, nitems, bitsize,
            base, nitems, nitems * bitsize)
    end
end

a = rand(10)

gradient(Reverse, func_llvm, Const(0.0), a)
# (nothing, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])

gradient(Reverse, func_ccall, Const(0.0), a)
# (nothing, [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])

b = @view a[1:5]
gradient(Reverse, func_llvm, Const(0.0), b)
# (nothing, [1.0, 0.0, 0.0, 0.0, 0.0])

gradient(Reverse, func_ccall, Const(0.0), b)
# ERROR: UndefVarError: `LVM` not defined in `Enzyme.Compiler`
# Suggestion: check for spelling errors or missing imports.
# Stacktrace:
#   [1] (::Enzyme.Compiler.var"#359#366"{LLVM.Module, LLVM.Function})(io::IOBuffer)
#     @ Enzyme.Compiler ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:4402
#   [2] sprint(::Function; context::Nothing, sizehint::Int64)
#     @ Base ./strings/io.jl:114
#   [3] sprint(::Function)
#     @ Base ./strings/io.jl:107
#   [4] lower_convention(functy::Type, mod::LLVM.Module, entry_f::LLVM.Function, actualRetType::Type, RetActivity::Type, TT::Union{…}, run_enzyme::Bool)
#     @ Enzyme.Compiler ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:4400
#   [5] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob{…})
#     @ Enzyme.Compiler ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:4713
#   [6] compile(target::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
#     @ GPUCompiler ~/.julia/packages/GPUCompiler/Gp8bZ/src/driver.jl:67
#   [7] compile
#     @ ~/.julia/packages/GPUCompiler/Gp8bZ/src/driver.jl:55 [inlined]
#   [8] _thunk(job::GPUCompiler.CompilerJob{…}, postopt::Bool)
#     @ Enzyme.Compiler ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:5946
#   [9] _thunk
#     @ ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:5944 [inlined]
#  [10] cached_compilation
#     @ ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:6001 [inlined]
#  [11] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{…}, A::Type{…}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, StrongZero::Bool, edges::Vector{…})
#     @ Enzyme.Compiler ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:6117
#  [12] thunk_generator(world::UInt64, source::Union{…}, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, StrongZero::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type, strongzero::Type)
#     @ Enzyme.Compiler ~/.julia/packages/Enzyme/LJjsP/src/compiler.jl:6312
#  [13] autodiff
#     @ ~/.julia/packages/Enzyme/LJjsP/src/Enzyme.jl:502 [inlined]
#  [14] autodiff
#     @ ~/.julia/packages/Enzyme/LJjsP/src/Enzyme.jl:542 [inlined]
#  [15] macro expansion
#     @ ~/.julia/packages/Enzyme/LJjsP/src/sugar.jl:286 [inlined]
#  [16] gradient(rm::ReverseMode{…}, f::typeof(func_ccall), x::Const{…}, args::SubArray{…})
#     @ Enzyme ~/.julia/packages/Enzyme/LJjsP/src/sugar.jl:273
#  [17] top-level scope
#     @ REPL[17]:1
# Some type information was truncated. Use `show(err)` to see complete types.

So it only happens when I use ccall and for that I need clang.

[vchuravy]

Perhaps you can get Clang from Clang_jll

[ymardoukhi]

Perhaps you can get Clang from Clang_jll

Worked like a charm! Will open the PR soon. Should I add the test in array.jl?

[wsmoses]

@ymardoukhi indepenent from the above, would you be able to open a PR to fix the typo?

[ymardoukhi]

@ymardoukhi indepenent from the above, would you be able to open a PR to fix the typo?

I will open it by the end of the day with the test.

[vchuravy]

Should I add the test

Or perhaps a new file embedded_bitcode.jl

[ymardoukhi]

See PR #2684

[vchuravy]

Fixed in #2722

@ymardoukhi your PR in #2684 is still very valuable since we had no test coverage of this feature.