Raise error when advisory lock cannot be acquired

[Farjaad]

Gitlab Issue: https://gitlab.com/jklabsinc/OpsLevel/-/issues/13122

I found traces where jobs fail with a lock timeout after 50s when inserting into entity_node_hierarchy so that means our jobs do fail if the actual insert is taking long.

But there are traces (example) where the job sits for minutes just doing SELECT GET_LOCK until it gets shutdown by sidekiq. The SELECT GET_LOCK comes from the closure_tree gem and the gem always calls with_advisory_lock without any options which means that it waits indefinitely for the lock.

So this MR uses with_advisory_lock! so that it raises an error if we can't get the lock instead of just waiting indefinitely so that the jobs will end up in the dead set if there's a lot of contention. https://github.com/ClosureTree/with_advisory_lock/blob/master/lib/with_advisory_lock/concern.rb#L16

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Farjaad]

ahh I forgot to squash my commits 😢