Laravel Horizon handling business logic double reporting on retried long running jobs

[sts-ryan-holton]

Hi 👋,

I was hoping to gain some insights from others on handling retried queued jobs in Horizon regarding to long running jobs that contain many HTTP requests and reporting logic.

Here's a scenario:

I've got a lead processing system which exposes an endpoint where people can submit leads to, when a lead enters the system it's status goes from "pending" to "processing" and then loops over various HTTP requests to send that lead to external endpoints, with an unknown timeout on each one. The whole lead process could take 2, 3, 4 minutes. At each point, I have a reporting table that increments different columns for administrators to see how many leads came in and how many external requests happened.

The problem:

When I want to deploy the project, I need to restart Horizon, (the fast_termination flag is already on) in order to get the new code onto the system. Since I've got thousands of leads coming in per hour, there's no ideal time to deploy.

When I deploy, Horizon generates a new unique name for it:

Whilst the first part is customisable and linked to the machine's hostname, the second part is not -0VOA and is generated at random (it appears).

This means that since the original Horizon where the job is processing is gone, and a new one boots, the job can be retried...

But, here's the problem:

1. If I retry the job, all of my business reporting logic is re-ran and incremented twice, I could do something with $this->attempts() but that's either a "yes/no", since I don't know where it got to in the job, I wouldn't know what business logic to re-increment.
2. HTTP requests have an unknown amount of execution time, so they can't be fixed.

Final thoughts

I'm not sure how you're handling this kind of scenario in your project with queued jobs and what business logic you have in place.

[henzeb]

Do you really need long running jobs? Can't they be split up in multiple jobs?

[sts-ryan-holton]

For our business use case, yes, they need to be long running unfortunately

[henzeb]

In that case, maybe you should look into a blue green like deployment strategy. After deployment your jobs should be pushed to a different cache instance, which is being processed by a different horizon instance, if it's just a few minutes, the first instance can idle out.