Offline runners are not removed but are selected for task execution #1796
Description
Last week I started the process of migrating my Semaphore instance to using remote runners. In the process of testing, I spun up a few temporary runners using Docker and didn't bother setting up a persistent storage volume for them. After doing this a few times I could no longer launch a task, as every task would hang indefinitely on "starting..." while waiting to be assigned to a runner. A quick investigation of the database showed that all the runners I launched and exited were still registered with Semaphore and presumably still being assigned tasks. I was able to resolve this by connecting to the database and manually deleting the runner registrations for the now-deactivated runners.
I know runners are still fairly new to the ecosystem, but I wanted to raise this as both an operational and administration issue.
Some potential solutions I thought of quick:
- Have Semaphore check whether a runner is online before assigning a task to it
- Have Semaphore periodically check whether all runners are online and remove any that aren't
- Runners should also periodically re-register to account for runners that may occasionally lose connectivity to the server
- Add an administration panel in the UI that shows the list of registered runners and allow admins to delete them
This is an amazing project and I'm really excited to work with it more!