|
| 1 | +=========== |
| 2 | +Task Queues |
| 3 | +=========== |
| 4 | + |
| 5 | +This application features a Redis-backed job/task queue thanks to the :doc:`SAQ <saq:index>` library. |
| 6 | +You can use it to run one-off tasks, or recurring tasks on a cron-like schedule. |
| 7 | + |
| 8 | +We are utilizing the `litestar-saq <https://github.com/cofin/litestar-saq>`_ plugin for native |
| 9 | +integration with Litestar. |
| 10 | + |
| 11 | +Background |
| 12 | +---------- |
| 13 | + |
| 14 | +Your application's task queue serves a variety of purposes, this section aims to explain when and why you would |
| 15 | +want to use a task queue. |
| 16 | + |
| 17 | +Cron jobs are essential for updating the database, performing regular cleanup, sending scheduled emails for various |
| 18 | +purposes, and conducting auditing or alerting activities. |
| 19 | + |
| 20 | +Startup jobs focus on initializing essential services, loading initial data, conducting health checks, starting |
| 21 | +monitoring services, and allocating necessary resources like database connections. |
| 22 | + |
| 23 | +Conversely, shutdown jobs ensure the graceful release of resources, synchronization of |
| 24 | +data with storage, completion of background jobs, and logging of shutdown activities. |
| 25 | + |
| 26 | +Additionally, one-off jobs handle on-demand processing of user-generated content, execution of ad-hoc database tasks, |
| 27 | +administrative operations like account resets or data migrations, and event-driven tasks triggered by |
| 28 | +specific user actions or system events. |
| 29 | + |
| 30 | +.. tip:: There are several alternatives to ``SAQ``, but we have chosen it for its speed, ease of use, and feature set. |
| 31 | + Some other options include: |
| 32 | + |
| 33 | + * Cron jobs via the OS |
| 34 | + * `Systemd timers <https://wiki.archlinux.org/title/systemd/Timers>`_ |
| 35 | + * `RQ <https://github.com/rq/rq>`_ |
| 36 | + * `ARQ <https://github.com/samuelcolvin/arq>`_ |
| 37 | + * `SAQ <https://github.com/tobymao/saq>`_ |
| 38 | + * `Celery <https://docs.celeryq.dev/en/stable/>`_ |
| 39 | + |
| 40 | +How |
| 41 | +--- |
| 42 | + |
| 43 | +Jobs are set up on a per-domain basis. The ``Worker`` class is instantiated when the application starts up. |
| 44 | +This calls :func:`litetar_saq.cli.run_worker_process` that has looks at certain directories to find jobs. |
| 45 | + |
| 46 | +Each domain has a directory in it for jobs, in the pattern of ``src/app/domain/$DOMAIN/jobs``. There are separate |
| 47 | +modules for each type of job (``scheduled``, ``startup``, ``shutdown``, and ``tasks``). |
| 48 | + |
| 49 | +It is self-explanatory, jobs in the ``startup`` directory are run on startup, etc. |
| 50 | +Scheduled jobs are the only slightly unique ones, in that they are a list of :class:`CronJob <saq.job.CronJob>`'s. |
| 51 | +That looks something like: |
| 52 | + |
| 53 | +.. code-block:: python |
| 54 | +
|
| 55 | + scheduled_tasks: list[CronJob] = [ |
| 56 | + CronJob( |
| 57 | + function=generate_analytics_report, |
| 58 | + cron="*/30 * * * *", |
| 59 | + heartbeat=30, |
| 60 | + unique=True, |
| 61 | + timeout=120, |
| 62 | + ), |
| 63 | + CronJob( |
| 64 | + function=send_daily_digest, |
| 65 | + cron="*/35 * * * *", |
| 66 | + heartbeat=30, |
| 67 | + unique=True, |
| 68 | + timeout=120, |
| 69 | + ), |
| 70 | + ] |
| 71 | +
|
| 72 | +So you see we have to scheduled jobs every 30 and 35 minutes. When the app starts up, these will run just as a |
| 73 | +system cron job would. |
0 commit comments