documentation updates, changelog

Author: bgentry

CHANGELOG.md

@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Breaking
+
+- **Breaking change:** The return type of `Client#insert_many` and `Client#insert_many_tx` has been changed. Rather than returning just the number of rows inserted, it returns an array of all the `InsertResult` values for each inserted row. Unique conflicts which are skipped as duplicates are indicated in the same fashion as single inserts (the `unique_skipped_as_duplicated` attribute), and in such cases the conflicting row will be returned instead. [PR #38](https://github.com/riverqueue/riverqueue-python/pull/38).
+- **Breaking change:** Unique jobs no longer allow total customization of their states when using the `by_state` option. The pending, scheduled, available, and running states are required whenever customizing this list.
+
+### Added
+
+- The `UniqueOpts` class gains an `exclude_kind` option for cases where uniqueness needs to be guaranteed across multiple job types.
+- Unique jobs utilizing `by_args` can now also opt to have a subset of the job's arguments considered for uniqueness. For example, you could choose to consider only the `customer_id` field while ignoring the other fields:
+
+  ```python
+  UniqueOpts(by_args=["customer_id"])
+  ```
+
+  Any fields considered in uniqueness are also sorted alphabetically in order to guarantee a consistent result across implementations, even if the encoded JSON isn't sorted consistently.
+
+### Changed
+
+- Unique jobs have been improved to allow bulk insertion of unique jobs via `Client#insert_many`.
+
+  This updated implementation is significantly faster due to the removal of advisory locks in favor of an index-backed uniqueness system, while allowing some flexibility in which job states are considered. However, not all states may be removed from consideration when using the `by_state` option; pending, scheduled, available, and running states are required whenever customizing this list.
+
 ## [0.7.0] - 2024-07-30
 
 ### Changed
@@ -79,4 +101,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Initial release, supporting insertion through [SQLAlchemy](https://www.sqlalchemy.org/) and its underlying Postgres drivers like [`psycopg2`](https://pypi.org/project/psycopg2/) or [`asyncpg`](https://github.com/MagicStack/asyncpg) (for async).
+- Initial release, supporting insertion through [SQLAlchemy](https://www.sqlalchemy.org/) and its underlying Postgres drivers like [`psycopg2`](https://pypi.org/project/psycopg2/) or [`asyncpg`](https://github.com/MagicStack/asyncpg) (for async).

README.md

@@ -95,12 +95,14 @@ insert_res.job
 insert_res.unique_skipped_as_duplicated
 ```
 
+Unique jobs can also be inserted in bulk.
+
 ## Inserting jobs in bulk
 
 Use `#insert_many` to bulk insert jobs as a single operation for improved efficiency:
 
 ```python
-num_inserted = client.insert_many([
+results = client.insert_many([
     SimpleArgs(job_num=1),
     SimpleArgs(job_num=2)
 ])
@@ -109,7 +111,7 @@ num_inserted = client.insert_many([
 Or with `InsertManyParams`, which may include insertion options:
 
 ```python
-num_inserted = client.insert_many([
+results = client.insert_many([
     InsertManyParams(args=SimpleArgs(job_num=1), insert_opts=riverqueue.InsertOpts(max_attempts=5)),
     InsertManyParams(args=SimpleArgs(job_num=2), insert_opts=riverqueue.InsertOpts(queue="high_priority"))
 ])

src/riverqueue/insert_opts.py

@@ -82,13 +82,8 @@ class UniqueOpts:
     args and queues. If either args or queue is changed on a new job, it's
     allowed to be inserted as a new job.
 
-    TODO update description ⚠ ⚠️ ⚠
-
-    Uniquenes is checked at insert time by taking a Postgres advisory lock,
-    doing a look up for an equivalent row, and inserting only if none was found.
-    There's no database-level mechanism that guarantees jobs stay unique, so if
-    an equivalent row is inserted out of band (or batch inserted, where a unique
-    check doesn't occur), it's conceivable that duplicates could coexist.
+    Uniqueness relies on a hash of the job kind and any unique properties along
+    with a database unique constraint.
     """
 
     by_args: Optional[Union[Literal[True], List[str]]] = None
@@ -98,6 +93,14 @@ class UniqueOpts:
 
     Default is false, meaning that as long as any other unique property is
     enabled, uniqueness will be enforced for a kind regardless of input args.
+
+    When set to true, the entire encoded args will be included in the uniqueness
+    hash, which requires care to ensure that no irrelevant args are factored
+    into the uniqueness check. It is also possible to use a subset of the args
+    by passing a list of string keys to include in the uniqueness check.
+
+    All keys are sorted alphabetically before hashing to ensure consistent
+    results.
     """
 
     by_period: Optional[int] = None

tests/client_test.py

@@ -295,12 +295,8 @@ def to_json(self) -> str:
     ordered_json = '{"a": 1, "b": 2, "c": 3}'
     reverse_ordered_json = '{"c": 3, "b": 2, "a": 1}'
 
-    insert_res1 = client.insert(
-        JsonArgs(json_str=ordered_json), insert_opts=insert_opts
-    )
-    insert_res2 = client.insert(
-        JsonArgs(json_str=reverse_ordered_json), insert_opts=insert_opts
-    )
+    client.insert(JsonArgs(json_str=ordered_json), insert_opts=insert_opts)
+    client.insert(JsonArgs(json_str=reverse_ordered_json), insert_opts=insert_opts)
 
     # Get the unique keys that were generated
     call_args1 = mock_exec.job_insert_many.call_args_list[0][0][0]  # type: ignore[index]