Skip to content

Commit 31a2cd1

Browse files
committed
use new unique jobs implementation
This moves the library to use the new unique jobs implementation from riverqueue/river#590 and migrates the sqlalchemy driver to use a unified insertion path, allowing bulk inserts to use unique jobs.
1 parent 580a39c commit 31a2cd1

File tree

20 files changed

+530
-1605
lines changed

20 files changed

+530
-1605
lines changed

.github/workflows/ci.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ jobs:
6363
run: psql --echo-errors --quiet -c '\timing off' -c "CREATE DATABASE ${TEST_DATABASE_NAME};" ${ADMIN_DATABASE_URL}
6464

6565
- name: river migrate-up
66-
run: river migrate-up --database-url "$TEST_DATABASE_URL" --max-steps 5 # temporarily include max steps so tests can pass with unique fixes
66+
run: river migrate-up --database-url "$TEST_DATABASE_URL"
6767

6868
- name: Test
6969
run: rye test
@@ -109,7 +109,7 @@ jobs:
109109
run: psql --echo-errors --quiet -c '\timing off' -c "CREATE DATABASE ${DATABASE_NAME};" ${ADMIN_DATABASE_URL}
110110

111111
- name: river migrate-up
112-
run: river migrate-up --database-url "$DATABASE_URL" --max-steps 5 # temporarily include max steps so tests can pass with unique fixes
112+
run: river migrate-up --database-url "$DATABASE_URL"
113113

114114
- name: Run examples
115115
run: rye run python3 -m examples.all

README.md

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,8 @@ class JobArgs(Protocol):
4848
pass
4949
```
5050

51-
* `kind` is a unique string that identifies them the job in the database, and which a Go worker will recognize.
52-
* `to_json()` defines how the job will serialize to JSON, which of course will have to be parseable as an object in Go.
51+
- `kind` is a unique string that identifies them the job in the database, and which a Go worker will recognize.
52+
- `to_json()` defines how the job will serialize to JSON, which of course will have to be parseable as an object in Go.
5353

5454
They may also respond to `insert_opts()` with an instance of `InsertOpts` to define insertion options that'll be used for all jobs of the kind.
5555

@@ -95,16 +95,6 @@ insert_res.job
9595
insert_res.unique_skipped_as_duplicated
9696
```
9797

98-
### Custom advisory lock prefix
99-
100-
Unique job insertion takes a Postgres advisory lock to make sure that its uniqueness check still works even if two conflicting insert operations are occurring in parallel. Postgres advisory locks share a global 64-bit namespace, which is a large enough space that it's unlikely for two advisory locks to ever conflict, but to _guarantee_ that River's advisory locks never interfere with an application's, River can be configured with a 32-bit advisory lock prefix which it will use for all its locks:
101-
102-
```python
103-
client = riverqueue.Client(riversqlalchemy.Driver(engine), advisory_lock_prefix=123456)
104-
```
105-
106-
Doing so has the downside of leaving only 32 bits for River's locks (64 bits total - 32-bit prefix), making them somewhat more likely to conflict with each other.
107-
10898
## Inserting jobs in bulk
10999

110100
Use `#insert_many` to bulk insert jobs as a single operation for improved efficiency:

requirements-dev.lock

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
# all-features: false
88
# with-sources: false
99
# generate-hashes: false
10+
# universal: false
1011

1112
-e file:.
1213
asyncpg==0.29.0

requirements.lock

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
# all-features: false
88
# with-sources: false
99
# generate-hashes: false
10+
# universal: false
1011

1112
-e file:.
1213
sqlalchemy==2.0.30

0 commit comments

Comments
 (0)