Ingester writer can die and not get revived, or if revived write jobs accumulate anyway

[scottyeager]

Seen on the staging bot ingester. No errors were logged, write jobs just accumulated in the queue. Need to see about making that more robust.

[scottyeager]

Here's another case where the writer proc was respawned but processing write jobs didn't start again:

2024-11-13 12:45:36.989369 processed 0 blocks in 30 seconds 13 blocks queued 5 processes alive 0 write jobs
Writer proc died, respawning it
2024-11-13 12:46:07.730442 processed 0 blocks in 30 seconds 0 blocks queued 5 processes alive 23 write jobs

[scottyeager]

We can also get into a state where the proc dies repeatedly:

Starting 5 workers to process 5088 blocks, with starting block number 14908601 and ending block number 15106975
2024-11-13 21:13:01.518279 processed 0 blocks in 30 seconds 5080 blocks queued 5 processes alive 10 write jobs
Writer proc died, respawning it
2024-11-13 21:13:32.653740 processed 0 blocks in 30 seconds 4309 blocks queued 5 processes alive 786 write jobs
Writer proc died, respawning it
2024-11-13 21:14:04.020105 processed 0 blocks in 30 seconds 3752 blocks queued 5 processes alive 1349 write jobs
Writer proc died, respawning it
2024-11-13 21:14:35.367496 processed 0 blocks in 30 seconds 3185 blocks queued 5 processes alive 1921 write jobs
Writer proc died, respawning it

[scottyeager]

These new cases noted today were actually caused by OOM killing. Deploying the new more efficient version of the ingester resolved that.