(2.12) Filestore async flush

[MauriceVanVeen]

The filestore's AsyncFlush setting can now be enabled for a JetStream stream using the AllowAsyncFlush field. Some initial benchmarks showed a 10-15% performance increase when using a R3 stream (3-node cluster with each node in a different availability zone).

Only enabling the filestore's AsyncFlush setting without additional code would be unsafe. That's why this PR introduces some new mechanisms to track async writes, and only allow the data from the log to be compacted once it has hit JetStream's store. This makes the performance improvement essentially "free", with no negative consequences for data safety/consistency.

• Raft now has ApplyWritePending to track any pending writes done by the filestore. ApplyWritePersisted is called when the write is actually persisted on disk. This allows async flushing of data, while still informing the Raft code which data it can safely compact. (This also requires passing the index of the append entry down into the filestore, so it can call back up with that index once the relevant data is persisted)
• Because of the above, there's some inherent state desync because the stream's (applied) state and snapshot will be ahead of what may be persisted on disk. In that case a snapshot is marked "async", which informs the server its state on recovery is not complete and the missing entries are safely stored in its Raft log and can be replayed.
• The AllowAsyncFlush stream setting can be freely/safely enabled and disabled. It will only be effective when using file storage, and only when the stream is backed by a Raft log, i.e. it's replicated.

Relates to #6784

Signed-off-by: Maurice van Veen [email protected]

[MauriceVanVeen]

Going to try if this can be simplified a bit more, in draft for now.

[alexbozhenko]

How is this related to the sync_interval setting in nats.conf?

[MauriceVanVeen]

How is this related to the sync_interval setting in nats.conf?

That means when fsync is called on the file. Either always or a specified interval. Currently writes are always synchronous, the file write is done. And is fsync-ed on an interval, or always after the write if sync_interval: always. This PR makes the writes asynchronous when enabled. If sync_interval: always, the writes may still happen asynchronously, but when they are written, fsync would be called right after as well.