File Sink and Async programs

[gwbres]

Hello,

First off: thanks for your crate! the auto rotating logger is a very nice feature.

My application is 100% asynchronous (based off Tokio + Tonic), does that mean I have to use the AsyncFileSink ?
I presume if you don't, you will create a bottleneck when internal threads try to access the Arc

[SpriteOvO]

tl;dr: You don't have to use AsyncPoolSink if you are just writing log to file (with not very verbose flush_level_filter value).

---

Full explanation

For file sinks, the bottleneck will not be happening on the Arc (accessing a Arc is almost zero-cost, just one more indirect pointer access), but will be in the internal locks for writing data to std::fs::File.

To avoid this bottleneck by default, spdlog-rs has already wrapped the File object with std::io::BufWriter. The effect of this behavior is that data writes are not actually performed to the filesystem, but are cached in the BufWriter until the buffer is full or flushed explicitly.

(Tip: Explicit flushes can be performed by manually calling Logger::flush or setting up auto-flush policies via Logger::set_flush_level_filter and Logger::set_flush_period)

For AsyncPoolSink, instead of using the Rust async runtime (e.g. tokio), it forwards operations log and flush to a ThreadPool to perform them asynchronously (in other threads). But for file sinks, which already use BufWriter, the log operation for file sinks is just writing data to a memory buffer, which is cheap enough, and according to my benchmarking, using AsyncPoolSink with a file sink sometimes doesn't improve much performance.

The best cases for using AsyncPoolSink are when you are sure you will be using some sinks with expensive log or flush operations. For example, you are using something like HttpSink, UdpSink, DiscordSink, EmailSink, etc. Yes, operations involving networks are always expensive, and they are some typical examples. AsyncPoolSink doesn't have to be limited to network related, something like a FileSink with flush_level_filter LevelFilter::All set (meaning that each log will be flushed in real time without caching) can be considered expensive as well.

[gwbres]

Thanks so much @SpriteOvO for such a detailed answer

You don't have to use AsyncPoolSink if you are just writing log to file (with not very verbose flush_level_filter value).

That is indeed my use case. I may use high verbosity as we're debugging and in early stage, but we also don't have that much information to log.

the bottleneck will not be happening on the Arc [..] but will be in the internal locks for writing data to std::fs::File.

👍

Tip: Explicit flushes can be performed by manually calling Logger::flush or setting up auto-flush policies via Logger::set_flush_level_filter and Logger::set_flush_period)

Yeah I had noticed those in the API, very good feature to have

The best cases for using AsyncPoolSink are when you are sure you will be using some sinks with expensive log or flush operations.

thank you for this other detail. I will stick to synchronous File sink, as it seems to do the job perfectly

[gwbres]

Thanks for the great lib ! it really helps 🚀