Exception "Resource temporarily unavailable" appears during executing rustup on Linux system with ulimit restrictions

[klim-iv]

For hosting my web projects I am using Linux on 3-rd party cloud provider
On that system installed 104 CPUs and also set maximum number of processes for user = 40 ( by ulimit -u )
Current std::thread::available_parallelism implementation always return 104 (instead 40), 3-rd party crates (for example rayon) try to creates thread pool for max , but host system restricts such attempts and rust throw to exception

Due to this exception I can't even install Rust (with using rustup) in my home at this system

Traceback for rustup

thread 'main' panicked at .../rayon-core-1.12.1/src/registry.rs:168:10:
The global thread pool has not been initialized.: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }
stack backtrace:
   0:     0x55919894eb6c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h388b5563225da53c
   1:     0x5591985ea79b - core::fmt::write::h7f11a232f16bffb4
   2:     0x559198911752 - std::io::Write::write_fmt::h6c69f7c334e53b10
   3:     0x559198951d48 - std::panicking::default_hook::{{closure}}::h33d794e7524ee3df
   4:     0x5591989519d1 - std::panicking::default_hook::h761c6a87e7786414
   5:     0x5591989529e9 - std::panicking::rust_panic_with_hook::h76eaaf856d4627ce
   6:     0x559198952755 - std::panicking::begin_panic_handler::{{closure}}::h822501910e3f1a1e
   7:     0x5591989526b9 - std::sys::backtrace::__rust_end_short_backtrace::hd0a94a8e1050aebb
   8:     0x5591989526a4 - rust_begin_unwind
   9:     0x5591985e9502 - core::panicking::panic_fmt::h20afa97c0700e760
  10:     0x5591985ee7b5 - core::result::unwrap_failed::hb85382bc192f0b4e
  11:     0x55919865e65b - rayon_core::registry::global_registry::hd507095517146e2d
  12:     0x55919886c04e - remove_dir_all::_impl::remove_dir_contents_recursive::h95c8f79df37c7add
  13:     0x559198810951 - remove_dir_all::_impl::remove_dir_all_path::h1cbfde4847891883
  14:     0x55919882f6ed - core::ptr::drop_in_place<rustup::dist::temp::Dir>::h6d8f29d59fbbbade
  15:     0x55919886619f - rustup::dist::component::package::TarPackage::new::hd25c5afdf0eea170
  16:     0x55919885c838 - rustup::dist::component::package::TarXzPackage::new::hc48a3c02f0beadd1
  17:     0x55919881a42f - rustup::dist::manifestation::Manifestation::update::h7b45faa9d05e9945
  18:     0x559198811f0c - rustup::dist::dist::try_update_from_dist_::h8c05956ab5371c1c
  19:     0x55919880f583 - rustup::install::InstallMethod::install::h09f0bf01539cc897
  20:     0x55919880c645 - rustup::toolchain::distributable::DistributableToolchain::install::hfc214fca9f3953f6
  21:     0x559198908489 - rustup::cli::self_update::install::h81aa5f4060099435
  22:     0x5591989005d2 - rustup::cli::setup_mode::main::h8ad5c7f7a0d657f3
  23:     0x55919854b221 - rustup_init::main::h5a60573c3e8e09fa
  24:     0x5591985479f3 - std::sys::backtrace::__rust_begin_short_backtrace::h80d522419e5738bf
  25:     0x55919854bc74 - main
  26:     0x7f947a429d90 - <unknown>
  27:     0x7f947a429e40 - __libc_start_main
  28:     0x559198547925 - _start
  29:                0x0 - <unknown>
thread 'main' panicked at core/src/panicking.rs:229:5:
panic in a destructor during cleanup
thread caused non-unwinding panic. aborting.

[fmease]

@rustbot transfer rustup

[klim-iv]

fix here:
rust-lang/rust#143614

[ChrisDenton]

I don't think this is a rustup issue. It's just one application where it surfaces.

We could attempt to workaround it in rustup somehow but that wouldn't fix it for anyone else.

[djc]

@tgross35 if you're considering merging the proposed

• take into account the num of processes by ulimit rust#143614

do you think it makes sense to move this issue back into rust-lang/rust, too, perhaps with a different title?

[cuviper]

In general, rayon can be controlled by the environment variable RAYON_NUM_THREADS to override the default use of available_parallelism. If you're in such a constrained environment, I would suggest setting that well below your actual resource limit, because consuming the entire limit in a threadpool means nothing else can start after that.

[klim-iv]

as for me, I am newbie in Rust, this issue is very very strange
when I faced with this crash, sure I found that possible to control threads with RAYON_NUM_THREADS , but root of issue in another place

Can you agree, that this is not normal when you start Rust installation and get so strange crash?
Why user should to know anything about RAYON_NUM_THREADS for start rustup?
If authors want to change rayon to another crate, needs to know another env variable?

Anyway, such errors, that have root cause in std lib, talks that rust is not for humans, sorry :-)

[cuviper]

I don't think this is a Rust-specific issue. For example, OpenMP is very similar in using the number of logical CPUs with an override OMP_NUM_THREADS. IMO, it's strange to have so many CPUs with such a low limit, whereas if they did that with cgroups instead, then available_parallelism() would account for that. (I'm sure it would be difficult to get your cloud provider to change though.)

In rustup's particular case, maybe it would make sense to initialize rayon's global pool with a maximum number of threads, because 104 threads is likely overkill for what it's doing.

[ChrisDenton]

I'd be ok with setting a reasonable upper limit in rustup.

[djc]

Maybe 8 or 16?

[ChrisDenton]

I would suspect we see diminishing returns fairly quickly if we're mostly I/O bound so 8 should be fine. But it is something it would be worth benchmarking on different systems. Still, even without benchmark result setting it to 16 would be, I think, a safe choice and would fix this issue.

[klim-iv]

I don't think this is a Rust-specific issue.

maybe you are right, but this is not first issue related to Rust and thread pools.
My first issue was with more strange behavior :-)

Listen to my story, maybe you understood how Rust threads looks from the outside.

I wanted to create DeltaChat bot, that will work on provider's host with ulimit restrictions (before I don't know about this restrictions)
Such bots is Python programs, that using another Pytnon packages. One of package - is python wrapper of Rust written lib.
And this rust lib using crate threadpool . That crate also try to create pull of threads for max number of CPUs = 104 (not ulimit = 40), but it created more 'stupid' (sorry) : in cycle, it starts new thread until number of thread will equal number of CPUs.
Sure, system kill all new threads when it quantity equal to 40 , that follow to infinite cycle and, in addition : impossible to stop python script, impossible to kill script from another terminal because impossible to start new bash process (limit exceeded),
no logs, no exceptions. It looks like python script kill all my sessions and processes on host without any info. WTF??? :-)

I found issue in DeltaChat and threadpool, but don't know how to fix it yet in Rust library.
So, I want to say : approach where library start to use max available CPU resources meets in many crates, this is Rust approach (to my mind) and it not a good. As I understood - you are Senior Rust Developers and have voice in community - please think about this and how to change such approach.

IMO, it's strange to have so many CPUs with such a low limit, whereas if they did that with cgroups instead, then available_parallelism() would account for that. (I'm sure it would be difficult to get your cloud provider to change though.)

This is provider chose, they know about cgroups but due business risks they not want to make so big change in servers management

Sorry for big comment, but maybe it can help to make Rust better!