Skip to content

Conversation

@giordano
Copy link
Member

@giordano giordano commented Nov 21, 2025

Reverts #60105. Nightly builds of aarch64-darwin Julia hang at startup on some systems, notably on GitHub Actions, making all nightly jobs timeout (after 6 hours...), see https://discourse.julialang.org/t/ci-testing-hangs-on-macos-nightly/133909 (previously reported on Slack at https://julialang.slack.com/archives/CPWJ5DGG1/p1763055610279379). See also julia-actions/julia-runtest#155. The error message when the process receives a SIGTERM signal is

in expression starting at none:0
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0) at (unknown file)
__psynch_mutexwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0) at (unknown file)
Allocations: 1 (Pool: 1; Big: 0); GC: 0

I can reliably reproduce it locally on a M1 box with MacOSX 12.6 21.6.0, I can provide more information as needed.

I bisected the issue to #60105. CC @xal-0 for your information.

@giordano giordano added the revert This reverts a previously merged PR. label Nov 21, 2025
@giordano
Copy link
Member Author

On the machine where I can reproduce the hang using builds from buildkite, I also systematically can't compile latest master (9af9b15) because I get the error

    JULIA usr/lib/julia/basecompiler-o.a
ERROR: rethrow() not allowed outside a catch block
make[1]: *** [/Volumes/STORAGE/mose/repo/julia/usr/lib/julia/basecompiler-o.a] Error 1
make: *** [julia-sysimg-release] Error 2

which @xal-0 mentioned on Slack is happening also on CI here quite frequently when building Julia. This revert PR also resolves that issue for me. Looks like #60105 had quite a few issues on aarch64-darwin

@giordano
Copy link
Member Author

giordano commented Nov 21, 2025

And just to further confirm this PR resolves the hang:

$ curl -LsfO https://buildkite.com/organizations/julialang/pipelines/julia-master/builds/52235/jobs/019aa86e-9eba-44e6-8a8f-09e9120a3250/artifacts/019aa87a-47dc-4ebc-b32a-d8a74145df20
$ tar xzf 019aa87a-47dc-4ebc-b32a-d8a74145df20 
$ julia-305ae33084/bin/julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.14.0-DEV.1278 (2025-11-21)
 _/ |\__'_|_|_|\__'_|  |  revert-60105-jitlink-cgmemmgr/305ae33084f (fork: 1 commits, 0 days)
|__/                   |

julia> 

Also, my local build, now successful:

$ ./julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.14.0-DEV.1278 (2025-11-21)
 _/ |\__'_|_|_|\__'_|  |  revert-60105-jitlink-cgmemmgr/305ae33084 (fork: 1 commits, 0 days)
|__/                   |

julia> 

@giordano giordano merged commit ddf1b02 into master Nov 22, 2025
11 checks passed
@giordano giordano deleted the revert-60105-jitlink-cgmemmgr branch November 22, 2025 01:20
@giordano
Copy link
Member Author

giordano commented Nov 22, 2025

For the record, prompted by a request from @xal-0 I found that System Integrity Protection is disabled on both systems where I could reproduce the hang (GitHub Actions and another remote M1 box I have access to):

$ csrutil status
System Integrity Protection status: disabled.

while it's enabled on other aarch64-darwin machines where I can't reproduced the issue (this is also the default, so most users will probably not be affected):

% csrutil status 
System Integrity Protection status: enabled.

The correlation between SIP being disabled and the issues is quite strong for the moment.

xal-0 added a commit to xal-0/julia that referenced this pull request Nov 24, 2025
Apple ARM CPUs treat the `ic ivau` as a memory read, which causes a confusing
crash in DualMapAllocator if we try using it on a wr_addr that has been
mprotected to `Prot::NO`, since we are still holding the allocator lock.

This re-lands JuliaLang#60105, after it was reverted in JuliaLang#60196.  Thanks @giordano!
xal-0 added a commit to xal-0/julia that referenced this pull request Nov 24, 2025
Apple ARM CPUs treat the `ic ivau` as a memory read, which causes a confusing
crash in DualMapAllocator if we try using it on a wr_addr that has been
mprotected to `Prot::NO`, since we are still holding the allocator lock.

For Apple aarch64 systems with SIP disabled, this will result in some memory
savings, since DualMapAllocator will now work there.  Like before, other JITLink
platforms, namely Linux aarch64 and RISC-V, will benefit too.

This re-lands JuliaLang#60105, after it was reverted in JuliaLang#60196.  Thanks @giordano!
Keno pushed a commit that referenced this pull request Nov 25, 2025
…ager/#60105) (#60230)

Apple ARM CPUs treat the `ic ivau` as a memory read, which causes a
confusing crash in DualMapAllocator if we try using it on a `wr_addr`
that has been mprotected to `Prot::NO`, since we are still holding the
allocator lock.

For Apple aarch64 systems with SIP disabled, this will result in some
memory savings, since DualMapAllocator will now work there. Like before,
other JITLink platforms, namely Linux aarch64 and RISC-V, will benefit
too.

This re-lands #60105, after it was reverted in #60196. Thanks @giordano!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

revert This reverts a previously merged PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants