Scanner stuck in ScanCode loop without any error

[voidpetal]

Describe the bug

The Scanner gets stuck for a very long time in scancode loop and doesn't output anything to help debug what's the issue.

To Reproduce

This is tied to a couple projects, still didn't find any connection between them so that I can say how to replicate...

Expected behavior

ORT outputs helpful debug statements and finishes scanning sooner.

Console / log output

10:45:47.537 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'scancode --copyright --license --info --strip-root --timeout 300 --processes 1 --license-references --json /tmp/ort-ScanCode18304449959201815913/result.json /tmp/ort-DefaultProvenanceDownloader2867394273174419430' in '/builds/ort'...
10:45:51.152 [HikariPool-1:housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=5/5, idle=5/5, active=0, waiting=0)
10:45:51.152 [HikariPool-1:housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled.
10:46:21.152 [HikariPool-1:housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=5/5, idle=5/5, active=0, waiting=0)
10:46:21.152 [HikariPool-1:housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled.
10:46:28.374 [HikariPool-1:housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - keepalive: connection org.postgresql.jdbc.PgConnection@58ba1760 is alive
10:46:29.108 [HikariPool-1:housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - keepalive: connection org.postgresql.jdbc.PgConnection@60a739ee is alive
...
11:07:45.180 [HikariPool-1:connection-closer] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection@23a5fa34: (connection has passed maxLifetime)
11:07:45.182 [HikariPool-1:connection-adder] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Attempting to create/setup new connection (a0b4057c-60dd-4526-bd6f-3b6f3fe6f117)
11:07:45.229 [HikariPool-1:connection-adder] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Established new connection (a0b4057c-60dd-4526-bd6f-3b6f3fe6f117)
11:07:45.229 [HikariPool-1:connection-adder] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@1ee571d7
11:07:51.163 [HikariPool-1:housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats 
...

Environment

• ORT version: 3b72c0e
• Java version: 21
• OS: linux alpine
• ScanCode version: 32.3.3
• Python-inspector version: 0.14.0

[sschuberth]

This goes a bit into the direction of #3296. ORT could make it easier to debug issues like this by streaming ScanCode's output to ORT's log, which relates to #4668. However, the underlying issue might be with ScanCode itself, so a third-party issue.

One thing to try @erhc is to upgrade to ORT 62.0.0 which comes with ScanCode 32.4.0 which in turn might fix the issue.

[alsutton]

As a data point, ORT 62.1.0 has been running for several days on one project for me, and it appears to be pegging a single core at 100%. I'm going to use jmap and jfr to see if I can identify the cause.

[voidpetal]

Hi @sschuberth, thanks for the reply. I tried running ORT 62.0.0 on the same project and there is no difference.

[fviernau]

Two ideas:

1. Scan the package source code where it gets stuck directly with scan-code from the command line and see what happens
2. Check how much / engough main memory is available

[sschuberth]

it appears to be pegging a single core at 100%.

At least "pegging a single core" should not be the case, as ScanCode 32.4.0 defaults to use N-1 cores. Are you maybe using a custom ScanCode version @alsutton?

In any case, it's important to know the package / source code being scanned, and whether you can reproduce the problem with ScanCode alone.

[alsutton]

@sschuberth This run took place with an out-of-the-box install (literally extract the archive, run ort analyze --input-dir . --output-dir ../ort/), so I believe that means the default version of ScanCode, with the memory options set to allow 128GB of Java Heap (-Xmx) on a machine with 256GB of total RAM by doing export JAVA_OPTS="$JAVA_OPTS -Xmx128g" ahead of the run.

The codebase is a monorepo with a mix of Servlet, CLI, Android, and iOS code. I'd say it's roughly 250k LoC, and a 150-200 build targets spread across gradle and iOS builds.

At no point were any warnings, or errors, printed about possible issues in the scan engine.

I ran some profiling utilities against it after 3 days, and jmap shows a lot of LinkedHashMap entries;

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:       3473155      194496680  java.util.LinkedHashMap$Entry (java.base@21.0.7)
   2:       2063563      181593544  org.gradle.tooling.internal.adapter.MethodInvocation
   3:       1074822      157397216  [Ljava.util.HashMap$Node; (java.base@21.0.7)
...

Java Flight Recorder (jfr) seems to indicate that there's a lot of GC going on;

$ jfr summary ort.jfr

 Version: 2.1
 Chunks: 12
 Start: 2025-07-07 07:09:31 (UTC)
 Duration: 1321 s

 Event Type                              Count  Size (bytes) 
=============================================================
 jdk.GCPhaseParallel                   4428655     124144302
 jdk.ObjectAllocationSample             166651       3098810
 jdk.ExecutionSample                     59068        806836
 jdk.TenuringDistribution                48600        693715
 jdk.ModuleExport                        19752        235128
 jdk.MetaspaceChunkFreeListSummary       12972        285384
 jdk.GCReferenceStatistics               12972        181609
 jdk.GCPhasePauseLevel1                   9750        435691
...

So I'm wondering if the default configuration is not very memory efficient, and the size of the monorepo is causing ORT to get stuck GC thrashing.

[fviernau]

so I believe that means the default version of ScanCode, with the memory options set to allow 128GB

ScanCode is not affected by these options: It runs in its own process and not inside a JVM.
Note that for setting memory options for ORT you can also use ORT_OPTS.

Could you run scan-code (without ORT) from the command line against the codebase and see how it behaves @alsutton ?

[alsutton]

ScanCode is not affected by these options: It runs in its own process and not inside a JVM.
Note that for setting memory options for ORT you can also use ORT_OPTS.

If ScanCode could be the issue, then it's probably worth updating https://oss-review-toolkit.org/ort/docs/getting-started/runtime-requirements to show how to affect it.

Could you run scan-code (without ORT) from the command line against the codebase and see how it behaves @alsutton ?

Can you point me at some instructions on how to do that? The ort install I have doesn't appear to have a separate execution point to run only ScanCode.