`repy.py` / `safe.py` memory efficiency issues

[aaaaalbert]

It's not really evident on platforms with more than 128 MB of memory, but a few things in repy.py and safe.py are less than efficient when dealing with memory, at least transiently. (I will use this issue to talk about these problems overall, and probably add specific sub-issues for the different parts that are affected).

The startup process of a Repy sandbox works like this. (When started through the nodemanager, the callargs will differ a little, but the process is unchanged.)

• Repy initializes, then forks resource manager and the actual sandbox. --- Better fork early while the process isn't that large yet, then init sandbox in child only. (See also nmmain self-daemonizing is memory inefficient nodemanager#117).
• safe.py which is called by the sandbox serves three mostly separate purposes:
  • Static code analysis (catches print, illegal var names)
  • Actual runtime execution
  • SafeDict
• safe.py Popen()s another process for safe_check.py which in turn imports safe, of which it only needs the static-checking part.
• safe.py's _check_node is tail recursive, but Python does no elimination, and so the call stack can grow large. --- Do breadth-first instead, visit children etc. of nodes later.

I'm working on these issues one by one and will report what the memory savings are for each.

[aaaaalbert]

First datapoint on safe.py's size and static check strategy: I ran <helloworld_plus_infinite_sleep.r2py python safe_check.py on my OpenWrt WGT634U. The RAM consumption, 5 MB, is by far dominated by the imports. Recursive/breadth-first checking of nodes has no reliably measurable impact.

[aaaaalbert]

Still looking at static safety checks for a hello-world type program:

• safe_check using plain vanilla safe.py consumes 4.9 MB of RAM
• Replacing nonportable.getruntime with time.time and thus saving one import saves 200kB
• Factoring out the static safety checks that this experiment uses (i.e. removing SafeDict, safe_run and the likes) saves another 50kB

[aaaaalbert]

After splitting static checks and the runtime parts of safe, running python repy.py restrictions.default test.r2py from the command line consumes 12.6 MB of RAM maximum during safety checking (was 15.2 using vanilla), and 8.1 MB in steady state (was 9.5).

This is a saving of 2.6 MB over vanilla max, and 1.4 MB better than vanilla steady state consumption.