Skip to content

Commit ea52306

Browse files
brandonwillardbrandonwillard
committed
Add stream vs. stack benchmarks to README
1 parent 9fef36c commit ea52306

File tree

4 files changed

+164
-20
lines changed

4 files changed

+164
-20
lines changed

‎README.md‎

Lines changed: 81 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,87 @@ See the full example in the [examples directory](https://github.com/pythological
120120

121121
## Performance and Reliability
122122

123-
`unification`'s current design allows for unification and reification of nested structures that break the Python stack recursion limit. This scalability incurs an overhead cost compared to simple stack-based recursive unification/reificiation.
123+
`unification`'s current design allows for unification and reification of nested structures that would otherwise break the Python stack recursion limit. It uses a generator-based design to "stream" the unifications and reifications.
124+
125+
Below are some stack vs. stream benchmarks that demonstrate how well the stream-based approach scales against the stack-based approach in terms of unifying and reifying deeply nested lists containing integers. These benchmarks were generated from the tests in `tests/test_benchmarks.py` using CPython 3.7.3.
126+
127+
<details><summary>Stack vs. stream benchmarks</summary>
128+
<p>
129+
130+
```python
131+
-------------------------------------------------------------------------------- benchmark 'reify_chain size=10': 2 tests -------------------------------------------------------------------------------
132+
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS (Kops/s) Rounds Iterations
133+
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
134+
test_reify_chain_stack[10] 41.0790 (1.0) 545.1940 (3.20) 52.9087 (1.07) 9.7964 (1.04) 50.8650 (1.08) 6.4301 (8.37) 11815;10849 18.9005 (0.93) 260164 1
135+
test_reify_chain_stream[10] 42.4410 (1.03) 170.5540 (1.0) 49.3080 (1.0) 9.3993 (1.0) 47.2400 (1.0) 0.7680 (1.0) 14962;102731 20.2807 (1.0) 278113 1
136+
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
137+
138+
------------------------------------------ benchmark 'reify_chain size=1000': 1 tests -----------------------------------------
139+
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
140+
-------------------------------------------------------------------------------------------------------------------------------
141+
test_reify_chain_stream_large[1000] 7.7722 28.2579 10.0723 2.5087 9.4899 0.3106 70;155 99.2820 1528 1
142+
-------------------------------------------------------------------------------------------------------------------------------
143+
144+
------------------------------------------------------------------------- benchmark 'reify_chain size=300': 2 tests --------------------------------------------------------------------------
145+
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
146+
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
147+
test_reify_chain_stack[300] 1.5183 (1.0) 22.1821 (1.19) 1.9826 (1.0) 1.5511 (1.16) 1.7410 (1.0) 0.0801 (1.0) 144;684 504.3878 (1.0) 7201 1
148+
test_reify_chain_stream[300] 1.7059 (1.12) 18.6020 (1.0) 2.1237 (1.07) 1.3389 (1.0) 1.9260 (1.11) 0.1020 (1.27) 118;585 470.8745 (0.93) 6416 1
149+
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
150+
151+
--------------------------------------------------------------------------------- benchmark 'reify_chain size=35': 2 tests --------------------------------------------------------------------------------
152+
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS (Kops/s) Rounds Iterations
153+
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
154+
test_reify_chain_stream[35] 129.2780 (1.0) 868.1510 (1.02) 190.0433 (1.11) 36.2784 (1.41) 179.5690 (1.08) 21.5360 (2.30) 1535;1455 5.2620 (0.90) 26072 1
155+
test_reify_chain_stack[35] 150.7850 (1.17) 853.7920 (1.0) 170.5166 (1.0) 25.7944 (1.0) 165.8500 (1.0) 9.3530 (1.0) 3724;5480 5.8645 (1.0) 81286 1
156+
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
157+
158+
------------------------------------------- benchmark 'reify_chain size=5000': 1 tests ------------------------------------------
159+
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
160+
---------------------------------------------------------------------------------------------------------------------------------
161+
test_reify_chain_stream_large[5000] 46.9073 86.9737 52.9724 6.6919 49.6787 3.9609 68;68 18.8778 292 1
162+
---------------------------------------------------------------------------------------------------------------------------------
163+
164+
------------------------------------------------------------------------------- benchmark 'unify_chain size=10': 2 tests -------------------------------------------------------------------------------
165+
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS (Kops/s) Rounds Iterations
166+
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
167+
test_unify_chain_stream[10] 77.6280 (1.0) 307.9130 (1.0) 86.7625 (1.0) 17.5355 (1.20) 82.7525 (1.0) 1.7290 (1.0) 809;1736 11.5257 (1.0) 15524 1
168+
test_unify_chain_stack[10] 92.9890 (1.20) 309.8770 (1.01) 104.2017 (1.20) 14.6694 (1.0) 101.0160 (1.22) 4.2368 (2.45) 3657;6651 9.5968 (0.83) 73379 1
169+
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
170+
171+
------------------------------------------- benchmark 'unify_chain size=1000': 1 tests ------------------------------------------
172+
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
173+
---------------------------------------------------------------------------------------------------------------------------------
174+
test_unify_chain_stream_large[1000] 27.3518 65.5924 31.1374 4.2563 29.5148 3.5286 38;35 32.1158 496 1
175+
---------------------------------------------------------------------------------------------------------------------------------
176+
177+
------------------------------------------------------------------------- benchmark 'unify_chain size=300': 2 tests --------------------------------------------------------------------------
178+
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
179+
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
180+
test_unify_chain_stream[300] 3.6957 (1.0) 13.1876 (1.0) 4.4439 (1.0) 1.0719 (1.42) 4.2080 (1.0) 0.2410 (1.67) 51;95 225.0298 (1.0) 1114 1
181+
test_unify_chain_stack[300] 4.2952 (1.16) 13.4294 (1.02) 4.7732 (1.07) 0.7555 (1.0) 4.6623 (1.11) 0.1446 (1.0) 36;136 209.5024 (0.93) 2911 1
182+
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
183+
184+
--------------------------------------------------------------------------------- benchmark 'unify_chain size=35': 2 tests ---------------------------------------------------------------------------------
185+
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS (Kops/s) Rounds Iterations
186+
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
187+
test_unify_chain_stream[35] 285.6880 (1.0) 934.9690 (1.0) 324.5402 (1.0) 40.8338 (1.0) 319.8520 (1.0) 20.4375 (1.0) 962;1159 3.0813 (1.0) 24331 1
188+
test_unify_chain_stack[35] 345.2770 (1.21) 1,088.3650 (1.16) 407.9067 (1.26) 52.2263 (1.28) 396.6640 (1.24) 20.6560 (1.01) 2054;3027 2.4515 (0.80) 37594 1
189+
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
190+
191+
--------------------------------------------- benchmark 'unify_chain size=5000': 1 tests ---------------------------------------------
192+
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
193+
--------------------------------------------------------------------------------------------------------------------------------------
194+
test_unify_chain_stream_large[5000] 555.2733 754.9897 605.4949 50.6124 591.1251 61.4030 2;2 1.6515 26 1
195+
--------------------------------------------------------------------------------------------------------------------------------------
196+
197+
Legend:
198+
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
199+
OPS: Operations Per Second, computed as 1 / Mean
200+
```
201+
202+
</p>
203+
</details>
124204

125205
## About
126206

‎tests/test_benchmarks.py‎

Lines changed: 48 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
import platform
2+
import sys
3+
14
import pytest
25

36
from tests.utils import gen_long_chain
@@ -47,8 +50,8 @@ def reify_stack(u, s):
4750
@pytest.mark.parametrize("size", nesting_sizes)
4851
def test_unify_chain_stream(size, benchmark):
4952
a_lv = var()
50-
form = gen_long_chain(a_lv, size)
51-
term = gen_long_chain("a", size)
53+
form, lvars = gen_long_chain(a_lv, size, use_lvars=True)
54+
term, _ = gen_long_chain("a", size)
5255

5356
res = benchmark(unify, form, term, {})
5457
assert res[a_lv] == "a"
@@ -58,8 +61,8 @@ def test_unify_chain_stream(size, benchmark):
5861
@pytest.mark.parametrize("size", nesting_sizes)
5962
def test_unify_chain_stack(size, benchmark):
6063
a_lv = var()
61-
form = gen_long_chain(a_lv, size)
62-
term = gen_long_chain("a", size)
64+
form, lvars = gen_long_chain(a_lv, size, use_lvars=True)
65+
term, _ = gen_long_chain("a", size)
6366

6467
res = benchmark(unify_stack, form, term, {})
6568
assert res[a_lv] == "a"
@@ -69,19 +72,54 @@ def test_unify_chain_stack(size, benchmark):
6972
@pytest.mark.parametrize("size", nesting_sizes)
7073
def test_reify_chain_stream(size, benchmark):
7174
a_lv = var()
72-
form = gen_long_chain(a_lv, size)
73-
term = gen_long_chain("a", size)
75+
form, lvars = gen_long_chain(a_lv, size, use_lvars=True)
76+
term, _ = gen_long_chain("a", size)
7477

75-
res = benchmark(reify, form, {a_lv: "a"})
78+
lvars.update({a_lv: "a"})
79+
res = benchmark(reify_stack, form, lvars)
7680
assert res == term
7781

7882

7983
@pytest.mark.benchmark(group="reify_chain")
8084
@pytest.mark.parametrize("size", nesting_sizes)
8185
def test_reify_chain_stack(size, benchmark):
8286
a_lv = var()
83-
form = gen_long_chain(a_lv, size)
84-
term = gen_long_chain("a", size)
87+
form, lvars = gen_long_chain(a_lv, size, use_lvars=True)
88+
term, _ = gen_long_chain("a", size)
8589

86-
res = benchmark(reify_stack, form, {a_lv: "a"})
90+
lvars.update({a_lv: "a"})
91+
res = benchmark(reify_stack, form, lvars)
8792
assert res == term
93+
94+
95+
@pytest.mark.benchmark(group="unify_chain")
96+
@pytest.mark.parametrize("size", [1000, 5000])
97+
def test_unify_chain_stream_large(size, benchmark):
98+
a_lv = var()
99+
form, lvars = gen_long_chain(a_lv, size, use_lvars=True)
100+
term, _ = gen_long_chain("a", size)
101+
102+
res = benchmark(unify, form, term, {})
103+
assert res[a_lv] == "a"
104+
105+
106+
@pytest.mark.skipif(
107+
platform.python_implementation() == "PyPy",
108+
reason="PyPy's sys.getrecursionlimit changes",
109+
)
110+
@pytest.mark.benchmark(group="reify_chain")
111+
@pytest.mark.parametrize("size", [sys.getrecursionlimit(), sys.getrecursionlimit() * 5])
112+
def test_reify_chain_stream_large(size, benchmark):
113+
a_lv = var()
114+
form, lvars = gen_long_chain(a_lv, size, use_lvars=True)
115+
term, _ = gen_long_chain("a", size)
116+
117+
lvars.update({a_lv: "a"})
118+
119+
res = benchmark(reify, form, lvars)
120+
121+
if size < sys.getrecursionlimit():
122+
assert res == term
123+
else:
124+
with pytest.raises(RecursionError):
125+
assert res == term

‎tests/test_core.py‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -218,18 +218,18 @@ def test_reify_recursion_limit():
218218

219219
a_lv = var()
220220

221-
b = gen_long_chain(a_lv, 10)
221+
b, _ = gen_long_chain(a_lv, 10)
222222
res = reify(b, {a_lv: "a"})
223-
assert res == gen_long_chain("a", 10)
223+
assert res == gen_long_chain("a", 10)[0]
224224

225225
r_limit = sys.getrecursionlimit()
226226

227227
try:
228228
sys.setrecursionlimit(100)
229229

230-
b = gen_long_chain(a_lv, 200)
230+
b, _ = gen_long_chain(a_lv, 200)
231231
res = reify(b, {a_lv: "a"})
232-
exp_res = gen_long_chain("a", 200)
232+
exp_res, _ = gen_long_chain("a", 200)
233233

234234
if platform.python_implementation().lower() != "pypy":
235235
# CPython has stack limit issues when comparing nested lists, but
@@ -248,8 +248,8 @@ def test_reify_recursion_limit():
248248
def test_unify_recursion_limit():
249249
a_lv = var()
250250

251-
b = gen_long_chain("a")
252-
b_var = gen_long_chain(a_lv)
251+
b, _ = gen_long_chain("a")
252+
b_var, _ = gen_long_chain(a_lv)
253253

254254
s = unify(b, b_var, {})
255255

‎tests/utils.py‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,36 @@
11
import sys
22

3+
from unification.variable import var
34

4-
def gen_long_chain(last_elem=None, N=None):
5+
6+
def gen_long_chain(last_elem=None, N=None, use_lvars=False):
7+
"""Generate a nested list of length `N` with the last element set to `last_elm`.
8+
9+
Parameters
10+
----------
11+
last_elem: object
12+
The element to be placed in the inner-most nested list.
13+
N: int
14+
The number of nested lists.
15+
use_lvars: bool
16+
Whether or not to add `var`s to the first elements of each nested list
17+
or simply integers. If ``True``, each `var` is passed the nesting
18+
level integer (i.e. ``var(i)``).
19+
20+
Returns
21+
-------
22+
list, dict
23+
The generated nested list and a ``dict`` containing the generated
24+
`var`s and their nesting level integers, if any.
25+
26+
"""
527
b_struct = None
628
if N is None:
729
N = sys.getrecursionlimit()
30+
lvars = {}
831
for i in range(N - 1, 0, -1):
9-
b_struct = [i, last_elem if i == N - 1 else b_struct]
10-
return b_struct
32+
i_el = var(i) if use_lvars else i
33+
if use_lvars:
34+
lvars[i_el] = i
35+
b_struct = [i_el, last_elem if i == N - 1 else b_struct]
36+
return b_struct, lvars

0 commit comments

Comments
 (0)