Skip to content

Commit 54b4e5c

Browse files
bravindrbravindr
authored
Merge pull request #7 from intel/zswap_iaa_batching_tracepoints
tests: Added madvise_tracepoints directory for running ZSWAP micro-benchmarks with IAA, ZSTD.
2 parents bd10871 + 145c997 commit 54b4e5c

34 files changed

+6984
-0
lines changed

‎tests/madvise_tracepoints/README

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
SPDX-License-Identifier: BSD-3-Clause
2+
Copyright (c) 2025, Intel Corporation
3+
4+
ZSWAP IAA and ZSTD [batch] [de]compress latency and compressed length/ratio micro-benchmarks:
5+
=============================================================================================
6+
7+
Micro-benchmarks Root Directory:
8+
--------------------------------
9+
10+
memory-usage-analyzer/tests/madvise_tracepoints
11+
12+
13+
Assumptions:
14+
------------
15+
1) We assume the kernel has been built with kernel tracepoints instrumentation
16+
to measure zswap_[de]compress and zswap_batch_[de]compress latencies.
17+
18+
2) We assume the platform has Intel IAA.
19+
20+
21+
Description:
22+
------------
23+
These tests use an madvise based micro-benchmark, along with kernel traces to
24+
get accurate measurements of ZSWAP [batch] [de]compress latencies, compressed
25+
lengths and compression ratios.
26+
27+
The "silesia.tar" dataset is used for running the micro-benchmarks. The tests
28+
will download and place "silesia.tar" in madvise_tracepoints/datasets using:
29+
30+
wget --no-check-certificate http://wanos.co/assets/silesia.tar
31+
32+
The tests are run with core frequency fixed at 2000 MHz, and measure the
33+
following:
34+
35+
1) IAA batch compression and decompression latencies, for these batch sizes
36+
respectively:
37+
38+
IAA parallel compressions:
39+
vm.compress-batchsize in [1 2 4 8 16 32]
40+
41+
IAA parallel decompressions:
42+
vm.page-cluster in [0 1 2 3 4 5]
43+
44+
"deflate-iaa-canned" (Canned mode) is the IAA compression mode used, but the
45+
scripts can be easily modified to run "deflate-iaa" (Fixed mode).
46+
47+
Single-page IAA compress/decompress latencies will be reported as
48+
"cb1"/"db1" rows in the summary.
49+
50+
cb = compress batch-size
51+
db = decompress batch-size
52+
53+
2) ZSTD compress and decompress latencies:
54+
55+
The tests will measure and report single-page zstd latencies for
56+
[de]compress, using:
57+
58+
vm.compress-batchsize in [1]
59+
vm.page-cluster in [0]
60+
61+
62+
3) IAA and ZSTD compress lengths, along with compression ratio computations.
63+
64+
65+
Running the ZSWAP micro-benchmarks with IAA and ZSTD:
66+
-----------------------------------------------------
67+
68+
We recommend rebooting the system so that vmstats, mthp stats, IAA wq_stats,
69+
etc. are cleared before running the tests.
70+
71+
72+
Step 1): Clone the tests git repo:
73+
74+
75+
git clone https://github.com/intel/memory-usage-analyzer.git
76+
77+
pip install -e memory-usage-analyzer
78+
79+
80+
Step 2): Go to the ZSWAP IAA micro-benchmarks directory:
81+
82+
83+
cd memory-usage-analyzer/tests/madvise_tracepoints
84+
85+
86+
Step 3): Configure IAA devices, WQs and iaa_crypto driver parameters.
87+
88+
89+
Important note: This should be done **only once**.
90+
91+
92+
cd memory-usage-analyzer/tests/madvise_tracepoints
93+
94+
./configure_iaa.sh
95+
96+
97+
Step 4): Run latency tests (can be repeated as needed):
98+
99+
100+
./zswap_latency.sh
101+
102+
103+
Step 5) Run compression lengths and compression ratio tests (can be repeated as
104+
needed):
105+
106+
107+
./zswap_lengths.sh
108+
109+
110+
Reports, summaries, logs:
111+
-------------------------
112+
The tests will generate these files/directories, all with timestamps:
113+
114+
115+
latency_summaries.txt:
116+
117+
Cumulative latency reports for test runs.
118+
119+
Important note: All latencies reported for batch_[de]compress are
120+
normalized to be per-page. Hence the latency distributions across all rows
121+
in the latency_summaries.txt can be accurately compared across compressors
122+
and batch-sizes in an apples-to-apples manner.
123+
124+
125+
compress_length_summaries.txt:
126+
127+
Cumulative reports compression length and ratio test summaries.
128+
129+
Important note: IAA batch compression lengths reported are sum of
130+
compressed lengths of all pages in the batch. The kernel tracepoints report
131+
aggregated batch compress lengths as well as aggregated original batch-size
132+
lengths, in order for the scripts to calculate the compression ratio.
133+
134+
135+
run.log:
136+
137+
Detailed stats collected from the tests, including ZSWAP parameters and
138+
iaa_crypto settings, vmstats, mTHP stats, IAA wq_stats and global_stats,
139+
etc.
140+
141+
142+
data/:
143+
144+
Timestamp-prefixed per-compressor and per-batch-size data directories
145+
containing kernel traces, csv files containing histograms, stdout, etc.
146+
147+
148+
reports/:
149+
150+
Timestamp-prefixed per-compressor and per-batch-size
151+
[de]compress_latencies.txt, compress_lengths.txt.
152+
153+
154+
Example reports:
155+
----------------
156+
The "madvise_tracepoints/sample_reports" directory has some example reports.
157+

‎tests/madvise_tracepoints/calcpercent.py

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/usr/bin/env python3
2+
# SPDX-License-Identifier: BSD-3-Clause
3+
# Copyright (c) 2025, Intel Corporation
4+
5+
import sys
6+
filename = sys.argv[1]
7+
subtest = sys.argv[2]
8+
nargs = len(sys.argv)
9+
10+
hits = 0
11+
percentile = 98
12+
thresh = hits * percentile / 100
13+
print("Filename: %s" % filename)
14+
15+
infile = open(filename, 'r')
16+
if nargs > 3:
17+
percentile = sys.argv[3]
18+
19+
print("Percentile: %s" % percentile)
20+
21+
lines = infile.readlines()
22+
23+
eq_0_map = {}
24+
25+
cur_event = ""
26+
ret0 = False
27+
28+
for line in lines:
29+
sline = line.strip()
30+
if sline.find("hists:") != -1:
31+
sline = sline.split(' ')
32+
cur_event = sline[0]
33+
elif sline.find("Hits:") != -1:
34+
sline = sline.split(':')
35+
eq_0_map[cur_event] = int(sline[1])
36+
37+
infile.close()
38+
39+
infile = open(filename, 'r')
40+
41+
lines = infile.readlines()
42+
43+
for line in lines:
44+
sline = line.strip()
45+
if sline.find("hists:") != -1:
46+
sline = sline.split(' ')
47+
cur_event = sline[0]
48+
elif sline.find("trigger info") != -1:
49+
if sline.find(f"{subtest}") != -1:
50+
hits = eq_0_map[cur_event]
51+
if hits == 0:
52+
continue
53+
thresh = hits * .98
54+
print("\nhist event: %s" % cur_event)
55+
print(" event hit count: %s" % hits)
56+
print(" {} percent of hit count = {}".format(int(percentile), thresh))
57+
sum = 0
58+
printed = False
59+
elif sline.find('{ 'f'{subtest}') != -1:
60+
print(sline)
61+
sline = sline.split(':')
62+
print(sline)
63+
print(sline[2])
64+
sys.stdout.flush()
65+
sum += int(sline[2])
66+
if sum >= thresh and not printed:
67+
print(" hit threshold of %d at line:" % sum)
68+
print(" %s" % line)
69+
printed = True
70+
71+
infile.close()
72+

‎tests/madvise_tracepoints/configure_iaa.sh

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: BSD-3-Clause
3+
# Copyright (c) 2025, Intel Corporation
4+
#
5+
# One-time setup of IAA devices and WQs. Please run this script once, after
6+
# reboot and before running tests/workloads with IAA as the zswap compressor.
7+
#
8+
# Arguments:
9+
#
10+
# iaa_devices - Number of IAA devices to enable: default 1.
11+
#
12+
################################################################################
13+
14+
iaa_devices=${1:-1}
15+
16+
# Configure IAA
17+
./enable_kernel_iaa.sh 0 1 ${iaa_devices} 8 2 async
18+
# Configure zswap.
19+
./enable_zswap.sh
20+
# Optional: swap disk can be used instead of zram.
21+
# This will configure 30% of available memory for zram.
22+
./enable_zram.sh
23+
24+

‎tests/madvise_tracepoints/datasets/README.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
2+
This directory is a placeholder for datasets.
3+

‎tests/madvise_tracepoints/disable_iaa

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/usr/bin/env bash
2+
#SPDX-License-Identifier: BSD-3-Clause
3+
#Copyright (c) 2025, Intel Corporation
4+
#Description: Disable all IAA devices
5+
6+
7+
IAX_CONFIG_PATH=/sys/bus/dsa/devices
8+
9+
IAX_BIND_PATH=/sys/bus/dsa/drivers/idxd/bind
10+
IAX_BIND_WQ_PATH=/sys/bus/dsa/drivers/crypto/bind
11+
12+
IAX_UNBIND_PATH=/sys/bus/dsa/drivers/idxd/unbind
13+
IAX_UNBIND_WQ_PATH=/sys/bus/dsa/drivers/crypto/unbind
14+
15+
echo lzo > /sys/module/zswap/parameters/compressor
16+
#
17+
# count iax instances
18+
#
19+
iax_dev_id="0cfe"
20+
num_iax=$(lspci -d:${iax_dev_id} | wc -l)
21+
num_wq=1
22+
echo "Found ${num_iax} IAX instances"
23+
24+
#
25+
# disable iax wqs and devices
26+
#
27+
echo "Disable IAX"
28+
29+
for ((i = 1; i < 2 * ${num_iax}; i += 2)); do
30+
for ((j = 0; j < ${num_wq}; j += 1)); do
31+
echo disable wq iax${i}/wq${i}.${j}
32+
echo wq${i}.${j} > $IAX_UNBIND_WQ_PATH
33+
done
34+
echo disable iax iax${i}
35+
echo iax${i} > $IAX_UNBIND_PATH
36+
done
37+
38+
rmmod iaa_crypto
39+
modprobe iaa_crypto
40+
41+
42+
43+
44+
45+

‎tests/madvise_tracepoints/dump-zswap-hists-lat

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: BSD-3-Clause
3+
# Copyright (c) 2025, Intel Corporation
4+
5+
echo "Dumping zswap latency hists"
6+
7+
echo -e "\n######\nzswap_compress hists:\n"
8+
# zswap_compress
9+
cat /sys/kernel/debug/tracing/events/zswap/zswap_compress/hist
10+
11+
echo -e "\n######\nzswap_decompress hists:\n"
12+
# zswap_decompress
13+
cat /sys/kernel/debug/tracing/events/zswap/zswap_decompress/hist
14+
15+
echo -e "\n######\nzswap_batch_compress hists:\n"
16+
# zswap_batch_compress
17+
cat /sys/kernel/debug/tracing/events/zswap/zswap_batch_compress/hist
18+
19+
echo -e "\n######\nzswap_batch_decompress hists:\n"
20+
# zswap_batch_decompress
21+
cat /sys/kernel/debug/tracing/events/zswap/zswap_batch_decompress/hist
22+

‎tests/madvise_tracepoints/dump-zswap-hists-len

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: BSD-3-Clause
3+
# Copyright (c) 2025, Intel Corporation
4+
5+
echo "Dumping zswap compress length hists"
6+
7+
echo -e "\n######\nzswap_compress_len hists:\n"
8+
# zswap_compress_len
9+
cat /sys/kernel/debug/tracing/events/zswap/zswap_compress_len/hist
10+
11+
echo -e "\n######\nzswap_batch_compress_len hists:\n"
12+
# zswap_batch_compress_len
13+
cat /sys/kernel/debug/tracing/events/zswap/zswap_batch_compress_len/hist
14+

‎tests/madvise_tracepoints/enable-zswap-traces-lat

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: BSD-3-Clause
3+
# Copyright (c) 2025, Intel Corporation
4+
5+
echo 0 > /sys/kernel/debug/tracing/events/zswap/zswap_compress/enable
6+
echo 0 > /sys/kernel/debug/tracing/events/zswap/zswap_decompress/enable
7+
echo 0 > /sys/kernel/debug/tracing/events/zswap/zswap_batch_compress/enable
8+
echo 0 > /sys/kernel/debug/tracing/events/zswap/zswap_batch_decompress/enable
9+
echo 0 > /sys/kernel/debug/tracing/events/zswap/zswap_compress_len/enable
10+
echo 0 > /sys/kernel/debug/tracing/events/zswap/zswap_batch_compress_len/enable
11+
12+
echo > /sys/kernel/debug/tracing/trace
13+
14+
echo 1 > /sys/kernel/debug/tracing/events/zswap/zswap_compress/enable
15+
echo 1 > /sys/kernel/debug/tracing/events/zswap/zswap_decompress/enable
16+
echo 1 > /sys/kernel/debug/tracing/events/zswap/zswap_batch_compress/enable
17+
echo 1 > /sys/kernel/debug/tracing/events/zswap/zswap_batch_decompress/enable
18+
19+
# Set trace buffer on CPU 1 to 5GB:
20+
echo 10485760 > /sys/kernel/debug/tracing/per_cpu/cpu1/buffer_size_kb
21+
22+
echo "Removing zswap hists"
23+
24+
# zswap_[de]compress
25+
echo '!hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_compress/trigger
26+
echo '!hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_decompress/trigger
27+
28+
# zswap_[de]compress_batch
29+
echo '!hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_batch_compress/trigger
30+
echo '!hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_batch_decompress/trigger
31+
32+
echo "Enabling zswap hists"
33+
34+
# zswap_[de]compress
35+
echo 'hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_compress/trigger
36+
echo 'hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_decompress/trigger
37+
38+
# zswap_[de]compress_batch
39+
echo 'hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_batch_compress/trigger
40+
echo 'hist:keys=cpu.buckets=10:sort=cpu:size=8192' >> /sys/kernel/debug/tracing/events/zswap/zswap_batch_decompress/trigger
41+

0 commit comments

Comments
 (0)