testcases/containers: use 'fio' utility #106

synarete · 2025-09-03T17:48:20Z

Use a wrapper shell script over 'fio' + a container to perform a set of I/O testing. Execute both I/O throughput workload as well as random I/O with data verification.

Shwetha-Acharya

lgtm!

spuiuk

ACK

Edit: Removing the ACK as we have seen issues when running.

spuiuk · 2025-09-10T22:59:03Z

built and pushed the container image to quay. Will retry tests to confirm that they are running as expected.

spuiuk · 2025-09-10T22:59:28Z

/retest all

spuiuk · 2025-09-10T23:01:01Z

@synarete, can you rebase and push. We should be able to see the output of the tests in the checks now that the fio containers have been setup.

synarete · 2025-09-11T05:50:41Z

@synarete, can you rebase and push. We should be able to see the output of the tests in the checks now that the fio containers have been setup.

Done

anoopcs9 · 2025-09-11T05:59:04Z

We have a problem w.r.t disk space.

run_fio.sh: fio --name=fio_simple_nproc --size=1G --directory=/testdir --runtime=120 --ioengine=pvsync2 --group_reporting --time_based --sync=1 --direct=0 --rw=readwrite --output-format=json --bs=8K --numjobs=35
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device

Our disks are configured with 10G. So either we increase it or limit the number of jobs. Or do both?

synarete · 2025-09-11T06:03:12Z

We have a problem w.r.t disk space.

run_fio.sh: fio --name=fio_simple_nproc --size=1G --directory=/testdir --runtime=120 --ioengine=pvsync2 --group_reporting --time_based --sync=1 --direct=0 --rw=readwrite --output-format=json --bs=8K --numjobs=35
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device
fio: pid=0, err=28/file:filesetup.c:253, func=fsync, error=No space left on device

Our disks are configured with 10G. So either we increase it or limit the number of jobs. Or do both?

I see numjobs=35 (which we get from nproc). I will add an upper limit to numjobs (say, 8) by why do we use such non-conventional odd number of CPUs?

anoopcs9 · 2025-09-11T06:24:50Z

Our disks are configured with 10G. So either we increase it or limit the number of jobs. Or do both?

I see numjobs=35 (which we get from nproc). I will add an upper limit to numjobs (say, 8)

Thanks, I hope we can use --max-jobs.

by why do we use such non-conventional odd number of CPUs?

This is not explicitly set rather comes from the host system within the fio container.

testcases/containers/images/fio/run_fio.sh

anoopcs9

@spuiuk Can you rebuild and push the fio container using the latest version?

testcases/containers/images/fio/run_fio.sh

spuiuk · 2025-09-11T11:10:18Z

Pushed the new version of the container.

spuiuk · 2025-09-11T11:10:31Z

/retest all

anoopcs9 · 2025-09-11T11:48:12Z

@synarete Does verify workloads consume more space than normal? Now it fails as below:

run_fio.sh: fio --name=fio_verify_4jobs --size=1G --directory=/testdir --ioengine=pvsync2 --group_reporting --sync=1 --direct=0 --rw=randwrite --do_verify=1 --verify_state_save=0 --verify=xxhash --output-format=json --bs=64K --numjobs=4
fio: pid=30, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device
fio: io_u error on file /testdir/fio_verify_4jobs.3.0: No space left on device: write offset=198246400, buflen=65536
fio: pid=29, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device
fio: io_u error on file /testdir/fio_verify_4jobs.0.0: No space left on device: write offset=608567296, buflen=65536
fio: pid=32, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device
fio: io_u error on file /testdir/fio_verify_4jobs.1.0: No space left on device: write offset=236388352, buflen=65536
fio: io_u error on file /testdir/fio_verify_4jobs.2.0: No space left on device: write offset=239665152, buflen=65536
fio: pid=31, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device

synarete · 2025-09-11T12:22:22Z

@synarete Does verify workloads consume more space than normal? Now it fails as below:

run_fio.sh: fio --name=fio_verify_4jobs --size=1G --directory=/testdir --ioengine=pvsync2 --group_reporting --sync=1 --direct=0 --rw=randwrite --do_verify=1 --verify_state_save=0 --verify=xxhash --output-format=json --bs=64K --numjobs=4
fio: pid=30, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device
fio: io_u error on file /testdir/fio_verify_4jobs.3.0: No space left on device: write offset=198246400, buflen=65536
fio: pid=29, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device
fio: io_u error on file /testdir/fio_verify_4jobs.0.0: No space left on device: write offset=608567296, buflen=65536
fio: pid=32, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device
fio: io_u error on file /testdir/fio_verify_4jobs.1.0: No space left on device: write offset=236388352, buflen=65536
fio: io_u error on file /testdir/fio_verify_4jobs.2.0: No space left on device: write offset=239665152, buflen=65536
fio: pid=31, err=28/file:io_u.c:1896, func=io_u error, error=No space left on device

It looks like it. I will reduce the size on verify runs

spuiuk · 2025-09-11T13:16:00Z

new version of container pushed.

spuiuk · 2025-09-11T13:16:10Z

/retest all

Noticed issues with test run.

anoopcs9 · 2025-09-12T05:25:26Z

XFS (and GPFS) tests are looking good but CephFS tests show different errors:

For default (without mgr) setup on a proxy enabled share (same should be the case without proxy):

run_fio.sh: fio --name=fio_simple_64k --size=1G --directory=/testdir --runtime=120 --ioengine=pvsync2 --group_reporting --time_based --sync=1 --direct=0 --rw=readwrite --output-format=json --bs=64K --numjobs=1
fio: io_u error on file /testdir/fio_simple_64k.0.0: Permission denied: write offset=0, buflen=65536
fio: pid=6, err=13/file:io_u.c:1896, func=io_u error, error=Permission denied
For mgr based setup on a normal vfs-ceph-new share without proxy (same should be the case with proxy):

run_fio.sh: fio --name=fio_simple_64k --size=1G --directory=/testdir --runtime=120 --ioengine=pvsync2 --group_reporting --time_based --sync=1 --direct=0 --rw=readwrite --output-format=json --bs=64K --numjobs=1
fio: io_u error on file /testdir/fio_simple_64k.0.0: Resource temporarily unavailable: write offset=0, buflen=65536
fio: pid=6, err=11/file:io_u.c:1896, func=io_u error, error=Resource temporarily unavailable

To me it looks like new ceph module doesn't complete the fio workload. Old vfs module based share could get past the fio test.

testcases/containers/test_containers.py::test_containers[192.168.123.10-share-cephfs-default-vfs-fio] PASSED [ 94%]

synarete · 2025-09-14T05:25:27Z

XFS (and GPFS) tests are looking good but CephFS tests show different errors:

For default (without mgr) setup on a proxy enabled share (same should be the case without proxy):

run_fio.sh: fio --name=fio_simple_64k --size=1G --directory=/testdir --runtime=120 --ioengine=pvsync2 --group_reporting --time_based --sync=1 --direct=0 --rw=readwrite --output-format=json --bs=64K --numjobs=1
fio: io_u error on file /testdir/fio_simple_64k.0.0: Permission denied: write offset=0, buflen=65536
fio: pid=6, err=13/file:io_u.c:1896, func=io_u error, error=Permission denied

For mgr based setup on a normal vfs-ceph-new share without proxy (same should be the case with proxy):

run_fio.sh: fio --name=fio_simple_64k --size=1G --directory=/testdir --runtime=120 --ioengine=pvsync2 --group_reporting --time_based --sync=1 --direct=0 --rw=readwrite --output-format=json --bs=64K --numjobs=1
fio: io_u error on file /testdir/fio_simple_64k.0.0: Resource temporarily unavailable: write offset=0, buflen=65536
fio: pid=6, err=11/file:io_u.c:1896, func=io_u error, error=Resource temporarily unavailable

To me it looks like new ceph module doesn't complete the fio workload. Old vfs module based share could get past the fio test.

testcases/containers/test_containers.py::test_containers[192.168.123.10-share-cephfs-default-vfs-fio] PASSED [ 94%]

Interesting -- I did not encounter those failures on my setup (without proxy). Will dig into it.

anoopcs9 · 2025-09-15T08:43:57Z

/retest centos-ci/cephfs

Use a wrapper shell script over 'fio' + a container to perform a set of I/O testing. Execute both I/O throughput workload as well as random I/O with data verification. Signed-off-by: Shachar Sharon <[email protected]>

synarete · 2025-09-15T13:52:03Z

I tried to reproduce on my local ceph cluster (without proxy) but failed to do so. Indeed, I had a crash because I used non-standard ceph images, but as soon as I switched back to normal ceph (image: quay.ceph.io/ceph-ci/ceph:main libcepfs: centos-release-ceph-reef-1.0-1.el9s.noarch libcephfs2-18.2.7-2.el9s.x86_64 everything works fine, repeated few time. Which version of libcephfs are we using for CI ?

anoopcs9 · 2025-09-15T17:53:03Z

Which version of libcephfs are we using for CI ?

Whatever is the latest build available from ceph main branch.

synarete requested a review from spuiuk September 3, 2025 17:48

Shwetha-Acharya approved these changes Sep 5, 2025

View reviewed changes

spuiuk previously approved these changes Sep 10, 2025

View reviewed changes

spuiuk requested a review from anoopcs9 September 10, 2025 22:58

synarete force-pushed the ss-container-fio branch from ed2e014 to be19a8e Compare September 11, 2025 05:50

synarete force-pushed the ss-container-fio branch from be19a8e to 4f1e8c7 Compare September 11, 2025 06:17

anoopcs9 reviewed Sep 11, 2025

View reviewed changes

testcases/containers/images/fio/run_fio.sh Show resolved Hide resolved

anoopcs9 reviewed Sep 11, 2025

View reviewed changes

testcases/containers/images/fio/run_fio.sh Show resolved Hide resolved

synarete force-pushed the ss-container-fio branch from 4f1e8c7 to 7616b48 Compare September 11, 2025 12:45

spuiuk self-requested a review September 11, 2025 13:12

testcases/containers: use 'fio' utility

5a3f2c1

Use a wrapper shell script over 'fio' + a container to perform a set of I/O testing. Execute both I/O throughput workload as well as random I/O with data verification. Signed-off-by: Shachar Sharon <[email protected]>

synarete force-pushed the ss-container-fio branch from 7616b48 to 5a3f2c1 Compare September 15, 2025 13:46

testcases/containers: use 'fio' utility #106

Are you sure you want to change the base?

testcases/containers: use 'fio' utility #106

Conversation

synarete commented Sep 3, 2025

Uh oh!

Shwetha-Acharya left a comment

Choose a reason for hiding this comment

Uh oh!

spuiuk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spuiuk commented Sep 10, 2025

Uh oh!

spuiuk commented Sep 10, 2025

Uh oh!

spuiuk commented Sep 10, 2025

Uh oh!

synarete commented Sep 11, 2025

Uh oh!

anoopcs9 commented Sep 11, 2025

Uh oh!

synarete commented Sep 11, 2025

Uh oh!

anoopcs9 commented Sep 11, 2025

Uh oh!

Uh oh!

anoopcs9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spuiuk commented Sep 11, 2025

Uh oh!

spuiuk commented Sep 11, 2025

Uh oh!

anoopcs9 commented Sep 11, 2025

Uh oh!

synarete commented Sep 11, 2025

Uh oh!

spuiuk commented Sep 11, 2025

Uh oh!

spuiuk commented Sep 11, 2025

Uh oh!

anoopcs9 commented Sep 12, 2025

Uh oh!

synarete commented Sep 14, 2025

Uh oh!

anoopcs9 commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

synarete commented Sep 15, 2025

Uh oh!

anoopcs9 commented Sep 15, 2025

Uh oh!

Uh oh!

spuiuk left a comment •

edited

Loading

anoopcs9 commented Sep 15, 2025 •

edited

Loading