SILO benchmarking CI system

These are the files for maintaining an evobench based benchmarking runner on a server.

Using a dedicated server has the advantage that

• the full datasets can be used (perhaps needing lots of RAM)
• no interference from other workloads, hence better measurements

evobench-run has a feature (see stop_start attribute in etc/evobench-run.ron) to temporarily shut down other workloads while running benchmarks, thus the same server can be used for other things at other times.

The system consists of a daemon (via evobench-run -v run daemon) that watches a set of job queues and executes them at the configured times, and regular execution of a check for new commits on configured branches (via evobench-run poll, see bin/evobench-run-poll).

We are currently running the system under the evobench user on gs-staging-1.

Installation

1. create a user account (e.g. evobench)

2. install evobench:

   1. install the current stable Rust toolchain in this account via rustup
   2. log out and in again (or source the shell startup files)
   3. git clone https://github.com/GenSpectrum/evobench
   4. cd evobench/evobench-evaluator/; cargo install --locked --path .

3. cd; git clone https://github.com/GenSpectrum/silo-benchmark-ci .silo-benchmark-ci

4. cd; ln -s .silo-benchmark-ci/bin; ln -s .silo-benchmark-ci/etc; ln -s etc/evobench-run.ron .evobench-run.ron

5. create ~/silo-benchmark-datasets directory, put subdirectories with the datasets (todo: where from?)

6. Get conan:

    python3 -m venv ~/venv
    source ~/venv/bin/activate
    pip install conan==2.8.1
   

7. Start daemon (for now):

    cd
    screen
    nohup evobench-run -v run daemon
   

8. Add poller and cleanup to crontab:

    mkdir ~/log
    echo "$PATH"
    crontab -e
   

   Add PATH=<output from echo $PATH above> to the top, and the following to the end:

    * * * * * bin/evobench-run-poll
    0 22 * * * evobench-run wd cleanup stale-for-days 7
   

More info

For the docs shown when logging into the account via ssh, see docs/README.md