Clean up non-latest output dir

I've noticed that the tests currently don't clean up past output dirs, causing rather quick growth of the folder:

<img width="857" alt="iTerm2 2024-09-08 20 30 51" src="https://github.com/user-attachments/assets/d440acb1-89d7-4598-a544-5edacb8fc56e">

<img width="978" alt="iTerm2 2024-09-08 20 31 04" src="https://github.com/user-attachments/assets/44c73bba-5d51-4199-b7ef-60f7b6a4cf3e">


