Qualitative tests for mesa and mesa-frames benchmarks

[adamamer20]

Comparability between mesa and mesa-frames models are crucial for validating the correctness of model implementations.
To achieve this given the same seed and initial conditions, we need:

1. A compatibility layer that transforms a mesa.Model into a mesa-frames.ModelDF (this can be implemented relatively easily, I have already done some work on a local branch)
2. Consistent randomized operations (also including shuffling of agents) across frameworks. This is more challenging because of the different random number generators (mesa uses random.Random and mesa-frames uses numpy.random.Generator) and different shuffling (mesa uses random.Random.shuffle and mesa-frames uses the native DFs shuffle operation). Maybe with an appropriate decorator, we could substitute random operations for mesa-frames models at runtime, at the cost of performance but gaining the reproducibility.

[rht]

For migrating then scaling up, I think writing qualitative test cases that ensure specific behavior outcomes (see e.g. https://github.com/projectmesa/mesa-examples/blob/main/examples/sugarscape_g1mt/tests.py) and check that both versions have the same outcome, is a more effective use of the modeler's time, than trying to reproduce down to the floating points of the original Mesa model. The latter gets much harder the more complex the model is. mesa-frames' activation is order-independent, and can only have an equivalent in Mesa if the latter uses staged activation that is carefully designed to be order independent. Any random activation won't be translatable due to the first mover advantage.

[adamamer20]

You raise a good point. I wanted something more automated and "safer", but qualitative tests may be better in practice.