MAGE-1109: Add Batching Optimizer feature

[damcou]

This PR contains the Batching Optimizer CLI command which include the following process:

• For each store that has the indexing enabled, performs a "scan" of a sample of products to get some figures about the size of the resulting records (stores can be specified with the store_id argument).
• The product sample is defined by the percentage of "simple products" (simple, virtual, downloadable, giftcard) and "complex products" (configurable, bundle, grouped) in the catalog. (for example, a sample of 20 products from a catalog composed of 40% of simple products and 60% of complex products will have 8 simple products and 12 complex products).
• With the sample, calculate some statistics regarding product records size (max, min, average, standard deviation)
• According to these values, determine the optimal value of batching size for indexing requests sent to Algolia
• Offer the possibility to update the "Maximum number of records processed per indexing job" configuration with this value with a prompt.

[cammonro]

Love where this is going. I know this is WIP - just a couple of small observations.

[cammonro]

This is just fantastic work @damcou !! 🙌

I do see some issues if you wouldn't mind taking a look and had a few suggestions.

I also think we should add some language that says something along the lines that these numbers are estimates only and that indexing activity should be monitored after making changes to ensure batches are not exceeding the recommended size of 10 MB.

[cammonro]

I added a POC for the issue you mentioned - if you think this can work then we could do something for margin as well.

Also noted one issue on the division by zero check.

Additional comments in Jira.

[cammonro]

Chebyshev for the win! 😆

Looking great. Let's go! 🚀