Switching to a fixed CFS threshold

[shubhamsrkdev]

Problem

Lucene uses CFS ratio (which is by default 10%) to determine whether to use Compound Files.

Solution

In this PR we are doing the following:

• Switching from a CFS ratio to a fixed threshold which is:

  • 64MB for byte-size-based merge policies
  • 65,536 docs for doc-based merge policies

• Moved CFS configuration to CompoundFormat.java from Merge policies.

Fixes #14959

[epotyom]

It would be great if we can avoid customizing it for specific policies, otherwise it might be tricky to maintain in the future, if e.g. there is another policy that is based on doc size not bytes.

Maybe we can add a enum and a method to MergePolicy which returns its unit (bytes/docs) , and use it here to decide which threshold to use?

Or do we want to always choose compound format based on size in bytes even for LogDocMergePolicy? In this case we might be able to use merge.getMergeInfo().sizeInBytes() when we call this method and avoid relying on MergePolicy#size all together?

[shubhamsrkdev]

+1 to the idea of Enums, I will wait if anyone else has other suggestions here but having an enum makes most sense to me.

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the [email protected] list. Thank you for your contribution!