db: edit default heuristic determining levels eligible for compaction

When deciding whether a level is even eligible for compaction picking, we
consider the 'compensated fill factor': a statistic about the size of a level
plus its garbage relative to the level's ideal size given the level multiplier
and database size. Previously, we considered a level eligible to be the input
level of a compaction if the compensated fill factor is >= 1.0 AND the
compensated fill factor divided by the next level's fill factor is also > 1.0.

We've preserved this behavior for historical consistency but have no clear
justification for it. Experimentally, we've observed instances where the LSM is
known to contain significant garbage (>14%) but no compactions are pursued
because most of the data is in L6 and L6's fill factor is positive (see
cockroachdb/cockroach#151633).

This commit adjusts the default behavior to only require a compensated score >=
1.0 for L0. Other levels only require a 'compensated fill factor' >= 1.0 to be
eligible for compaction.

A new experimental UseDeprecatedCompensatedScore Option is introduced to allow
opting out of the new behavior at runtime. This option is intended as an escape
hatch if we observe unintended side effects in production. Otherwise, it'll
ultimately be removed.

Author: jbowens

compaction_picker.go

@@ -1052,15 +1052,29 @@ func (p *compactionPickerByScore) calculateLevelScores(
 			score /= denominator
 			compensatedScore /= denominator
 		}
-		// The level requires compaction iff both compensatedFillFactor and
+
+		// We calculated a compensated score above by dividing the
+		// compensatedFillFactor by the next level's fill factor. Previous
+		// versions of Pebble had a default behavior of only considering levels
+		// with a compensatedScore >= 1.0 eligible for compaction. This wasn't a
+		// principled decision and has been experimentally observed to limit
+		// productive compactions that would reclaim disk space, but were
+		// prohibited because the output level's fill factor was > 1.0.
+		//
+		// We allow the use of old behavior through the
+		// UseDeprecatedCompensatedScore option, which if true, only considers
+		// the level eligible for compaction iff both compensatedFillFactor and
 		// compensatedScore are >= 1.0.
 		//
-		// TODO(radu): this seems ad-hoc. In principle, the state of other levels
-		// should not come into play when we're determining this level's eligibility
-		// for compaction. The score should take care of correctly prioritizing the
-		// levels.
+		// Otherwise, only L0 requires the compensatedScore to be >= 1.0; all
+		// other levels only require the compensatedFillFactor to be >= 1.0. L0
+		// is treated exceptionally out of concern that a large, LBase that's
+		// being compacted into the next level may prevent L0->Lbase compactions
+		// and attempting to pick L0 compactions may result in intra-L0
+		// compactions.
 		const compensatedScoreThreshold = 1.0
-		if compensatedScore < compensatedScoreThreshold {
+		if (level != 0 || p.opts.Experimental.UseDeprecatedCompensatedScore()) &&
+			compensatedScore < compensatedScoreThreshold {
 			// No need to compact this level; score stays 0.
 			continue
 		}

options.go

@@ -609,6 +609,15 @@ type Options struct {
 		// due to garbage.
 		CompactionGarbageFractionForMaxConcurrency func() float64
 
+		// UseDeprecatedCompensatedScore is a temporary option to revert the
+		// compaction picker to the previous behavior of only considering
+		// compaction out of a level if its compensated fill factor divided by
+		// the next level's fill factor is >= 1.0. The details of this setting
+		// are documented within its use within the compaction picker.
+		//
+		// The default value is false.
+		UseDeprecatedCompensatedScore func() bool
+
 		// IngestSplit, if it returns true, allows for ingest-time splitting of
 		// existing sstables into two virtual sstables to allow ingestion sstables to
 		// slot into a lower level than they otherwise would have.
@@ -1565,6 +1574,9 @@ func (o *Options) EnsureDefaults() {
 	if o.WALFailover != nil {
 		o.WALFailover.FailoverOptions.EnsureDefaults()
 	}
+	if o.Experimental.UseDeprecatedCompensatedScore == nil {
+		o.Experimental.UseDeprecatedCompensatedScore = func() bool { return false }
+	}
 	if o.Experimental.LevelMultiplier <= 0 {
 		o.Experimental.LevelMultiplier = defaultLevelMultiplier
 	}