db: don't smoothe compaction picker compensated fill factor

When deciding whether a level is even eligible for compaction picking, we
consider the 'compensated fill factor': a statistic about the size of a level
plus its garbage relative to the level's ideal size given the level multiplier
and database size. Currently, we consider a level eligible to be the input
level of a compaction if the compensated fill factor is > 1.0 AND the
compensated fill factor divided by the next level's fill factor is also > 1.0.

We've preserved this behavior for historical consistency but have no clear
justification for it. Experimentally, we've observed instances where the LSM is
known to contain significant garbage (>14%) but no compactions are pursued
because most of the data is in L6 and L6's fill factor is positive (see
cockroachdb/cockroach#151633).

This commit removes the separate concept of a 'compensated score,' instead
allowing the compaction picker to pursue a compaction out of a level so long as
its compensated fill factor is > 1.0.

Author: jbowens

compaction_picker.go

@@ -1038,30 +1038,25 @@ func (p *compactionPickerByScore) calculateLevelScores(
 			continue
 		}
 		const compensatedFillFactorThreshold = 1.0
+		// The level requires compaction iff the compensatedFillFactor is >= 1.0.
 		if scores[level].compensatedFillFactor < compensatedFillFactorThreshold {
 			// No need to compact this level; score stays 0.
 			continue
 		}
 		score := scores[level].fillFactor
-		compensatedScore := scores[level].compensatedFillFactor
 		if level < numLevels-1 {
 			nextLevel := scores[level].outputLevel
 			// Avoid absurdly large scores by placing a floor on the factor that we'll
 			// adjust a level by. The value of 0.01 was chosen somewhat arbitrarily.
 			denominator := max(0.01, scores[nextLevel].fillFactor)
 			score /= denominator
-			compensatedScore /= denominator
-		}
-		// The level requires compaction iff both compensatedFillFactor and
-		// compensatedScore are >= 1.0.
-		//
-		// TODO(radu): this seems ad-hoc. In principle, the state of other levels
-		// should not come into play when we're determining this level's eligibility
-		// for compaction. The score should take care of correctly prioritizing the
-		// levels.
-		const compensatedScoreThreshold = 1.0
-		if compensatedScore < compensatedScoreThreshold {
-			// No need to compact this level; score stays 0.
+		}
+		// L0 doesn't get compensated, so its compensated and uncompensated fill
+		// factors are equal. We avoid compacting L0 if its score (the value
+		// after dividing by the next level's fill factor) is less than 1.0.
+		// This helps avoid over eager L0 compaction picking (and intra L0
+		// compactions) when LBase is large.
+		if level == 0 && score < 1.0 {
 			continue
 		}
 		scores[level].score = score