Fix: Optimizer Missing Final Step - Combine Remaining WHERE Clauses (#732)

* fix: Optimize queries without joins by combining multiple WHERE clauses

Addresses issue #445 - performance slowdown when using multiple .where() calls.

## Problem
When using multiple .where() calls on a query without joins:
```javascript
query.from({ item: collection })
  .where(({ item }) => eq(item.gridId, gridId))
  .where(({ item }) => eq(item.rowId, rowId))
  .where(({ item }) => eq(item.side, side))
```

The optimizer was skipping these queries entirely, leaving multiple WHERE
clauses in an array. During query compilation, each WHERE clause was applied
as a separate filter() operation in the D2 pipeline, causing a 40%+ performance
degradation compared to using a single WHERE clause with AND.

## Solution
Modified the optimizer to combine multiple WHERE clauses into a single AND
expression for queries without joins. This ensures only one filter operator is
added to the pipeline, improving performance while maintaining correct semantics.

The optimizer now:
1. Detects queries without joins that have multiple WHERE clauses
2. Combines them using the AND function
3. Reduces pipeline complexity from N filters to 1 filter

## Testing
- Updated existing optimizer tests to reflect the new behavior
- All 42 optimizer tests pass
- Added new test case for combining multiple WHERE clauses without joins

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* docs: Add changeset and investigation report for issue #445

- Added changeset for the WHERE clause optimization fix
- Documented root cause analysis and solution details

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* fix: Complete optimizer fix - combine remaining WHERE clauses after pushdown

This completes the fix for issue #445 by implementing the missing "step 3" of
the optimizer process.

## Problem (Broader than Initially Identified)
The optimizer was missing the final step of combining remaining WHERE clauses
after optimization. This affected:

1. Queries WITHOUT joins: All optimization was skipped, leaving multiple
   WHERE clauses as separate array elements
2. Queries WITH joins: After predicate pushdown, remaining WHERE clauses
   (multi-source + unpushable single-source) were left as separate elements

Both cases resulted in multiple filter() operations in the pipeline instead
of a single combined filter, causing 40%+ performance degradation.

## Solution
Implemented "step 3" (combine remaining WHERE clauses) in two places:

1. **applySingleLevelOptimization**: For queries without joins, combine
   multiple WHERE clauses before returning

2. **applyOptimizations**: After predicate pushdown for queries with joins,
   combine all remaining WHERE clauses (multi-source + unpushable)

## Testing
- Added test: "should combine multiple remaining WHERE clauses after optimization"
- All 43 optimizer tests pass
- Updated investigation report with complete analysis
- Updated changeset to reflect the complete fix

Thanks to colleague feedback for catching that step 3 was missing!

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* style: Run prettier on markdown files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* docs: Add PR body update for issue #445 fix

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* docs: Remove specific 40% performance claim

The original issue compared TanStack db with Redux, not the bug itself.
Changed to more general language about performance degradation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* docs: Remove temporary investigation and PR body files

These were used for context during development but aren't needed in the repo.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* fix: Flatten nested AND expressions when combining WHERE clauses

Addresses reviewer feedback - when combining remaining WHERE clauses after
predicate pushdown, flatten any nested AND expressions to avoid creating
and(and(...), ...) structures.

Changes:
- Use flatMap(splitAndClausesRecursive) before combineWithAnd to flatten
- Added test for nested AND flattening
- Added test verifying functional WHERE clauses remain separate

All 45 optimizer tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* style: Remove issue reference from code comment

As requested by @samwillis - issue references in code comments can become
stale. The comment is self-explanatory without the reference.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

---------

Co-authored-by: Claude <[email protected]>

Author: KyleAMathews

.changeset/optimize-multiple-where-clauses.md

@@ -0,0 +1,10 @@
+---
+"@tanstack/db": patch
+---
+
+Fixed performance issue where using multiple `.where()` calls created multiple filter operators in the query pipeline. The optimizer now implements the missing final step (step 3) of combining remaining WHERE clauses into a single AND expression. This applies to both queries with and without joins:
+
+- Queries without joins: Multiple WHERE clauses are now combined before compilation
+- Queries with joins: Remaining WHERE clauses after predicate pushdown are combined
+
+This reduces filter operators from N to 1, making chained `.where()` calls perform identically to using a single `.where()` with `and()`.

packages/db/src/query/optimizer.ts

@@ -330,9 +330,22 @@ function applySingleLevelOptimization(query: QueryIR): QueryIR {
     return query
   }
 
-  // Skip optimization if there are no joins - predicate pushdown only benefits joins
-  // Single-table queries don't benefit from this optimization
+  // For queries without joins, combine multiple WHERE clauses into a single clause
+  // to avoid creating multiple filter operators in the pipeline
   if (!query.join || query.join.length === 0) {
+    // Only optimize if there are multiple WHERE clauses to combine
+    if (query.where.length > 1) {
+      // Combine multiple WHERE clauses into a single AND expression
+      const splitWhereClauses = splitAndClauses(query.where)
+      const combinedWhere = combineWithAnd(splitWhereClauses)
+
+      return {
+        ...query,
+        where: [combinedWhere],
+      }
+    }
+
+    // For single WHERE clauses, no optimization needed
     return query
   }
 
@@ -674,6 +687,20 @@ function applyOptimizations(
     // If optimized and no outer JOINs - don't keep (original behavior)
   }
 
+  // Combine multiple remaining WHERE clauses into a single clause to avoid
+  // multiple filter operations in the pipeline (performance optimization)
+  // First flatten any nested AND expressions to avoid and(and(...), ...)
+  const finalWhere: Array<Where> =
+    remainingWhereClauses.length > 1
+      ? [
+          combineWithAnd(
+            remainingWhereClauses.flatMap((clause) =>
+              splitAndClausesRecursive(getWhereExpression(clause))
+            )
+          ),
+        ]
+      : remainingWhereClauses
+
   // Create a completely new query object to ensure immutability
   const optimizedQuery: QueryIR = {
     // Copy all non-optimized fields as-is
@@ -692,8 +719,8 @@ function applyOptimizations(
     from: optimizedFrom,
     join: optimizedJoins,
 
-    // Only include WHERE clauses that weren't successfully optimized
-    where: remainingWhereClauses.length > 0 ? remainingWhereClauses : [],
+    // Include combined WHERE clauses
+    where: finalWhere.length > 0 ? finalWhere : [],
   }
 
   return optimizedQuery