merge: 'main' into core/JsonString

Author: jason-ha

.changeset/slick-pillows-cheer.md

@@ -0,0 +1,137 @@
+---
+"fluid-framework": minor
+"@fluidframework/tree": minor
+"@fluid-experimental/tree-react-api": minor
+"__section": tree
+---
+Added APIs for tracking observations of SharedTree content for automatic invalidation
+
+`TreeAlpha.trackObservations` and `TreeAlpha.trackObservationsOnce` have been added.
+These provide a way to run some operation which reads content from [TreeNodes](https://fluidframework.com/docs/api/tree/treenode-class), then run a call back when anything observed by that operation changes.
+
+This functionality has also been exposed in the form of React hooks and React higher order components via the `@fluid-experimental/tree-react-api` package.
+It is now possible to use these utilities to implement React applications which pass TreeNodes in their props and get all necessary invalidation from tree changes handled automatically.
+The recommended pattern for doing this is to use `treeDataObject` or `TreeViewComponent` at the root, then `withTreeObservations` or `withMemoizedTreeObservations` for any sub-components which read from TreeNodes.
+Alternatively more localized changes can be made by using `PropNode` to type erase TreeNodes passed in props, then use one of the `usePropTreeNode` or `usePropTreeRecord` hooks to read from them.
+
+These APIs work with both hydrated and [un-hydrated](https://fluidframework.com/docs/api/tree/unhydrated-typealias) TreeNodes.
+
+### React Support
+
+Here is a simple example of a React components which has an invalidation bug due to reading a mutable field from a TreeNode that was provided in a prop:
+
+```typescript
+const builder = new SchemaFactory("example");
+class Item extends builder.object("Item", { text: SchemaFactory.string }) {}
+const ItemComponentBug = ({ item }: { item: Item }): JSX.Element => (
+	<span>{item.text}</span> // Reading `text`, a mutable value from a React prop, causes an invalidation bug.
+);
+```
+
+This bug can now easily be fixed using `withTreeObservations` or ``withMemoizedTreeObservations`:
+
+```typescript
+const ItemComponent = withTreeObservations(
+	({ item }: { item: Item }): JSX.Element => <span>{item.text}</span>,
+);
+```
+
+For components which take in TreeNodes, but merely forward them and do not read their properties, they can use `PropTreeNode` as shown:
+
+```typescript
+const ItemParentComponent = ({ item }: { item: PropTreeNode<Item> }): JSX.Element => (
+	<ItemComponent item={item} />
+);
+```
+
+If such a component reads from the TreeNode, it gets a compile error instead of an invalidation bug.
+In this case the invalidation bug would be that if `item.text` is modified, the component would not re-render.
+
+```typescript
+const InvalidItemParentComponent = ({
+	item,
+}: { item: PropTreeNode<Item> }): JSX.Element => (
+	// @ts-expect-error PropTreeNode turns this invalidation bug into a compile error
+	<span>{item.text}</span>
+);
+```
+
+To provide access to TreeNode content in only part of a component the `usePropTreeNode` or `usePropTreeRecord` hooks can be used.
+
+
+### TreeAlpha.trackObservationsOnce Examples
+
+Here is a rather minimal example of how `TreeAlpha.trackObservationsOnce` can be used:
+
+```typescript
+cachedFoo ??= TreeAlpha.trackObservationsOnce(
+	() => {
+		cachedFoo = undefined;
+	},
+	() => nodeA.someChild.bar + nodeB.someChild.baz,
+).result;
+```
+
+That is equivalent to doing the following:
+
+```typescript
+if (cachedFoo === undefined) {
+	cachedFoo = nodeA.someChild.bar + nodeB.someChild.baz;
+	const invalidate = (): void => {
+		cachedFoo = undefined;
+		for (const u of unsubscribe) {
+			u();
+		}
+	};
+	const unsubscribe: (() => void)[] = [
+		TreeBeta.on(nodeA, "nodeChanged", (data) => {
+			if (data.changedProperties.has("someChild")) {
+				invalidate();
+			}
+		}),
+		TreeBeta.on(nodeB, "nodeChanged", (data) => {
+			if (data.changedProperties.has("someChild")) {
+				invalidate();
+			}
+		}),
+		TreeBeta.on(nodeA.someChild, "nodeChanged", (data) => {
+			if (data.changedProperties.has("bar")) {
+				invalidate();
+			}
+		}),
+		TreeBeta.on(nodeB.someChild, "nodeChanged", (data) => {
+			if (data.changedProperties.has("baz")) {
+				invalidate();
+			}
+		}),
+	];
+}
+```
+
+Here is more complete example showing how to use `TreeAlpha.trackObservationsOnce` invalidate a property derived from its tree fields.
+
+```typescript
+const factory = new SchemaFactory("com.example");
+class Vector extends factory.object("Vector", {
+	x: SchemaFactory.number,
+	y: SchemaFactory.number,
+}) {
+	#length: number | undefined = undefined;
+	public length(): number {
+		if (this.#length === undefined) {
+			const result = TreeAlpha.trackObservationsOnce(
+				() => {
+					this.#length = undefined;
+				},
+				() => Math.hypot(this.x, this.y),
+			);
+			this.#length = result.result;
+		}
+		return this.#length;
+	}
+}
+const vec = new Vector({ x: 3, y: 4 });
+assert.equal(vec.length(), 5);
+vec.x = 0;
+assert.equal(vec.length(), 4);
+```

.changeset/stupid-ties-raise.md

@@ -0,0 +1,16 @@
+---
+"@fluidframework/driver-definitions": minor
+"@fluidframework/driver-web-cache": minor
+"@fluidframework/odsp-driver": minor
+"@fluidframework/odsp-driver-definitions": minor
+"__section": deprecation
+---
+Move IPersistedCache types to driver-definitions
+
+In an effort to decouple the driver web cache from the odsp driver a number of types have been moved from `@fluidframework/odsp-driver-definitions` to `@fluidframework/driver-definitions`. The moved types have been deprecated in `@fluidframework/odsp-driver-definitions`, and any usages should be moved to  `@fluidframework/driver-definitions`.
+
+The moved types are:
+ - `IEntry`
+ - `IFileEntry`
+ - `ICacheEntry`
+ - `IPersistedCache`

docs/docs/data-structures/tree/schema-definition.mdx

@@ -11,6 +11,10 @@ It also has three methods for specifying internal nodes; `object()`, `array()`,
 Use the members of the class to build a schema.
 See [defining a schema](../../start/tree-start.mdx#defining-a-schema) for an example.
 
+:::note
+Once you have documents using your schema, see [Schema Evolution](./schema-evolution/index.mdx) for guidance on safely updating your schema over time.
+:::
+
 ## Object Schema
 
 Use the `object()` method to create a schema for a note. Note the following about this code:

docs/docs/data-structures/tree/schema-evolution/index.mdx

@@ -0,0 +1,200 @@
+---
+title: Schema Evolution
+sidebar_position: 8
+---
+
+# Schema Evolution
+
+As your application grows and evolves, you may need to modify the structure of your [SharedTree](../index.mdx) data.
+SharedTree makes it possible to update your [schemas](../schema-definition.mdx) in ways that allow an app to load or upgrade documents created or edited by earlier versions of the app (see [understanding compatibility](./index.mdx#understanding-compatibility)).
+
+:::note
+For detailed information about which types of changes are safe versus breaking, see [types of schema changes](./types-of-changes).
+:::
+
+## Stored vs View Schema
+
+When updating your schema, it's important to understand the difference between the two types of schemas:
+
+- **View Schema**: The schema defined in your application code using [`SchemaFactory`](../../../api/fluid-framework/schemafactory-class)
+- **Stored Schema**: The schema persisted with each document. It describes what type of tree can be stored in the document.
+
+When a document is first created, its stored schema is set using the given view schema.
+Afterwards, it is immutable until explicitly [upgraded](./index.mdx#schema-upgrade-process).
+
+### Stored Schema Invariants
+
+The stored schema follows additional constraints that don't apply to view schemas:
+
+**1. Complete Description**
+The stored schema completely describes all possible data in the tree.
+Every node and field in the document is known to the stored schema.
+
+**2. Additive Only**
+The stored schema can grow over time but cannot shrink.
+All nodes types present in the stored schema will exist forever, even if they stop being used by the view schema.
+
+**3. Field Key Stability**
+Field keys on object nodes are permanent once introduced.
+Therefore, a field key on an object node in the stored schema can never be changed/reused/repurposed.
+
+### Why These Constraints?
+
+It's assumed that a Fluid application developer does not have absolute control over all user documents.
+Documents can persist for an unbounded length of time before they are re-opened by an application.
+Therefore, a Fluid application should be prepared to encounter any version of a document, even a very old one, and handle it appropriately.
+
+## Schema Compatibility
+
+When you open a document, SharedTree compares your application's view schema with the document's stored schema to determine compatibility:
+
+- **Compatible**: Document can be opened and used normally
+- **Upgradeable**: Document can be upgraded to match your view schema using [`view.upgradeSchema()`](../../../api/fluid-framework/treeview-interface#upgradeschema-methodsignature)
+- **Incompatible**: Document cannot be opened with or upgraded to match your current view schema
+
+They must be compatible or upgradeable in order for the application to work - otherwise, it cannot read or write data from/to the document.
+SharedTree will throw an assert if the application attempts to access the document's data with an incompatible view schema.
+
+Initially, a document's stored schema is created to match the view schema at the time of creation, and therefore the view schema is compatible with the stored schema.
+However, later on the application might change the view schema.
+Such a change might be backwards compatible, forwards compatible, or neither or both.
+
+### Understanding Compatibility
+
+#### Backwards Compatibility
+
+A change to the view schema is **backwards compatible** if the *new view schema* is compatible with the *old stored schema*.
+In practice, this means that the new version of the application will be able to [upgrade](./index.mdx#schema-upgrade-process) documents to the new schema (note that the process of upgrading might affect forwards compatibility - see below).
+
+If a change is made that is not backwards compatible, then existing documents will "break", that is, they can no longer be loaded by the application.
+
+#### Forwards Compatibility
+
+A change to the view schema is **forwards compatible** if an *old view schema* is compatible with the *new stored schema* that results from the upgrade.
+In practice, the older version of the application must be able to "view" the new document after it has been upgraded.
+
+If a change is made that is backwards compatible, but not forwards compatible, then old versions of the application will "break" when trying to load newer/upgraded documents.
+To prevent this, the application can employ a [staged rollout](./index.mdx#staged-rollouts).
+
+### Schema Change Examples
+
+#### 😎 Chill
+| Backwards Compatible | Forwards Compatible | Rollout Process |
+|-----------------------|----------------------|-----------------|
+| ✅                    | ✅                   | Normal          |
+
+**Examples:**
+- Adding an optional field when compatibility flag is enabled
+
+#### 🌶️ Spicy
+| Backwards Compatible | Forwards Compatible | Rollout Process |
+|-----------------------|----------------------|-----------------|
+| ✅                    | ❌                   | ⚠️ Staged       |
+
+**Examples:**
+- Adding a new allowed type (TODO link to how to doc)
+
+#### 👮 Go to Jail
+| Backwards Compatible | Forwards Compatible | Rollout Process |
+|-----------------------|----------------------|-----------------|
+| ❌                    | ❌                   | 🚫 Forbidden    |
+
+**Examples:**
+- Adding/removing a required field
+- Removing an allowed type from a field
+- Renaming a node identifier
+
+
+### Checking Compatibility
+
+Use [`TreeView.compatibility`](../../../api/fluid-framework/treeview-interface#compatibility-propertysignature) to check if your schema changes are compatible with existing documents.
+
+Generally, the most important properties of the `compatibility` object to monitor are [`canView`](../../../api/fluid-framework/schemacompatibilitystatus-interface#canview-propertysignature) and [`canUpgrade`](../../../api/fluid-framework/schemacompatibilitystatus-interface#canupgrade-propertysignature).
+`canView` indicates that the view schema is [compatible](./index.mdx#schema-compatibility) with the stored schema while `canUpgrade` indicates that it is valid to [upgrade the schema](./index.mdx#schema-upgrade-process).
+
+### Monitoring Remote Schema Changes
+
+In collaborative applications, other clients may upgrade the document's schema while you're working.
+Listen for the `schemaChanged` event to handle these remote upgrades:
+
+```typescript
+// Monitor for remote schema changes
+view.events.on("schemaChanged", () => {
+    // Check view.compatibility here and handle accordingly
+});
+```
+
+## Schema Upgrade Process
+
+When you make [backwards compatible](./index.mdx#understanding-compatibility) changes to your schema, you can upgrade existing documents to use the new schema using [`view.upgradeSchema()`](../../../api/fluid-framework/treeview-interface#upgradeschema-methodsignature):
+
+```typescript
+import { SchemaFactory, TreeViewConfiguration } from "@fluidframework/tree";
+
+const factory = new SchemaFactory("WhiteboardApp");
+
+// Updated schema with new optional field
+class Note extends factory.object("Note", {
+    id: factory.string,
+    x: factory.number,
+    y: factory.number,
+    color: factory.optional(factory.string), // New optional field
+}) {}
+
+// Create tree view with updated schema
+const config = new TreeViewConfiguration({ schema: Note });
+const view = tree.viewWith(config);
+
+// Check compatibility and upgrade if needed
+if (!view.compatibility.canView) {
+    if (view.compatibility.canUpgrade) {
+        view.upgradeSchema(); // Upgrade document to new schema
+        console.log("Document upgraded successfully");
+    } else {
+        throw new Error("Schema is incompatible and cannot be upgraded");
+    }
+}
+
+// Now you can use the upgraded document
+view.root.color = "#FF0000";
+```
+
+For detailed information about which types of changes are safe versus breaking, see [Types of Schema Changes](./types-of-changes).
+
+### Staged Rollouts
+
+When deploying schema changes in production, use staged rollouts to ensure compatibility between clients running different application versions.
+Specifically, staged rollouts allow you to handle changes that are [backwards compatible](./index.mdx#backwards-compatibility) but not [forwards compatible](./index.mdx#forwards-compatibility).
+
+**Why Staged Rollouts Are Necessary**
+
+In collaborative applications, multiple users with different client versions may work on the same document simultaneously.
+If a client upgrades a document to a new schema that older clients cannot read, those users will be locked out until they update their application.
+
+**Two-Phase Deployment Strategy**
+
+**Phase 1: Deploy Schema Reading Support**
+- Deploy application versions that can read the new schema without upgrading documents
+- Wait until all (or "tolerably most") users have updated to the new application version. This is called **client saturation**.
+- Monitor deployment metrics to ensure coverage
+
+**Phase 2: Enable Document Upgrades**
+- Deploy code that automatically upgrades documents to the new schema
+- All clients can now collaborate since they support reading the upgraded schema
+- Monitor for any compatibility issues
+
+Changes that are both [backwards compatible](./index.mdx#backwards-compatibility) and [forwards compatible](./index.mdx#forwards-compatibility) are capable of cross-version collaboration without requiring a staged rollout.
+In that case, new clients can read and write to the same document as old clients without either one being fundamentally incompatible.
+However, staged rollouts are not easily avoidable.
+Even seemingly simple schema changes (for example, adding a new type of node to a field) may not be forwards-compatible, cannot support cross-version collaboration and therefore require a staged rollout.
+
+**Best Practices:**
+- Allow appropriate amount of time between release cycles to ensure client version adoption
+- Monitor client version distribution before enabling upgrades
+- Test compatibility scenarios with mixed client versions
+
+## See Also
+
+- [Types of Schema Changes](./types-of-changes) - Detailed guide on safe vs breaking changes
+- [Schema Definition](../schema-definition.mdx) - How to define schemas
+- [Node Types](../node-types.mdx) - Understanding different node types