Skip to content

Ensure schemas can apply defaults when inserting #209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
Jul 15, 2025

Conversation

DawidWraga
Copy link
Contributor

@DawidWraga DawidWraga commented Jun 25, 2025

see #207 (comment) for details

Copy link

changeset-bot bot commented Jun 25, 2025

🦋 Changeset detected

Latest commit: 7b86a18

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@DawidWraga DawidWraga marked this pull request as draft June 25, 2025 22:37
Copy link

pkg-pr-new bot commented Jun 25, 2025

@tanstack/db-example-react-todo

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@209

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@209

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@209

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@209

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@209

commit: 7b86a18

@DawidWraga DawidWraga marked this pull request as ready for review June 26, 2025 12:42
@KyleAMathews
Copy link
Collaborator

Hey it's looking good! A few failing tests to fix still...

@DawidWraga
Copy link
Contributor Author

Hey I've fixed the previously failing tests but after merging in the new changes there's a bunch of different test failures. I'm moving houses tomorrow so I won't be able to address this for some time. If anyone has the moment to look into these conflicts I'd appreciate your help! Thanks

@KyleAMathews
Copy link
Collaborator

👍 I'll pick it up after lunch

@KyleAMathews KyleAMathews requested a review from samwillis June 26, 2025 21:41
@KyleAMathews
Copy link
Collaborator

@samwillis could you review my type changes? Particularly to the query builder? Don't want to mess with any guarantees you wanted. Our types are getting complicated... :-\

@KyleAMathews KyleAMathews changed the title fix: use validated schema to ensure defaults are passed on insert Ensure schemas can apply defaults when inserting Jun 26, 2025
@DawidWraga
Copy link
Contributor Author

Thanks for picking this up. I have an idea that could simplify some of the collection types:

export class CollectionImpl<
  T extends object = Record<string, unknown>,
  TKey extends string | number = string | number,
  // Replace `TExplicit` `TSchema` `TFallback` with TInsertInput:
  TInsertInput extends object = T,
> {
  // ...
}


export class Collection<
  T extends object = Record<string, unknown>,
  TKey extends string | number = string | number,
  TUtils extends UtilsRecord = {},
  // Replace `TExplicit` `TSchema` `TFallback` with TInsertInput:
  TInsertInput extends object = T,
> {
  // ...
}

export function createCollection<
  TExplicit = unknown,
  TKey extends string | number = string | number,
  TUtils extends UtilsRecord = {},
  TSchema extends StandardSchemaV1 = StandardSchemaV1,
  TFallback extends object = Record<string, unknown>,
>(options): Collection<
  ResolveType<TExplicit, TSchema, TFallback>,
  TKey,
  TUtils,
  // Pass the resolved type:
  CollectionInsertInput<TExplicit, TSchema, TFallback>
> {
  // ...
const collection = new CollectionImpl<
    ResolveType<TExplicit, TSchema, TFallback>,
    TKey
    // Pass the resolved type:
    CollectionInsertInput<TExplicit, TSchema, TFallback>
  >(options)

  return collection as Collection<
    ResolveType<TExplicit, TSchema, TFallback>,
    TKey,
    TUtils
    // Pass the resolved type:
    CollectionInsertInput<TExplicit, TSchema, TFallback>
  >
}

This is similar to how ResolveType already works, only passing the resolved type instead of the 3 generics.


// Minimal data
const tx1 = createTransaction({ mutationFn })
tx1.mutate(() => collection.insert({ text: `task-1` }))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you test asserting on tx.mutations[0] (changes, etc.)?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also add some type tests in the collection.test-d.ts file

@DawidWraga
Copy link
Contributor Author

Hey @KyleAMathews I've written some tests but having trouble deciding what to do with the mutation types

If we want full type safety on the PendingMutation - including the changes type being different for delete/update/insert based on the TOperation, then it adds a significant amount of complexity for a little bit more type safety, I'm not sure whether the trade off is worth it.

All of these helpers:

export type UpdateMutationFnParams<T extends object = Record<string, unknown>> =
  {
    transaction: TransactionWithMutations<T>
  }

export type InsertMutationFnParams<T extends object = Record<string, unknown>> =
  {
    transaction: TransactionWithMutations<T>
  }

export type DeleteMutationFnParams<T extends object = Record<string, unknown>> =
  {
    transaction: TransactionWithMutations<T>
  }

export type InsertMutationFn<T extends object = Record<string, unknown>> = (
  params: InsertMutationFnParams<T>
) => Promise<any>

export type UpdateMutationFn<T extends object = Record<string, unknown>> = (
  params: UpdateMutationFnParams<T>
) => Promise<any>

export type DeleteMutationFn<T extends object = Record<string, unknown>> = (
  params: DeleteMutationFnParams<T>
) => Promise<any>

// ...

/**
 * Utility type for a Transaction with at least one mutation
 * This is used internally by the Transaction.commit method
 */
export type TransactionWithMutations<
  T extends object = Record<string, unknown>,
> = Transaction<T> & {
  mutations: NonEmptyArray<PendingMutation<T>>
}

export interface TransactionConfig<T extends object = Record<string, unknown>> {
  /** Unique identifier for the transaction */
  id?: string
  /* If the transaction should autocommit after a mutate call or should commit be called explicitly */
  autoCommit?: boolean
  mutationFn: MutationFn<T>
  /** Custom metadata to associate with the transaction */
  metadata?: Record<string, unknown>
}

(and more)

All these helpers ultimately wrap around PendingMutation. This is what it looks like in main right now:

export interface PendingMutation<
  T extends object = Record<string, unknown>,
  TOperation extends OperationType = OperationType,
> {
 // ...
  original: TOperation extends `insert` ? {} : T
  // The result state of the object after all mutations.
  modified: T
  // The actual changes made.
  changes: TOperation extends `insert`
    ? T
    : TOperation extends `delete`
      ? T
      : Partial<T>
      
   // without TInsertInput
    collection: Collection<T, any>
   //...
      }

So if you wanted to make the changes property be fully type safe you could do something like this:

export type ResolveTransactionData<
  T extends object = Record<string, unknown>,
  TOperation extends OperationType = OperationType,
  TInsertInput extends object = T,
> = TOperation extends `insert`
  ? TInsertInput
  : TOperation extends `delete`
    ? T
    : Partial<T>

export interface PendingMutation<
  T extends object = Record<string, unknown>,
  TOperation extends OperationType = OperationType,
  TInsertInput extends object = T,
  TCollection extends Collection<T, any, any, any, TInsertInput> = Collection<
    T,
    any,
    any,
    any,
    TInsertInput
  >,
> {
  mutationId: string
  // The state of the object before the mutation.
  original: TOperation extends `insert` ? {} : T
  // The result state of the object after all mutations.
  modified: T
  // The actual changes made.
  changes: ResolveTransactionData<T, TOperation, TInsertInput>

  collection: TCollection
  
  //....
}

But then the trouble is - you would have to pass many generics to all the mutation helpers.

I tried to refactor following this pattern to minimize complexity:
Inside createCollection we simplify TExplicit, TSchema and TFallback on Collection into just TInput
Similarly, inside createTransaction I tried to simplify T, TOperation and TInsertInput into just T

However, unlike Collection, PendingMutations explicitly requires all 3 to be type safe (T for modified , TOperation for type and TInsertInput for changes). If we just relied on the generic defaults then it kinda defeats the point - no benefit but adds lots of extra code.

On collection the complexity is lower and definitely worth it to make the operation params fully type safe.
However, on the mutations I'm not sure if it's worth the complexity trade off.

It could just be left as TOperation extends delete ? T : Partial<T> like it was before to keep things simple.

Not sure, it's up to you. I'll push my attempt for visibility but it's just a draft currently.

Current state of the code:

  • I've added some tests
  • Initially, added the new generics to most the type utils (most tests passing, but led to lots of extra types)
  • Tried to simplify using ResolveTransactionData inside createTransaction but then realised PendingMutation requires all 3.
  • Uncertain about trade-off between complexity and typesafety, stopped development, wrote this post instead

You're welcome to pick it up from here if you like, or I can try something more later this week, my brain is pretty fried for a moment from reading all those types 😆

@KyleAMathews
Copy link
Collaborator

This might be beyond me 😅 @kevin-dp any thoughts?

@KyleAMathews
Copy link
Collaborator

In general I think type complexity is worth it — I've spent more days than I care for working on typescript for this lib but we expect 100s of thousands of engineers to use this lib so even marginal improvements to type safety are useful.

@kevin-dp
Copy link
Contributor

kevin-dp commented Jul 9, 2025

I've read through #207 and the comments in this PR. Regarding #207, i could fully understand the use case and agree with the solution. However, i've not played enough with it to fully understand the benefits of the fully type-safe changes as explained in #209 (comment). From the perspective of developers, what would be the benefits of this approach? i.e. can we look at some concrete examples with and without the type safety of changes, to see how impacting it is. I generally agree that the more type safe the better, but having to juggle around 3 type parameters everywhere instead of 1 makes the code a lot more complex, hard to understand and maintain. So it really depends how big the benefit will be for developers.

@kevin-dp
Copy link
Contributor

kevin-dp commented Jul 9, 2025

I've been chatting this through with @KyleAMathews. Here are my thoughts:

  • i agree that we want to be able to omit columns that have defaults when inserting
  • regarding the changes i think it's fine to type them as Partial<T> which keeps the code simpler and more maintainable at the cost of a slightly less precise type.

Having precise types for user input is really important because e.g. when developers pass an argument to a function and the argument type checks, you want the function to work on that input. That's why Partial<T> is not acceptable for the argument to insert because if the user would omit a required field it will typecheck but won't work at runtime.

However, the type of outputs is a different story. When the output of a function (or the changes field in our example) is imprecise like Partial<T> that is still correct because all of the changes will fulfill that type. So if the developer's code can handle changes of type Partial<T> (i.e. the code typechecks) then at runtime it will be able to handle any of the actual changes it gets (because all changes are of type Partial<T> even if that's not the most precise type for it).

I don't think the slightly improved typing for changes is worth the added complexity. In essence, the difference with your suggested approach would be that instead of typing the changes as Partial<T> (which makes every column optional) you would only make the columns that have defaults optional.

@DawidWraga
Copy link
Contributor Author

Thanks for your input @kevin-dp that's exactly what I was thinking, completely agree.

I'll finish this up now 😄

@KyleAMathews
Copy link
Collaborator

KyleAMathews commented Jul 14, 2025

@DawidWraga I merged in main — could you check it over to see if it looks correct still?

@KyleAMathews
Copy link
Collaborator

Actually did another through go over and it looks good to go — please do check though @DawidWraga when you have a chance and PR anything I missed!

@KyleAMathews KyleAMathews merged commit 056609e into TanStack:main Jul 15, 2025
5 checks passed
@github-actions github-actions bot mentioned this pull request Jul 15, 2025
@DawidWraga
Copy link
Contributor Author

DawidWraga commented Jul 15, 2025

Hi @KyleAMathews

Thanks for merging this in and fixing the last few tweaks. I've reviewed the PR and everything looks good.

One technical decision I wanted to run by you is what data to pass into the changes field for insert mutations. The current implementation uses a specific approach, but there are valid alternatives worth considering.

The Question: What data should the insert mutation changes field contain?

See packages/db/src/collection.ts - CollectionImpl.insert() method:

items.forEach((item) => {
  const validatedData = this.validateData(item, `insert`)
  
  mutations.push({
    original: {},
    modified: validatedData,
    
    // Option A: raw input
    changes: item,

    // Option B: validated input
    changes: validatedData,

    // Option C: validated fields, without default fields (current approach)
    changes: Object.fromEntries(
      Object.keys(item).map((k) => [k, validatedData[k]])
    ),
  })
})

Example scenario:

const schema = z.object({
  title: z.string(),
  dueDate: z.coerce.date().optional(),
  completed: z.boolean().default(false)
})

taskCollection.insert({ 
  title: "lorem",
  dueDate: "2025-07-20" // string, not Date
}

Options:

  • A) Raw input: { title: "lorem", dueDate: "2025-07-20" } (no default, no transform)
  • B) Full validated: { title: "lorem", dueDate: Date(...), completed: false } (default & transform)
  • C) Validated without defaults: { title: "lorem", dueDate: Date(...) } (current) (no default, but transformed)

Rationale for C: Shows user intent (no defaults) with transformed values (for consistency with stored data)

Case for A: Simpler, pure user input representation

Let me know if you'd prefer a different approach - happy to adjust it, or feel free to tweak it directly if that's easier!

@KyleAMathews
Copy link
Collaborator

Hmm good question. I think c still makes the most sense as "changes" are how the model was changed not the raw inputs. I'd find it odd if people wouldn't want the cleaned up data. If they didn't, they should probably change their schema? I'm not super confident or feel strongly about this. I suspect either way would be fine but keeping everything at the end state sounds good right now.

cursor bot pushed a commit that referenced this pull request Jul 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants