Skip to content

SQLite INSERT ON CONFLICT RETURNING does not return conflicts #773

@sheaf

Description

@sheaf

With the SQLite backend, it seems that runInsertReturningList $ insertOnConflict ... does not work correctly. It will return the newly inserted entries into the table but it does not return any of the entries that were updated on conflict. That seems like a footgun to me although I am very far from being a database expert.

Show/hide reproducer
{-# LANGUAGE DeriveAnyClass #-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE StandaloneDeriving #-}
{-# LANGUAGE TypeFamilies #-}

module Main where

-- base
import Data.Int
  ( Int32 )

-- text
import Data.Text
  ( Text )

-- beam-core
import Database.Beam
import Database.Beam.Backend.SQL.BeamExtensions
  ( onConflictUpdateSetWhere
  , insertOnConflict
  , conflictingFields
  , runInsertReturningList
  )

-- beam-migrate
import Database.Beam.Migrate
  ( defaultMigratableDbSettings )
import Database.Beam.Migrate.Simple
  ( CheckedDatabaseSettings, autoMigrate )

-- beam-sqlite
import Database.Beam.Sqlite
  ( Sqlite, runBeamSqliteDebug )
import Database.Beam.Sqlite.Migrate
  ( migrationBackend )

-- sqlite-simple
import Database.SQLite.Simple
  ( open )

--------------------------------------------------------------------------------

data TestDb f
    = TestDb
    { usersTable :: f (TableEntity User)
    }
    deriving stock Generic

deriving anyclass instance Database be TestDb

testDb :: DatabaseSettings be TestDb
testDb = defaultDbSettings

checkedDb :: CheckedDatabaseSettings Sqlite TestDb
checkedDb = defaultMigratableDbSettings

data User f
    = User
    { userId   :: Columnar f Int32
    , userName :: Columnar f Text
    }
    deriving stock Generic

deriving stock instance Show ( PrimaryKey User Identity )
deriving stock instance Show ( User Identity )

instance Table User where
  newtype PrimaryKey User f = UserId ( Columnar f Int32 )
    deriving stock Generic
  primaryKey = UserId . userId

deriving anyclass instance Beamable User
deriving anyclass instance Beamable ( PrimaryKey User )

main :: IO ()
main = do

  conn <- open "test.sqlite"

  conflicts <-
    runBeamSqliteDebug putStrLn conn $ do

      -- Use beam-migrate to create the db for simplicity
      autoMigrate migrationBackend checkedDb

      -- Insert some users
      runInsert $
        insert ( usersTable testDb )
          ( insertValues [ User 0 "user0", User 2 "user2", User 5 "user5" ] )

      let newUsers = [ User 1 "user1", User 2 "different_user2" ]

      -- Insert some conflicting users
      runInsertReturningList $
        insertOnConflict ( usersTable testDb )
          ( insertValues newUsers )
          ( conflictingFields userId )
          ( onConflictUpdateSetWhere
            ( \ ( User { userName = fld  } )
                ( User { userName = excl } ) ->
                  fld <-. excl
            )
            ( \ ( User { userName = fld  } )
              ( User { userName = excl } ) ->
                current_ fld /=. excl
            )
          )

  print conflicts

This returns the conflicts as [User {userId = 1, userName = "user1"}]. User 1 is newly added so that's valid, but it's missing the conflicting User 2. Of course the new User 2 with name different_user2 is properly updated in the database, overwriting the old User 2, but it's not returned as a conflict.

I think this may be due to the way that beam-sqlite emulates runInsertReturningList. I think it would be good to get rid of that code and to instead use RETURNING, as that is supported since SQLite version 3.35.0 (2021-03-12). That should also enable support for DELETE RETURNING in the SQLite backend, which would be quite useful as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions