Upgrade bot data handling

[scottyeager]

Currently the bot uses PicklePersistence provided by the python-telegram-bot library to store bot data on disk. The data stored this way includes information about the subscribed users, nodes, and farmerbot related minting violations. If the data is lost, users stop getting the alerts they subscribed for and must subscribe again, some extra alerts regarding violations will likely fire, and overall it's a bad situation.

So far the deployment of the bot has included syncing this bot data file to remote storage as a backup. At least one recovery from the remotely synced file has been successful, but overall this approach is not so robust. Here's a summary of issues:

• Python's pickle is a binary file format, and any corruption will likely result in the file being completely unreadable. Such corruption might occur if the machine running the bot crashes or goes offline while data is being synced
• We are pickling some custom classes and this is also fragile. Moving these classes to a different module, for example, requires a sort of migration for the data to remain readable (this could be solved by using generic data structures like lists and dicts only)
• Examining bot data is not very straightforward because it relies on creating an appropriate environment where the unpickling can take place

There are some alternatives, which are broadly:

• DictPersistence and store the data as JSON text on disk
  • Is better than binary data, in that at least some manual recovery should be possible in case of corruption
• Database
  • Database is best for on disk consistency
  • Get access to replication mechanisms that can also provide consistency guarantees in case of crashes
  • We have to deal with schema updates/migrations, updating data structure becomes a pain (maybe this can be eased by just storing serialized JSON objects in the database rather than mapping
  • More complicated implementation, see below

When it comes to databases, there are further sub options:

• A third party persistence class that writes to a database, like python-telegram-bot-django-persistence
  • These projects tend to be small and not terribly active, decent chance they get abandoned one day
  • Could be very convenient, just drop in replacement
• Write a subclass of BasePersistence
  • Bot code needs little to no modification
  • Need to study/integrate with the existing persistence framework
• Handle data persistence separately (stop using built in persistence feature)
  • Will need to update code wherever data is stored
  • Best flexibility

In the case of a custom implementation, there's also the question of how to handle interfacing with the database, as well as what database to use:

• SQLite
  • Good fit for this use case
  • Live replication options available (Litestream)
• peewee
  • Simple Python ORM
• Etc...

[scottyeager]

I built a persistent dictionary implementation that can be used more or less as a drop in for the dict(s) we store within bot data currently. A bit more work is needed to handle the inherently threaded nature of the bot library. Check the receipts handler in Peppermint for an idea of how to do this.

[scottyeager]

After much consideration, I think the best approach is to store the bot data in a database that can operate in a distributed and HA fashion. This is essentially the only missing component to having an overall HA bot (multiple bots can simply compete to grab the single connection allowed by Telegram as a form of "leader election" and with luck it doesn't need to be any more complicated than that).

Currently my pick for a database is rqlite, based on its design priorities of being simple, lightweight, and simple to operate. We don't have any huge requirements. Current bot data is about 500kb and most of the writes happen during the once per minute cycle where node data is pulled. Relational features aren't really important at this stage, but the data is a more natural fit to tables than key and value.

The work to be done:

• Define a schema:
  • Users
  • Nodes
  • ?
• Find all instances where the bot interacts with bot data and replace them with db calls (for now I think we can skip the ORM)
• Check if there's any issues around concurrency. Maybe this is just okay since rqlite is a network service with its own concurrency handling
• Check for any gotchas with the fact that rqlite has its own transaction feature. Are we doing anything transactional? If so, is the feature engaged properly via the way we use the rqlite Python library?

[scottyeager]

Work is completed with initial testing on rqlite branch.

Notes:

• Concurrency isn't an issue I think because we never had concurrency enabled in the first place
• My idea about using Telegram connection errors for "leader election" didn't work out. Turns out a second instance can disrupt the connection of the first and so on leading to constant churn
• Give the above point, I've implemented a basic leader election leveraging the fact that rqlite gives us consistent shared data