Ansible playbooks

[scottyeager]

Since the bot is now an optionally distributed application, it's especially relevant to have some deployment automation. The idea here it to provide some Ansible playbooks that can more or less automate the following:

• Initial deployment into a cluster of grid micro VMs (or other Zinit based machines/containers)
• Updates
• Healing the cluster by bringing up a new member when there's still a quorum of rqlite processes alive (automating the recovery process if quorum is lost is not a goal at this time)

These are the overall components:

• Rqlite
  • Allow configuring automatic backups to S3
• Bot
  • Prometheus monitoring drop in (bot must expose metrics first)
• Bot bootstrap from old style bot data
  • Create a script that migrates the data into rqlite from a bot_data file
• Ingester
  • Prometheus monitoring drop in
• Ingester database bootstrap
  • Bootstrap from file
  • Bootstrap from active remote database

Checked items are completed but not deeply tested.

Ingester bootstrap

The ingester database (SQLite) bootstrap presents a particular challenge. At this point, the chosen architecture is that each machine in the cluster runs its own ingester and keeps its own ingester database. This works because the ingesters only collect and organize external data, so they can all produce the same outcome without coordination. On the other hand, collecting historic data is both time consuming and computationally expensive. This means that we'd always prefer to bootstrap an ingester from an existing database if possible, to ensure that maximum historic data is available to services that consume the ingester's output.

Two cases are of interest:

1. We have a database file that's not in active use (a backup or database of an ingester that's been temporarily paused) that can just be copied into the cluster
2. One or more machines in the cluster has an up to date database that we want to sync to another machine in the cluster (such as when joining a new member into the cluster)

Case 1 is rather simple using standard Ansible features. Case 2 can be assisted by the new rqlite_rsync tool that can sync a live database from a remote target. Details about implementation and usage scenarios are in ansible/README.md.