provide GraphQL explorer module for rivine (shorthand: q)

[GlenDC]

As a first phase of working on the big R&D story #605, it would be a nice first step to provide a new module that works completely independent and provides a full query/subscription (so no mutation) scheme for all our queries.

This first phase would allow us to develop a complete scheme to query on our data, play with these queries, experiment with the subscription model (useful for light clients) and test its limits, all without breaking or changing any existing features.

It would also allow, once completed, light clients to already start supporting this new GraphQL endpoint (served still over HTTP(S)) as the phases, also for #605, that follow after this will be changes in the internals of rivine and should not impact the API for the users (see: light clients).

[GlenDC]

For the technology I think https://github.com/graphql-go/graphql will be the way to go.

A very useful resource to bootstrap myself into GraphQL was https://graphql.github.io/learn/

Over the course of this issue I think it would also be useful to try to apply some best practices from https://graphql.github.io/learn/best-practices/:

• pagination, json (with gzip), HTTP, nullability (don't go overboard with the enforced non-nulls), batching and caching (there is even a Go implementation of Facebook's Dataloader: https://github.com/graph-gophers/dataloader).

[GlenDC]

Added an initial, still pure theory, attempt in defining a schema, using static queries only for now:

https://github.com/threefoldtech/rivine/blob/3f48bd994ff106292a2c0354ec324ac2dc730e35/modules/explorergraphql/schema.graphql

I think it should be enough to start playing with a first quick implementation, with the goal to as soon as possible be able to have a small web frontend that allows me to play with queries and iterate on this initial attempt as such.

[GlenDC]

https://github.com/99designs/gqlgen looks like it has potential. It allows us to generate the server code directly from the GraphQL schema, allowing us to focus on the business logic and ensuring it is in sync with our API definition.

[GlenDC]

Since 70e1229 the first phase of the GraphQL explorer starts to be feature complete, at least on a database level.

Objects can be looked up (wallets, contracts, blocks, transactions and outputs). Information can be requested. All seems to work and resolves in a lazy manner.

What are the next steps?

1. We need to have some regular meetings @robvanmieghem, @LeeSmet, @DylanVerstraete and anyone who is interested (e.g. @zaibon), starting with a first meeting today, where I can show the current state, we can have a a Q&A and provide feedback to where to go from that moment.

2. I need to iterate on the API, currently the API is quite complete, but I do not have pagination for big lists yet, there is no filtering yet, and any other advanced query parameters I could add to fields. This I will start doing today.

3. @LeeSmet will help me with setting up a TFChain std network explorer node that will have the current version of the GraphQL explorer running so we can also test it in a real environment. It will be redeployed each time I make substantial updates. This will be done in parallel with task (2).

4. GraphQL also supports updates of (web) sockets, called subscriptions. It has been added since 2015 I think, so I would think it is quite established by now (or so I hope). It would be good to provide already 3types of subscriptions: subscription to a wallet, contract or new (latest) block.

5. There are still "important" improvements that can be done to the current MVP implementation, despite the work already done to it, and than I'm not even talking about supporting extensions yet:
   - The chain data is currently already stored as a network of data, linking objects together rather than storing them embedded (e.g. outputs of a transaction are stored separately, and in a transaction these outputs are simply stored by ID). The exception being the inputs, which only store the spenditure information as optional data of an input. This can be improved upon by linking by the already used DataID, an incrementing 64 integer that I use already for storing all objects (wallets, contracts, blocks, transactions, outputs). This DataID could also be used internally to link, and would reduce all things where ID links are made by a factor of 4 (and slightly more for wallet links, e.g. ms wallet references in ss wallets). To achieve this we'll need to make it on that level also slightly more lazy, where identifiers (the real ones) are given as a function instead of a flat structure. But if so we might as well look if it isn't better to make most of those exposed models from the explorer db interface as resolver interfaces instead of data structures. Something to investigate.

   • Having data identifiers as links will already reduce the size of wallets (especially the heavily-used) ones by a lot. And reduce them by a factor of 4 (more or less, used as a limit). These wallets still do pose a big burden on the disk I/O as each time a wallet is to be updated it requires reading the entire wallet and storing it back. This could be resolved in 2 steps:
     • storing the wallet primitive and aggregated (e.g. balance) information can be stored as one object and the object links (coin outputs, blockstake outputs, transactions, blocks) could be stored as 4 different objects, all using that same data ID). This fits in the proposal to turn the explorer db models into resolvers, so these links only require to be resolved in case the they are requested by the user. If on top of that, but this one might be more fantasy than ever possible in storm DB, we could simply append to it directly on disk, instead of having to read and rewrite it completly, that would be great. If that is possible, and we handle our data processing smart, we should be able to reverse just as easily by trimming the last (x) identifiers from the lists directly on disk as well.
   • Currently we already, since yesterday, use a single (RW) transaction per apply consensus change, this already made our (initial) syncing 3 times faster on my machine. The reading is currently still done however using a (R) transaction per call. Not sure how much performance impact that has, but if somehow we could keep it to one transaction per call, that might be nice too. How that would fit in the API provided by the generated GraphQL resolver code is a mystery to me though. Not sure it also matters as much for reading as it does for writing.

6. Currently the GraphQL explorer does not support unconfirmed data yet (transaction pool data). This can be easily resolved by providing an explorer.DB interface implementation, which subscribes to the Transaction Pool, and wraps another explorer DB. It could then be made so, that if the wrapped DB returns ErrNotFound for an object that could be in the pool theoretically, that it would check its current pool state if that is the case, and if so, return it with an unconfirmed flag to true. All objects that can be unconfirmed would from that moment also have an optional Unconfirmed Bool field. Doing this task is pretty easy, and so trivial that for me it is less urgent, and is only required to be done once we want to start using this module into production.

[GlenDC]

For task (5) I might have an idea. Currently a wallet contains some identification data, aggregated data (such as balance), and mostly linked data (blockids, txnids, coinoutputids, blockstakeoutputids). In the most storage-cheap approach we could instead perhaps link all used wallets in each block (by data ID). This would make the updates and storage very cheap. It would make the querying for outputs, transactions and blocks a lot more expensive though. Blocks would still be kinda OK, as that can be indexed directly, but for transactions we would need to resolve each block transaction ID, and than for outputs we would on top of that need to resolve each output to know if that output is linked to that address. Perhaps that is OK, but perhaps not. A balance that needs to be investigated I guess.

I propose that once we make some time for this task, that we setup some quick stand alone examples that we can benchmark, so we know what gives the best results, in terms of storage as well as update speed and query speed.

[GlenDC]

For Task (3) @LeeSmet tried to setup a std standard tfchain daemon, for which I provided a branch, but we seem to have an issue on the data storage side. It seems to take about 1 GB of disk space per 1000 blocks, not a very good trade-off.

So I am fairly sure that I'll need to look into optimising this heavily starting Monday next week.

[GlenDC]

Schema documentation has also been added. Our playground supports this also by default. If we want it is also possible to use third-party tools to generate documentation from this, should we want to display it in a stand alone way.

Example in our playground:

[GlenDC]

Last two days I was trying to optimise some stuff, both in terms of speed and in terms of storage.

We currenly require, with wallet data included, require about 260MB per 1000 blocks, which is a reduction of factor 4, compared to the initial implementation. Still a lot though.

We also already sync 12% faster (more or less).

Both optimisations have been mostly achieved by using msgpack instead of rivbin for the explorer graphql db storage.

Still a lot of room for optimisation though. Biggest one will be to not rely on the query feature for the unlocking of outputs. Besides that we should try to batch the applied/revert bocks in bigger groups, might also help (but not sure here). For sure not relying on the query feature for unlocking of outputs will help a lot already (or so is a theory of mine).

We're still using StormDB for now. If there are other embedded DB suggestions that I should try to use instead, feel free to hit me up.

[DylanVerstraete]

My findings on Graphql implementations for frontend applications.

Apollo Graphql : https://github.com/apollographql

Which can be used in React and Vue. For the explorer frontend we will be focussing on the Vue implementation.

Apollo Graphql Vuejs: https://github.com/vuejs/vue-apollo

Its straightforward to integrate in our existing explorer frontend project. https://github.com/threefoldtech/rivine-chain-explorer.

Apollo Graphql Vuejs uses vue and typescript. Which fits our needs. It also has local state and cache management functionality. https://apollo.vuejs.org/guide/local-state.html#local-state

For a preview of how this will work: https://apollo.vuejs.org/guide/apollo/#usage-in-vue-components

[GlenDC]

Graphql explorer db now syncs pretty fast. On my laptop with wallet updates it can do about 20 blocks per second. Which is a lot faster than we started with. Disk space is still an issue, we'll need to do the aggregated wallet data different, that is for sure.

Without wallet updates we can go up to 60 blocks per seconds on my laptop.

API is also cleared up, removed the reference point thing, and kept it at height only. Getting a block by height can be done using the blockAt query, and you can also get one starting from the end using a negative index, where -1 is the last block (same as omitting the height), -2` the second last block, etc...